The tracking technology depicted on the TV show Person of Interest is now real. Technology developed at the University of Washington detects and tracks people across multiple cameras, even when the cameras’ observation frames do not overlap.
The system distinguishes individuals by drawing a numbered box around them, the box following them as they move. The image from a surveillance camera pointing north on Avenue A can be linked to one pointing east on 1st St. When someone seen on the Avenue A camera turns left onto 1st, the system picks up the person on the second camera and assigns the same box and number to that image.
If you have ever tried to locate someone in a crowd and then keep track of them as they moved through the crowd, you can appreciate how difficult the task is. Cops trained in surveillance techniques take years to develop this kind of expertise, and some never get the knack of it.
From Fiction to Fact
Person of Interest has been using a mockup of this technology to introduce chapters in its story. The output from a surveillance camera on the streets of New York City shows superimposed boxes around each person and vehicle, moving as the person or vehicle moves.
In the TV show, the computer that is drawing the boxes also knows the name, address and blood type of everyone it tracks (which is everybody), and feeds this to the all-seeing government. The real-life system skips the identification feature, but the tracking technology is still amazing.
Lead researcher Jenq-Neng Hwang expanded on a computational task called Visual Simultaneous Localization and Mapping, or V-SLAM. V-SLAM is best known for its use in the development of autonomous vehicles. These include the driverless cars that Google is experimenting with, and the more rugged vehicles produced for the DARPA Grand Challenge . The latter was a competition to make a vehicle that could negotiate a difficult obstacle course, unknown to the designers, with no human assistance.
One of the tasks performed by V-SLAM systems is the identification of pedestrians, animals, and other driving hazards that appear unexpectedly and move in unpredictable ways. It’s one thing for a system to see the center median of a highway and avoid driving over it; it’s quite another for a computer to distinguish between a stationary tree and a child who might dash out from the sidewalk without warning.
The new research expands this technology beyond distinguishing between a person and an inanimate object. Once the person is identified as a person, the computer assigns a tracking label (usually a number) to them.
On the display, a colored box with that number stays drawn around the person as they move across the camera field. When the same person appears in the field of another camera integrated into the system, they get the same box and number assigned to them. This will happen even if the cameras don’t overlap, and when there is a gap between when the person disappears from one image and appears in another.
The system works just as well with cameras that are moving themselves, as with those mounted on a moving vehicle.
Terrorism Tracking
The transportation industry is interested in this, not just because of the association with autonomous cars, but because transportation terminals like airports, train, and bus stations may have more than 100,000 people moving through them daily. Picking a criminal fugitive or terrorist out of the crowd on a security monitor is tough enough. It becomes close to impossible to keep track of the person as they leave the area covered by one camera and enter another. Being able to follow people like this could be the difference between locating and disarming an IED, or dealing with an incident like the 2013 Boston Marathon bombing.
You can see some examples of the system in action on the University of Washington’s website. The paper explaining the research in full was published by the Institute of Electrical and Electronics Engineers (IEEE).
