![](https://www.popsci.com/wp-content/uploads/2019/03/18/3TN3BADXYQJSO3YU4XPQNUJEBE.jpg?quality=85&w=1024)
![](https://www.popsci.com/wp-content/uploads/2019/03/18/3TN3BADXYQJSO3YU4XPQNUJEBE.jpg?quality=85&w=1024)
Cops tracking gang activity frequently need to decipher graffiti, decoding messages, identifying gangs and even monitoring individual people by looking at their bubbly letters and spray-painted scrawls. New software can help speed the process by automatically checking graffiti against a library of street art.
The system works by filtering existing images, which are currently captured and labeled by hand. Given a specific graffiti query, the system finds a list of similar images based on visual and content similarity, and then returns the gang names associated with those similar images. Michigan State University computer scientists led by Anil Jain have been working on the software for a couple years now and plan to present their findings at the ACM Multimedia conference in Scottsdale, Ariz., later this month.
Like other image-recognition programs, the graffiti matching system searches for matches based on commonalities, according to Jain and his colleagues. This helps filter different graffiti that depicts the same logo or tag, but may look dramatically different based on the person or gang who created it. First, the system extracts visual features from a new image, and then letters, numbers and symbols are manually annotated. Then, the system finds candidate images that share similarities with this new image.
It would be nice to have a system that automatically recognizes numbers and letters, but this is too difficult with existing technology because graffiti is so variable, Jain and his colleagues say. Instead, Jain uses crowdsourcing to identify words, using Amazon’s Mechanical Turk service, according to New Scientist.
When using both image and text similarity, the system is able to pinpoint a gang or individual with about 65 percent accuracy, New Scientist says.
The system still needs some tweaking so it can be applied to vast graffiti image databases, Jain and colleagues explain. “In the future, we plan to explore additional information about graffiti other than the textual features, such as the time stamp and location of graffiti,” the authors write.