MIT chemists cover the whiteboards and even windows of their with graphical diagrams of molecular bonds, but they need not rely upon dry-erase markers much longer. Their computer science colleagues have begun developing sketch-recognition for tablet computers such as Apple’s iPad that can interpret stylus drawings of chemical bonds and element symbols for chemists, circuit components such as batteries or capacitors in an electrical engineering diagram, and even family trees.
That could transform tablet computers into smart sketchpad interfaces that make interaction with computers more natural for humans. MIT’s researchers also envision speech or gesture recognition eventually coming into play.
“We want to interconnect this with some of the other things we’ve done with speech and web-based lookup so that one could walk up to the whiteboard and sketch a molecule and say, ‘Has anybody published anything like this?'” said Randall Davis, an MIT computer scientist.
“And then there’s the multimodal aspect of that, which is, I draw it, ask if it’s ever appeared, and the system says, ‘I can’t find anything like it.’ And I point at the corner of the molecule and I say, ‘What if I put a methyl group there?’ Not draw it, but just gesture at it.”
Tablet computers and smart phones already have software that can recognize handwriting based on language patterns and the relatively limited strokes for each letter. But creating better sketch-recognition software required MIT to combine information about the physical appearance of a final sketch with information about how it was drawn — the system can recall which way a stylus was moving when it made any given stroke.
The MIT software then breaks down the drawn symbol into different parts such as horizontal, vertical or diagonal elements. Algorithms help clean up stray marks and enhance intentional strokes. And a database allows the system to compare drawn symbols to known symbols.
There’s a lot of work still ahead for this project. The MIT software has yet to work for electrical-engineering diagrams of circuits, and it still needs training to recognize chemistry abbreviations such as AC for acetyl groups. But it could help sketching become a more natural way of interacting with computers, and especially as the iPad or similar devices become popular.
“Previously, the technology was looking for places to be used,” said Tom Stahovich, a mechanical engineer at the University of California in Riverside. “Now, there’s hardware everywhere in need of this technology.”