MIT's Sketch-Interpreting Software Turns Tablet Computers into Smart Whiteboards

Unleash your inner geek artiste

By Jeremy Hsu Posted 02.22.2010 at 2:29 pm 14 Comments

Sketch-Recognition Can you see me now? MIT

MIT chemists cover the whiteboards and even windows of their with graphical diagrams of molecular bonds, but they need not rely upon dry-erase markers much longer. Their computer science colleagues have begun developing sketch-recognition for tablet computers such as Apple's iPad that can interpret stylus drawings of chemical bonds and element symbols for chemists, circuit components such as batteries or capacitors in an electrical engineering diagram, and even family trees.

That could transform tablet computers into smart sketchpad interfaces that make interaction with computers more natural for humans. MIT's researchers also envision speech or gesture recognition eventually coming into play.

"We want to interconnect this with some of the other things we've done with speech and web-based lookup so that one could walk up to the whiteboard and sketch a molecule and say, 'Has anybody published anything like this?'" said Randall Davis, an MIT computer scientist.

"And then there's the multimodal aspect of that, which is, I draw it, ask if it's ever appeared, and the system says, 'I can't find anything like it.' And I point at the corner of the molecule and I say, 'What if I put a methyl group there?' Not draw it, but just gesture at it."

Tablet computers and smart phones already have software that can recognize handwriting based on language patterns and the relatively limited strokes for each letter. But creating better sketch-recognition software required MIT to combine information about the physical appearance of a final sketch with information about how it was drawn -- the system can recall which way a stylus was moving when it made any given stroke.

The MIT software then breaks down the drawn symbol into different parts such as horizontal, vertical or diagonal elements. Algorithms help clean up stray marks and enhance intentional strokes. And a database allows the system to compare drawn symbols to known symbols.

There's a lot of work still ahead for this project. The MIT software has yet to work for electrical-engineering diagrams of circuits, and it still needs training to recognize chemistry abbreviations such as AC for acetyl groups. But it could help sketching become a more natural way of interacting with computers, and especially as the iPad or similar devices become popular.

"Previously, the technology was looking for places to be used," said Tom Stahovich, a mechanical engineer at the University of California in Riverside. "Now, there's hardware everywhere in need of this technology."

Previous Article: Bloom Energy Promises Cheap, Emissions-Free Power From a Small Box

Next Article: An Astronaut Peeks Out from the Space Station's Lovely New 360-Degree Window

14 Comments

Link to this comment

boka

02/22/10 at 3:15 pm

This is perfect for non-artists or people that can't draw. But it would be awful for artists and art in general.

Link to this comment

Cuishi14

02/22/10 at 6:04 pm

Chris

ahh the desk from enders game is coming soon

Link to this comment

PhilInYork

02/22/10 at 10:32 pm

This looks like a Windows program running on a PC tablet. Why - oh - why does Pop Sci feel the need to plug the iPad every chance they get? I doubt software like this will even be able to run on the iPad.

Link to this comment

Meshca

02/23/10 at 2:20 am

lol I mentioned the this being the side-effect of releasing the iPad. go figure. It prob can run on iPad and be useless at the same time. The iPad is here for us to out-do it, not for it to be a convenience. This isn't about the maxipad.

This is a great way to provide easier high-tech communication between engineers, drafters, architects, your school, and your daily coffee shop debates that you just can't explain with words.

Link to this comment

ford2go

02/23/10 at 3:11 am

I can see this as being a real 'devil is in the details' project. If it can generate useful results for the majority of users, it will be great. If it's finicky and error prone then it will be crap.

Plus, it will have to be customized for each discipline.

I agree with PhillinYork -- lay off of the Apple refs. Tablets have been around for a while, and some probably even work. Also, IIRC Apple is pushing the app store concept for their new wunderkind -- that would certainly complicate getting it on the TheyPad.

Link to this comment

sbryfcz

02/23/10 at 9:07 am

Clemson University has been developing a similar tool for use in a classroom. The tool is called OrganicPad and it allows users to draw Organic chemistry Lewis structures with a stylus. OrganicPad also allows for teachers and students to connect in or out of class and the students as the teacher can ask students to complete problems. OrganicPad will automatically check the students work for correctness and can give feedback to students if the teacher desires. Furthermore since all the submissions are recorded into a database, OrganicPad provides a number of post-analysis features to teachers to help facilitate research. OrganicPad is free to download. They are current working on a web version of the tool that can run in the browser.

www.clemson.edu/organicpad/

-Sam Bryfczynski

Link to this comment

Meshca

02/23/10 at 10:45 am

sbryfcz. That's some good info. thanks.

Link to this comment

JonathanDavies

02/23/10 at 11:46 am

hi folks

You may be interested in a chemical drawing tool available on the iPhone/iPod touch that recognizes structure by gesture. Available on the App store

www.Chemjuice.Com

Link to this comment

cafegurl

02/23/10 at 5:48 pm

this will make software a lot more fluid and easy to use. I hope that they create a simply application that will allow you to program the software the way software like typinator works.

Link to this comment

Meshca

02/25/10 at 9:24 pm

I like how there are so many 1 time posters that advertise like jonathandavis, but each time it's actually relevant to the blog post. kudos.

Even though I'm against adds online period, that was actually good advert. etiquette.

Link to this comment

BrynTheSkits

03/12/10 at 1:39 am

Thing is the iPad doesn't have a touch screen that will recognising a stylus, only one that recognises the touch of something that conducts an electric current big enough for it to detect (eg human finger), so there goes that theory for the iPad.

Link to this comment

atlantic-ec

09/21/10 at 9:44 pm

I sense a lot of criticism for this product and for what it is worth I cannot see the mad scientist working on a tiny screen when working through his or her creation.

www.atlantic-ec.com

Popular Tags

Technology

All Tags All Photos All Videos

Technology

+ More Photo Galleries

June 2013: American Energy Independence

Five amazing, clean technologies that will set us free, in this month's energy-focused issue. Also: how to build a better bomb detector, the robotic toys that are raising your children, a human catapult, the world's smallest arcade, and much more.