Building A Gesture-Driven Digital Interface

John Underkoffler looks at his desktop computer and sees an outmoded, inadequate tool. Trackpads and touchscreens, the former MIT instructor says, are limiting. Don’t even get him started on the mouse. The future, he says, is in gesture recognition-based interfaces, which he calls “spatial operating environments,” or SOEs.

Underkoffler has already introduced millions to his vision of a working SOE. Remember the scene where Tom Cruise pieces together a crime, gesticulating wildly in front of interactive screens in Minority Report? That was Underkoffler’s idea. Now he’s building and selling working models of his Hollywood version. “This is what the future has to be,” he says. “Otherwise we’re going to end up as little mouse-clutching gnomes, tethered to our desktops.”

In an SOE, infrared sensor strips lining the ceiling read hand gestures and translate them into commands. Point your finger like a gun at a screen, for instance, and the object on the screen sticks with it. The SOE in Minority Report was rendered in such convincing detail that it’s still referenced as a model. Two years after the movie premiered, Underkoffler co founded Oblong Industries. He now serves as its chief scientist and constructs customized SOEs for such clients as Boeing and Aramco. His latest invention is an SOE capable of recognizing finger movements of less than a tenth of a millimeter, so hand commands are much subtler than Tom Cruise’s (since nobody wants to wave his hands around throughout the workday). Installation costs about $200,000.

Underkoffler’s end goal is nothing short of a universal gesture-based language that will allow us to communicate with our ever-growing number of smart devices across multiple platforms. “The world we’re building,” he says, “is one where you can walk up to any screen, anywhere in the world, point to it, and take control.”