Video: Kinect Hack Recognizes Buttons Drawn With Ink and Paper

Draw your own instrument, then play it

Hand-Drawn Kinect Accessory

Garratt Gallagher

Willow Garage recently held a contest to see who could use their open-source Robot Operating System in conjunction with a hacked Microsoft Kinect in the most interesting or useful way. The first place winner is this Customizable Buttons hack. In short: draw buttons with a marker on a piece of paper, then press them. The Kinect recognizes the buttons and the pushing of said buttons, and executes commands--in this case, music. It's like a childhood dream come true.

Willow Garage is a California-based robotics research lab that's probably best known for their creation of the PR2 robot, a very advanced robot that can autonomously fold laundry, open doors, shoot pool, and, best of all, fetch its creators a beer of their choosing. This ROS contest was designed to find the most creative and useful hacks the Kinect community could come up with, and Garratt Gallagher's customizable buttons hack won handily.

The interface requires a white or just barely off-white paper and a black or very dark pen that can draw in thick strokes. It uses ROS and the OpenNI Kinect driver, and seems like a fairly simple hack, as far as these things go. (Willow Garage is nice enough to provide instructions and demos on their contest page.) It uses the Simple Fast Media Library for sounds, though it's not unreasonable to think other versions could be made that use just about any sound you want.

What we really like about this hack, besides its combination of childlike whimsy and modern technology, is that it actually capitalizes on one of the Kinect's advertised but unused abilities. In early demos, Microsoft boasted of the Kinect's ability to recognize objects as well as human bodies--you could hold up a baseball bat, the Kinect would map it, and then it could show the (highly dangerous) movement of your real-life bat in a game. But no currently available Kinect game, to my knowledge, takes advantage of this ability, so it's great to be reminded of it. Hopefully it'll spur Microsoft to make a version of their own--the possibilities are pretty much endless, especially if you can draw your own in-game tools.