Hacking Microsoft’s Kinect: Two Weeks In, Six Amazing Projects

Microsoft’s Kinect is amazing. The first time you try the “crank that” Soulja Boy dance on Dance Central, or slam a ping-pong ball in Kinect Sports using only your awkward, flailing arms–those are moments of sheer futuristic glee. The Kinect, as we noted in our review, is definitely lacking in must-have games, but the potential of Kinect is way bigger than merely video games. Luckily, hackers have almost immediately stepped up to free it from the shackles of the Xbox. Here are six of the hacks we’ve seen already.

The Kinect’s combination of hardware and software allows it to do some pretty incredible things. The video sensors basically spit out thousands of little light dots to measure distance and movement, and thanks to the analysis software, are able to do some incredibly impressive things. Just try out _Your Shape: Fitness Evolved_–the game tracks your tiniest movement with near-total precision. Unfortunately, the audio side of things has proved harder to take advantage of, largely because it’s Microsoft’s software that makes the Kinect able to filter out ambient noise, distinguish between speakers, and adjust volume so you never have to yell. But surely that will come–the Kinect has only been available for two weeks, remember.

Still, the Kinect is a limited device at the moment. You can play the few Kinect-specific games that are available and control a few parts of the Xbox experience, but most of the Kinect launch games are underwhelming, and Kinect can’t be used to control the vast majority of the Xbox dashboard. Netflix, especially, is begging for it–just imagine being able to say “Xbox! Play 30 Rock, season two, episode seven, on Netflix,” or swipe through seasons with a wave of the hand. That kind of control is totally possible, but hasn’t been implemented yet.

Luckily, Microsoft’s plans for Kinect, as well as Kinect’s processing software, won’t stop at just a few video games. Kotaku explored some of the things Microsoft’s got in store, which include implementation in other devices entirely (computers, tablets, smartphones, cars), controlling live television, and all kinds of other stuff. But since nobody wants to wait for Microsoft to roll out these new applications, hackers have stepped up to see what can be done with Kinect right now. (And don’t worry–it’s been hacked to work with Mac OS X, too.)

The Hacking

Adafruit, a New-York-based electronics DIY company, put up a bounty for the first person who could successfully hack the Kinect. Less than two weeks later, Microsoft sold more than a million Kinects, and the futuristic sensor bar was officially hacked. The winning hack managed to provide the open source code for the camera’s depth and RGB sensor, and Adafruit later posted their own in-depth guide to hacking the Kinect. Be warned: This is no simple job. To be honest, unless you’ve at one point been described as a “code monkey,” the hack probably isn’t for you.

But getting at the Kinect’s code is imperative to unlocking all the cool stuff it can be made to do. So while this hack may not be something 99.9% of the population can even think about understanding, it’s the key that’ll open the door to everything else. The five hacks that follow all stem from the securing of these drivers.

The KinectBot

The KinectBot might be the most obviously practical creation to come out of Kinect hacking, even if it’s mostly a proof of concept. Created by essentially strapping a hacked Kinect to an iRobot Create (a sort of blank Roomba platform used by robotics developers) along with a gutted Intel Atom computer, the KinectBot uses the Kinect bar’s 3D sensors in ways that’d make an ordinary Roomba hopelessly jealous.

Philipp Robbel over at MIT’s Personal Robotics Group is behind the KinectBot, a concept with possibilities far beyond sweeping up Cheetos crumbs. The KinectBot creates impressively detailed 3D maps of its environment, and beams this data wirelessly to a computer. Thanks to the Kinect’s myriad sensors, KinectBot can detect humans and respond to commands. So not only can it effectively recognize obstacles like furniture and properly maneuver around them, it can understand what you want it to do.

The video above shows that Robbel really gets what the Kinect is about: Using its combination of modern hardware and hyper-advanced software to do incredibly simple, intuitive things. The KinectBot recognizes and responds when you do something as simple as pointing at a corner of the room. This has implications for advanced engineering and military applications, but we’d be satisfied with a consumer model of the KinectBot, really.

The Obligatory (but Nevertheless Awesome) Minority Report Reference

I know, I know. It’s the most tired reference point in computing. But that doesn’t lessen the effect when you actually see gesture-based controls in action. The Kinect may not be sensitive enough to detect individual fingers, but in this early demo, we can see how well it responds to the kinds of multitouch (multi-gesture?) movements we expect. Yeah, it’s simple, and the hacker doing the demo almost seems scornful of how basic this kind of photo manipulation is, but it’s amazing stuff.

It’s highly likely that Microsoft will implement this sort of gesture control into Windows at some point–possibly even in Windows 7, given how mature the gesture-sensing technology is already. It would really liven up PowerPoint presentations, that’s for sure.

[via Gizmodo]

3D Holographic Videos

These Kinect applications are so cool not because there was no way to do this stuff before, but because now these projects can be undertaken with little more than a $150 piece of hardware you can find in any Best Buy. Take this video: I’m sure this kind of 3D holographic mapping has been done before, but I very much doubt it’s been done by one dude in his bedroom using a gadget that’ll be on the wish lists of the world’s teenagers.

The depth sensor and camera combination allow Oliver Kreylos to retain a 3D view of his bedroom while spinning the entire room like a top. Of course, since the camera remains stationary, it can’t see through objects (leading to an incomplete picture), but it’s still an amazing effect. Next, Kreylos says he’s going to attempt to insert computer-generated graphics into real-world 3D video.

Create a Multitouch Surface

Here’s a little Microsoft-aping-Microsoft hack: As it turns out, the Kinect’s sensors allow it to turn any flat surface into, well, a Surface. Microsoft’s Surface is a multitouch display that, despite being undeniably cool, has, due to impracticality, languished as a gimmicky tool in CNN studios and Las Vegas casinos. This video, from Patten Studio, shows how to use the open-source drivers to use the Kinect as a sort of impromptu Surface.

You project a video at any surface, and then point the Kinect at that projection. This allows you to not only move digital images around with a swipe, but also to place real-world objects on the surface and also be recognized.

[via Crunchgear]

Shadow Puppets

The combination of a hacked Kinect and a projector has made for some pretty creative hacks, including the Surface-like example above. A hack just popped up this morning that shows off this combination in a decidedly charming way: Kinect shadow puppets. Theo Watson has created a program with a hacked Kinect that recognizes the elbow, forearm, wrist, fingers, and thumb in the classic “bird” shape.

The project interprets your bird-like arm posture as if you were making a shadow puppet: Opening and closing your hand like a beak causes the cute, bird-from-Up-like avian to do the same, and let off a chirp. This kind of hack is super simple, but surprisingly polished and, obviously, adorable.

[via Crunchgear]