Deep in Microsoft's lairs, the Xbox 360 team is working on more than just a new video-game system. They're actually trying to solve an incredibly difficult problem in artificial intelligence. Their prototype Project Natal lets you control a game just with your body movements—no buttons or Wii-like wands—by watching you with a 3-D video camera. Sounds simple enough, but most cameras just snap images without having any idea what they're looking at. To make Natal work, Microsoft has to teach its camera to understand what it sees.
Here at CES, Microsoft announced last night that Natal will go on sale "by the holidays." Before the show, we were given an exclusive look at the smarts that make Natal tick.
The part of Natal that players see looks like a webcam. (Microsoft's not divulging details about this hardware yet, presumably because the release is many months in the future, but we do know that it measures relative distances using a black-and-white camera sensor and an near-infrared beam.) But it's the software inside, which Microsoft casually refers to as "the brain," that makes sense of the images captured by the camera. It's been programmed to analyze images, look for a basic human form, and identify about 30 essential parts, such as your head, torso, hips, knees, elbows, and thighs.
In programming this brain--a process that's still going on—Microsoft relies on an advancing field of artificial intelligence called machine learning. The premise is this: Feed the computer enough data—in this case, millions of images of people—and it can learn for itself how to understand it. That saves programmers the near-impossible task of coding rules that describe all the zillions of possible movements a body can make.
The process is a lot like a parent pointing to many different people's hands and saying "hand," until a baby gradually figures out what hands looks like, how they can move, and that, for instance, they don't vanish into thin air when they're momentarily out of sight.
How To Teach A Machine To See
Microsoft is currently training and improving the version of the brain that will ultimately go into the final product. How? By painstakingly gathering pictures of people in many different poses, and then running all this data through huge clusters of computers (as shown in the gallery) where the learning brain resides.
The process of gathering the data actually requires a lot of manual labor. First, reps went into homes around the world and recorded people moving in front of a specially built rig. The images captured are real people moving the way any ordinary person would. But those recordings can't tell the computer anything useful about joints and limbs on their own, so programmers dive into the raw data and hand-code it to label each body part (at each frame!).
Microsoft also uses professionally staged motion-capture scenes, which provides similar data but without all the manual labor of coding by hand (since the systems use sensors that mark individual body parts). And Microsoft has a mini mo-cap studio of its own, where staff can make a quick recording when a new chunk of data is needed.
All of these marked-up images comprise tens of terabytes of information. Microsoft's computer farms sift through this huge data set, letting the brain come up with probabilities and statistics about the human form. Once the brain is done learning, it and its stats get packaged into the Natal system. An early version is now making the rounds of trade shows, and later, more-accurate versions will eventually show up in your living room. Next, read about how it applies its hard-earned knowledge to decipher your game-playing moves.
Inside Natal's Thought Process
What's the brain thinking as it watches you jump around, swinging imaginary bats or head-butting imaginary soccer balls? The above screenshot shows what's going on in it's head—the different images represent different stages of Natal's computational process. Here's the step-by-step:
Step 1: As you stand in front of the camera, it judges the distance to different points on your body. In the image on the far left, the dots show what it sees, a so-called "point cloud" representing a 3-D surface; a skeleton drawn there is simply a rudimentary guess. (The image on the top shows the image perceived by the color camera, which can be used like a webcam.)
Step 2: Then the brain guesses which parts of your body are which. It does this based on all of its experience with body poses—the experience described above. Depending on how similar your pose is to things it's seen before, Natal can be more or less confident of its guesses. In the color-coded person above [bottom center], the darkness, lightness, and size of different squares represent how certain Natal is that it knows what body-part that area belongs to. (For example, the three large red squares indicate that it's highly probable that those parts are "left shoulder," "left elbow" and "left knee"; as the pixels become smaller and muddier in color, such as the grayish pixels around the hands, that's an indication that Natal is hedging its bets and isn't very sure of its identity.)
Step 3: Then, based on the probabilities assigned to different areas, Natal comes up with all possible skeletons that could fit with those body parts. (This step isn't shown in the image above, but it looks similar to the stick-figure drawn on the left, except there are dozens of possible skeletons overlaid on each other.) It ultimately settles on the most probable one. Its reasoning here is partly based on its experience, and partly on more formal kinematics models that programmers added in.
Step 4: Once Natal has determined it has enough certainty about enough body parts to pick the most probable skeletal structure, it outputs that shape to a simplified 3D avatar [image at right]. That's the final skeleton that will be skinned with clothes, hair, and other features and shown in the game.
Step 5: Then it does this all over again—30 times a second! As you move, the brain generates all possible skeletal structures at each frame, eventually deciding on, and outputting, the one that is most probable. This thought process takes just a few milliseconds, so there's plenty of time for the Xbox to take the info and use it to control the game.
(If you want to get into more details on the science, check out the machine-learning papers of Microsoft researcher Andrew Blake, on whose work Natal is partly based.
Wow. I thought this would just be the next Eye Toy, but this is really promising to deliver a lot more than that. Of course, it's going to be hard to make a really accurate golf or bowling simulation if there's nothing to hold on to. Especially for bowling - it'd be hard to get the proper amount of spin on the ball.
You know what I love to do? Run three miles in place in my living room. (Sarcasm)
It is true that this is quite amazing in its design, but it is not really practicle for gamers.
Halo, one of the most popular and reconized games, would be impossible on this system. Alot of it would have been on rail, as in a path has already been set for you, and you just control the upper body movements.
how would you play traditional video games? you can't race without a way to control the brakes, gas and steering. and all the war games? its hard to fire a gun without a trigger. like thor0997 said, its an amazing design, but the only thing you could do is run. video game controllers might be annoying because they are always running out of battery, or losing them, or something, but that allow you to play lots of the games you love.
Are you guys kidding me? Are you really that close minded to think that you would have to use this tech without a controller? I am sure some niche games will come out that use no controller, but the possibilities of whats possible with this AND a controller and incredible.
People made such a big deal out of the six-axis controller, but this puts that to shame...
Lets give Microsoft a chance and see if they can really pull this off. (Hopefully its not just vaporware and they include a controller)
I don't think its true to say that you CAN'T play racing games and action games (halo) using natal. In one of Microsoft's first videos showed a family playing a racing game. This video was a great demonstration of gestures that could be used with natal. I don't think you will be able to playing older video games (halo 3)that well using natal. but maybe what ever new games come out in the near future (halo) will have new gesture controls to use with natal. This is amazing technology at work, after all its just a video console accessory. Lets just hope Microsoft gets the hardware, gestures, and marketing right.
I would also like to add to benny476's comment. Natal can also scan objects such as a ball, toy, stick, or even a skate bored you like. But lets not forget facial recognition that natal has to offer. This again adds to the possibility's natal has to offer.
Well said, Bizzy.
Saying you won't be able to play halo on this is like asking the inventor of the automobile "But how will it stay on the rails?"
Do we forget why controllers exist in the first place? They are a highly imperfect translator of our wants. If we could have controlled Mario with just our hands, do you think Nintendo would have given us a little tethered plastic box to mash buttons on? Are analog sticks, shift buttons, and switches somehow intuitive?
Think outside the (x)box!
I have to comment again because I'm pretty excited about this and just thought of something else.
I don't know how detailed it is, but the potential is that if you can scan anything, couldn't ANYTHING be a controller? I can imagine (assuming detailed movement detection) playing a guitar game, a shooting game, a melee fighting game, etc. with just a tennis racquet.
In fact, 50$ says when it comes out, they'll just show someone picking up a regular object and using it as a guide in whatever game they're playing.
Can I see where this is going? Assisting a robot to walk or run. It's rather somewhat the same as a child learning without the long learning curve if it can be programmed along with an assisted balance controler!!!
While I doubt we get much from this piece of tech, it is an important step into a deeper imersive system. The reason we won't get too much from it is some of the limitations in factors outside of the device itself - it demands that other advances be made to match up with it.
Obviously, this will record and impliment movement (jumping, swinging a bat or sword, etc). What it wants to do, however, is also control perspective - the direction you avatar is looking and the display in front of you. Unfortunately, while this device could likely render that easily, display isn't there yet - needs either a display helemet (VR style) or 4 wall display. Niether of those is near production.
You will still need a controller to pull a trigger, of course, but why wouldn't you want one? If realism is your goal, you want a trigger to pull to mimic your trigger pull. I would like mine with a suped-up vibrator for some real world kick.
You will also still need a controller for movement. While a program could likely tell if your leg movements indicated forward, backward, or sideways movement, the limitations of room space dictate that you must be able to translate leg movements into character movements without physically moveing around the room. Giant hamster balls floating in kiddie pools are not a solution.
While at home, 4 wall projection, 360 degree, helmetless, motion captured, 1st person gaming is still far away, this is a very large step to that becoming a plausable technology in an arcade environment.
The first gen games will likely be cliche and weak, the tech in the software will open up signifigant advances in a few gens.
MS rocks at virtual technology.
the most this is going to do is compete with the wiiFit crap that came out. DDR revolution! tennis! uh.. uh.. what else? It's a mistake to apply so much effort and resources to a casual game market. Here's an idea, if you need exercise, you can get it free!! Just go outside! Play some real basketball or tennis. durrrr
Why does everyone think movement with natal such a daunting task. Does your thumb run out of space? Why can't it just recognize you in say a five foot radius. The neutral spot would be center. The close you go to the edge of the radius the faster it moves. Trust me after awhile everyone would be trained to recognize the distance instinctively. It's like your hand moving to play different chord shapes on a guitar.
You throw both hands to jump but should be program for physical jumping as well.
As for shooting, while the tactile part isnt there. Hold your hand up as if but comfortably. What do you see? Most will never point a finger out but the thumb is the point they could track. Now shooting is as easy as actually moving your trigger finger. Rapid fire middle and trigger(pointer) in an altering pattern.
I will be honest with you wiimotes and that wand while great for tennis and sword-fighting things of that nature is wrong for shooting games. The best thing about wii is the design allowed for easy attachments. If sony isn't thinking about that i see carpal injuries occurring. Grab anything in your hand and point straight. You will notice that it can be actually painful to point downward. See Ulnar Deviation.
In other words it's easier and less stressful to hold nothing than something that's ergonomically incorrect. Heck if you need to hold something hold a stress sponge
I really like the comments here. I can't really think out of the box for situations like this, so it's interesting to see what the rest of you think about it.
Personally I play RPGs so facial recognition might be cool or gestures if it can recognize all that. I'm sure they'd have some other gesture for running than actually running, similar to what gameonbro said, maybe if you're closer it auto moves forward and further moves back and so forth. I personally really like that idea.
Outside of gaming, this interface could be used (I'm thinking Milo here) to make interfaces seem more personal, like electronically buying movie tickets at the theatre or checking out at Target in the self checkout line. If MS performs this right, I foresee many profits.
I'm surprised that some people here cannot think outside the box, or even into the realm of common logic! It would be rather obvious that they wouldn't just sell Natal without some sort of external controller. While I'm not saying that this controller would be anything akin to Microsoft Xbox 360's hand-held controller, it may involve physical manipulation for certain games such as first-person shooters. I admit that creating a shooter would pose a few challenges, but if you think about it, the answers are fairly simple. According to a few other sources, and some from CES itself, there are controllers now being made as a glove prototype (i.e. gyros mounted on the back of your hand, connected to small pieces of material running the length of your fingers to incidate tension), that could easily be combined with Natal to create as accurate a shooter as anyone would want. Everything else would be simple (aiming, running, turning, etc.). I understand that there would be problems, but I would hedge my bets that if you came up with a hardware, software, etc. problem, most likely the technicians at Microsoft did too.
personally, I wouldn't mind holding a small, perhaps nunchock(wii) like object if it allowed me to play shooters. obviously the first few shooters will be awful- much like the first few shooters that came to the wii, but some innovative team- maybe even an indie group will create a shooter that will truly shine.
also, many people underrate the capability of the natal camera. chances are good that it will be able to see your finger pulling a trigger if you make it obvious enough
another thing that natal could make great is RPGs. I could really see bethesda softworks taking a leap and making oblivion V really incredible using natal.
I also thought it would be just an itoy like the playstations. But WOW!!!! That looks so amazing. Would also like to know how smooth it will work. I'm guessing that since they are taking so long and have so much hype, that it will actually work. Very cool though I must say. Will really put the Wii to shame. If of course it actually works. The Wii just lets you moves your hands, but this is the real deal. Nintendo will have to think of a comeback. (If this piece of technology actuall works and is successful, then Project Natal 2 should be even better. Pinpointing every little part of your body. Possibly even have twice the amount of sensors or even adding new ones.)
how will the wii compete??
I think that the key problem here is that some people are thinking a little bit to literally. While it would feasibly be possible to exactly control the movement of your character in shooters, such as looking around and moving this is not necessary. There is a happy median between controllers and exact duplication where one simple has to step forward to begin your character moving, changing foot position for turning, and return to a baseline to stop just as you use the joysticks on your controllers now. I know that some might say that this seems like going nowhere since you still have to use approximations to a achieve your goals, but this can allow for much finer interpretations of your wishes and for a much wider variety of movements than can be interpreted from a standard button and joystick controller. Project Natal may not be everything that we imagine for the future of gaming, but then again, we imagine a lot to become of gaming, and this is a lot closer than any other system to be sure.
If you get the chance, go to YouTube and look up the demos from E3 2009 and Late Night with Jimmy Fallon. In both demos they play a version of Burnout Paradise modified to work with Natal. Steering is controlled by holding your hands as you would a steering wheel. You turn the same way you would in a regular car, acceleration is performed by moving your right foot forward(as if pressing the gas peddle) and braking is controlled pretty much the same way. With a little practice, I could actually see a game like Burnout being MORE fun with Natal than a regular controller. If the tech can pick up on really subtle movements and adjustments to your body like it appears to be able to from the PaintParty and 3D Brickout demos, this could really be the next evolution of the video game controller. I even plan to pay whatever the ridiculous launch price is just to be able to get the experience of the early games and see how game design either gets better or worse with the add on.
It looks cool. If it works, I play mostly RTS and FPS games seeing how it performs in these areas will be interesting
One thing I'm excited about is the uses for this beyond simple gaming. Just think:
-Golf tutor - swing analysis
-Teaching aid - do dangerous chemical experiments without the danger.
-Surgery teaching - perform surgery without 'dead' patient.
I could go on - there are so many real world applications to technology like this...taking it further....
-Control a robot - the robot could be anywhere - Mars, in a war zone etc - I'm sure the Army will be very interested in this techonology if they aren't already using it!!!
Project Natal's general method for full-body motion capture is described. However, for me, one question still needs to be answered.
How will we swing a sword?
See, nothing in that article implies that Natal will be capable of determining the twist angle of the player's wrist relative to their arm. Unless we know that, it's impossible to accurately determine what direction a virtual sword is pointing relative to the "hand" block shown in the 3D primitive model Natal builds.
While it appears to be true that if your fingertips are significantly far away and off-angle from your wrist that the virtual hand's box will move slightly, this is likely not indicative of an ability to determine the twist angle mentioned above.
What I want to know is: Did the people who were treated to the "exclusive look" at Natal happen to notice if it could accurately determine that twist angle?
Somehow I expect this to not work as well in real life as it does in the demos.
It'll probably go the way of voice commands in games which have been tried off an on since the 80s.
This is microsoft we're talking about.
For FPS, what if you used a fake gun that came with Natal, and could tell the console when you pull the trigger
I'll tell you how you would fire a gun, because it would recognize your stance in a rifle holding position. or gun holding position. Certain games would obviously come with a way to tell the computer to look for more war game skeletons. The sensor may even be able to see you mime the trigger being pulled with your fingers. As to walking and running in place that craps for the birds but could be solved with a simple floor mat that has a dead zone center and 360 degree sensor around it. step forward to walk forward. or the sensor could tell you of a predetermined standing area that it sees. you step so far forward it will make your guy walk till you step back. Camera direction would follow the swivel of your own head. It really is just a matter of patience and clever recognition software. Lets just hope they put the time into it that it deserves. Hope it doesnt get rushed to market with to many limitations.
what all of yall are forgetting is that this is a first generation platform. Now granted i am a young college kid who has plenty of time in life to see this technology evolve to be even better and even more time to waste using it, I still do have my doubts but the fact is we already have a press release on page one of the Gaming section on this website which a literal human track ball exists now comprise that technology with at minimum of a 180 degree panorama projection inside the virtual ball and top it off with natal technology you will be looking at the pinnacle of video gaming i mean all the technology involved in that idea does exist and god willing the gamer in me is dying to experience that and yall are being narrow minded on the idea of how to control a fps or even the obsession of a sword? haven't any of yall watched the movie "Gamer" i mean its really not that hard to believe that Microsoft hasn't figured at least one solution to these problems it probably has a team of at least a hundred people that just sit in a room thinking of the same things you are right now and i guarantee that they will do there best to solve and improve your ideas i wouldn't be surprised if they haven't even read this article just to see yalls concerns and if they had thought about those issues themselves. just remember this is only a door opening to what could be so don't be so hasty to dismiss the idea of this being a leap forward in the gaming world.
Ok, I personally think this is kick ass. Except I like to get out of the house and bang my girlfriend... I don't think I can do that through video. And show a motion video with an actual good game like BFBC2.
thank you very much
<a href="http://www.tran33m.com/vb/">منتديات</a>,<a href="http://www.tran33m.com/vb/">منتدى</a>,<a href="http://www.tran33m.com/vb/tags/2011.html">2011</a>,<a href="http://www.tran33m.com/vb/tags/2010.html">2010</a>,<a href="http://www.tran33m.com/vb/tags/1431.html">1431</a>,<a href="http://www.tran33m.com/vb/">1432</a>,<a href="http://www.tran33m.com/vb/f69.html">مباراة</a>,<a href="http://www.tran33m.com/vb/f69.html">اهداف</a>,<a href="http://www.tran33m.com/vb/f10.html">برنامج</a>,<a href="http://www.tran33m.com/vb/f37.html">يوتيوب</a>,<a href="http://www.tran33m.com/vb/f18.html">للمسن</a>,
<a href="http://www.tran33m.com/vb/f10.html">تحميل</a>,<a href="http://www.tran33m.com/vb/f37.html">انشودة</a>,<a href="http://www.tran33m.com/vb/f24.html">صورة</a>,<a href="http://www.tran33m.com/vb/f69.html">الهلال</a>,<a href="http://www.tran33m.com/vb/f69.html">النصر</a>,<a href="http://www.tran33m.com/vb/f69.html">الاتحاد</a>,<a href="http://www.tran33m.com/vb/f69.html">الاهلي</a>,<a href="http://www.tran33m.com/vb/f69.html">اهداف الهلال</a>,<a href="http://www.tran33m.com/vb/f69.html">اهداف النصر</a>,<a href="http://www.tran33m.com/vb/f69.html">اهداف الاتحاد</a>,
<a href="http://www.tran33m.com/vb/f69.html">اهداف الاهلي</a>,<a href="http://www.tran33m.com/vb/f69.html">اهداف مباراة</a>,<a href="http://www.tran33m.com/vb/f21.html">دوري زين</a>,<a href="http://www.tran33m.com/vb/f69.html">الدوري الاسباني</a>,<a href="http://www.tran33m.com/vb/f69.html">الدوري الايطالي</a>,<a href="http://www.tran33m.com/vb/f69.html">الدوري الانجليزي</a>,
<a href="http://www.tran33m.com/vb/f69.html">دوري ابطال اوروبا</a>,<a href="http://www.tran33m.com/vb/f21.html">بث مباشر</a>,<a href="http://www.tran33m.com/vb/f21.html">رابط نقل</a>