Week 5-Kinect software

The Kinect Software

There are many elements to the software that is part of the Kinect game program, but with out a doubt, the software that frees the user from input devices that must be handled  is what gives true meaning to the device.  When the developers were looking for ways to maximize the users’ experience they make a conscious decision NOT to pre-program motion into canned actions and reactions into the software.  Instead, they did something amazing.  They decided to “teach” the program to learn by categorizing and classifying real people in the real world. As a result:

“Every single motion of the body is an input,”

– Alex Kipman, Microsoft’s director of Project Natal

The developers used countless people of all different ages, sizes, even wearing different kinds of clothing to create the massive database the software would “learn” from.  The algorithyms for this program were written by Jamie Shotton, a member of Project Natal, the Cambridge based Microsoft research team which developed the Kinect software.  The process by which it does that is outlined below.

Step 1: As you stand in front of the camera, it judges the distance to different points on your body. In the image on the far left, the dots show what it sees, a so-called “point cloud” representing a 3-D surface; a skeleton drawn there is simply a rudimentary guess. (The image on the top shows the image perceived by the color camera, which can be used like a webcam.)

Step 2: Then the brain guesses which parts of your body are which. It does this based on all of its experience with body poses—the experience described above. Depending on how similar your pose is to things it’s seen before, Natal can be more or less confident of its guesses. In the color-coded person above [bottom center], the darkness, lightness, and size of different squares represent how certain Natal is that it knows what body-part that area belongs to. (For example, the three large red squares indicate that it’s highly probable that those parts are “left shoulder,” “left elbow” and “left knee”; as the pixels become smaller and muddier in color, such as the grayish pixels around the hands, that’s an indication that Natal is hedging its bets and isn’t very sure of its identity.)

Step 3: Then, based on the probabilities assigned to different areas, Natal comes up with all possible skeletons that could fit with those body parts. (This step isn’t shown in the image above, but it looks similar to the stick-figure drawn on the left, except there are dozens of possible skeletons overlaid on each other.) It ultimately settles on the most probable one. Its reasoning here is partly based on its experience, and partly on more formal kinematics models that programmers added in.

Step 4: Once Natal has determined it has enough certainty about enough body parts to pick the most probable skeletal structure, it outputs that shape to a simplified 3D avatar [image at right]. That’s the final skeleton that will be skinned with clothes, hair, and other features and shown in the game.

Step 5: Then it does this all over again—30 times a second! As you move, the brain generates all possible skeletal structures at each frame, eventually deciding on, and outputting, the one that is most probable. This thought process takes just a few milliseconds, so there’s plenty of time for the Xbox to take the info and use it to control the game.

Sources:

http://electronics.howstuffworks.com

Exclusive: Inside Project Natal’s Brain

The artificial intelligence behind Microsoft’s Xbox 360 motion-sensing game controller

By Jill Duffy (http://www.popsci.com/gadgets/article/2010-01/exclusive-inside-microsofts-project-natal)