Kinect in A.I.

Akihiro Eguchi

What is Kinect

a motion sensing input device by Microsoft for the Xbox 360 video game console, which has

◦ RGB Camera

◦ 3D Depth Sensors

◦ Motorized Tilt

◦ Multi-Array Microphone

Kinect Hacks

3D Video Capture with Kinect

http://www.youtube.com/watch?v=7QrnwoO1-8A

KinectFusion - 3D reconstruction

and Augmented Reality

http://www.youtube.com/watch?v=quGhaggn3cQ

Kinect Enabled Autonomous

Mini Robot Car Navigation

http://www.youtube.com/watch?v=NmnepqAQIlk

Some More Links

Top ten Kinect hacks

◦ http://spectrum.ieee.org/automaton/robotics/diy/top-10-robotic-kinect-hacks

Robots use Kinect to understand our world (object recognition)

◦ http://www.newscientist.com/article/dn20723-robots-use-kinect-to-understand-our-world.html

3d models from a Kinect on a car

◦ http://blog.decoratorpattern.com/2011/01/23/real-world-mapping-with-the-kinect/

Kinect RGB Demo v0.3.1

◦ http://nicolas.burrus.name/index.php/Research/KinectRgbDemoV3

Drawing in the air with Kinect

◦ http://vimeo.com/16818988

Teaching Kinect to recognize objects

◦ http://www.youtube.com/watch?v=fQ59dXOo63o

Microsoft Official SDK

Microsoft has released of a non-

commercial Kinect software development

kit for Windows on June 16, 2011.

C++/CLI, C#, or Visual Basic .NET.

Tutorial of how to use the SDK

◦ Kinect for Windows SDK Quickstarts

Object learning

How human learn name of objects

Human children use mainly two biases

when learning name of objects:

◦ Shape bias (Landau, Smith, and Jones (1988))

Generalize name of the object if the shape is similar

◦ Function bias (Nelson, Russsel, Duke, and Jones (2000))

Generalize name of the object if the function of the

use seem to be the same

Previous work in object name

learning Simulation of biases (Grabner1, Gall, Gool (2011))

◦ Shape bias:

Grab 3D models that are labeled with a same name (e.g., chair) and run a machine learning algorithm to train the classifier.

◦ Function bias:

First define the use of the object; e.g., chair is a object to sit on.

Then, let the program learn the posture of sitting.

If the object is sittable, the object is more likely to be a chair.

Shape Bias Implementation with

Kinect Take the depth info. of the environment.

Use RANSAC algorithm to remove the ground the object is placed.

Name the objects and run machine learning to train the classifiers.

Google 3D warehouse should enhance the learning (Lai and Fox (2010))

E.g., of machine learning techniques

K-NN algorithm

◦ Simplest machine learning technique

◦ Store all the input associated with a name.

◦ To identify a new object, take k closest match and take a majority vote to find the most probable name of the object.

Problem:

◦ Memory consumption and speed limitation

One solution:

◦ Use SVM, which is a lot better in the way it does not store every exact data of the input.

Function Bias implementation with

Kinect Kinect has a feature to track 20 different

joints of human body.

◦ Easily be able to draw skeletal body image

So, program can learn a posture of human (like sitting) easily.

Like Grabner1, Gall, Gool (2011), we can add a simple function bias with this way.

Activity learning

How human learn name of action

Children distinguish actions based on

◦ Manner bias

Based on the manner the action includes

Shrink-expand, movement of the hands, and etc.

◦ Path bias:

Based on the path the action takes

E.g., left to right, up to down, and etc.

Pulverman, Hirsh-Pasek, Golinkoff, Pruden and Salkind (2006)

Previous work in action learning

Laptev, and Caputo (2004)

◦ used video recorded action data like walking,

jogging, and running, which is four seconds in

average, and run SVM to train the classifier to

recognize those actions.

Implementation of the action

learning with Kinect records 10 seconds of each action and name

it.

motion data can be represented as a relative distances between all of 20 joints tracked by Kinect.

By running SVM on the cumulated data, program can recognize what kind of action humans are performing in front of the Kinect sensor based on a manner bias.

Path bias can be implemented easily as well.

Activity learning demo on youtube

http://www.youtube.com/watch?v=AxCn0eKWkiQ