Kinect & Leap Motion Overview

Nicholas Gillian

Responsive Environments, MIT Media Lab

Thursday, September 5, 13

Leap Demo

Thursday, September 5, 13

Kinect Libraries, APIs & Tools

• OpenNI SDK

• Openframeworks

• Microsoft Official Kinect SDK

• Synapse

Thursday, September 5, 13

Microsoft Official Kinect SDK

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

Advantages

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Calibration-free skeleton tracking

Advantages

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Calibration-free skeleton tracking

Advantages

• Standing and seated tracking modes

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Calibration-free skeleton tracking

Advantages

• Standing and seated tracking modes

• Joint Orientation in addition to Joint Position

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Calibration-free skeleton tracking

Advantages

• Standing and seated tracking modes

• Joint Orientation in addition to Joint Position

• Tracks up to 6 users, 2 users full skeleton tracking

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Calibration-free skeleton tracking

Advantages

• Standing and seated tracking modes

• Joint Orientation in addition to Joint Position

• Speech Recognition

• Tracks up to 6 users, 2 users full skeleton tracking

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Calibration-free skeleton tracking

Advantages

• Standing and seated tracking modes

• Joint Orientation in addition to Joint Position

• Speech Recognition

• Face tracking (feature tracking - i.e. mouth, nose, eyes, etc. and direction estimation)

• Tracks up to 6 users, 2 users full skeleton tracking

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Calibration-free skeleton tracking

Advantages

• Standing and seated tracking modes

• Joint Orientation in addition to Joint Position

• Speech Recognition

• Face tracking (feature tracking - i.e. mouth, nose, eyes, etc. and direction estimation)

• Good examples (c++, c#) and user documentation

• Tracks up to 6 users, 2 users full skeleton tracking

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

AdvantagesThursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

Disadvantages

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Only works on Windows

Disadvantages

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Only works on Windows

Disadvantages

• Very CPU intensive

Thursday, September 5, 13

Microsoft Official Kinect SDK

• http://www.microsoft.com/en-us/kinectforwindows/

• Only works on Windows

Disadvantages

• Very CPU intensive

• Skeleton tracking can fail drastically if not used in ideal scenarios

Thursday, September 5, 13

Open NI

Thursday, September 5, 13

Open NI

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

Thursday, September 5, 13

Open NI

Advantages

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

Thursday, September 5, 13

Open NI

• Cross Platform

Advantages

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

Thursday, September 5, 13

Open NI

• Cross Platform

Advantages

• Works with a large number of depth sensors

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

Thursday, September 5, 13

Open NI

• Cross Platform

Advantages

• Works with a large number of depth sensors

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

• Robust Skeleton Tracking

Thursday, September 5, 13

Open NI

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

• Cross Platform

Advantages

• Works with a large number of depth sensors

• Robust Skeleton Tracking

• Calibration-free skeleton tracking

Thursday, September 5, 13

Open NI

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

AdvantagesThursday, September 5, 13

Open NI

Disadvantages

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

Thursday, September 5, 13

Open NI

Disadvantages

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

• Installation requires a few independent drivers, libraries, etc.

Thursday, September 5, 13

Open NI

Disadvantages

• Installation requires a few independent drivers, libraries, etc.

• API requires some time to digest

• http://openni.org/

• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.

Thursday, September 5, 13

Synapse

Thursday, September 5, 13

Synapse

• http://synapsekinect.tumblr.com/

Thursday, September 5, 13

Synapse

Advantages

• http://synapsekinect.tumblr.com/

Thursday, September 5, 13

Synapse

• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)

Advantages

• http://synapsekinect.tumblr.com/

Thursday, September 5, 13

Synapse

• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)

Advantages

• http://synapsekinect.tumblr.com/

• Very easy to use - OS X version requires no setup, Windows version needs drivers to be installed

Thursday, September 5, 13

Synapse

• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)

Advantages

• http://synapsekinect.tumblr.com/

• Very easy to use - OS X version requires no setup, Windows version needs drivers to be installed

Disadvantages

Thursday, September 5, 13

Synapse

• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)

Advantages

• http://synapsekinect.tumblr.com/

• Very easy to use - OS X version requires no setup, Windows version needs drivers to be installed

Disadvantages

• Skeleton tracking needs calibration pose

Thursday, September 5, 13

Synapse

• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)

Advantages

• http://synapsekinect.tumblr.com/

• Very easy to use - OS X version requires no setup, Windows version needs drivers to be installed

• Only works with “Kinect XBox360” hardware

Disadvantages

• Skeleton tracking needs calibration pose

Thursday, September 5, 13

Openframeworks

Thursday, September 5, 13

Openframeworks

• http://www.openframeworks.cc/

• https://github.com/ofTheo/ofxKinect

• Uses the libfreenect library

Thursday, September 5, 13

Openframeworks

Advantages

• http://www.openframeworks.cc/

• https://github.com/ofTheo/ofxKinect

• Uses the libfreenect library

Thursday, September 5, 13

Openframeworks

Advantages

• http://www.openframeworks.cc/

• Easiest way to access the raw depth image from the Kinect

• https://github.com/ofTheo/ofxKinect

• Uses the libfreenect library

Thursday, September 5, 13

Openframeworks

Advantages

• http://www.openframeworks.cc/

• Easiest way to access the raw depth image from the Kinect

• https://github.com/ofTheo/ofxKinect

• Supports multiple Kinects

• Uses the libfreenect library

Thursday, September 5, 13

Openframeworks

Advantages

• http://www.openframeworks.cc/

• Easiest way to access the raw depth image from the Kinect

• https://github.com/ofTheo/ofxKinect

• Supports multiple Kinects

• Uses the libfreenect library

Thursday, September 5, 13

Openframeworks

Advantages

• http://www.openframeworks.cc/

• Easiest way to access the raw depth image from the Kinect

• https://github.com/ofTheo/ofxKinect

• Supports multiple Kinects

• Works on OS X

• Uses the libfreenect library

Thursday, September 5, 13

Openframeworks

Advantages

• http://www.openframeworks.cc/

• Easiest way to access the raw depth image from the Kinect

• https://github.com/ofTheo/ofxKinect

Disadvantages

• Supports multiple Kinects

• Works on OS X

• Uses the libfreenect library

Thursday, September 5, 13

Openframeworks

• No skeleton tracking

Advantages

• http://www.openframeworks.cc/

• Easiest way to access the raw depth image from the Kinect

• https://github.com/ofTheo/ofxKinect

Disadvantages

• Supports multiple Kinects

• Works on OS X

• Uses the libfreenect library

Thursday, September 5, 13

Kinect Gesture Demo

Thursday, September 5, 13

Kinect Gesture Demo

Built Using:

Thursday, September 5, 13

Kinect Gesture Demo

• OpenNI API to interface with the Kinect

Built Using:

Thursday, September 5, 13

Kinect Gesture Demo

• OpenNI API to interface with the Kinect

• NITE library for calibration-free skeleton tracking

Built Using:

Thursday, September 5, 13

Kinect Gesture Demo

• OpenNI API to interface with the Kinect

• NITE library for calibration-free skeleton tracking

• OSC (Open Sound Control) to send skeleton data to external apps

Built Using:

Thursday, September 5, 13

Kinect Gesture Demo

• OpenNI API to interface with the Kinect

• NITE library for calibration-free skeleton tracking

• OSC (Open Sound Control) to send skeleton data to external apps

Built Using:

OSC

Thursday, September 5, 13

Kinect Gesture Demo

• OpenNI API to interface with the Kinect

• NITE library for calibration-free skeleton tracking

• OSC (Open Sound Control) to send skeleton data to external apps

Built Using:

• Openframeworks as a wrapper for drawing and networking

OSC

Thursday, September 5, 13

Kinect Gesture Demo

• OpenNI API to interface with the Kinect

• NITE library for calibration-free skeleton tracking

• OSC (Open Sound Control) to send skeleton data to external apps

Built Using:

• Openframeworks as a wrapper for drawing and networking

• Gesture Recognition Toolkit: www.nickgillian.com/software/grt

OSC

Thursday, September 5, 13

www.nickgillian.com/09-05-13.html

Slides & Software Links

Thursday, September 5, 13