Connect with your

Connect with your

• Hi I’m Justin Weinberg• I’m a Carnegie Mellon Graduate• Manager Consultant at Sogeti• Founder of the North Texas Silverlight User

Group

Wow!

Motorized TiltRGB Camera

3D Depth SensorsMicrophone Array

That’s not USB?

You’ll need this..

http://openkinect.org/ http://www.openni.org/

http://research.microsoft.com/kinectsdk/

http://research.microsoft.com/kinect/

http://www.microsoft.com/express

Color Camera Streams

Depth / Depth with Player Streams

Skeletal Tracking

Audio Functionality

http://research.microsoft.com/kinect/

Demo 0Hello Tilt

Kinect NUI Big Picture Runtime nui = new Runtime();

nui.VideoFrameReady += ...

nui.VideoStream.Open(...)

void nui_VideoFrameReady(object sender,  ImageFrameReadyEventArgs e)   {                          //Process e.ImageFrame.       }

1 2 3 4

9 10 11 12

8765

13 14 15 16

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

ImageFrame.Image.BitsWith camera data

you might expect Byte[,]

What you actually get is Byte[]:

BGR32 Formatwidth in pixels

heig

ht in

pix

els Stride: Bytes Per Row

Bytes per pixel * Width

4 Bytes Per pixel

Byte Array:

Demo 1Hello Colorful World

Depth data500mm 500mm500mm 500mm

2000mm2030mm2010mm2000mm

500mm500mm500mm2500mm

Depth dataMax resolution for raw depth data is 640 x 480

2 bytes per pixel of depth data

distance(0,0) = byte0 | byte1 << 8

11011110 pixel pixel pixel

0000101011101010000001011110101000001110

11011110

00001010

Demo 2Depth

Depth + Player Data500mm 500mm500mm 500mm

2000mm2030mm2010mm2000mm

500mm500mm500mm2500mm

player 1

player 1

player 1 player 1player 1

player 1 player 1

Depth + Player DataMax resolution is 320 x 240

11011001 00001011

Player: byte0 & 7

00000001

0000011111011001

&

Depth(0,0) = byte0 >> 3 | byte1 << 5

Still 2 bytes per pixel

But player is last 3 bits in first byte

11011 00001011001

Depth + Player Indexdistance

In mm11 bits per pixel

distance In mm

distance In mm

11011001 XXXX0111player player player

distance In mmplayer

distance In mmplayer

distance In mmplayer

distance In mmplayer player

distance In mmplayer

distance In mm

distance In mm

player

distance In mmplayer

distance In mmplayer

000 = No player001 = Player 1010 = Player 2

11011001 XXXX010111011001 XXXX0111

Demo 3Depth + Player Data

Skeletal Tracking

skeletonFrame.Skeletons                                where skeleton.TrackingState == SkeletonTrackingState.Tracked

skeletonFrame.Skeletons

from skeleton in skeletonFrame.Skeletons                where skeleton.TrackingState == SkeletonTrackingState.PositionOnly

hand_left

wrist_left

elbow_left

shoulder_left

hand_right

wrist_right

elbow_right

shoulder_right

hip_left

head

hip_right

knee_left knee_right

ankle_rightankle_left

foot_left foot_right

hip center

spine

shoulder_center

Skeleton.Joints[JointId enum]Joint.Position (X,Y,Z,W) Skeletons are mirrored

X, Y, Z in left handed cartesian system+Z is out from sensor.W always returns 1 at this moment.

Demo 3Skeletal Tracking

Demo 4Putting Visuals

Together

Kinect Audio

Kinect Audio API

Microphone Array DMO

Implementation

Beamforming and Sound Localization

Features

Beam Forming

Beam Forming-0.875 radians-50 degrees

0.875 radians 50 degrees

.0175 radians

10 degrees

MicArrayModeMicArrayAdaptiveBeam Adaptive beam forming. Raises

beamforming eventsMicArrayExternBeam Your app chooses a beam (0 -10), May be

broke. Use with MicArrayBeamAngle.MicArrayFixedBeam Always use the center beam.MicArraySimpleSum All inputs are used (No beamforming)MicArraySingleBeam Perform beam forming. An internal source

localizer selects the beam.MicArraySingleChan One input is used. Set last 8 bits to choose

the input source.

SoundSourcePosition

SoundSourcePositionConfidence BeamChanged Event

Accousting Echo Cancellation

AEC

bool NoiseFill // Clip removes residual audio but leaves gaps. This fills with noise if true

bool CenterClip // After AEC, some echo residuals will remain. Setting to true removes.

int EchoLength – Sets an adaptive filter to the AEC algorithm (in MS). Recommended values are 128, 256, 512, 1024. For most offices and homes, use 256ms

int FrameSize // Size of a frame in samples Allowed values are : 80, 128, 160, 240, 256, 320. System default for microphone array only is 320All other modes default is 160

int SpeakerIndex – Device index of speaker apply AEC process to

Noise Suppression

Noise Supressionbool NoiseSuppression – True enables, false disables.

Automatic Gain Control

Automatic Gain Control

bool AutomaticGainControl – True enables, false disables.

Demo 5Recording Audio

Audio @!%!#!! Gotchas

Use Microsoft.Speech not System.Speech

Must run in an MTA

AEC SystemModes crash if no speaker output.

Must plug Kinect into Wall

DevicePairGuid / Qualitymetrics = crash

Should run in ThreadMode Priority Highest

Demo 6Speech recognition

Demo 7Kinect Effect

(Thanks SoulSolutions!)

Creative Commons Acknowledgments• Flickr by Ian Lewis• Flickr by major nelson• Flickr by markodesign• Flickr by stephantom• Flickr by Abstract Machine• Flickr By Aixo.co.kr• Flickr By Andresmh• Flickr by Antony Bennison• Flickr by Ars Electronica• Flickr By bm.iphone• Flickr by campuspartymexico• Flickr by Chris Jagers• Flickr By ci_polla Paola Sucato• Flickr by creepyhalloweenimages• Flickr by cvander• Flickr by IMAL.org• Flickr by Institut Douglas• Flickr by Joshua and AmberFlickr by Logon Sakai• Flickr by openexhibits• Flickr by Paul_Clarke

• Flickr by petyr.rahl• Flickr by pt• Flickr by Stilldavid• Flickr by y3rdua• Flickr by yesyesnono• Flickr by yum9me• Flickr by Jeff Howard• Flickr by jhscreenwm• Flickr by Kyle McDonald• Flickr by Morten Rand-Hendriksen• Flickr by Stuart Herbert• Flickr by Visiophone• Flickr by orkomedix• Michael Zöllner and Stephan Huber at the University of Konstanz• Openclip by J_Alves• Openclip by jetxee• Openclip by molumen• Openclip by shokunin• Openclip by Jhnri• Openclip by Machovka• Adafruit image of Hector MartinThanks!!