KINECT SENSOR

Group members

Bhoomit Morkar 14103180Sachin Keniya 14103170Vaishnavi Shah 14103182

Ears and Eyes of your application

Content Introduction

• Basic Construction• Basic Specifications• Kinect’s Potential and applications

Working of Kinect• Classification of Depth Measurement Techniques

Triangulation Principles of TOF Imaging Principle of structured light

• Skeletal Tracking

Applications

References

Introduction

• It was Designed by Microsoft for the Xbox 360 video game console and windows PC’s.

• While making the Kinect sensor , Microsoft’s aim was to enable interactions between users and a game without the need to touch a controller.

• Kinect is an RGB-D sensor providing synchronized color and depth images.

• It’s greatest differentials is depth sensing technology that could be extended far beyond gaming and at a much lower cost than traditional 3-D cameras.

Why Kinect is revolutionary ?

• Kinect has proven to be an incredible piece of technology.

• It is the first practical natural user interface for machines communicating with people.

• It is combining acoustical and visual sense.

• It is recognizing full body movements, even complicated ones.

• It finds application not only in games but also in the field of medical sciences and robotics.

• After selling a total of 8 million units in its first 60 days, the Kinect holds the Guinness World Record of being the "fastest selling consumer electronics device".

Basic Construction

Power Comes from the Sum

• The Kinect Sensor is the outcome of several devices(or sensors) working together.

• It is comprised of a color camera (also known as RGB), a depth sensor, a 4-microphone array, and a motorized tilt.

• The depth sensor, in itself, consists of an infrared (IR) projector and an IR camera.

• Due to that Kinect being a motion sensing input device, able to capture, track, and interpret full-body movements(3D), and recognize gestures and voices.

• The depth sensor is a monochrome complementary metaloxide semiconductor (CMOS) sensor.

• One of Kinect’s greatest differentials is its relatively low cost depth sensor.

• The depth-sensing technology is licensed from the Israeli company PrimeSense. Although the exact technology is not disclosed, it is based on the structured light principle.

• The IR projector is an IR laser that passes through a diffraction grating and turns into a set of IR dots.

• The sensor has an RGB (red-green-blue) camera for color video.

• It has four microphones that enable your application to respond to verbal input, in addition to responding to movement.

Basic Specifications

Hardware

• Depth resolution : 640x480 px • RGB resolution : 1600x1200

px• FrameRate : 60 FP

Software• Depth;

30 FPS: 80x60 , 320x240 , 640x480

• Color;12 FPS : 1280x980 RGB15 FPS : Raw YUV 640x48020 FPS : 640x480

Kinect’s Potential and applications

Depth sensor Skeletal tracking Microphone array

Background removalObject recognition

Multi-user Easy Gesture Recognition

Sound source detection Speech recognition

How it works ?

Audio input

Microphone array

Sound threshold

How does Kinect hear ?

Track me !

Or

Speak louder .

what does it see?

Near mode depth ranges

Default mode depth ranges

Angle of vision(RGB-D)

Area under Observation

Classification of Depth Measurement Techniques

Triangulation

• In trigonometry and geometry, triangulation is the process of determining the location of a point by forming triangles to it from known points.

• Here rather than measuring distances to the point directly as in trilateration; we use of both angles and distance

Basic Mathematical model for Kinect

Principles of TOF Imaging

• Time-of-Flight (TOF) Imaging refers to the process of measuring the depth of a scene by quantifying the changes that an emitted light signal encounters when it bounces back from objects in a scene.

Pulsed ModulationContinuous Wave Modulation

Principle of structured light

• Project a known light pattern into the 3D scene, viewed by cameras.

• Distortion of light pattern allows computing the 3D structure.

• This pattern has to be unique per position in order to recognize each point in the pattern.

• Kinect uses pseudo random pattern.

speckles

Based on distance and orientation

First Region

High accurate depth.

(0.8 – 1.2 m)

Second Region

Medium accurate depth.

(1.2 – 2.0 m)

Third Region

Low accurate depth.

(2.0 – 3.0 m)

Depthobserved patternCalibration

IR camera observes the scene

known pattern

projected

3D map of the

beginning frame

New 3D map of the

scene

Renew 3D map of the

scene

Between projector

and camera&

x-direction speckle shift

Precision

Spatial x/y resolution

3mm @ 2m distance

Depth Z resolution

1cm @ 2m distance 10 cm @ 4m distance

operation range

0.8m ~ 4m | 0.5m ~ 3m

Skeletal Tracking

In the beginning, it was justhands;

• Head and hands,• Added shoulder and elbows.

Today!• It can track 20 joints of our body and

rans fast.

Process

Modes

Applications

• The Kinect fusion project allows a user to use an off-the-shelf Kinect camera as a 3D scanner for producing high-quality 3D scans of small or large objects.

• Researchers in China, in collaboration with Microsoft Research, developed the Kinect

Sign Language Translator, a system that can translate sign language into spoken and written language in near real time.

• The Holoflector is an augmented reality mirror. Behind the mirror is a large LCD monitor and right above the mirror is the Kinect.

• Combining a Kinect, a few hacking tools and a 3D environment, Kinectasploit allows a

user to break into a computer’s security system using body gestures. The system works almost like a video game. In one demo, a hacker ‘shoots’ at Wi-Fi targets to break into them.

• UBI Interactive developed their software UBI which combines your Windows 8 computer to Kinect and projector to create an interactive touchscreen off any surface.

• In another case of copying from the movies, many users have hacked their Kinect to allow users to control robots via body movement.

• We can pick out cloths and try them all out virtually with the help of Fitnect, an interactive dressing room.

References http://mms.ecs.soton.ac.uk/2014/papers/2.pdf

http://lshao.staff.shef.ac.uk/pub/KinectReview_TC2013.pdf

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Microsoft20Kinect20Sensor20and20Its20Effect20-20IEEE20MM202012.pdf

http://mms.ecs.soton.ac.uk/2014/papers/2.pdf

http://campar.in.tum.de/twiki/pub/Chair/TeachingSs11Kinect/2011-DSensors_LabCourse_Kinect.pdf

http://www.slideshare.net/MatteoValoriani/introduction-to-kinect-update-v-18?qid=e70a04cc-282d-4c11-a0d0-cff7f58c2435&v=&b=&from_search=9

http://www.slideshare.net/rojizofrio/xbox-one-kinect?qid=e70a04cc-282d-4c11-a0d0-cff7f58c2435&v=&b=&from_search=15

Videos https://youtu.be/uq9SEJxZiUg

http://youtu.be/bRgEdqDiOuQ?t=10m16s

http://www.google.com/patents/US20090

http://campar.in.tum.de/twiki/pub/Cha... (p. 33)

http://en.wikipedia.org/wiki/Range_im... (Stereo triangulation)

https://www.youtube.com/watch?v=-RELJD0LBg8&spfreload=5

https://youtu.be/uq9SEJxZiUg

http://www.hongkiat.com/blog/innovative-uses-kinect/