Natural User Interfaces in a Nutshel

www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission

Global Services

Michael Heydt Principal Global Technologist Natural User Interfaces in a Nutshell October 15, 2011


Agenda

! Introduction ! What is NUI/NUX ! NUX In the media and the wild ! Real-world devices ! Tools and SDK’s ! Demos using Kinect ! Come visit our booth for more!


About Mike

! Principal global technologist, Advanced Technology Services, SunGard Global Services

! Technology focus on .NET, cloud, parallel systems, rich interfaces and NUX

! Executive Masters Technology Management and Innovation, UPenn / Wharton

! [email protected], [email protected] ! @mikeheydt ! 42spikes.com

mailto:[email protected]

www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission 4

SGS Advanced Technology

What is Advanced Technology?

SunGard Consulting Services Approach

Typical Client Engagements

! Our consultants offer deep domain experience and a broad set of leading edge technology skills, allowing companies to optimize their business processes

! We use a variety of Agile project management and execution techniques to facilitate quick delivery and reduce risk

! We take an objective approach in the selection of tools and products, and are comfortable working with a variety of vendor products and platforms

! We have applied our advanced technology skills to provide solutions to customers in finance, energy and media sectors

! Typical assignments involve deploying highly skilled specialists with relevant domain knowledge who work with customers in a collaborative way to design and build custom solutions

! MapReduce/Hadoop

! Complex Events Processing

! GPU Mathematical Pricing

! Mashups

! Web 2.0

! Cloud & Grid Computing

! Functional Languages

! Mobile Applications

! RIA (Silverlight/Flex)

! Semantic Web

! REST/JSON

! Grails


Opening Questions…

! Who here has used a Kinect on the Xbox? ! Who has programmed a Kinect? ! Who has used a WiiMote on the Wii? ! Who has programmed a wiimote in their application? ! Who has seen Minority Report?


What is NUI/NUX?

! In computing, a natural user interface, or NUI, is the common parlance used by designers and developers of computer interfaces to refer to a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users.

! NUI Covers the class of interfaces where the user is enabled to interact with the computer in a more natural manner.

http://en.wikipedia.org/wiki/Computing

http://en.wikipedia.org/wiki/User_interface


Business

! Interactive data manipulation and visualization ! Free-form visual application development ! Media presentations ! Home control ! Oil reserve exploration


Genesis

! Conceptually has been thought of for several decades ! Made popular by Minority Report ! iPad has changed the game with single-task gestural

apps ! Xbox Kinect has broken open the flood gates for NUX ! NUX is starting to show adoption in operating systems

through the use of tablet and trackpad gestures


in the Media and the Wild

! XBox360 ! Operating systems ! Movies and television ! Science


XBox360’s New Dashboard


OS X Lion


Windows 8 / Metro


TV: Caprica


Movies: Minority Report






Science: World Wide Telescope


Real World NUX Devices

! Surface ! Kinect ! PrimeSense ! Wiimote ! Emotiv


Surface


Surface Capabilities

! Multi-user ! Multi-touch ! Also camera based identification and tracking ! Built in wireless to devices


Surface 2.0


Kinect

! Laser Depth Imager ! Video Camera ! Microphone Array ! Very XBox Specific Interface (non-standard USB) ! Microsoft SDK, non-commercial license ! $150


Kinect: An Evolution?


PrimeSense

! Only laser depth finder ! Standard USB ! OpenNI drivers ! Semi-open source ! commercial license


Johnny 5


Wiimote


Emotiv

http://www.vitodibari.com/en/tech-entrepreneur-tan-le-explains-emotiv-turn-thoughts-actions-2.html


Elements of NUX

! Depth Imaging ! Skeleton Tracking ! Hand and Finger Tracking ! Gesture Identification ! Voice Recognition ! Thought Recognition (yep, you heard it here first)


Kinect: What You Need

! A Kinect ! Fastest selling consumer electronics device ever

! A PC or Mac (or Linux) ! SDKs


Kinect SDK

http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/


OpenNI

• www.openni.org/• Pseudo-open source

http://www.openni.org/


Data from the Kinect

! Multiple streams of data: ! Video ! Audio ! Depth ! Skeleton(s)


The How of the Kinect: Depths

! You are scanned and tracked by an IR laser ! Depths are measured to the millimeter at 320 x 240 ! From depths skeletons are identified and tracked


Kinect: What You Don’t See


Depth Mapping

• Distance to the millimeter

• Used to create maps for “skeletons”


Skeleton Tracking

• Built from the depth map

• 1..n skeletons

• Done in software not the hardware


Hand and Finger Tracking

Not a part of any SDK


Speech

! Kinect has a very good set of microphones ! Speech recognition is not in the Kinect SDK - use

the .NET libraries ! Must be bridged with Microsoft Speech Platform SDK

! I can’t stress this enough – this is excellent augmentation


Gesture Recognition

! Not built into any of the devices ! Typically implemented with

Dynamic Time Warping algorithms (DTW)

! Record sequences of skeleton locations, match against live streams


Hand Recognition

! Not built into any of the devices

! General Algorithm: ! Get depth data ! Point cloud identification ! Center of mass determination ! K-Curvature Analysis ! Least squares fitting ! Rendering and event processing


Difficulties

! Many devices ! Immature API’s and libraries ! Small body of work ! Nonstandard USB ! IR data is noisy ! Low resolution ! Single device focus

! No off the shelf gesture engines

! No standard pattern languages for gestures

! Point clouds don’t map 100% to video

! Setting are finicky… Tend to be very distance and angle specific


What I’m Building: NuiDotNet

! NuiDotNet (nuidotnet.codeplex.com, nuidot.net) ! Abstracts devices from the application ! Allows configuration and location based construction of services ! Multiple and prioritized services and interactions mapped to

application functions ! Gesture engine and domain specific language for gestures ! Abstraction of details of gestures and algorithms

! Release date? Soon…

! I’ll show some of it in a little bit…


Directions

! Define useful set of gesture patterns for data manipulation

! Build finger / hand tracking capabilities ! Make the DTW gesture engine robust ! Tracking of specific parts of the body ! Architecture and software to plug into business

applications ! Provide a library of pluggable UI elements and gestures ! Easing of the issues with noise, distance and angles


Interactive Demos…

! Hand Tracking ! Finger Pointing ! Hand Gestures ! Voice Recognition ! Skeleton Tracking ! Gestures


NuiDotNet Basics: Hand Tracking / Visualization


Gesture Models in RavenDB