View
205
Download
0
Category
Preview:
Citation preview
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Global Services
Michael Heydt Principal Global Technologist Natural User Interfaces in a Nutshell October 15, 2011
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Agenda
! Introduction ! What is NUI/NUX ! NUX In the media and the wild ! Real-world devices ! Tools and SDK’s ! Demos using Kinect ! Come visit our booth for more!
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
About Mike
! Principal global technologist, Advanced Technology Services, SunGard Global Services
! Technology focus on .NET, cloud, parallel systems, rich interfaces and NUX
! Executive Masters Technology Management and Innovation, UPenn / Wharton
! mike@heydt.org, michael.heydt@sungard.com ! @mikeheydt ! 42spikes.com
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission 4
SGS Advanced Technology
What is Advanced Technology?
SunGard Consulting Services Approach
Typical Client Engagements
! Our consultants offer deep domain experience and a broad set of leading edge technology skills, allowing companies to optimize their business processes
! We use a variety of Agile project management and execution techniques to facilitate quick delivery and reduce risk
! We take an objective approach in the selection of tools and products, and are comfortable working with a variety of vendor products and platforms
! We have applied our advanced technology skills to provide solutions to customers in finance, energy and media sectors
! Typical assignments involve deploying highly skilled specialists with relevant domain knowledge who work with customers in a collaborative way to design and build custom solutions
! MapReduce/Hadoop
! Complex Events Processing
! GPU Mathematical Pricing
! Mashups
! Web 2.0
! Cloud & Grid Computing
! Functional Languages
! Mobile Applications
! RIA (Silverlight/Flex)
! Semantic Web
! REST/JSON
! Grails
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Opening Questions…
! Who here has used a Kinect on the Xbox? ! Who has programmed a Kinect? ! Who has used a WiiMote on the Wii? ! Who has programmed a wiimote in their application? ! Who has seen Minority Report?
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
What is NUI/NUX?
! In computing, a natural user interface, or NUI, is the common parlance used by designers and developers of computer interfaces to refer to a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users.
! NUI Covers the class of interfaces where the user is enabled to interact with the computer in a more natural manner.
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Business
! Interactive data manipulation and visualization ! Free-form visual application development ! Media presentations ! Home control ! Oil reserve exploration
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Genesis
! Conceptually has been thought of for several decades ! Made popular by Minority Report ! iPad has changed the game with single-task gestural
apps ! Xbox Kinect has broken open the flood gates for NUX ! NUX is starting to show adoption in operating systems
through the use of tablet and trackpad gestures
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
in the Media and the Wild
! XBox360 ! Operating systems ! Movies and television ! Science
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
XBox360’s New Dashboard
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
OS X Lion
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Windows 8 / Metro
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
TV: Caprica
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Movies: Minority Report
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Movies: Minority Report
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Movies: Minority Report
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Science: World Wide Telescope
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Real World NUX Devices
! Surface ! Kinect ! PrimeSense ! Wiimote ! Emotiv
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Surface
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Surface Capabilities
! Multi-user ! Multi-touch ! Also camera based identification and tracking ! Built in wireless to devices
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Surface 2.0
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect
! Laser Depth Imager ! Video Camera ! Microphone Array ! Very XBox Specific Interface (non-standard USB) ! Microsoft SDK, non-commercial license ! $150
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect: An Evolution?
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
PrimeSense
! Only laser depth finder ! Standard USB ! OpenNI drivers ! Semi-open source ! commercial license
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Johnny 5
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Wiimote
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Emotiv
http://www.vitodibari.com/en/tech-entrepreneur-tan-le-explains-emotiv-turn-thoughts-actions-2.html
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Elements of NUX
! Depth Imaging ! Skeleton Tracking ! Hand and Finger Tracking ! Gesture Identification ! Voice Recognition ! Thought Recognition (yep, you heard it here first)
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect: What You Need
! A Kinect ! Fastest selling consumer electronics device ever
! A PC or Mac (or Linux) ! SDKs
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect SDK
http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
OpenNI
• www.openni.org/• Pseudo-open source
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Data from the Kinect
! Multiple streams of data: ! Video ! Audio ! Depth ! Skeleton(s)
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
The How of the Kinect: Depths
! You are scanned and tracked by an IR laser ! Depths are measured to the millimeter at 320 x 240 ! From depths skeletons are identified and tracked
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect: What You Don’t See
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Depth Mapping
• Distance to the millimeter
• Used to create maps for “skeletons”
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Skeleton Tracking
• Built from the depth map
• 1..n skeletons
• Done in software not the hardware
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Hand and Finger Tracking
Not a part of any SDK
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Speech
! Kinect has a very good set of microphones ! Speech recognition is not in the Kinect SDK - use
the .NET libraries ! Must be bridged with Microsoft Speech Platform SDK
! I can’t stress this enough – this is excellent augmentation
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Gesture Recognition
! Not built into any of the devices ! Typically implemented with
Dynamic Time Warping algorithms (DTW)
! Record sequences of skeleton locations, match against live streams
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Hand Recognition
! Not built into any of the devices
! General Algorithm: ! Get depth data ! Point cloud identification ! Center of mass determination ! K-Curvature Analysis ! Least squares fitting ! Rendering and event processing
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Difficulties
! Many devices ! Immature API’s and libraries ! Small body of work ! Nonstandard USB ! IR data is noisy ! Low resolution ! Single device focus
! No off the shelf gesture engines
! No standard pattern languages for gestures
! Point clouds don’t map 100% to video
! Setting are finicky… Tend to be very distance and angle specific
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
What I’m Building: NuiDotNet
! NuiDotNet (nuidotnet.codeplex.com, nuidot.net) ! Abstracts devices from the application ! Allows configuration and location based construction of services ! Multiple and prioritized services and interactions mapped to
application functions ! Gesture engine and domain specific language for gestures ! Abstraction of details of gestures and algorithms
! Release date? Soon…
! I’ll show some of it in a little bit…
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Directions
! Define useful set of gesture patterns for data manipulation
! Build finger / hand tracking capabilities ! Make the DTW gesture engine robust ! Tracking of specific parts of the body ! Architecture and software to plug into business
applications ! Provide a library of pluggable UI elements and gestures ! Easing of the issues with noise, distance and angles
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Interactive Demos…
! Hand Tracking ! Finger Pointing ! Hand Gestures ! Voice Recognition ! Skeleton Tracking ! Gestures
www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
NuiDotNet Basics: Hand Tracking / Visualization
Recommended