NUX Presentation from TechMixer Birmingham 2011

NUX in a

nutshell

agenda• Introduction• What is NUI/NUX• NUX In the media and the wild• Real-world devices• Tools, SDK’s, …• Where is this going?• Demo using Kinect and Wiimote

about mike• Principal global technologist, Advanced Technology Services, SunGard Global Services

• Technology focus on .NET, cloud, parallel systems, rich interfaces and NUX

• mike@heydt.org, michael.heydt@sungard.com• @mikeheydt• 42spikes.com

55

Advanced TechnologyWhat is Advanced Technology?

SunGard Consulting Services Approach

Typical Client Engagements

Our consultants offer deep domain experience and a broad set of leading edge technology skills, allowing companies to optimize their business processes

We use a variety of Agile project management and execution techniques to facilitate quick delivery and reduce risk

We take an objective approach in the selection of tools and products, and are comfortable working with a variety of vendor products and platforms

We have applied our advanced technology skills to provide solutions to customers in finance, energy and media sectors

Typical assignments involve deploying highly skilled specialists with relevant domain knowledge who work with customers in a collaborative way to design and

build custom solutions

MapReduce/HadoopComplex Events ProcessingGPU Mathematical Pricing

Mashups

Web 2.0Cloud & Grid ComputingFunctional LanguagesMobile Applications

RIA (Silverlight/Flex)Semantic Web

REST/JSONGrails

opening questions…

• Who here has used a Kinect on the Xbox?• Who has programmed a Kinect?• Who has used a WiiMote on the Wii?• Who has programmed a wiimote in their application?

• Who has seen Minority Report?

what is nui/x?

• In computing, a natural user interface, or NUI, is the common parlance used by designers and developers of computer interfaces to refer to a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users.

• NUI Covers the class of interfaces where the user is enabled to interact with the computer in a more natural manner.

genesis• Conceptually has been thought of for several decades

• Made popular by Minority Report• iPad has changed the game with single-task gestural apps

• Xbox Kinect has broken open the flood gates for NUX

• NUX is starting to show adoption in operating systems through the use of tablet and trackpad gestures

in the media and the wild

• Xbox• Operating systems• Movies and television• Science

XBox

os and mobility: osx lion

os and mobility: Windows 8

tv: Caprica

movie: District 9

movie: Disclosure

movie: Minority Report

movie: Minority Report

movie: Minority Report

science: world wide telescope

other: smart cubes

real world devices

• Surface• Kinect• PrimeSense• Wiimote• Tablets and SmartPhones

surface

surface capabilities

• Multi-user• Multi-touch• Also camera based identification and tracking• Built in wireless to devices

surface 2.0

kinect• Laser Depth Imager• Video Camera• Microphone Array• Very XBox Specific Interface (non-standard USB)• Microsoft SDK, non-commercial license• $150

kinect: an evolution

primesense• Only laser depth finder• Standard USB• OpenNI drivers• Semi-open source• commercial license

johnny 5

wiimote

tablets and smartphones• Have driven

adoption of NUX• Will be an

integral “augmentation” to NUX

elements of nux

• Skeleton Tracking• Gesture (Body, fingers) Identification• Voice Recognition

the how of the kinect

• Perform initialization gesture (helps, but not always necessary)

• You are scanned and tracked by an IR laser• Depths are measured to the millimeter at 320 x 200 +

• From depths skeletons are identified and tracked

Kinect SDK

http://research.microsoft.com/en-us/um/

redmond/projects/

kinectsdk/

OpenNI• http://www.o

penni.org/

• Pseudo-open source

data from the kinect

• Multiple streams of data• Video• Audio• Depth• Skeletons

kinect - the invisible

depth mapping• Distance to the millimeter• Used to create maps for “skeletons”

skeleton tracking

• Built from the depth map• 1..n skeletons• Done in software not the hardware

the “psi” gesture

hand and finger tracking

Not a part of any SDK

the how of wiimote

• Connects to PC with Bluetooth• With .NET, use WiimoteLib (http://wiimotelib.codeplex.com/)

Wiimotelib

http://www.ted.com/talks/johnny_lee_demos_wii_remote_hacks.html

Thoughts on the Wiimote• An interesting device, but…

• It’s first gen• Looked good prior to the Kinect• All capabilities can be duplicated with Kinect (point tracking) and a cell phone (accelerometer, gyro)

• And a cell phone can provide additional uses such as robust data entry

speech• Kinect has a very good set of microphones• Speech recognition is not in the kinect - use the .NET libraries

• Must be bridged with Microsoft Speech Platform SDK

• I can’t stress this enough – this is excellent augmentation

gesture recognition• Not built into any of the devices

• Typically implemented with Dynamic Time Warping algorithms (DTW)

• Record sequences of skeleton locations, match against live streams

difficulties• Many devices• Immature libraries• Small body of work• Many API’s, even for each individual device

• Kinect is non-standard USB

• All devices are separate and don’t interoperate

• All work on a single system – No support for moving physical location

• No standard pattern language for NUX

• No “canned” gesture engines

what i’m building• NuiDotNet (nuidotnet.codeplex.com, nuidot.net)

• Abstracts devices from the application• Allows configuration and location based construction of services

• Multiple and prioritized services and interactions mapped to application functions

• Gesture engine and domain specific language for gestures

directions• Define useful set of gesture patterns for data manipulation

• Build finger / hand tracking capabilities• Make the DTW gesture engine robust• Tracking of specific parts of the body• Architecture and software to plug into business applications

• Provide a library of pluggable UI elements and gestures

uses for business

• Interactive data manipulation and visualization• Free-form visual application development• Media presentations• Home control• Many still to be determined

demo and the tools

• Hardware: Kinect, WiiMote• Libraries: Kinect SDK, Wiimotelib• Language and GUI: C# / WPF

resources• ^ Buxton, Bill (2010-01-06). "CES 2010: NUI with Bill Buxton". Microsoft Research.

• http://www.microsoft.com/surface/en/us/default.aspx

• http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/

• http://www.youtube.com/watch?v=1-tMp4WkQjA

• http://openni.org/

• http://silverlightkinect.codeplex.com/

• http://mono-mig.sourceforge.net/

• https://42spikes.com

Resources

• http://en.wikipedia.org/wiki/NUI• http://kinectdtw.codeplex.com/• http://en.wikipedia.org/wiki/Dynamic_time_warping

• http://tinyurl.com/6a64sqr

• http://wiimotelib.codeplex.com/

Resources

• http://ctmarionette.wordpress.com/ (marionette)• http://www.ted.com/talks/johnny_lee_demos_wii_remote_hacks.html

• http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/docs/speech_walkthrough.pdf

• http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=14373

resources

• http://www.springer.com/cda/content/document/cda_downloaddocument/9783540740476-c1.pdf?SGWID=0-0-45-452103-p173751818

• http://www.youtube.com/watch?v=tlLschoMhuE&feature=player_embedded

• http://www.youtube.com/watch?v=lCuItHQEgEQ&NR=1

resources

• http://www.microsoft.com/presspass/press/2011/jan11/01-06mssurfacesamsungpr.mspx

• http://blogs.oracle.com/mprove/entry/reipadsnui• http://channel9.msdn.com/Shows/HanselminutesOn9/Hanselminutes-on-9-Johnny-Lee-on-Computer-Vision