Embed Size (px)
Citation preview
NUX in a
nutshell
agenda• Introduction• What is NUI/NUX• NUX In the media and the wild• Real-world devices• Tools, SDK’s, …• Where is this going?• Demo using Kinect and Wiimote
about mike• Principal global technologist, Advanced Technology Services, SunGard Global Services
• Technology focus on .NET, cloud, parallel systems, rich interfaces and NUX
• [email protected], [email protected]• @mikeheydt• 42spikes.com
55
Advanced TechnologyWhat is Advanced Technology?
SunGard Consulting Services Approach
Typical Client Engagements
Our consultants offer deep domain experience and a broad set of leading edge technology skills, allowing companies to optimize their business processes
We use a variety of Agile project management and execution techniques to facilitate quick delivery and reduce risk
We take an objective approach in the selection of tools and products, and are comfortable working with a variety of vendor products and platforms
We have applied our advanced technology skills to provide solutions to customers in finance, energy and media sectors
Typical assignments involve deploying highly skilled specialists with relevant domain knowledge who work with customers in a collaborative way to design and
build custom solutions
MapReduce/HadoopComplex Events ProcessingGPU Mathematical Pricing
Mashups
Web 2.0Cloud & Grid ComputingFunctional LanguagesMobile Applications
RIA (Silverlight/Flex)Semantic Web
REST/JSONGrails
opening questions…
• Who here has used a Kinect on the Xbox?• Who has programmed a Kinect?• Who has used a WiiMote on the Wii?• Who has programmed a wiimote in their application?
• Who has seen Minority Report?
what is nui/x?
• In computing, a natural user interface, or NUI, is the common parlance used by designers and developers of computer interfaces to refer to a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users.
• NUI Covers the class of interfaces where the user is enabled to interact with the computer in a more natural manner.
genesis• Conceptually has been thought of for several decades
• Made popular by Minority Report• iPad has changed the game with single-task gestural apps
• Xbox Kinect has broken open the flood gates for NUX
• NUX is starting to show adoption in operating systems through the use of tablet and trackpad gestures
in the media and the wild
• Xbox• Operating systems• Movies and television• Science
XBox
os and mobility: osx lion
os and mobility: Windows 8
tv: Caprica
movie: District 9
movie: Disclosure
movie: Minority Report
movie: Minority Report
movie: Minority Report
science: world wide telescope
other: smart cubes
real world devices
• Surface• Kinect• PrimeSense• Wiimote• Tablets and SmartPhones
surface
surface capabilities
• Multi-user• Multi-touch• Also camera based identification and tracking• Built in wireless to devices
surface 2.0
kinect• Laser Depth Imager• Video Camera• Microphone Array• Very XBox Specific Interface (non-standard USB)• Microsoft SDK, non-commercial license• $150
kinect: an evolution
primesense• Only laser depth finder• Standard USB• OpenNI drivers• Semi-open source• commercial license
johnny 5
wiimote
tablets and smartphones• Have driven
adoption of NUX• Will be an
integral “augmentation” to NUX
elements of nux
• Skeleton Tracking• Gesture (Body, fingers) Identification• Voice Recognition
the how of the kinect
• Perform initialization gesture (helps, but not always necessary)
• You are scanned and tracked by an IR laser• Depths are measured to the millimeter at 320 x 200 +
• From depths skeletons are identified and tracked
Kinect SDK
http://research.microsoft.com/en-us/um/
redmond/projects/
kinectsdk/
OpenNI• http://www.o
penni.org/
• Pseudo-open source
data from the kinect
• Multiple streams of data• Video• Audio• Depth• Skeletons
kinect - the invisible
depth mapping• Distance to the millimeter• Used to create maps for “skeletons”
skeleton tracking
• Built from the depth map• 1..n skeletons• Done in software not the hardware
the “psi” gesture
hand and finger tracking
Not a part of any SDK
the how of wiimote
• Connects to PC with Bluetooth• With .NET, use WiimoteLib (http://wiimotelib.codeplex.com/)
Wiimotelib
http://www.ted.com/talks/johnny_lee_demos_wii_remote_hacks.html
Thoughts on the Wiimote• An interesting device, but…
• It’s first gen• Looked good prior to the Kinect• All capabilities can be duplicated with Kinect (point tracking) and a cell phone (accelerometer, gyro)
• And a cell phone can provide additional uses such as robust data entry
speech• Kinect has a very good set of microphones• Speech recognition is not in the kinect - use the .NET libraries
• Must be bridged with Microsoft Speech Platform SDK
• I can’t stress this enough – this is excellent augmentation
gesture recognition• Not built into any of the devices
• Typically implemented with Dynamic Time Warping algorithms (DTW)
• Record sequences of skeleton locations, match against live streams
difficulties• Many devices• Immature libraries• Small body of work• Many API’s, even for each individual device
• Kinect is non-standard USB
• All devices are separate and don’t interoperate
• All work on a single system – No support for moving physical location
• No standard pattern language for NUX
• No “canned” gesture engines
what i’m building• NuiDotNet (nuidotnet.codeplex.com, nuidot.net)
• Abstracts devices from the application• Allows configuration and location based construction of services
• Multiple and prioritized services and interactions mapped to application functions
• Gesture engine and domain specific language for gestures
directions• Define useful set of gesture patterns for data manipulation
• Build finger / hand tracking capabilities• Make the DTW gesture engine robust• Tracking of specific parts of the body• Architecture and software to plug into business applications
• Provide a library of pluggable UI elements and gestures
uses for business
• Interactive data manipulation and visualization• Free-form visual application development• Media presentations• Home control• Many still to be determined
demo and the tools
• Hardware: Kinect, WiiMote• Libraries: Kinect SDK, Wiimotelib• Language and GUI: C# / WPF
resources• ^ Buxton, Bill (2010-01-06). "CES 2010: NUI with Bill Buxton". Microsoft Research.
• http://www.microsoft.com/surface/en/us/default.aspx
• http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/
• http://www.youtube.com/watch?v=1-tMp4WkQjA
• http://openni.org/
• http://silverlightkinect.codeplex.com/
• http://mono-mig.sourceforge.net/
• https://42spikes.com
Resources
• http://en.wikipedia.org/wiki/NUI• http://kinectdtw.codeplex.com/• http://en.wikipedia.org/wiki/Dynamic_time_warping
• http://tinyurl.com/6a64sqr
• http://wiimotelib.codeplex.com/
Resources
• http://ctmarionette.wordpress.com/ (marionette)• http://www.ted.com/talks/johnny_lee_demos_wii_remote_hacks.html
• http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/docs/speech_walkthrough.pdf
• http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=14373
resources
• http://www.springer.com/cda/content/document/cda_downloaddocument/9783540740476-c1.pdf?SGWID=0-0-45-452103-p173751818
• http://www.youtube.com/watch?v=tlLschoMhuE&feature=player_embedded
• http://www.youtube.com/watch?v=lCuItHQEgEQ&NR=1
resources
• http://www.microsoft.com/presspass/press/2011/jan11/01-06mssurfacesamsungpr.mspx
• http://blogs.oracle.com/mprove/entry/reipadsnui• http://channel9.msdn.com/Shows/HanselminutesOn9/Hanselminutes-on-9-Johnny-Lee-on-Computer-Vision
