View
218
Download
0
Category
Tags:
Preview:
Citation preview
T4G The Intelligent Application of TechnologyTM
Natural User Interfaces &
Microsoft Surface Development
T4G The Intelligent Application of TechnologyTM
Why are we here?
• Obligatory corporate plug• Obligatory personal plug• The real content:
– What is a Natural User Interface (NUI)?– What is the Microsoft Surface?– MS Surface development environment– Demos and code:
• Hello, Surface!• Raw Images in WPF• Contact Data Visualization• Surface Controls• Tag Visualizations
– The Future: Windows Touch and Surface 2.0
T4G The Intelligent Application of TechnologyTM
Our Company, Culture and Success• Based on our philosophy of The Intelligent
Application of TechnologyTM, we design, develop, and implement technology solutions that help our customers run their businesses better.
T4G The Intelligent Application of TechnologyTM
T4G at a glance
1996Year T4G was
established
260+# of employees
in 7 locations across North
America
80% of customers
in 2010 who invited us back for
multiple projects
3# of consecutive years on the Great Place to Work® list
T4G The Intelligent Application of TechnologyTM
OUR VALUES Integrity Passion Communication Teamwork
Adaptive Innovation Entrepreneurialism Value Conscious Social Responsibility
T4G The Intelligent Application of TechnologyTM
Delivering leadership and expertise
Travel and Hospitality
Retail
Energy
Financial Services
Professional Services
Healthcare
Government
Communications
Our Vertical ExpertiseOur Supporting Services
Business Intelligence
Infrastructure & Security
Software Development & Integration
Content Delivery
Mobile Solutions
Social Media Strategy & Measurement
Strategic Planning
Hosting & Application Management
Portals & Collaboration
User Experience & Design
T4G The Intelligent Application of TechnologyTM
Who am I?
• Principal Consultant at T4G• Focused on Portal Technologies (SharePoint)• Also work on NUIs and Multi-touch technologies, among
other things• 25+ years on software and various technology things
T4G The Intelligent Application of TechnologyTM
What is a Natural User Interface?
A natural user interface is a user interface designed to use natural human behaviours for interacting
directly with content.
T4G The Intelligent Application of TechnologyTM
NUI is much more than Touch
A New Way of Interacting
Augmented Reality
Speech
Object Recognition
Gestures
Eye Tracking
User eXperie
nce
Natural Learning
Social Interaction
T4G The Intelligent Application of TechnologyTM
A New Way of Interacting
SocialImmersive
“The content is the application”
T4G The Intelligent Application of TechnologyTM
Implications
Pervasive UIsDon’t replace human interaction, enhance it
Extend the real world, blur the lines between UI and real life
T4G The Intelligent Application of TechnologyTM
What is the Surface?
“Microsoft Surface is a revolutionary multi-touch computer that responds to natural hand
gestures and real-world objects, helping people interact with digital content in a simple and
intuitive way.”
T4G The Intelligent Application of TechnologyTM
Surface is not just multi-touch
• Windows 7 & WPF 4 are a great baseline for touch
• Surface creates unique opportunities with specialized HW– Robust top– Horizontal form factor– Infrared camera array
T4G The Intelligent Application of TechnologyTM
Surface: Beyond Multi-Touch
• Massive multi-touch
• Multi-user experiences
• Object recognition
• Optical engineering
T4G The Intelligent Application of TechnologyTM
Surface: Multi-User
• Trend: Software continues to reduce the need for face-to-face human interactions
• Problem: Happiness and creativity depend on social interactions
• Surface brings people together– Industrial design encourages users to gather– Input capabilities allow simultaneous use– Apps facilitate collaboration / competition
T4G The Intelligent Application of TechnologyTM
Surface: Massive Multi-Touch
• Surface can efficiently track 50+ simultaneous fingers and objects
T4G The Intelligent Application of TechnologyTM
Surface: Objects
• Uniquely identify & react to billions of tagged objects
Byte Tags256 unique values
Identity Tags>340,282,366,920,938,000,000,000,000,000,000,000,000
unique values
• Tags include orientation indictors dots
T4G The Intelligent Application of TechnologyTM
Surface: Optics
• Determine the size and shape of every input• See into and project onto physical objects
T4G The Intelligent Application of TechnologyTM
What makes a GOOD Surface Application?
• The app must be right for the Surface paradigm• It should be social• Should not involve large amounts of “traditional” content• Needs to be implemented thinking about touch
T4G The Intelligent Application of TechnologyTM
Who is using it?
• Retailers (Microsoft, Wind Mobile, etc.)• Hospitality (Hard Rock Café, Rio iBar, Sheraton, etc.)• Museums (Smithsonian)• Media (MSNBC)• Healthcare, Education, Others
T4G The Intelligent Application of TechnologyTM
Development Environment
Architecture of the Surface Development Platform
T4G The Intelligent Application of TechnologyTM
Development Environment
• Windows Vista 32-bit• .NET 3.5• WPF or XNA
All stuff you know…
T4G The Intelligent Application of TechnologyTM
Development Environment
On a unit, you can use the Microsoft Surface SDK to develop and test Microsoft Surface applications directly. You can run the unit in two modes. In administrator mode, you can access all of the Windows Vista functionality. User mode demonstrates how the unit appears in a commercial venue. That is, the unit suppresses the Windows Vista user interface, automatically starts required applications, and makes sure that those applications are restarted if they need to. Typically, you develop applications in administrator mode.
Get your own Surface…
T4G The Intelligent Application of TechnologyTM
Development Environment
On a separate workstation, you can install the Microsoft Surface SDK and use the Surface Simulator tool to simulate a Microsoft Surface test environment.
– Runs the same Surface Shell as the Surface does– Use 1 or more mice to simulate multi-touch– Tools for simulating Tags or Blobs– Record and Replay interactions to support testing and debugging
Limitations:– No cameras (obviously) so if you use raw image data, cannot test it on simulator– Should have a pretty good sized screen– The SDK is only supported on Windows Vista Business, 32-bit! But, there are
ways around that! (http://www.brianpeek.com/blog/archive/2009/03/10/install-the-surface-sdk-on-windows-7-and-or-x64.aspx)
If you do not have a Surface or you share one)…
T4G The Intelligent Application of TechnologyTM
Surface SDK Features for Multi-Touch
• Common controls optimized for Multi-Touch
• Controls design primarily for Multi-Touch
• Essential Multi-Touch UX Functionality
SurfaceWindow SurfaceButton SurfaceSlider SurfaceScrollViewer SurfaceInkCanvas
SurfaceCheckBox SurfaceRadioButton SurfaceListBox SurfaceListBoxItem
SurfaceThumb SurfaceScrollBar SurfaceRepeatButton SurfaceToggleButton
ScatterView ScatterViewItem
LibraryContainer LibraryBar LibraryStack
LibraryBarItemLibraryStackItem
Rich async drag & drop Input visualizations
T4G The Intelligent Application of TechnologyTM
Lessons Learned
• There is a lot to learn about NUIs• There is just as much to unlearn• Forget most of what you know about what makes a
“good” UI• Visuals/graphics/interaction design more important
than code• Less is more• Development is easier on a workstation
T4G The Intelligent Application of TechnologyTM
Hello, World!
Plain Old WPF XAML
<Window x:Class="WpfApplication1.MainWindow“xmlns=http://schemas.microsoft.com/winfx/2006/xaml/presentationxmlns:x=http://schemas.microsoft.com/winfx/2006/xamlTitle="MainWindow" Height="350" Width="525">
<Grid>
</Grid></Window>
T4G The Intelligent Application of TechnologyTM
Hello, World!
Surface XAML
<s:SurfaceWindow x:Class="HelloSurface.SurfaceWindow1“xmlns=http://schemas.microsoft.com/winfx/2006/xaml/presentationxmlns:x=http://schemas.microsoft.com/winfx/2006/xamlxmlns:s=http://schemas.microsoft.com/surface/2008Title="Hello Surface">
<Grid>
</Grid></s:SurfaceWindow>
T4G The Intelligent Application of TechnologyTM
Code Sample 1
• Basic Surface Application• Responding to Touch Events• Responding to multiple Touch Events• Handling orientation information
Hello, Surface!
T4G The Intelligent Application of TechnologyTM
Code Sample 2
• Types of Contact events Surface sees (fingers, tags, and blobs)• Information about contacts
Contact Data Visualization
T4G The Intelligent Application of TechnologyTM
Code Sample 3
Surface Controls• Common controls optimized for Multi-Touch
• Controls design primarily for Multi-Touch
SurfaceWindow SurfaceButton SurfaceSlider SurfaceScrollViewer SurfaceInkCanvas
SurfaceCheckBox SurfaceRadioButton SurfaceListBox SurfaceListBoxItem
SurfaceThumb SurfaceScrollBar SurfaceRepeatButton SurfaceToggleButton
ScatterView ScatterViewItem
LibraryContainer LibraryBar LibraryStack
LibraryBarItemLibraryStackItem
T4G The Intelligent Application of TechnologyTM
Code Sample 3Surface Controls
• Common controls optimized for Multi-Touch
• Controls design primarily for Multi-Touch
SurfaceWindow SurfaceButton SurfaceSlider SurfaceScrollViewer SurfaceInkCanvas
SurfaceCheckBox SurfaceRadioButton SurfaceListBox SurfaceListBoxItem
SurfaceThumb SurfaceScrollBar SurfaceRepeatButton SurfaceToggleButton
ScatterView ScatterViewItem
LibraryContainer LibraryBar LibraryStack
LibraryBarItemLibraryStackItem
T4G The Intelligent Application of TechnologyTM
Code Sample 3
• ScatterView Control• Adding ScatterViewItems in XAML• Adding ScatterViewItems in code (in response to Contact
Events)• ScatterViewItems through Binding (with simple item
template)• Handling contact events when ScatterViewItem content is
“active”
Surface Controls
T4G The Intelligent Application of TechnologyTM
Surface: Objects
• Uniquely identify & react to billions of tagged objects
Byte Tags256 unique values
Identity Tags>340,282,366,920,938,000,000,000,000,000,000,000,000
unique values
• Tags include orientation indictors dots
T4G The Intelligent Application of TechnologyTM
Code Sample 4
• Adding a Tag Visualizer to recognize a tag• Extracting the information in the tag• Displaying a Tag Visualization
Tag Visualizations
T4G The Intelligent Application of TechnologyTM
Surface Hardware
Windows VistaWindows 7
NativeWin32
Application
Managed Wrapper and Interop
Touch Development Roadmap
WinForms Application
Windows 7 (2009)
Multi-Touch Controls
Multi-Touch API
WPF 3.5 SP1
WPF Application
Surface SDK1.0
Multi-Touch Controls & API
Surface Application
T4G The Intelligent Application of TechnologyTM
Surface Hardware
Windows VistaWindows 7
NativeWin32
Application
Touch Development Roadmap
WPF 3.5 SP1
Surface SDK1.0
Managed Wrapper and Interop
WinForms Application
.NET 4.0 & Surface Toolkit (Q1 2010)
WPF 4.0 w/ Multi Touch APIs
Multi-Touch ControlsSurface
Multi-Touch Controls &API
Multi-Touch API
Surface Toolkit for Windows Touch
Multi-Touch Controls & API
WPF Application
Surface Application
T4G The Intelligent Application of TechnologyTM
Surface Hardware
Windows 7Windows 7
NativeWin32
Application
Touch Development Roadmap
Managed Wrapper and Interop
WPF 4.0 w/ Multi Touch APIs
Surface Toolkit for Windows Touch
Surface SDKvNext
WinForms Application
Surface 2.0
Multi-Touch ControlsSurface
Multi-Touch Controls &API
Multi-Touch API
Surface-specificControls & API
WPF Application
Surface Application
T4G The Intelligent Application of TechnologyTM
Surface 2.0 Integration & Extension of WPF 4
• All input is routed through the WPF InputManager
• Surface input provider can send to multiple windows
• Surface extension methods provide access to additional input data (tag values, orientation, shape, etc)
WPF InputManager
Surface input provider
Windows Touch input
providerApps
& Controls
Manipulation & Touch Events
Surface extension methods
T4G The Intelligent Application of TechnologyTM
Surface 2.0
• Announced at CES 2011• Available later this year• SDK Available April 2011 (?)• Embedded Windows 7 Professional 64-bit • .NET Framework 4.0 • Windows Presentation Foundation (WPF) 4.0 • Microsoft XNA® Framework 4.0 • Windows PowerShell and DMTF DASH support, and enhanced
administrator tools
T4G The Intelligent Application of TechnologyTM
Resources
On the web:Surface Web Site (http://www.surface.com)Deconstructing the NUI (Joshua Blake, Surface MVP)My Blog (http://fyeomans.com)
On Twitter:@Surface – the official MS Surface twitter identity#Surface@fyeomans
T4G The Intelligent Application of TechnologyTM
Questions?
Recommended