Davide Spano lucio.davide.spano@isti.cnr.it lucio.davide.spano@isti.cnr.it spano@di.unipi.it CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 1 56124 Pisa, Italy.

  • Published on
    02-Apr-2015

  • View
    213

  • Download
    0

Transcript

<ul><li>Slide 1</li></ul> <p>Davide Spano lucio.davide.spano@isti.cnr.it lucio.davide.spano@isti.cnr.it spano@di.unipi.it CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 1 56124 Pisa, Italy Slide 2 Gestures 16 May 2012Davide Spano Gesture Interfaces Slide 3 Remotes (1) Nintendo Wiimote, 2006 Buttons Three-axis linear accelerometer IR camera Three axis gyroscope extension (Wii Motion Plus) Game market opened to casual gamers 16 May 2012Davide Spano Gesture Interfaces Slide 4 Remotes (2) Playstation Move, 2010 Buttons Spherical RGB light emitter plus camera Three-axis accelerometer and gyroscope Compass 16 May 2012Davide Spano Gesture Interfaces Slide 5 Floor Devices (1) Dance pads Konami Dance Dance Revolution (1998) Huge directional pads Buttons 16 May 2012Davide Spano Gesture Interfaces Slide 6 Floor Devices (2) Wii Balance Board Four pressure sensors Snowboard games Fitness games 16 May 2012Davide Spano Gesture Interfaces Slide 7 Motion Tracking (1) CamSpace, 2009 Object recognition No haptic feedback 16 May 2012Davide Spano Gesture Interfaces Slide 8 Motion Tracking (2) Microsoft Kinect, 2009 RGB camera Depth sensor Microphone array Still no haptic feedback 16 May 2012Davide Spano Gesture Interfaces Slide 9 Multitouch Two or more touch-sensed position on the screen Patents already on 1984 iPhone 2007 Support in Mac OS X, Windows 7, Ubuntu 16 May 2012Davide Spano Gesture Interfaces Slide 10 Gestures Overview Gesture types: Static Dynamic Spatio-temporal Conceptual: Gestures are continuous Gestures exploit time and space as arguments Gestures enable single user parallel interaction Need for a suitable programming paradigm 16 May 2012Davide Spano Gesture Interfaces Slide 11 The Midas Touch 16 May 2012Davide Spano Gesture Interfaces TrackingEngaged Out Of range Tracking Not Tracking Mouse Touches, Skeleton Tracking Accelerometers, Gyroscope Slide 12 Current Programming Model Do gestures really fit this pattern? 16 May 2012Davide Spano Gesture Interfaces Slide 13 Multitouch Frameworks 16 May 2012Davide Spano Gesture Interfaces Gesture Events Touch Events EndMoveStartRotatePinchSlide Slide 14 Example: Pinch for zoom (1) 16 May 2012Davide Spano Gesture Interfaces Slide 15 Example: Pinch for zoom (2) 16 May 2012Davide Spano Gesture Interfaces Start Move End Slide 16 Full body tracking 16 May 2012Davide Spano Gesture Interfaces Slide 17 Microsoft Kinect SDK 16 May 2012Davide Spano Gesture Interfaces Slide 18 Primesense NITE Controls State of the Art toolkit for full body gesture interaction Reusable controls (recognizers) for NUI Control list Push detector Swipe detector Steady detector Wave detector Circle detector Selectable slider (1D/2D) All based on the observer pattern 16 May 2012Davide Spano Gesture Interfaces Slide 19 Circle Control Sample 16 May 2012Davide Spano Gesture Interfaces Slide 20 One Event is not enough! 16 May 2012Davide Spano Gesture Interfaces Pan Move Start Handler End 1 1 Gesture Touch ID Samples Handler 1 Slide 21 Gesture Definition Meta Model Declarative Compositional Ground Terms (basic building blocks) Composition Operators Behaviour Association Entire Gesture Any other sub-component Abstract Gesture Recognition Support Formally defined through Petri Nets 16 May 2012Davide Spano Gesture Interfaces Slide 22 Break 16 May 2012Davide Spano Gesture Interfaces http://www.foxtrot.com/ Slide 23 Gesture Support Abstraction 16 May 2012Davide Spano Gesture Interfaces Slide 24 Ground Terms and Composition Operators Low-level device events as ground terms Optional conditions on support state Composition operators 16 May 2012Davide Spano Gesture Interfaces Slide 25 Multitouch Gesture Support 16 May 2012Davide Spano Gesture Interfaces Slide 26 Multitouch Gesture Description 16 May 2012Davide Spano Gesture Interfaces Slide 27 Proton (1) 16 May 2012Davide Spano Gesture Interfaces Slide 28 Proton (2) 16 May 2012Davide Spano Gesture Interfaces Slide 29 Full Body Gesture Platform Skeleton tracked as a set of joints 2 user full tracking + 4 only body centre Features: Joint (user id, joint id, position, orientation) Current time 16 May 2012Davide Spano Gesture Interfaces Slide 30 Full Body Example (1) 16 May 2012Davide Spano Gesture Interfaces Front Not Front Right hand Both hands Slide 31 Full Body Example 16 May 2012Davide Spano Gesture Interfaces Slide 32 A different programming model Describe declaratively the gesture Create new ones using the building blocks Reuse existing ones through composition Code the event handlers Attach them to the events at the desired granularity It is called only when it is needed (no further checks) Temporal description and interface behaviour are completely separated Complex Gestures can be stored as classes They can still expose their low level events 16 May 2012Davide Spano Gesture Interfaces Slide 33 Library Support 16 May 2012Davide Spano Gesture Interfaces Slide 34 Multitouch with Full Body Experiment 16 May 2012Davide Spano Gesture Interfaces Slide 35 Proof of Concept Applications 16 May 2012Davide Spano Gesture Interfaces Slide 36 Demo Videos Parallel Interaction with iPhone videovideo Multitouch with Kinect videovideo Same Gesture Definition! 16 May 2012Davide Spano Gesture Interfaces Slide 37 Conclusion 16 May 2012Davide Spano Gesture Interfaces http://www.mojocomic.com/ Slide 38 References Spano, L.D. A Model Based Approach for gesture interfaces (EICS 2011) pdfpdf Kin K., Hartmann B., DeRose T., Agrawala M. Proton: Multitouch Gestures as Regular Expressions pdfpdf Kinect for Windows Kinect for Windows OpenNI OpenNI Primesense NITE Primesense NITE 16 May 2012Davide Spano Gesture Interfaces </p>

Recommended

View more >