Gesture InterfacesDavide Spano

lucio.davide.spano@isti.cnr.itspano@di.unipi.it

CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 156124 Pisa, Italy

Davide Spano Gesture Interfaces

Gestures

16 May 2012

Davide Spano Gesture Interfaces

Remotes (1)Nintendo Wiimote, 2006

Buttons Three-axis linear accelerometer IR camera Three axis gyroscope extension (Wii Motion Plus)

Game market opened to casual gamers

16 May 2012

Davide Spano Gesture Interfaces

Remotes (2)Playstation Move, 2010

Buttons Spherical RGB light emitter plus camera Three-axis accelerometer and gyroscope Compass

16 May 2012

Davide Spano Gesture Interfaces

Floor Devices (1) Dance pads

Konami Dance Dance Revolution (1998) Huge directional pads Buttons

16 May 2012

Davide Spano Gesture Interfaces

Floor Devices (2)Wii Balance Board

Four pressure sensors Snowboard games Fitness games

16 May 2012

Davide Spano Gesture Interfaces

Motion Tracking (1)CamSpace, 2009

Object recognition No haptic feedback

16 May 2012

Davide Spano Gesture Interfaces

Motion Tracking (2)Microsoft Kinect, 2009

RGB camera Depth sensor Microphone array Still no haptic feedback

16 May 2012

Davide Spano Gesture Interfaces

MultitouchTwo or more touch-sensed position on the screen

Patents already on 1984 iPhone 2007 Support in Mac OS X, Windows 7, Ubuntu …

16 May 2012

Davide Spano Gesture Interfaces

Gestures Overview

Gesture types: Static Dynamic Spatio-temporal

Conceptual: Gestures are continuous Gestures exploit time and space as arguments Gestures enable single user parallel interaction Need for a suitable programming paradigm

16 May 2012

Davide Spano Gesture Interfaces

The Midas Touch

16 May 2012

Tracking EngagedOutOf

range

TrackingNotTracking

Mouse

Touches,Skeleton

Tracking

Accelerometers,Gyroscope

Davide Spano Gesture Interfaces

Current Programming Model

Do gestures really fit this pattern?

16 May 2012

Davide Spano Gesture Interfaces

Multitouch Frameworks

16 May 2012

Gesture Events

Touch Events

End

Move

Start

Rotate

Pinch

Slide

Davide Spano Gesture Interfaces

Example: Pinch for zoom (1)

16 May 2012

Davide Spano Gesture Interfaces

Example: Pinch for zoom (2)

16 May 2012

Start Move

Move End

Davide Spano Gesture Interfaces

Full body tracking

16 May 2012

Davide Spano Gesture Interfaces

Microsoft Kinect SDK

16 May 2012

Davide Spano Gesture Interfaces

Primesense NITE ControlsState of the Art toolkit for full body gesture interaction

Reusable controls (recognizers) for NUI

Control list Push detector Swipe detector Steady detector Wave detector Circle detector Selectable slider (1D/2D)

All based on the observer pattern

16 May 2012

Davide Spano Gesture Interfaces

Circle Control Sample

16 May 2012

Control Event Skeleton Tracking

Davide Spano Gesture Interfaces

One Event is not enough!

16 May 2012

Pan

MoveStart

Handler

End1 1

Gesture

Touch IDSamples

Handler

Handler

Handler

1

Davide Spano Gesture Interfaces

Gesture Definition Meta ModelDeclarative

Compositional Ground Terms (basic building blocks) Composition Operators

Behaviour Association Entire Gesture Any other sub-component

Abstract Gesture Recognition Support

Formally defined through Petri Nets

16 May 2012

Davide Spano Gesture Interfaces

Break…

16 May 2012

http://www.foxtrot.com/

Davide Spano Gesture Interfaces

Gesture Support AbstractionFeature: an observable property

Gesture Support: a set of features

Gesture Support State: feature values at a given time

Gesture Support Sequence: gesture support state values through the time

16 May 2012

Davide Spano Gesture Interfaces

Ground Terms and Composition OperatorsLow-level device events as ground terms

Optional conditions on support state

Composition operators

16 May 2012

Davide Spano Gesture Interfaces

Multitouch Gesture SupportFeatures:

2D position of n touches, in pixel coordinates If a touch is not currently detected current detection time

Multi-Touch recognition ground terms:

16 May 2012

Davide Spano Gesture Interfaces

Multitouch Gesture DescriptionTap

Double Tap

Pan

Slide

Pinch

Rotate

16 May 2012

Davide Spano Gesture Interfaces

Proton (1)Kin et al. Proton: Multitouch Gestures as Regular

Expressions, CHI 2012

Multitouch Gestures Described as Regular Expression

Ground Terms

Composition operators

*Matcher for handling ambiguity

One gesture at time assumption

Tablature graphical representation for gestures

16 May 2012

Davide Spano Gesture Interfaces

Proton (2)

16 May 2012

Davide Spano Gesture Interfaces

Full Body Gesture PlatformSkeleton tracked as a set of joints

2 user full tracking + 4 only body centre Features: Joint (user id, joint id, position, orientation) Current time

16 May 2012

Davide Spano Gesture Interfaces

Full Body Example (1)

16 May 2012

Front Not Front Right hand Both hands

Davide Spano Gesture Interfaces

Full Body Example

(𝐹𝑟𝑜𝑛𝑡≫ ( h𝑅𝑖𝑔 𝑡𝐻𝑎𝑛𝑑 [ ] h𝐵𝑜𝑡 𝐻𝑎𝑛𝑑𝑠 ) )∗¿

16 May 2012

Davide Spano Gesture Interfaces

A different programming modelDescribe declaratively the gesture

Create new ones using the building blocks Reuse existing ones through composition

Code the event handlers Attach them to the events at the desired granularity It is called only when it is needed (no further checks)

Temporal description and interface behaviour are completely separated Complex Gestures can be stored as classes They can still expose their low level events

16 May 2012

Davide Spano Gesture Interfaces

Library Support

16 May 2012

Davide Spano Gesture Interfaces

Multitouch with Full Body Experiment Hands point tracking

Basic Touches Description

16 May 2012

Davide Spano Gesture Interfaces

Proof of Concept Applications

16 May 2012

Davide Spano Gesture Interfaces

Demo VideosParallel Interaction with iPhone video

“Multitouch” with Kinect video

Same Gesture Definition!

16 May 2012

Davide Spano Gesture Interfaces

Conclusion

16 May 2012

http://www.mojocomic.com/

Davide Spano Gesture Interfaces

ReferencesSpano, L.D. A Model Based Approach for gesture interfaces

(EICS 2011) pdf

Kin K., Hartmann B., DeRose T., Agrawala M. Proton: Multitouch Gestures as Regular Expressions pdf

Kinect for Windows

OpenNI

Primesense NITE

16 May 2012