Embed Size (px)
Citation preview
Gesture-‐based interaction Characterization
Recognition Typical approach
Design challenges, advantages, drawbacks Applications
Conclusion
2
Gesture-‐based interaction, why? The gestures are a natural way to interact with object, tools and other
people As substitution for other forms of communication when other
interactions are not possible ▪ Impaired people ▪ Special context
As complement to other types of interaction modalities
3
A motion of the limbs or body made to express or help express thought or to emphasize speech.
The act of moving the limbs or body as an expression of thought or emphasis.
An act or a remark made as a formality or as a sign of intention or attitude.
A succession of postures.
Own definition (for this seminar): An intentional sign made with the body or limbs to communicate intention or information
4
Gestures Vs Gesticulation Also the gesticulation provides information
Static Vs Dynamic Gestures Static gestures (aka postures, poses,…) Dynamic gestures: a sequence of postures/positions
Multi-‐dimensional gestures 2D gestures 3D gestures Pointing gesture
5
2D gesture 3D gesture Pointing gesture
6
Gesture-‐based interaction Characterization
Recognition Typical approach
Design challenges, advantages, drawbacks Applications
Conclusion
7
Dynamic gesture recognition (through computer vision) can be divided in the following main phases: Detection Tracking Gesture segmentation Gesture recognition
Features extraction
8
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
Two sub-‐steps: Image acquisition Preprocessing
9
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
Image acquisition Mono-‐camera, multi-‐camera, stereo-‐camera, or 3D camera
Camera resolution (low Vs high resolution)
Frames per second
10
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
Preprocessing Pixel level segmentation ▪ Color segmentation ▪ Hand detection ▪ Color marker detection
Motion segmentation ▪ Background subtraction ▪ Works good on known background (static background)
▪ Cannot detect stationary hands or determine which moving object is the hand
11
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
Preprocessing Contour detection ▪ Not directly depending on skin color and lighting conditions ▪ Can be a large number of objects (even in the background)
Correlation ▪ Problems when objects are rotated or scaled ▪ Problem can be avoided with continuously updating the template
12
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
13
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
1
1
2
2
Frame 1
Frame 2
Approaches Kalman filter ▪ Easily computable in real-‐time ▪ Basic form of Kalman filters cannot track objects
on unknown background
Condensation ▪ One of the most used technique for tracking ▪ Detect and track contour of moving objects in a
cluttered environment
CAMshift ▪ Fast, real-‐time ▪ It may be possible to improve accuracy by using
different color representation ▪ There are quite a few parameters
… System without tracking
In controlled environment with a special gesture vocabulary
14
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
Gesture segmentation Initial (final) posture When hands are not moving -‐> end of gesture
Gesture decomposition ▪ Preparation, stroke and retraction
Statistical approach ▪ Hidden Markov Model
15
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
Potential features: Position, acceleration, velocity Spatial – temporal width FFT of the position …
The features can be extracted in three steps of the process chain
Post-‐processing should be done before providing the features to the GR block
16
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
Classification Algorithms: Hidden Markov Model (HMM) ▪ Ergodic HMM, Left-‐Right HMM, Left-‐Right
Banded Hierarchic HMM, Input-‐Output HMM, Parametric HMM, etc. etc. etc.
Conditional Random Fields (CRF) ▪ Hidden CRF, Latent-‐dynamic Discriminative
CRF, etc.
Neural Networks (NN), Decision Trees, Support Vector Machine (SVN), KNN, Dynamic time warping (DTW), etc.
Boosting Algorithms, etc. Hybrid algorithms
17
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
Gesture-‐based interaction Characterization
Recognition Typical approach
Design challenges, advantages, drawbacks Applications
Conclusion
18
Design challenges Lighting conditions Provide a feedback to the user Gesture vocabulary (small – large, kind of gesture used, etc.)
Real-‐time interaction Wearable gesture interfaces Multimodality
19
Skinput project
Advantages: Natural way of interaction “Space” effective interaction modality (compared with keyboard and mouse) ▪ Removes the user’s dependency on a surface ▪ Remote interaction
Drawbacks: Tiring (e.g. gorilla arm) User dependent gestures – few universal understandable gestures
Computationally expensive 20
21
Sixthsense G-speak
Natal project
The Project Natal sensor device
Project natal: http://www.xbox.com/en-‐US/live/projectnatal/
Oblong g-‐speak: http://www.oblong.com/ Sixth Sense:
http://www.pranavmistry.com/projects/sixthsense/ Touchless: http://www.codeplex.com/touchless Wiiremotes
(and soon the Play Station Move)
22
Gesture-‐based interaction Characterization
Recognition Typical approach
Design challenges, advantages, drawbacks Applications
Conclusion
23
Gesture based interaction Gesture as interface Gesture characterization ▪ Gesticulation & gesture ▪ Dynamic Vs static gesture ▪ Multi-‐dimensional gesture
Typical approach Challenges, advantages, and drawback
24
25
