Gesture Recognitionhomepage.cs.uiowa.edu/~ochipara/classes/sensing_the... · Motion-Based Gesture Recognition Based on the Paper: Kazuya Murao, Tsutomu Terada, Ai Yano, Ryuichi Matsukura,

Gesture Recognition

Presented By: Baoluo Meng

Chutian Gao George F Judisch Renjitha Nair

Project Proposal Group

Gestures

•  User Interaction in apps today are moving away from mouse and pen

•  Smartphones or Video game controllers using gesture recognition are easy to use and provide intuitive human computer interaction

•  Pervasive , more physical and tangible

Our Project and Gestures

•  What is a Gesture ? Gesture is a promising mobile user-interface modality that ensures eyes-free interaction without stopping or impeding movements

•  And our Project : an educational fun game would definitely need to incorporate a variety of these gesture recognition techniques to be intuitive and appealing to the user.

Gesture Recognition on Mobile Devices

Latest smartphones and similar mobile devices are capable of basically identifying three types of Gestures

•  Motion Based Gestures •  Touch Based Gestures

•  Vision Based Gestures

Touch-Based Gesture Recognition

Gesture Coder : A Tool for Programming Multi-Touch Gestures by Demonstration

Hao Lu(University of Washington)

Yang Li ( Google Research )

Multi-Touch Gestures

•  Increasing number of modern devices are equipped with multi-touch capability

•  They are intuitive, efficient and often have physical metaphors

•  However they are difficult to implement

•  Challenge : Complicated touch state transitions from multiple fingers and their simultaneous movements

Existing System •  Framework supports predefined gestures with one

recognizer for each gesture

•  Higher Level Specification Languages-Burden on Developers

•  Recognizing gestures from trajectories generated with a single finger or stylus

•  Mostly treats gestures as a shortcut for triggering a discrete , one-shot action

Contributions

•  interaction techniques and architectural support for programming multi-touch by demonstration

•  conceptualization of multi-touch gestures and enables effective machine learning from small no:of gestures

•  methods for training the recognizers from gestures and generating user-modifiable code for invoking app specific actions

Gesture Coder

> Automatically generates code that recognizes multi-touch gestures,tracks their state changes and invokes corresponding application actions

Available as an Eclipse Plug-In for Android SDK !!!

Using a Gesture Coder

•  Demonstrating gestures on a multi-touch device

•  Testing the generated recognizer anytime

•  Integrating generated code into developer's project

Demonstrating Gestures

Testing the Generated Recognizer

Integrating Generated Code Into Project

Life-Cycle of a Gesture in Gesture-Coder

•  Possible

•  Failed

•  Began

•  Changed

•  Cancelled

•  Ended

Essentially Implementing a machine Learning Algorithm with decision trees (using C4.5 algorithm from weka-toolkit for training the decision tree)

Demonstration

Defining Multi-touch

Lü and Li define multi-touch as:

"a sequence of finger configura/on changes, and each each finger configura/on might produce a series of repe//ve mo/ons. A mo/on is repe//ve when any of its segments triggers the same type of ac/on as its whole."

Multi-Touch Gestures as State Machines

How an Algorithm May Generate State Machines for the Gestures it is Given

Callbacks

Each gesture has 6 callbacks: •  Possible •  Fail •  Begin •  Changed •  Cancel •  End

These callbacks are invoked when a transition takes place.

Evaluation

Discussion and Scope For Future Work

•  Tuning Parameters such as motion segment length and the threshold for low-pass filtering

•  Add more features such as the contour of the finger touches to the model, as it is extensible

•  Fine-tune recognition further by adding constraints like absolute location of a gesture on the screen

•  Of Course training with higher quality and more training data

Motion-Based Gesture Recognition

Based on the Paper:

Kazuya Murao, Tsutomu Terada, Ai Yano, Ryuichi Matsukura, Evaluating Gesture Recognition by Multiple-Sensor-Containing Mobile Devices

Jussi Tohka, Tampere University of Technology Department of Signal Processing, Introduc>on to Pa?ern Recogni>on

•  What is Pa?ern Recogni>on?

Introduc*on to Pa-ern Recogni*on


Placing objects to certain classes

•  That thing is a dog. •  That thing is a car. •  That thing is a dog. •  That thing is a chicken.

•  That thing is a dog. •  That thing is a cat.

h?p://www.improvedoutcomes.com/docs/WebSiteDocs/Classifica>on_and_Predic>on/SLAM/An_Introduc>on_to_Classifica>on.htm

Introduction to Pattern Recognition



•  Why Recogni>on?

•  Examples Op>cal Character Recogni>on (OCR) Ac>vity Recogni>on Gesture Recogni>on

Because is not clever enough.

1. Sensing 2. Pre-‐processing and segmenta>on 3. Feature extrac>on 4. Classifica>on 5. Post-‐processing


Five stage of Pa?ern recogni>on:


1. Sensing 2. Preprocessing and segmenta>on 3. Feature extrac*on 4. Classifica*on 5. Post-‐processing


Five stage of Pa?ern recogni>on:


•  That thing is a dog. •  That thing is a car. •  That thing is a dog. •  That thing is a chicken.

•  That thing is a dog. •  That thing is a cat.

h?p://www.improvedoutcomes.com/docs/WebSiteDocs/Classifica>on_and_Predic>on/SLAM/An_Introduc>on_to_Classifica>on.htm


Ac*vity Recogni*on from Accelerometer Data

Eight ac>vi>es: •  Standing •  Walking •  Running •  Climbing up stairs •  Climbing down stairs •  Sit-‐ups

•  Vacuuming •  Brushing teeth

Nishkam Ravi and Nikhil Dandekar and Preetham Mysore and Michael L. Li?man Department of Computer Science, Rutgers University

X-‐axis readings for different ac>vi>es

Ac*vity Recogni*on from Accelerometer Data

Data collec*on

Four features: •  Mean •  Standard Devia>on •  Energy •  Correla>on


Results

Classifier

Accuracy(%)

Setting1

Setting2

Setting3

Setting4

Boosted 98.86 98.71 89.96 64

kNN

98.15

99.26

72.93

49.67

Plurality Voting

99.57

99.82

90.61

65.33


Another experiment on Gesture recognition

Overview

•  Experiment - Capturing data for 27 kinds of gestures on 8 subjects - Investigated the effects on recognition accuracy of changing the number and positions of sensors, the number and kinds of gestures.

•  Results

- The use of multiple sensors and of sensors positioned at specific positions affects accuracy.

- Gestures are interdependent and selecting specific gestures improves recognition accuracy

Equipment

Experimental board with 9 accelerometers and 9 gyroscopes

Board: W117×H155×D16 (mm) weighed 200 g

Sensors: WAA-006, made by Wireless-T Inc a wireless 3-axis accelerometer

a 3-axis gyroscope

Gestures

Experiment

1. How? All the subjects were right-‐handed and

performed the gestures while holding the lower right of the board with their right hands. 2. Data?

Mean and variance over the sensor value of the gesture were extracted over the whole gesture and over the first 1 second(the la?er is be?er).

Experiment cont.

3. Analysis? •  Intra-subject analysis -- measures the

averaged accuracy over eight test subjects after 4-fold cross-validation is conducted for each subject

•  Inter-subject analysis -- conducts 4-fold cross validation after the data for the eight subjects are merged

Algorithm Support Vector Machine (SVM)

•  Basic idea of SVM -‐ Op>mal hyperplane for linearly separable

pa?erns -‐ Extend to pa?erns that are not linearly separable by transforma>ons of original data to map into new space

Support Vectors

•  Support vectors are the data points that lie closest to the decision surface

•  They are the most difficult to classify

•  They have direct bearing on the optimum location of the decision surface

Support Vectors(cont.)

Support Vector Machine (SVM)

•  Assume linear separability for now: – in 2 dimensions, can separate by a line

– in higher dimensions, need hyperplanes

•  Can find separating hyperplane by linear programming

– separator can be expressed as ax + by = c

Support Vector Machine (SVM)

SVMs maximize the margin around the separa>ng hyperplane.

Results and consideration

Number of Sensors

-‐ Using mul>ple homogeneous sensors improves accuracy

-‐ Mul>ple homogeneous sensors can capture the diversity among the subjects

Results and consideration cont.

Posi>on of Sensors

-‐ Accelerometer : further from the holding posi>on is be?er

-‐ Gyro : on the cross-‐ point of the axes of gyra>on for each gesture works well



Kind of gestures

- Gesture that involves parallel displacement and is not consistent are rarely recognized -‐ A gyro is inferior to an accelerometer


Number of gestures

-‐ Higher accuracy can be achieved without excluding a lot of gestures by selec>ng gestures that do not conflict with each other

Conclusion

This paper investigates the effects on gesture recognition accuracy by changing the number and positions of sensors and the kinds and number of gestures

Sources

•  http://yangl.org/pdf/gcoder.pdf •  http://www.cs.ucf.edu/courses/cap6412/

fall2009/papers/Berwick2003.pdf •  http://ieeexplore.ieee.org/

ielx5/5958741/5959573/05959595.pdf?tp=&arnumber=5959595&isnumber=5959573

•  http://www.improvedoutcomes.com/docs/WebSiteDocs/Classification_and_Prediction/SLAM/An_Introduction_to_Classification.htm

Documents

Gesture Recognitionhomepage.cs.uiowa.edu/~ochipara/classes/sensing_the... · Motion-Based Gesture Recognition Based on the Paper: Kazuya Murao, Tsutomu Terada, Ai Yano, Ryuichi Matsukura,