uWave: Accelerometer-based Personalized Gesture

uWave: Accelerometer-based Personalized Gesture Recognition and Its ApplicationsRecognition and Its Applications

Jiayang Liu, Zhen Wang, and Lin Zhong Jehan Wickramasuriya and Venu Vasudevany g , g, gDepartment. Of Electrical Computer

Engineering Rice University, Houston TX 77005

jiayang@rice.edu, wangzhen127@gmail.com, lzhong@rice edu

yPervasive Platforms & Architectures Lab

Applications & Software Research Center, Motorola Labs

{jehan,venu}@motorola.com lzhong@rice.edu

Matthew Knappmknapp@wpi.eduCS 525w Mobile Computing

Introduction to uWaveIntroduction to uWave

• “efficient recognition algorithm”efficient recognition algorithm– Focus on Gestures/Physical Manipulation– User-dependent Gesture Recognitionp g– Dynamic Time Warping

• Goal: Support efficient personalized gesture recognition on a wide range of devices

2

Worcester Polytechnic Institute

2

Related WorkRelated Work

• Computer Vision/Vision BasedComputer Vision/Vision Based Techniques– Translates a “gesture” into “handwriting”– Fundamentally Limited by Hardware

Requirements• Hidden Markov Models

– Require extensive training data to be effectiveR i k l d f th b l i d– Require knowledge of the vocabulary in order to configure the model

3

Worcester Polytechnic Institute

3

Related WorkRelated Work

• Dynamic Time Warping (DTW)Dynamic Time Warping (DTW)– Algorithm for measuring similarity between

two sequences which may vary in time or speed

– Allows a computer to find an optimal match between two given sequences with certainbetween two given sequences with certain restrictions

4

Worcester Polytechnic Institute

4

Technical ChallengesTechnical Challenges

• Gesture Recognition lacks a standardizedGesture Recognition lacks a standardized “vocabulary”

• Spontaneous interaction requires p qimmediate engagement

5

Worcester Polytechnic Institute

5

uWave Algorithm Design: O iOverview

• Premise: “Human gestures can bePremise: Human gestures can be characterized by the time series of forces applied to the handheld device”

• Template Library– Store of one or more time series of known

identities for every vocabulary gesture• Input: Time series of acceleration

id d b th i l tprovided by a three-axis accelerometer

6

Worcester Polytechnic Institute

6

uWave Algorithm Design: O iOverview

7

Worcester Polytechnic Institute

7

uWave Algorithm Design: Q i iQuantization

8

Worcester Polytechnic Institute

8

uWave Algorithm Design: D i Ti W iDynamic Time Warping

9

Worcester Polytechnic Institute

9

uWave Algorithm Design:T l Ad iTemplate Adaptation

• Variation between gesture samplesVariation between gesture samples by same user

• Should adapt templates to• Should adapt templates to accommodate variationsU d ti S h• Updating Schemes:– Positive Update– Negative Update

10

Worcester Polytechnic Institute

10

Prototype ImplementationPrototype Implementation

• Wii remote prototypeWii remote prototype– Accelerometer range: -3g to 3g– Noise below 3.5mgg

• Recognition results returned without perceptible delay on PCs (template library of 8 gestures)– 2ms on Lenovo T60– 4ms on T-Mobile MDA Pocket PC– 300ms on 16-bit microcontroller in the Rice

Orbit sensor 11Orbit sensor

Worcester Polytechnic Institute

11

Gesture VocabularyGesture Vocabulary

12

Worcester Polytechnic Institute

12

Evaluation: SetupEvaluation: Setup

• Uses the gesture vocabulary fromUses the gesture vocabulary from previous slide

• 8 Participants• 8 Participants– 2 undergraduate, 8 graduate

7 l 1 f l– 7 male, 1 female– All 20s or early 30s, right handed

13

Worcester Polytechnic Institute

13

Evaluation: Data CollectionEvaluation: Data Collection

• Gestures are collected from 7 daysGestures are collected from 7 days within a period of about 3 weeks

• Each day the participant uses the Wii• Each day the participant uses the Wii remote and performs the 8 gestures, 10 times each10 times each

• Database at the end consists of 4480 t t t l d 560 f hgestures total and 560 for each

participant14

Worcester Polytechnic Institute

14

Evaluation: Recognition i h Ad iwithout Adaptation

• Evaluate uWave using the gestures fromEvaluate uWave using the gestures from each subject separately

• Use Bootstrapping to improve statistical pp g psignificance

• Use the collected samples to generate 70 p gtests of uWave– Produces 70 confusion matrixes– Averaged into 1 confusion matrix per subject– Average confusion matrixes of the 8 subjects

combined into a final confusion matrix 15combined into a final confusion matrix

Worcester Polytechnic Institute

15

Evaluation: Recognition i h Ad iwithout Adaptation

16

Worcester Polytechnic Institute

16

Evaluation: Recognition i h Ad iwithout Adaptation

• Average Accuracy of 93.5%g y– Gestures 1,2,6 and 7 have lower accuracy due

to similar hand movements• Large variation (9%) among participants

– “The participant with the highest accuracy performed the gestures in larger amplitude andperformed the gestures in larger amplitude and slower speed compared to other participants”

• Temporal Compression of the data speeds p p pup recognition by more than 9 times without negatively affecting accuracy

17

Worcester Polytechnic Institute

17

Evaluation: Recognition i h Ad iwithout Adaptation

Evaluation Using Samples from theEvaluation Using Samples from the Same Day

• Significantly Higher Accuracy (98 4%) when usingSignificantly Higher Accuracy (98.4%) when using only samples from the same day

• Results reported in previous reports may have been overly optimistic

• “The difference between Figure 4 (Left) and Figure 4 (Right) highlights the possible variationsFigure 4 (Right) highlights the possible variations for the same gesture from the same user over multiple days and the challenge it poses to

iti ” 18recognition.”

Worcester Polytechnic Institute

18

Evaluation: Recognition with Ad iAdaptation

19

Worcester Polytechnic Institute

19

Evaluation: Recognition with Ad iAdaptation

• Produced 7 confusion matrixes for each participants

• Averaged into confusion matrix on previous slide

• Accuracy:– Positive Update: 97.4%– Negative Update: 98.6%

• Accuracy is much better than without adaptation

Cl t d 20– Close to same day accuracy

Worcester Polytechnic Institute

20

uWave-Enhanced Applications: Gesture based Light WeightGesture-based Light-Weight

User Authentication• Prioritizes Ease-of-use over hard security• Privacy InsensitivePrivacy Insensitive• Enables authentication based on physical

manipulation of the devicep• Ran studies that showed uWave can

recognize user-defined gestures with g ghigher than 99.5% accuracy

21

Worcester Polytechnic Institute

21

uWave-Enhanced Applications: Gesture based 3D Mobile UserGesture-based 3D Mobile User

Interface• Intuitive and Convenient to navigate a 3D

interface with 3D hand gesturesS i l N ki b d id h i• Social Networking-based video-sharing serviceR t ti Ri I t f• Rotating Ring Interface– Employed uWave to navigate the interface

Uses a series of specific movements such as– Uses a series of specific movements such as tilting or slight shaking

22

Worcester Polytechnic Institute

22

Discussion of uWaveDiscussion of uWave

• Gestures and Time Series of ForcesGestures and Time Series of Forces– Diverse opinions on what is a unique gesture– Closer to speech than handwritingp g

• Challenge of Tilt– uWave uses a single three-axis accelerometer– Tilt can change the readings of force applied– Opportunity for detecting tilt is limited with a

i l l tsingle accelerometer– Extra Sensors needed to fully address problem

23

Worcester Polytechnic Institute

23

Discussion of uWaveDiscussion of uWave

• User-Dependent vs. User IndependentUser Dependent vs. User Independent Recognition– Much Lower Accuracy for User Independent

Recognition (75.4% down from 98.4%)– No commonly accepted gestures for

InteractionsInteractions• Gesture Vocabulary Selection

More Complicated Gestures may have higher– More Complicated Gestures may have higher accuracy

– Number of Complicated Gestures Users can 24use may be small

Worcester Polytechnic Institute

24

ConclusionsConclusions

• Employs a single accelerometer so it canEmploys a single accelerometer so it can be readily implemented on current devices

• Uses DTW to measure similarities between two time series of forces

• Tests show uWave achieves 98.6% accuracy with one training sample– Comparable to HMM-based methods with 12

i i ltraining samples• Challenges of Variation across Time and

Users 25Users

Worcester Polytechnic Institute

25

Video DemonstrationVideo Demonstration

• uWave DemonstrationuWave Demonstration

26

Worcester Polytechnic Institute

26

Questions?

27

Worcester Polytechnic Institute

27