Virtual “Valipilla” : Air Gesture Based Tool for

Practicing the English Alphabet Writing

U.V Vandebona

Reg No. : 2013/MCS/072 Index No. : 13440722

Supervisor : Dr. G.D.S.P. Wimalaratne

Master of Computer Science - UCSC - Final Year Project - 2016Date : 2016-Feb-9

Air Writing• Yoshihiro Itaguchi et al, “Writing in the Air: Contributions

of Finger Movement to Cognitive Processing”, Journal of Public Library of Science - PLoS One, June 2015

• Human Computer Interaction - Writing interaction with the computer

• Machine Learning - Recognize what was written

HCI ML

Popular Consumer Vision Sensor Devices

Microsoft Kinect [1]

Image Reference:[1] http://www.xbox.com/en-US/xbox-one/accessories/kinect-for-xbox-one [2] http://asia.creative.com/p/web-cameras/creative-senz3d [3] http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html [4] https://www.leapmotion.com/

Leap Motion [4]

Normal Web Cams and 3D Cams

Intel RealSense [3]Creative Senz3D [2]

Air Written Character Recognition Techniques

Warping Methods

Dynamic Time Warping

(DTW)

Hilbert Warping (HW)

Statistical Methods

Hidden Markov Model (HMM)

Artificial Neural Network (ANN)

Template Matching

$P Cloud Point

Related Work• Kinect Sensor with DTW and SVM Approach

– Chengzhang Qu et al, “Online Kinect Handwritten Digit Recognition Based on Dynamic Time Warping and Support Vector Machine”, Journal of Information & Computational Science, pp 413 - 422, January 2015

• Kinect Sensor with Neural Gas Network Approach– MRA Heidari et al, “Writing in the Air Using Kinect and Growing Neural Gas Network”, Jurnal Teknologi

Universiti Teknologi Malaysia, vol. 72, no. 5, 2014

• Leap Motion Sensor with DTW Approach– Vikram Sharad et al, "Writing and sketching in the air, recognizing and controlling on the fly.," in ACM

Conference on Human Factors in Computing Systems (CHI), 2013.

• Leap Motion Sensor with HMM Approach– Mingyu Chen, "Universal Motion Based Control," School of Electrical and Computer Engineering, Georgia

Institute of Technology, PhD. Dissertation 2013.

• Webcam with Neural Network Approach– Aditya G. Joshi et al, "Touchless Writer a hand gesture recognizer for Englsih characters," in Proceedings of

22nd IRF International Conference, Pune, India, January 2015.

• Webcam with HW Approach– Hiroyuki Ishida, Tomokazu Takahashi, Ide Ichiro, and Murase Hiroshi, "A Hilbert warping method for

handwriting gesture recognition," Pattern Recognition, vol. 43, no. 0031-3203, pp. 2799-2806, August 2010.

Proposed Prototype

Writing Capturing Recognition Feedback

Left Handed

Right Handed

Gesture Type

Finger Tip

Tool Tip• Pencil

tip• Pen tip

Left Handed

Right Handed

•Start

•Clear

•Check

•Back

•Sound

•Next

Motion Segmentation

Motion Segmentation Techniques

Explicit Delimitation

Key/Button Press

Break by Gesture

Virtual Touch Zone

Automatic Detection

Spotting Approach

Sliding Window

Approach

• Vision input devices constantly streams the location of the fingers within its field of view.

Virtual Touch Input Zone

+1

0

-1Touch Zone

Hover Zone

Cursor Pointer

Proposed Approach

Writing Capturing Recognition Feedback

2D Virtual Writing Interface

Proposed Approach

Writing Capturing Recognition Feedback

Normalization Process

• Resample• Scale with

Shape Preservation

• Translation to Origin

Measure Cloud Distance

• Use Euclidean Distance

Greedy Cloud Match

• Recognized Template

$P Point Cloud Framework

$P Point Cloud

• “The $P Point-Cloud Recognizer is a 2-D gesture recognizer designed for rapid prototyping of gesture-based user interfaces. In machine learning terms, $P is an instance-based nearest-neighbor classifier with a Euclidean scoring function, i.e., a geometric template matcher [1]. ”

Reference[1] (2016, January), MAD Lab - University of Washington, $P Point-Cloud Recognizer, [Online] http://depts.washington.edu/aimgroup/proj/dollar/pdollar.html

$P Point Cloud - Advantages

• Independent of– Scale– Number of Strokes– Stroke Direction– Stroke Order – Stroke Type (Uni-Stroke & Multi Stroke)

$P Point Cloud - Normalization

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 50 100 150 200 2500

50

100

150

200

250

Stroke 2

Template Defined for letter ‘R’

Normalized Template for Matching

• Resample• Scale with

Shape Preservation

• Translation to Origin

Stroke 1

Stroke 3

$P Point Cloud - Normalization

Air Writing for letter ‘R’

Normalized Air Writing for Matching

• Resample• Scale with

Shape Preservation

• Translation to Origin

0 100 200 300 400 500 600 700 800 900 10000

100

200

300

400

500

600

700

800

900

1000

-0.600000000000001 0.399999999999999

-0.600000000000001

-0.400000000000001

-0.200000000000001

-5.55111512312578E-16

0.199999999999999

0.399999999999999

0.6

$P Point Cloud - Recognition

Template Library

(T)Candidate

(C)

(Tt1)

(Tt2)

(Tt3)

d1

d2

d3

Cloud Distance : d1 < d2 < d3

-0.600000000000001 -0.100000000000001 0.399999999999999

-0.600000000000001

-0.400000000000001

-0.200000000000001

-5.55111512312578E-16

0.199999999999999

0.399999999999999

0.6

Template Point Cloud (Normalized)

Candidate Air Writing Point Cloud (Normalized)

Matching as an Assignment Problem

?

$P Point Cloud - Greedy Cloud Matching

Reference[1] (2016, January), MAD Lab - University of Washington, $P Point-Cloud Recognizer, [Online] http://depts.washington.edu/aimgroup/proj/dollar/pdollar.html

123

$P Point Cloud - Cloud Distance

Template Point Cloud (Normalized)

Candidate Air Writing Point Cloud (Normalized)

3 421

123

3 421

44

Tj

Ci

Tj

Ci

Sum of Euclidean distances with a confidence weight

Loop 1.1 Loop 1.1.2

123

3 421

4

Tj

Ci

Loop 1.2

123

3 421

4

Tj

Ci

Loop 2….

Min

imum

Clou

d Di

stan

ce

$P Point Cloud - Cloud Distance

Reference[1] (2016, January), MAD Lab - University of Washington, $P Point-Cloud Recognizer, [Online] http://depts.washington.edu/aimgroup/proj/dollar/pdollar.html

Writing Capturing Recognition Feedback

• Cloud Distance is normalize as a score between 0 and 1 and presented as a percentage which is beneficial for a general user to understand.

Proposed Approach

Recognized As

Feedback

Evaluation - Character Recognition

No of English letters tested :

26

No of samples from each letter tested :

24

Path On Mode : 12

Left Handed : 6

Right Handed : 6

Path Off Mode : 12

Left Handed : 6

Right Handed : 6

Total tested samples for each gesture type : 26 24 = 624 [312 samples on each mode]

Results - Character Recognition

• Accuracy on Tooltip Air Writing

0 5 10 15 20 250.00

20.00

40.00

60.00

80.00

100.00

120.00

Series1

PrecisionSensitivity

• Path On Mode = 100% • Path Off Mode = 90.7%

Ergonomics

• Involve more muscles; As a result cause more arm fatigue• Educational Intervals in-between• Rest the elbow and writes with the movement of

the upper arm and wrist.• Workstation/Leap Motion positioning

adjustments.• Preferred tool based air writing

Conclusion

• For Practice Writing : Touch Vs Vision– Wide Interaction Area– Invulnerable to Scratches• User doesn’t need to carry an expensive item

which prone to accidental damages.– With improvements can be used as a touch

based system• Using physical touch plane obtained from the

environment (ex: a wall, a board, etc.)

Thank You