Design & Implementation of a Gesture Recognition

System

Isaac Gerg

B.S. Computer EngineeringThe Pennsylvania State University

Necessity

KiosksVehicle ControlVideo Gaming

Large Screen OS ControlNovelty

Types of Gestures

Static Gestures

Dynamic Gestures

MTrackSoftware Characteristics

• Runs in Windows• COTS Hardware Support• Utilizes DirectX

Classifier Characteristics• Recognize four fundamental gestures plus

variations for a total of 9 actions.

System Architecture

RGB to HSVConversion

Image Thresholding(pdf)

Feature VectorGenerator

CAMSHIFT

MicrostateAssignment

Action Engine

MacrostateAssignment Win32 API

VideoStream

Display to User

5 Stages

System Architecture

Stages (in order or processing)1. RGB to HSV Colorspace conversion.2. Image Thresholding (pdf)3. CAMSHIFT4. Microstate Assignment5. Action Engine

• Macrostate Assignment• Win 32 API

Thresholding

Dealing with Noise

Mathematical Morphology Operations

DiscriminantHu Invariant Moments

Scale, Rotation, and Translation Invariant

20321

2123003211230

20321

21230123003217

03211230112

03212

123002206

20321

2123003210321

20321

21230123012305

20321

212304

20321

212303

2202202

02201

)()(3))(3()(3)())(3(

))((4)()()(

)()(3))(3()(3)())(3(

)()(

)3()3(

4)(11

I

I

I

I

I

I

I

Classification1st & 2nd Order Invariant Moments

0

100

200

300

400

600 700 800 900

1st Order

2nd

Ord

er

Open Hand, Open Fingers

Open Hand, Closed Fingers

Fist

1 Finger

Classification

The need for a Distance Metric.

ClassifierThe Mahalanobis Distance

Minimum Distance Classifier

)()()( 12 mxSmxxD ttt

xt = feature vector at time t of unknown class.

m = mean vector of samples.S = covariance matrix of samples.

Micro/MacrostatesStatistical physics paradigm

Last chance to correct before taking action

Provides contextual analysis

Implemented using order statistics

MTrack in Action

 

MTrack in Action

Tracker Settings

The FutureVideo Filtering (Wiener Filtering, Kalman Filtering)

Morphological Filtering

Trainable Data Sets

Macrostate Improvement

Referenceshttp://www.galactic.com/Algorithms/discrim_mahaldist.htm

J. Flusser and T. Suk, "Affine Moment Invariants: A New Tool for Character Recognition, " Pattern Recognition Letters, Vol. 15, pp. 433-436, Apr. 1994.

Bradski, G. R., “Computer Vision Face Tracking For Use In A Perceptual User Interface.” Intel Technology Journal, 1998(2).