Human Activity Recognition using

Smartphone

Amin Rasekh, Chien-An Chen, Yan Lu

CSCE 666 Project Presentation

Introduction◦ Human Activity Recognition◦ Active Learning

Goals Literature Review Methods

◦ Data Collection and Feature Extraction◦ Classification Techniques◦ Query Strategies of active learning

Results Conclusions

Outline

Using sensors to identify human activities such as walking, jogging, limping.

Motivation◦ Human survey (study human daily activities)◦ Medical care (diabetes, elderly, rehabilitation)

Sensors types◦ Inertial sensors (accelerometer, gyroscope)◦ Camera◦ GPS

Smartphone is small and convenient to carry around and its computational resource is powerful enough for our purpose.

Introduction: Activity Recognition

Passive Learning: What we have studied in classWe can achieve greater accuracy with fewer training labels if we choose the data from which we learn

Motivation: To minimize the time and labor for labeling abundant data

Introduction: Active Learning

Design a simple, light weight, and accurate system that can learn human activity with minimum user interaction.

◦ Compare and find a model that best fit our system in terms of accuracy and efficiency.

◦ Reduce the labeling time and labor works using active learning.

Goals

Use one or multiple camera to do a vision-based recognition [5,6].

Install multiple inertial sensors on the body. [1, 2, 3,4]

A mixture between vision-based and inertial sensor system.[7]

Classifiers such as Bayesian Decision Making, KNN, SVM, ANN were studied before. [10,11]

Features from time domain, frequency domain and wavelet analysis have been studied.[8,9]

Literature review

Data Collection◦ Smartphone: HTC EVO 4G◦ Sensor: 3D accelerometer,50 Hz ◦ Cellphone in pockets around waist◦ 3 people 5 activities: walking, biking, walking upstairs,

walking downstairs, jogging, limping

Feature Generation (Total 31 features)◦ Sampling Window: 256 samples (5.12 seconds)◦ Time Domain:

Variance, Mean, 25% Percentile, 75% Percentile, Correlation, Average Resultant Acceleration

◦ Frequency Domain: Energy, Entropy, Centroid Frequency, Peak Frequency

Methods

Classification Techniques◦ Quadratic ◦ K-Nearest Neighbors◦ Support Vector Machines◦ Artificial Neural Networks

Query Strategies based on Uncertainty◦ Quadratic: Distance from discriminant curve◦ KNN: Entropy◦ SVM: Distance from the boundary◦ ANN Discriminant function values

Methods

Query is performed for the unlabeled instance that is nearest to the discriminant curve or SVM boundary

Query Strategy: Distance Measure

Random Query

Active Query

Query is performed for the unlabeled instance that has the maximum entropy:

Query Strategy: Entropy Measure

Results: LDA Subspace

-1.8 -1.6 -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0-2.6

-2.4

-2.2

-2

-1.8

-1.6

-1.4

-1.2

-1

-0.8

LDA component 1

LD

A c

ompo

nen

t 2

walkinglimpingjoggingdownstairupstair

8.4 8.6 8.8 9 9.2 9.4 9.6 9.8 10 10.2 10.47

7.5

8

8.5

LDA component 3

LD

A c

ompo

nen

t 4

walkinglimpingjoggingdownstairupstair

Results: Passive Learning

◦ Sequential Forward Selection (Wrapper)

◦ Algorithm: SVM

◦ 10-Fold Cross Validation for each feature subset

◦ Best Features

Variance, 25% Percentile, Frequency-Domain Entropy, Peak Frequency

◦ Classification Rate of SVM+LDA: 78%

◦ Classification Rate of SVM+SFS: 84%

Results: Feature Subset Selection

Results: LDA Space for Hw2 Data

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

First LDA Component

Seco

nd L

DA

Com

ponent

Results: Active Learning on Hw2 Data

0 50 100 1500.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

Number of Instance Queries

Cla

ssifi

catio

n ra

te

Active LearningRandom Sampling

KNN

SVM

Quadratic

Results: Active Learning on Activity Data

0 50 100 150 200 250 3000.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

Number of instance queries

Cla

ssifi

catio

n ra

te

Active LearningRandom Sampling

-1.8 -1.6 -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0-2.6

-2.4

-2.2

-2

-1.8

-1.6

-1.4

-1.2

-1

-0.8

LDA component 1

LD

A c

ompo

nen

t 2

-1.8 -1.6 -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0-2.6

-2.4

-2.2

-2

-1.8

-1.6

-1.4

-1.2

-1

-0.8

LDA component 1

LD

A c

ompo

nen

t 2

Active learning with SVM

Random sampling with SVM

QuadraticKNN SVM

Improving the performance of active learning for activity recognition problem◦ Clustering◦ Hybrid query strategies

Adding more activities such as biking

Works in Progress

We achieved a classification rate of over 80% on 5 human activities using a smartphone.

The result is robust to common positions and orientations of cellphone.

SVM+SFS gives the best performance and is promising to run on mobile devices.

Performance of active learning is highly sensitive to the type of problem

Conclusions

Thank you!

Questions?

1) L. Bao and S. S. Intille, “Activity recognition from user-annotated acceleration data,” Pers Comput., Lecture Notes in computer Science, vol. 3001, pp. 1–17, 2004.

2) U. Maurer, A. Rowe, A. Smailagic, and D. Siewiorek, “Location and activity recognition using eWatch: A wearable sensor platform,” Ambient Intell. Everday Life, Lecture Notes in Computer Science, vol. 3864, pp. 86–102, 2006.

3) J. Parkka, M. Ermes, P. Korpipaa, J. Mantyjarvi, J. Peltola, and I. Korhonen, “Activity classification using realistic data from wearable sensors,” IEEE Trans. Inf. Technol. Biomed., vol. 10, no. 1, pp. 119–128, Jan. 2006.

4) N.Wang, E. Ambikairajah,N.H. Lovell, and B.G. Celler, “Accelerometry based classification of walking patterns using time-frequency analysis,” in Proc. 29th Annu. Conf. IEEE Eng. Med. Biol. Soc., Lyon, France, 2007, pp. 4899–4902.

5) T.B.Moeslund,A.Hilton,V.Kr ¨ uger, Asurveyofadvancesinvision-based human motioncaptureandanalysis,Comput.VisionImageUnderstanding 104 (2–3)(2006)90–126.

6) T.B. Moeslund, E. Granum, A survey of computer vision-based human motion capture, Comput. Vision Image Understanding 81 (3) (2001) 231–268.

7) Y. Tao, H. Hu, H. Zhou, Integration of vision and inertial sensors for 3D arm motion tracking in home-based rehabilitation, Int. J. Robotics Res. 26 (6) (2007) 607–624.

8) Preece S J, Goulermas J Y, Kenney L P J and Howard D 2008b A comparison of feature extraction methods for the classification of dynamic activities from accelerometer data IEEE Trans. Biomed. Eng. at press

9) N. Ravi, N. Dandekar, P. Mysore, and M. L. Littman. Activity recognition from accelerometer data. In AAAI, pages 1541–1546, 2005.

10) S.J. Preece, J.Y. Goulermas, L.P.J. Kenney, D. Howard, K. Meijer and R. Crompton, Activity identification using body-mounted sensors—a review of classification techniques. Physiol Meas,  30  (2009), pp. R1–R33.

11) Altun, K., Barshan, B., Tun¸cel, O.: Comparative study on classifying human activities with miniature inertial and magnetic sensors. Pattern Recogn. 43(10), 3605–3620 (2010), doi:10.1016/j.patcog.2010.04.019

Reference

Support Vector Machine

Results (can be removed)

Results: Active Learning on Hw2 Data

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30

-50 -45 -40 -35 -30 -25-30

-20

-10

0

10

20

30