Activity recognition based on a multi-sensor hierarchical-

classifier

IWANN 2013, 12-14 June, Tenerife (Spain)

Oresti Baños, Miguel Damas, Héctor Pomares and Ignacio Rojas Department of Computer Architecture and Computer Technology, CITIC-UGR,

University of Granada, SPAIN

DG-Research Grant #228398

Introduction

• Activity recognition concept

– “Recognize the actions and goals of one or more agents from a series of observations on the agents' actions and the environmental conditions”

• Applications (among others)

– eHealth (AAL, telerehabilation)

– Sports (performance improvement, injury-free pose)

– Industrial (assembly tasks, avoidance of risk situations)

– Gaming (Kinect, Wii Mote, PlayStationMove)

• Categorization by sensor modality

– Ambient

– On-body

2

Sensing Activity

3

• Ambient sensors

Sensing Activity

• Ambient sensors

Limitations*

3rd Generation (and beyond…)

2nd Generation 1st Generation

Sensing Activity

5

• On-body sensors

Activity Recognition Chain (ARC)

6

Activity Recognition Chain (ARC)

7

Activity Recognition Chain (ARC)

8

Activity Recognition Chain (ARC)

9

Activity Recognition Chain (ARC)

10

Activity Recognition Chain (ARC)

11

Activity Recognition Chain (ARC)

12

Activity Recognition Chain (ARC)

13

[-0

.14

, 3.4

1, 4

,21

, … ,

6.1

1]

[-0

.84

, 3.2

1, 4

.21

, … ,

6.1

1]

[-0

.81

, 5.7

1, 4

.21

, … ,

6.2

2]

[-0

.14

, 3.9

2, 4

.23

, … ,

7.8

2]

S u p s1,s2,…,sk fℝ(s1,s2,…,sk) c

Activity Recognition Chain (ARC)

14

Activity Recognition Chain (ARC)

15

Activity Recognition Chain (ARC)

16

SENSOR FUSION

ARC Fusion: Feature Fusion

17

[-0

.14

, 3.4

1, 4

,21

, … ,

6.1

1]

[-0

.84

, 3.2

1, 4

.21

, … ,

6.1

1]

[-0

.81

, 5.7

1, 4

.21

, … ,

6.2

2]

[-0

.14

, 3.9

2, 4

.23

, … ,

7.8

2]

S1

S2

SM

u1 p1 s11,s12,…,s1k fℝ(s11,s12,…,s1k)

c u2 p2 s21,s22,…,s2k fℝ(s21,s22,…,s2k)

uM pM sM1,sM2,…,sMk fℝ(sM1,sM2,…,sMk)

fℝ(s11,s12,…,s1k,

s21,s22,…,s2k,…,

sM1,sM2,…,sMk)

ARC Fusion: Decision Fusion

18

[-0

.14

, 3.4

1, 4

,21

, … ,

6.1

1]

[-0

.84

, 3.2

1, 4

.21

, … ,

6.1

1]

[-0

.81

, 5.7

1, 4

.21

, … ,

6.2

2]

[-0

.14

, 3.9

2, 4

.23

, … ,

7.8

2]

S1

S2

SM

u1 p1 s11,s12,…,s1k fℝ(s11,s12,…,s1k) c1

c=φ(c1,c2,…,cM)

u2 p2 s21,s22,…,s2k fℝ(s21,s22,…,s2k) c2

uM pM sM1,sM2,…,sMk fℝ(sM1,sM2,…,sMk) cM

Multi-Sensor Hierarchical Classifier

19

SM

S2

S1 α11

∑ C12

C1N

C11

∑

C21

C22

C2N

∑

CM1

CM2

CMN

∑

Decisio

n

Class level Source level Fusion

β11

α12 β12

α1N β1N

α21 β21

α22 β22

α2N β2N

αM1 βM1

αM2 βM2

αMN βMN

γ11,…,1N δ11,…,1N

γ21,…,2N δ21,…,2N

γM1,…,MN δM1,…,MN

[-0

.14

, 3.4

1, 4

,21

, … ,

6.1

1]

[-0

.84

, 3.2

1, 4

.21

, … ,

6.1

1]

[-0

.81

, 5.7

1, 4

.21

, … ,

6.2

2]

[-0

.14

, 3.9

2, 4

.23

, … ,

7.8

2]

S1

S2

SM

u1 p1 s11,s12,…,s1k fℝ(s11,s12,…,s1k)

u2 p2 s21,s22,…,s2k fℝ(s21,s22,…,s2k)

uM pM sM1,sM2,…,sMk fℝ(sM1,sM2,…,sMk)

Multi-Sensor Hierarchical Classifier

20

N activities M sensors & Class level Source level Fusion

Multi-Sensor Hierarchical Classifier

21

N activities M sensors & Class level Source level Fusion

Multi-Sensor Hierarchical Classifier

22

N activities M sensors & Class level Source level Fusion

Multi-Sensor Hierarchical Classifier

23

N activities M sensors & Class level Source level Fusion

Experimental setup: dataset

• Fitness benchmark dataset

• Up to 33 activities

• 9 IMUs (XSENS) ACC, GYR, MAG

• 17 subjects

24 Baños, O., Toth M. A., Damas, M., Pomares, H., Rojas, I., Amft, O.: A benchmark dataset to evaluate sensor displacement in activity recognition. In: 14th International Conference on Ubiquitous Computing (Ubicomp 2012), Pittsburgh, USA, September 5-8, (2012)

Results

• Segmentation: sliding window (6 seconds) • Feature extraction: FS1={mean}, FS2={mean,std}, FS3={mean,std,max,min,cr} • Classification: Decision tree (C4.5) (10-fold cross-validated, 100 repetitions)

25 10 activities 20 activities 33 activities

FS1 FS2 FS3 FS1 FS2 FS3 FS1 FS2 FS360

65

70

75

80

85

90

95

100

Accura

cy (

%)

Feature Fusion Weighted Majority Voting Multi-Sensor Hierarchical Classifier

Experimental Parameters

Conclusions

• We propose a multi-sensor hierarchical classifier that allows data fusion of multiple sensors

– Its assymetric decision weighting (SEinsertions/SPrejections) leverages the potential of the classifiers either for classification/rejection or both

– Specially suited for complex scenarios

• Feature Fusion and MSHC are quite in line in terms of performance however

– Our method outperforms the former when a more informative feature set is used

– Particularly notable for complex recognition scenarios

• Our model is expected to be particularly suited to deal with sensor anomalies (work-in-progress)

26

On-going work…

• Our model is expected to be particularly suited to deal with sensor anomalies (work-in-progress)

27

FEAT-FUSION MSHC0

20

40

60

80

100

Accura

cy (

%)

Ideal Self Induced

Thank you for your attention. Questions?

Oresti Baños Legrán Dep. Computer Architecture & Computer Technology

Faculty of Computer & Electrical Engineering (ETSIIT) University of Granada, Granada (SPAIN)

Email: oresti@atc.ugr.es Phone: +34 958 241 516 Fax: +34 958 248 993

Work supported in part by the HPC-Europa2 project funded by the European Commission - DG Research in the Seventh Framework Programme under grant agreement no. 228398, the Spanish CICYT Project SAF2010-20558, Junta de Andalucia Project P09-TIC-175476 and the FPU Spanish grant AP2009-2244.

28