1/35

UCAmI2015

DeustoTech - Deusto Institute of Technology, University of Deustohttp://www.morelab.deusto.es

December 2, 2015

Facing up social activity recognition using smartphone sensors

Pablo Curiel, Ivan Pretel, Ana B. Lago

2/35

UCAmI2015

Outline

Introduction

System Design

Evaluation

Conclusion

3/35

UCAmI2015

Introduction

System Design

Evaluation

Conclusion

4/35

UCAmI2015

Introduction

Introduction

AT HOME

.

.

.

5/35

UCAmI2015

Introduction

.

.

.

6/35

UCAmI2015

Introduction

► Location-based services► Foursquare, Twitter, Google Keep,…

► Low-level inference► Physical activity: walking, running,

cycling,…► High-level inference

► High-level user activities: cooking, reading novel,…

► Environments or surroundings: home, bar, public transport

.

.

.

7/35

UCAmI2015

Introduction

► Socialization as a high-level user activity► based on environment recognition► provides “social reminders”

.

.

.

@

@@

8/35

UCAmI2015

Introduction

System Design

Evaluation

Conclusion

9/35

UCAmI2015

System Design: Context capture

► Environments► Bar, café, sports bar, disco and restaurant

► Characteristics► Noisy places► Stationary positions► Artificially lighted places

10/35

UCAmI2015

System Design: Context capture

► Captured Data► Audio

►RMS power and dBs►Microphone

► Acceleration►3-axial acceleration►Acceleration,

gyroscope and geomagnetic sensors

► Ambient luminosity►Luxes►Luminosity sensor.

► Screen status

► Used devices► LG Nexus 4 (100

hours)► HTC Desire 816 (20

hours)

11/35

UCAmI2015

Data processing

► 3 steps► 1. Data fusion► 2. Data transformation► 3. Feature extraction

1. Data Fusion2. Data transformation3. Feature extraction

12/35

UCAmI2015

Data processing

► 1. Data fusion► Timestamps► Gathering halts► Sample rate

►50Hz, 20Hz, 10Hz, 5Hz, 2Hz and 1Hz

1. Data Fusion2. Data transformation3. Feature extractionRMS, dBs

Acceleration, gyroscope, compassLuminosity, screen

13/35

UCAmI2015

Data processing

► 1. Data fusion► 2. Data transformation

► Raw to processed characteristics1. Data Fusion2. Data transformation3. Feature extractionRMS, dBs

Acceleration, gyroscope, compassLuminosity, screen

LPF(RMS), LPF(dBs)Lineal-acc., earth-acc.log(lum), fixedLum, log(fixedLum)

+

14/35

UCAmI2015

Data processing

► 1. Data fusion► 2. Data transformation► 3. Feature extraction

1. Data Fusion2. Data transformation3. Feature extractionRMS, dBs

Acceleration, gyroscope, compassLuminosity, screen

Max, min, mean, median, standard deviation LPF(RMS),

LPF(dBs)Lineal-acc., earth-acc.log(lum), fixedLum, log(fixedLum)

+

15/35

UCAmI2015

Introduction

System Design

Evaluation

Conclusion

16/35

UCAmI2015

Evaluation

► Training Set► 10x5-fold cross

validation► Nexus 4 70h

► Test Set► Nexus 4 30h► HTC Desire 20h

► Classifiers► Random forest► Support vector

machine (SVM) -Gaussian radial basis function kernel

► k-Nearest Neighbours (k-NN)

► Naive Bayes classifier

► Parameters► The best features to

use► The most suitable

window sizes► Classifier comparison► Sensor sampling rate

comparison► Performance

► Recall► Specificity► AUC► Accuracy

What is the best combination of parameters to detect bar-like environments?

17/35

UCAmI2015

Evaluation

► Feature comparison► Acceleration features comparison

►Vector norm -> Random forest, SVM and k-NN leads to better results

►Types of acceleration– Linear = “Earth-acceleration” – Base acc. better than Linear&“Earth-acceleration”

(Random forest and SVM, 4%)► Audio features comparison

►dB better than RMS:– SVM(4% - 9%), k-NN(6% - 15%), Naive Bayes (2% -

8%)►Filtered better than Unfiltered (k-NN is the only

exception)► Luminosity features comparison

►Combination of log transformation and the fixed version is the best choice– Random Forest (1%), SVM(3%), k-NN(-), Naive

Bayes (11%)

18/35

UCAmI2015

Evaluation

► Contribution of each sensor► Training with the best performing feature of each

sensor ►concluded in the previous comparisons

► Results►Audio exclusion declines from 15% to 20%►Acceleration exclusion declines from 1% to 10%►Luminosity only useful for SVM and Naive Bayes

19/35

UCAmI2015

Evaluation

► Window size comparison► Common pattern: The smaller the window size, the

worse the results► Random Forest

►240 seconds ►120 or 90 -> 2% performance lost

► SVM►120 seconds►60 seconds -> 2% performance lost

►k-NN classifier►180 seconds ►60 seconds-> less than 2% performance lost

►Naive Bayes►240 seconds ►120 seconds -> 2% performance lost

20/35

UCAmI2015

Evaluation

► Sample rate comparison►Smaller window sizes suffer more than bigger

ones when this parameter is decreased

21/35

UCAmI2015

Evaluation

► Classifier comparison►The best is SVM

►+ recall►+ AUC►+ accuracy

► Random Forest►+ specificity

22/35

UCAmI2015

Evaluation

► The best performing configuration► SVM► Features

►Linear acceleration

►Filtered dBs►Log-

transformed fixed luminosity

► Capable of generalizing to new environments►User and

device dependencies

Bar-like TPFN

Other FPTN

Bar-like TPFN

Other FPTN

► Results

23/35

UCAmI2015

Introduction

System Design

Evaluation

Conclusion

24/35

UCAmI2015

Conclusion

► Findings► The preliminary results obtained seem promising

regarding the recognition of new locations for the same user.

► However, generalization to new users seems to be more troublesome.

► Future work► New data collection campaign which involves more

users in order to better study these aspects►Study what is the most descriptive value for

each feature (mean, median, standard deviation, minimum and maximum)

► Search for better recognition results with separate classes for each type of bar-like environment, as this could potentially enable to better capture the particular characteristics each of these environments has.

25/35

UCAmI2015

Thank you for your attention

26/35

UCAmI2015

DeustoTech - Deusto Institute of Technology, University of Deustohttp://www.morelab.deusto.es

Facing up social activity recognition using smartphone sensors

Pablo Curiel, Ivan Pretel, Ana B. Lago{pcuriel@deusto.es} {ivan.pretel@deusto.es} {anabelen.lago@deusto.es}

27/35

UCAmI2015

All rights of images are reserved by the original owners*, the rest of the content is licensed

under a Creative Commons by-sa 3.0 license.

*• http://mami.uclm.es/ucami-iwaal-amihealth-2015• https://flic.kr/p/enRrs9• https://www.iconfinder.com/yudha_ap• https://www.iconfinder.com/iconsets/stash• https://www.iconfinder.com/DemSt• https://www.iconfinder.com/paomedia• https://flic.kr/p/eD7GR• https://flic.kr/p/8G1yiU