1

Mobile Phones based Continuous Sensing Systems

Kiran Rachurikkr27@cam.ac.uk

Computer LaboratoryUniversity of Cambridge

2

• Microphone

• Magnetometer

• GPS

• Bluetooth

• Accelerometer

• Camera

• Ambient light

• Proximity

Sensors in a Smart Phone

3

Microphone – Speaker Recognition

What Can be DoneCamera – Face Recognition

Accelerometer – Activity Recognition

4

Health monitoring

Application ScenariosSocial sensing

Location based services

5

Sense continuously

Energy-Accuracy trade-offs

Continuous Sensing Mobile Systems

6

Mobile Phone Limitations

Energy, processing, and memory constraints

Google Nexus One, and HDC HD2 are equipped with1GHz processor and 512MB RAM

Energy is still a scarce resource

7

EmotionSense

8

Emotions of users and how they are influenced

Social Psychology - Emotions

9

Speech patterns of users

Social Psychology - Speech Patterns

10

Oh yes, I was very happy today

Self reports ◦ But they are biased towards positive emotions

One-time behavioural study in laboratory◦ Users hide their natural behaviour

Social Psychology - Existing Methods

11

• Use Mobile phones• Powerful sensors• Ubiquitous• Unobtrusive

Our Solution

• Challenges• Sensors not built for this purpose• Battery powered• Processing• Main memory limitations• Privacy concerns

12

EmotionSense - Architecture

EmotionSense Manager

Inference Engine

SpeakerMonitor

HTTP Module

Declarative Database

Remote ServerColocationMonior

MovementMonitor

LocationMonitor

HTK

HTK: Hidden markov ToolKit

Implemented in PyS60 and Symbian C++

13

EmotionSense – Flow of Data

Classifiers Facts Base Inference Engine Actions Base

EmotionSense Manager

Sensor Monitors

Raw data

E.g. X Y Z of Accelerometer

E.g. User is MovingE.g. fact(Moving, True)E.g. fact(action, LocationSamplingInterval, 2)

14

Emotion & Speech Recognition

Collect voice data User/Emotion specific modelsTrain background GMM

[2] M. Liberman, K. Davis, M. Grossman, N. Martey, and J. Bell. Emotional prosody speech and transcripts, 2002.

Training Procedure

At Runtime

Record voice on phone Extract PLPs using HCopy Compare using HERest

[1] http://htk.eng.cam.ac.uk

GMM: Gaussian Mixture ModelPLP: Perceptual Liner PredictionHcopy, HERest: Tools of Hidden markov ToolKit (HTK)

Speaker Recognition: Participants voice dataEmotion Recognition: from libraryLoad speaker and emotion models on phone

15

Clustering of emotions

Emotion Categories

Broad Emotion Narrow Emotion

Happy Elation, Interest, Happy

Sad Sadness

Fear Panic

Anger Disgust, Dominant, Hot anger

Neutral Neutral normal, Neutral conversation, Neutral distant, Neutral tete, Boredom, Passive

(a) Used by psychologists (b) Improves accuracy

Why?

16

Optimizations

Adaptive framework

What About Energy?

17

Silence detection◦ train an additional GMM using silence audio

Comparisons driven by co-location information◦ A recorded audio sequence is compared only

with the models co-located users◦ This improves accuracy and saves energy

Speaker Recognition - Optimizations

18

Implemented using Pyke, a knowledge based inference engine [http://pyke.sourceforge.net/]

Activate GPS only when user is moving

Adaptation Framework

set_location_sampling_intervalforeach facts.fact($factName, $value) check $factName == 'Activity' facts.fact($actionName, $currentInterval) check $actionName == 'LocationInterval' $interval = update($value, $currentInterval)assert facts.fact('action', 'LocationInterval', $interval)

Facts Base Inference Engine Actions Base

E.g. fact(Moving, True)E.g. fact(action, GPS, ON)

19

Sensor Sampling

Time

Sleep Sense Sleep Sense Sleep

Events

Time

Sleep Sense

Events

Sense SleepSleep

20

Sensor Sampling Issues Continuous sampling

degrades battery life

Long sleep durations result in loss of sensor data

Not all sensors are similar

Accuracy varies with sensors and classifiers

21

Sampling Interval

Sleep Sense Once

Sleep 0 ∞Minimum Sampling Interval

Maximum Sampling Interval

Constant

22

Design Methodology

Missable Event

Sleep

Not all events are important

E.g.: Microphone recording when there is no audible sound

•Classify events as Unmissable and Missable•Use functions to control the sleep interval

Back-off Function

E.g.: f(x) = 2x, where x is sleep interval

UnMissable Event

Advance Function

E.g.: f(x) = x/2, where x is sleep interval

Sense

23

Back-off and Advance FunctionsType Back-off function Advance function

Linear

Quadratic

Exponential

Minimum N/A Minimum interval

Maximum Maximum interval N/A

€

kxkx

2xxe

€

ln(x)

x

x: sleep interval

24

Dynamic Adaptation

Dynamically switch functions from least to most aggressive

Missable Event Sleep

Sequence Count

Sense

Linear back-off function

Quadratic back-off function

Exponential back-off function

Update Sleep

Interval

< Linear Threshold

< Quadratic Threshold

> Quadratic Threshold

25

Micro-benchmarks

Social psychology experiment

Meeting experiment

Evaluation

26

Speaker Recognition - Benchmarks

Accuracy Effect of noise on accuracy

27

Speaker Recognition – Benchmarks

Energy consumption Latency

Why ?

a) Computationally intensive processingb) High latency

Then why compute locally ?

a) Privacy concernsb) Users can use their own sim cards

28

Emotion Recognition - Benchmarks

Accuracy Energy consumption

Clustering helps

29

Nokia 6210 Navigator mobile phones

18 participants, 10 days

Users filled in daily diary questionnaire

Voice data is discarded immediately

All computation performed locally on phone

Social Psychology Experiment

30

Emotion distribution similarity EmotionSense Self reports

Social Psychology Experiment - Results

Users indicated ``happy'' emotion to represent their mental state, and not necessarily verbal expression

31

Social Psychology Experiment - Results

Correlation with time of day and co-location

32

EmotionSense can also be used to analyze the speech patterns

Considerable amount of consistency in verbal behaviour

Social Psychology Experiment - Results

33

11 participants , 30 minutes discussion

We identified conversation leaders in each time slot of length 5 minutes

The result shows the top five most active speakers

Meeting Experiment

34

Continuous sensing mobile systems

EmotionSense◦ Speaker and Emotion Recognition◦ Sensor Monitors◦ Adaptive and Programmable Framework

Evaluation◦ Micro-benchmarks◦ Social Psychology Experiment◦ Meeting Experiment

Summary

35

Thank YouKiran Rachuri

Computer LaboratoryUniversity of Cambridge

kkr27@cam.ac.uk

EmotionSense http://www.cl.cam.ac.uk/research/srg/netos/emotionsense/