Using Sensors to Infer how People Feel

how can sensor data be used toinfer how people feel?

@neal_lathiauniversity of cambridge

how many smartphones do you have?

how often do you carry your phone with you (%) ?

how often do you carry your phone with you (%) ?

research: 90% of the time it is in the same room as you

making it a powerful, personal device that can sense, monitor, and interact with you

sense (infer): activities, social interactions, mobility, ambient sound

in the moment

monitor: green transportation habits and modes, fitness levels

individuals over time

interact: social network broadcasts, ubiquitous technology evaluation, persuasive interfaces

understand & persuade

now: a tool for psychology research“ubiquitous, intimate, sensor-rich, computationally

powerful, remotely accessible”

sense

interact

mental processes

infer

sensed context

reported mood

“in the moment”

infers

(responds)

system → challenges

user → challenges

inference → challenges

system → challenges

(kiran rachuri)

system → challenges

battery

processing

memory

network

smartphone sensors weren't built for continuous monitoring

system → challenges:: adaptive sensor sampling

sense sleep

continuous interval (50%)adaptive

-45% battery -5 % battery-2% accuracy - 40% accuracyco-location

-44% battery -8 % battery-1% accuracy - 34% accuracyaccelerometer

inference → challenges

(ubicomp 2010)

inference → challenges:: machine learning {sensed context} → emotion

microphone sample

emotional prosody-based model

daily diary inferred emotion

+ 71% accuracy

MAP likelihood

5 broad emotions

user → challenges

user → challenges:: protocols for experience sampling

signal respond

compliance, bias, delay

experience sampling

system → challenges

user → challenges

putting it together

inference → challenges

adaptive sampling

experience sampling

putting it together?

emotion inference learning

conflict?

conflict?

conflict?

{sensed context} → {current emotion}

in the moment

{sensed context} → {current emotion}

{ } → {current emotion}

{sensed context} → { }

{sensed context} → {current emotion}

design for participant compliance

design for sensor data

{sensed context} → {current emotion}

design for participant compliance

design for sensor data

just finishing: a/b test with a variety of triggers. results?

so, to conclude..

who canmisinterpret

your emotions?

you

your social circlesa psychologist

who canmisinterpret

your emotions?

you

your social circlesa psychologist

your phone

sensed context

reported mood

“in the moment”

infers

responds

emotionsense.org

@neal_lathiauniversity of cambridge