[email protected]@mikehurst
Imagine a WorldMichael Hurst
Department of Information Science and Computer Science
Supervisors: Dr Tom W Jackson, Dr Mashhuda Glencross
[email protected]@mikehurst
Motivation
Imagine a world where a computer
can interact with you with the same
quality and fluidity with that as
another human being
[email protected]@mikehurst
The Current Situation is Very Different
[email protected]@mikehurst
Issues
• Can only work on what information they have
• Hand Location (Mouse)
• Written Text (Keyboard)
• Sight
• Smell
• Taste
• Hearing
• Touch
[email protected]@mikehurst
Addressing the Issues
• Historically, design was the answer
• However, one size fits all only goes so far
• Give computers the information people have
• Advancements mean technology is now capable
• Key question:
1. What is the key variable
involved in interaction?
[email protected]@mikehurst
The Research
Using a combination of readily available and
affordable devices that yield sensory data in the
hope to correctly predict the emotional state of a
user at any one time
• Why?
1. How can the Data can be Used (Motivation)
[email protected]@mikehurst
Research Questions
1. How do we collect the data needed to
understand emotional state?
2. What do we do with the sensory data once
collected?
[email protected]@mikehurst
How do we Collect the Data?
Give computers the information people have
• Facial Expressions
• Gestures
• Movement
• Vocal Tone
• Vocal Context
• Gaze Velocity
• Gaze Fixations
• Pupil Dilation
• Eye Movement
[email protected]@mikehurst
How do we Collect the Data?
Give computers the information people have
…And possibly further
• Heart Rate
• Heart Rate Variability
• Location of electrical
activity (EEG)
[email protected]@mikehurst
What to do with Sensory Data?
• Compare sensory data with participants self-
assessment of own emotional state to establish
relationships
• For each metric
• Find somatic marker, create a rule
• To combat participant subjectivity, deviation from mean
used
• Rules can be used by computers to assist user/alter
behaviour
[email protected]@mikehurst
What to do with Sensory Data?
• Development of an Emotional Salient Model
• Works in real time
• Determines emotional saliency at time t
• Based on likeliness of emotional state from somatic markers
• Researcher friendly
• Field Standard
• Expandable to incorporate new technologies, task data, other input
methods (twitter), personalisation etc.
• User friendly
• Model is dynamic using only technology that is available to user
[email protected]@mikehurst
What to do with Sensory Data?
Distressed(11%)
Aroused(11%)
Excited(11%)
Displeased(11%)
Neutral(11%)
Pleased(11%)
Depressed(11%)
Sleepy(11%)
Contented(11%)
Aro
usa
l
Valence
Russell’s Circumplex Model of Affect
19
1 9
• Baseline
probabilities
• Presence of
somatic markers
makes
probabilities
fluctuate
[email protected]@mikehurst
What to do with Sensory Data?
MicrosoftKinect
Heart Rate Sensor
EEG Sensor
Eye Tracking
Time t
[email protected]@mikehurst
What to do with Sensory Data?
MicrosoftKinect
Heart Rate Sensor
EEG Sensor
Eye Tracking
Time t
• Increase in Pupil Dilation • Rule Triggered• Emotional Probabilities
recalculated
[email protected]@mikehurst
What to do with Sensory Data?
MicrosoftKinect
Heart Rate Sensor
EEG Sensor
Eye Tracking
Time t
• Pupil Dilation Returns to Mean
• As t Progresses, Likeliness Reverts to Baseline
[email protected]@mikehurst
What to do with Sensory Data?
40% 15% 5%
15% 10% 3%
5% 3% 2%
Time t
Predicted Emotion (Salient Emotion)
Distressed
CPU Alters Behaviour(e.g. … )
• Tell user to take a break• If showing video/image turn off
• Inform friends and/or family• Pause game
[email protected]@mikehurst
To Conclude
• If your research involves people, could emotion
be used to further enhance your findings?
• Study Participation: bit.ly/AffectingHCI
• Questions?