Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
1
18th November, 2014 Assistant Professor Dr. Bert ARNRICH
Wearable Computing
Lecture 5: Affective Computing
Assistant Professor Dr. Bert ARNRICH 218th November, 2014
Intro
� If we want computers to interact naturally with us, then we
need Affective Computing, i.e. computers which recognize
and express emotions
2
Assistant Professor Dr. Bert ARNRICH 318th November, 2014
Intro
� Sometimes we believe that computers notice our emotional
state� We repeatedly type the same wrong command or click on something
that does not work, as if the computer would notice our frustration and react in some way
Assistant Professor Dr. Bert ARNRICH 418th November, 2014
Intro
� Sometimes we even have a conversation with a computer
program which has almost no information about human
thought or emotion
3
Assistant Professor Dr. Bert ARNRICH 518th November, 2014
Eliza
� In 1966, Joseph Weizenbaum
implemented ELIZA
� Simple parsing and substitution
of key words into phrases� "Who is your favorite composer?“
� "What about your own favorite composer?" or
� "Does that question interest you?“
� ELIZA effect: tendency to
assume that computer behave
like humans
Assistant Professor Dr. Bert ARNRICH 618th November, 2014
Application Scenarios I
� Call center: � If a user is frustrated/confused, connects him to a human operator
� Emotional music player: � Music selection according to user’s mood
� Teleconference support: � Display information about participants’ emotional states
� Affective electronic communication: � Automatically tag messages with Emoticons ☺ � �
� Computer-aided learning: � Adapt teaching in case the student is bored or frustrated
4
Assistant Professor Dr. Bert ARNRICH 718th November, 2014
Application Scenarios II
� Media content tagging: � Identify and tag emotional pictures/sequences automatically
� Affective disorders: � Monitor daily variations of patient’s state to improve treatment
� Personal Stress Assistant:� Keep track of stressful situations and help creating a better “Work-Life-
Balance”
� “Out-of-the-lab” psychological studies� Understand the role of emotions in daily life, e.g. development of
relationships or group behavior
Assistant Professor Dr. Bert ARNRICH 818th November, 2014
What are emotions?
� Consensual definition derived from 92 different notions:
� Emotion is a complex set of interactions among subjective and objective factors, mediated by neural/hormonal systems, which can � Give rise to affective experiences such as feelings of arousal,
pleasure/displeasure
� Generate cognitive processes such as emotionally relevant perceptual effects, appraisals, labeling processes
� Activate widespread physiological adjustments to the arousing conditions
� Lead to behavior that is often, but not always, expressive, goal-directed, and adaptive.
[Kleinginna et al., 1981] A categorized list of emotion definitions, with suggestions for a consensual definition
5
Assistant Professor Dr. Bert ARNRICH 918th November, 2014
Darwin was one of the first to study emotions
[Darwin, 1872] Expression of the emotions
Assistant Professor Dr. Bert ARNRICH 1018th November, 2014
Emotion Categorization into Valence – Arousal
6
Assistant Professor Dr. Bert ARNRICH 1118th November, 2014
Physiological adjustments to arousing conditions
� Apparent to others:� Facial expression
� Voice intonation
� Gestures, Movement
� Posture
� Pupillary dilation
� Less apparent to others:� Respiration
� Heart rate
� Temperature
� Electrodermal response
� Muscle action potentials
� Blood pressure
[Picard, 2000] Affective Computing
Most of the physiological adjustments can be monitored with wearable devices.
Assistant Professor Dr. Bert ARNRICH 1218th November, 2014
Activation of physiological adjustments means …
� … that we can measure and model emotions
HR = heart rate, FT = finger temperature
[Setz, 2012] Multimodal Emotion and Stress Recognition
7
Assistant Professor Dr. Bert ARNRICH 1318th November, 2014
Emotion Recognition
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 1418th November, 2014
Content of this Lecture
Sensing of Physiological adjustments
Emotion Elicitation Ground Truth
8
Assistant Professor Dr. Bert ARNRICH 1518th November, 2014
Content of this Lecture
Sensing of Physiological adjustments
Emotion Elicitation Ground Truth
Assistant Professor Dr. Bert ARNRICH 1618th November, 2014
Facial Expression
� Paul Ekman et al. have shown that facial
expressions across the globe fall roughly into few
categories:
� SADNESS: The eyelids droop as the inner corners of the brows rise and, in extreme sadness, draw together. The corners of the lips pull down, and the lower lip may push up in a pout.
� SURPRISE: The upper eyelids and brows rise, and the jaw drops open.
� ANGER: Both the lower and upper eyelids tighten as the brows lower and draw together. Intense anger raises the upper eyelids as well. The jaw thrusts forward, the lips press together, and the lower lip may push up a little.
9
Assistant Professor Dr. Bert ARNRICH 1718th November, 2014
Facial Expression
� Paul Ekman et al. have shown that facial
expressions across the globe fall roughly into few
categories:
� DISGUST: The nose wrinkles and the upper lip rises while the lower lip protrudes.
� FEAR: The eyes widen and the upper lids rise, as in surprise, but the brows draw together. The lips stretch horizontally.
� HAPPINESS: The corners of the mouth lift in a smile. As the eyelids tighten, the cheeks rise and the outside corners of the brows pull down.
Assistant Professor Dr. Bert ARNRICH 1818th November, 2014
� Specify how to recognize and score action units (AUs) which
represent the muscular activity that produces momentary
changes in facial appearance
� Computer Vision methods available for automatic coding
Facial Action Coding System
AU1: Inner Brow Raiser AU2: Outer Brow Raiser AU4: Upper Lid Raiser
10
Assistant Professor Dr. Bert ARNRICH 1918th November, 2014
Voice
� The most important is often not what is said but how it is
said.
� Even pets are able to “understand” how something is said
� Try it by your own: infer emotions from pseudo sentences � “Hätt sandig prong nju wentsie”
� “Vi gott leich jean kill gos terr”
Happiness Interest Anger Boredom
I22103
F22205
L22203
K11111
Assistant Professor Dr. Bert ARNRICH 2018th November, 2014
Voice
� The most important is often not what is said but how it is
said.
� Even pets are able to “understand” how something is said
� Try it by your own: infer emotions from pseudo sentences � “Hätt sandig prong nju wentsie”
� “Vi gott leich jean kill gos terr”
Happiness Interest Anger Boredom
I22103
F22205
L22203
K11111
11
Assistant Professor Dr. Bert ARNRICH 2118th November, 2014
Voice Feature Calculation
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 2218th November, 2014
Body Movement
� Body movements are related to the level of arousal or the
degree of intensity of an affective experience
� Happiness: relatively jerky, loose, fast, hard, expanded, and full of action.
� Anger: very jerky, stiff, fast, hard, expanded, and full of action.
� Sadness: very smooth, loose, slow, soft, contracted, inactive.
� Neutral: relatively smooth, loose, slow, soft, very contracted, and inactive
12
Assistant Professor Dr. Bert ARNRICH 2318th November, 2014
Body Movement: Stress vs. Cognitive Load
� Motivation:� Body movements carry
information about affective states.
� Chair Sensor:� Tekscan pressure mat
� 32x32 senor elements
� Sampling 25Hz
� Two conditions under study� Stress
� Cognitive Load
[Arnrich et al., 2010] What does your chair know about your stress level?
Assistant Professor Dr. Bert ARNRICH 2418th November, 2014
Body Movement: Center of Pressure
5 10 15 20 25 30
51
02
03
0
1:dim.x
1:d
im.y
5 10 15 20 25 30
51
02
03
0
1:dim.x
1:d
im.y
5 10 15 20 25 30
51
02
03
0
1:dim.x
1:d
im.y
[Arnrich et al., 2010] What does your chair know about your stress level?
Compute Center of Pressure (CoP) out of each pressure frame and
investigate CoP movements.
13
Assistant Professor Dr. Bert ARNRICH 2518th November, 2014
Body Movement: Measure of Nervousness
� Variation of the x-
CoP is higher during
stress for most of the
subjects (p<0.05)
� Might be a measure
of nervousness
� 6 subjects behave
differently to stress
[Arnrich et al., 2010] What does your chair know about your stress level?
Assistant Professor Dr. Bert ARNRICH 2618th November, 2014
Posture
� Naturally occurring postures convey information about
affective states and interest level
� Symbolic postures convey a specific meaning about the
actions of a user, e.g., leaning forward toward a computer
screen might be a sign of attention
14
Assistant Professor Dr. Bert ARNRICH 2718th November, 2014
Respiration
� Normal respiration rate at rest: 12-18 breaths per minute
� Increase of respiration rate is often observed during anxiety
� Often used measurement principle: strain sensor
Assistant Professor Dr. Bert ARNRICH 2818th November, 2014
Respiration: Measurement via Seat Belt
15
Assistant Professor Dr. Bert ARNRICH 2918th November, 2014
Respiration
� Respiration signal is sometimes affected by motion artifacts
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 3018th November, 2014
Respiration
� Usually, the feature of interest is the respiration rate
� Peak detection allows to identify inhalation/exhalation events
16
Assistant Professor Dr. Bert ARNRICH 3118th November, 2014
Heart Rate
� The electrocardiogram (ECG)
measures the electrical activity of the
heart muscle
� The characteristic segments and
peaks of the ECG pattern are named
by the letters P to T
� The R-peak represents the most
prominent attribute of the ECG and is
used to compute the heart rate
Assistant Professor Dr. Bert ARNRICH 3218th November, 2014
Heart Rate
� Nowadays a broad variety of wearable chest belt exists
which allow to record ECG data
� Usually, the chest belts provide additional measures on body
acceleration (important for artifact detection), breathing rate
and skin temperature
17
Assistant Professor Dr. Bert ARNRICH 3318th November, 2014
ECG integrated into Airplane Seat
[Schumm et al., 2012] ECG Monitoring in an Airplane Seat
Assistant Professor Dr. Bert ARNRICH 3418th November, 2014
Heart Rate: RR intervals
� Important feature of interest is the time duration between
successive heart beats, called RR interval
18
Assistant Professor Dr. Bert ARNRICH 3518th November, 2014
Heart Rate: RR intervals
� Important feature of interest is the time duration between
successive heart beats, called RR interval
� RR intervals over time reveal heart rate variability
[Schumm et al., 2012] ECG Monitoring in an Airplane Seat
Assistant Professor Dr. Bert ARNRICH 3618th November, 2014
� Heart rate variability decreases with increasing load levels
RR intervals during workload load levels
[Cinaz et al., 2013] Monitoring of Mental Workload Levels during an Everyday Life Office-Work Scenario
19
Assistant Professor Dr. Bert ARNRICH 3718th November, 2014
Heart Rate Variability: Frequency Domain
� Compute power spectral density of the RR-intervals
� Usually, the Lomb-Scargle Periodogram is used for
computation in order to account for then uneven sampled
heart beat events
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 3818th November, 2014
Heart Rate Variability: Frequency Domain
� Two important frequency ranges are often considered� Low frequency (LF) range: 0.04-0.15 Hz
� High frequency (HF) range: 0.15-0.4 Hz
20
Assistant Professor Dr. Bert ARNRICH 3918th November, 2014
Common used Heart Rate Variability Features
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 4018th November, 2014
Temperature
� Peripheral body temperature is affected by emotional states
� Usually measured with a temperature sensor attached to the
finger or the arm
� It was shown that average finger temperature � increases between 0.1 and 0.2 degree Celsius due to anger
� decreases between 0.01 and 0.08 degree Celsius due to fear
21
Assistant Professor Dr. Bert ARNRICH 4118th November, 2014
Electrodermal Response
� Psychological or physiological arousal results in increased
sweat gland activity
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 4218th November, 2014
Electrodermal Response
� Electrodermal response measurement is a method of
measuring the electrical conductance of the skin which
varies with its moisture level
� Also known as skin conductance, galvanic skin response
(GSR), electrodermal response, electrodermal activity
(EDA), psychogalvanic reflex
� Electrodermal response is usually measured at the palmar
sites of the hands or the feet where the density of sweat
glands is highest (> 2000/cm2).
[Setz, 2012] Multimodal Emotion and Stress Recognition
22
Assistant Professor Dr. Bert ARNRICH 4318th November, 2014
Skin Conductance Reaction
� Ideal Skin Conductance Reaction to a stimulus
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 4418th November, 2014
EDA signal example
� Smoothed EDA signal consists of � a slowly changing part (Skin Conductance Level: SCL)
� overlaid by short, fast conductance changes (Skin Conductance Responses SCRs)
[Setz, 2012] Multimodal Emotion and Stress Recognition
23
Assistant Professor Dr. Bert ARNRICH 4518th November, 2014
EDA signal example
� High-pass filtered EDA signal with threshold and detected
peaks used for detection of skin conductance responses
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 4618th November, 2014
EDA and Affective State
� Slowly changing skin conductance level (SCL) is a measure
for general psychophysiological activation
� Fast conductance changes (Skin Conductance Responses
SCRs) as reaction to a stimulus or general activation
[Setz, 2012] Multimodal Emotion and Stress Recognition
24
Assistant Professor Dr. Bert ARNRICH 4718th November, 2014
Emotional sweating under stress
10 20 30 40 50 60 70 800
0.5
1
1.5
2
2.5
3
3.5
x 10−6 EDA level data (stress condition)
Time [min]
Co
nd
ucta
nce [
S]
Baseline (Instruction) MIST
C+SF2C2C1 F1
Recovery
[Setz et al., 2009] Discriminating stress from cognitive load using a wearable EDA device
Assistant Professor Dr. Bert ARNRICH 4818th November, 2014
Electromyogram
� Electrical activity of a muscle can be measured using
electromyography (EMG)
� Usually measured via surface electrodes applied to the skin,
e.g. on the “smiling muscle”
[Setz, 2012] Multimodal Emotion and Stress Recognition
25
Assistant Professor Dr. Bert ARNRICH 4918th November, 2014
Electrooculogram
� Electrooculography (EOG) is a technique
to measure the resting potential of the
retina
� Used for eye movement analysis
� Measured between two electrodes
� Attached at the right and left side of the eye for the horizontal eye movements
� Attached below and above the eye for the vertical eye movements
Assistant Professor Dr. Bert ARNRICH 5018th November, 2014
Electrooculogram and Affective State
� In the vertical EOG, the blinks are visible
� Eye blinks are related to startle events
� The blink rate can be affected by the emotional state of a
person
[Setz, 2012] Multimodal Emotion and Stress Recognition
26
Assistant Professor Dr. Bert ARNRICH 5118th November, 2014
Lecture Content
Sensing of Physiological adjustments
Emotion Elicitation Ground Truth
Assistant Professor Dr. Bert ARNRICH 5218th November, 2014
Emotion Elicitation: Pictures
27
Assistant Professor Dr. Bert ARNRICH 5318th November, 2014
Emotion Elicitation: Pictures
� Applicable to elicite a range of emotions, simple, highly
standardized experiment setup allows for replication
� Only short duration of emotions (~6 s) can be induced
� Some emotions (e.g. fear, anger) are difficult to induce with
pictures because subjects are not personally threatened.
� International Affective Picture System (IAPS)� Database of pictures used to elicit a range of emotions.
� Contains over 1,000 colored pictures
Assistant Professor Dr. Bert ARNRICH 5418th November, 2014
Emotion Elicitation: Movies
28
Assistant Professor Dr. Bert ARNRICH 5518th November, 2014
Emotion Elicitation: Movies
Assistant Professor Dr. Bert ARNRICH 5618th November, 2014
Emotion Elicitation: Movies
� Wide range of emotions which develop over time (~1-10 min)
� Simple, highly standardized experiment setup allows for replication
� Allows to elicit cognitively sophisticated emotional states such as nostalgia
� Some emotions (fear, anger) are difficult to induce because subjects are not personally threatened
� Recommended film clips for eliciting discrete emotions (amusement, anger, disgust, fear, plain neutral, pleasant neutral, sadness, surprise) are available
[Setz, 2012] Multimodal Emotion and Stress Recognition
29
Assistant Professor Dr. Bert ARNRICH 5718th November, 2014
Emotion Elicitation: Music
� Emotions develop over time (15-20 min.)
� Simple, highly standardized experiment setup allows for
replication
� Music taste might influence experienced emotions.
� Moods (positive or negative) rather than discrete emotions
� Recommended classical music pieces for inducing happy
and sad moods are available
[Setz, 2012] Multimodal Emotion and Stress Recognition
Assistant Professor Dr. Bert ARNRICH 5818th November, 2014
Emotion Elicitation: Psychological Approaches
� Masking of experiment purpose fosters realistic responses
� Emotions occur in social context
� Effective technique for eliciting anger and stress
� Time-consuming and complex experiment setup� development of convincing cover story
� training of experimenters
� thorough debriefing
� Might be ethically critical
[Setz, 2012] Multimodal Emotion and Stress Recognition
30
Assistant Professor Dr. Bert ARNRICH 5918th November, 2014
Psychological Approach: Load + Social Evaluation
Arithmetic
Task
Time
Pressure
Performance of
norm collectivePerformance of
subject (always
less than norm)
Answer field
Assistant Professor Dr. Bert ARNRICH 6018th November, 2014
Psychological Approach: Social Stress
1. Mild� Do you have problems with the keyboard?
� Can you read the tasks on the screen well?
2. Moderate� Did you sleep bad?
� Are you feeling bad today?
3. Strong� Do you take drugs?
� Did you already had math problems in school?
31
Assistant Professor Dr. Bert ARNRICH 6118th November, 2014
� 5 minute presentation in front of a jury
� 5 minutes mental arithmetic in front of the jury:
count backwards from 1,022 in steps of 13. If a mistake is
made, then start again from the beginning.
Psychological Approach: Trier Stress Test
Assistant Professor Dr. Bert ARNRICH 6218th November, 2014
Emotion Elicitation: Social Interaction
� Recommended approach to invoke a conflict discussion in
interacting subjects (e.g. couples)
� Elicited emotions closely resemble the emotions that occur in
everyday life
� Temporal characteristics of emotions can be investigated
� Time-consuming and complex experiment setup, limited
standardization since experimenter can not totally control
interactions and participants may change the topic
[Setz, 2012] Multimodal Emotion and Stress Recognition
32
Assistant Professor Dr. Bert ARNRICH 6318th November, 2014
Emotion Elicitation: Daily Life
� Lab stress, movie and soccer
[Wilhelm et al., 2010] Emotions beyond the laboratory
Assistant Professor Dr. Bert ARNRICH 6418th November, 2014
Emotions Elicitation during Soccer
[Setz, 2012] Multimodal Emotion and Stress Recognition
33
Assistant Professor Dr. Bert ARNRICH 6518th November, 2014
Lecture Content
Sensing of Physiological adjustments
Emotion Elicitation Ground Truth
Assistant Professor Dr. Bert ARNRICH 6618th November, 2014
� When eliciting emotions through pictures, movies or music a
ground truth is given by the content of the media
� However, self-assessment questionnaires is the preferred
method to gather ground truth
� There exists a variety of self-assessment questionnaires
developed by psychologists
� Depending on the research question, one has to decide
when and what to ask
Groundtruth
34
Assistant Professor Dr. Bert ARNRICH 6718th November, 2014
When to ask?
� In standardized experiments when using
pictures/movies/music, usually after each stimulus the
subject is asked for a self-assessment on experienced
emotions
� In more real-life experiments, more practical self-
assessments are used� Experience Sampling
� Day Reconstruction
Assistant Professor Dr. Bert ARNRICH 6818th November, 2014
Experience Sampling
� Ask the subjects at random time
points how they feel right now
� Allows to obtain experienced
feelings in the current situation
� Subjects should not be aware
when the next questionnaire is
given
� Consider a minimal time span
between successive time points
35
Assistant Professor Dr. Bert ARNRICH 6918th November, 2014
� Retrospective self-assessment, usually done at the end of a
day
� Ask the subjects to define episodes of daily activities and
rate their feelings within the episodes
� Self-assessments might be biased, e.g. in the evening you
might forget a stressful event occurred during the day
Day Reconstruction
Episode Name Time Start Time End How did you feel?
Preparing breakfast 7:30 8:00 Tired
Commuting 8:00 8:30 Boring
Read email 8:30 9:00 Exited
… … … …
Assistant Professor Dr. Bert ARNRICH 7018th November, 2014
What to ask?
� Mood and Feelings Questionnaire� Available for child and adults
� Long (33 items) and short versions (13 items) available
[http://devepi.duhs.duke.edu/mfq.html]
36
Assistant Professor Dr. Bert ARNRICH 7118th November, 2014
What to ask?
� Positive Affect Negative Affect Scale� Up to 60 items
� Measures the two scales positive/negative affect and specific affects
[http://www2.psychology.uiowa.edu/faculty/Clark/PANAS-X.pdf]
Assistant Professor Dr. Bert ARNRICH 7218th November, 2014
� Geneva Emotion Wheel� Emotions are arranged in the arousal – valence space
What to ask?
[http://www.affective-sciences.org/gew]
37
Assistant Professor Dr. Bert ARNRICH 7318th November, 2014
What to ask?
� Self-assessment manikin� First row: valence, second row: arousal
� Useful to allow fast and simple response, e.g. when collecting self-assessments with the smart phone
Assistant Professor Dr. Bert ARNRICH 7418th November, 2014
� Biological marker like Cortisol useful in stress experiments
� Experts label video sequences recorded from the subjects� Often it is required that several experts rate the same sequences in
order to estimate inter-rater reliability
Stress
Control
Alternative Groundtruths
38
Assistant Professor Dr. Bert ARNRICH 7518th November, 2014
Lessons learned
� Emotions activate widespread physiological adjustments which can be monitored with wearable devices
� Sensing of Physiological adjustments to arousing conditions
� Emotional face expressions can be analyzed via Facial Action Coding
� Voice feature calculation helps to understand how something is said
� Body movements carry information about level of arousal
� Postures convey information about interest level
� Respiration rate is usually measured with strain sensors
� Heart rate and its variability is a powerful modality in emotion recognition
� Body temperature is affected by emotional states
� Arousal results in increased sweat gland activity observable via electrodermal response measurement
� Electromyogram to measure electrical activity of a muscle, e.g. “smiling muscle”
� Electrooculography for measuring eye blinks that are affected by emotional state
Assistant Professor Dr. Bert ARNRICH 7618th November, 2014
Lessons learned
� Emotion Elicitation
� Pictures: short duration, wide range of emotions
� Movies: wide range of emotions which develop over time
� Music: eliciting mood rather than discrete emotions
� Psychological Approaches: complex setup, effective technique for
eliciting anger and stress
� Social interaction: closely resemble emotions that occur in everyday
life, limited standardization
� Daily life: ultimate goal
39
Assistant Professor Dr. Bert ARNRICH 7718th November, 2014
Lessons learned
� Ground Truth
� When to ask: Experience Sampling, Day Reconstruction
� What do ask: Mood and Feelings Questionnaire, Positive Affect
Negative Affect Scale, Geneva Emotion Wheel, etc.
� Biological marker
� Video labeling