Lecture 3 Emotion Theory (Part 2) - Institute for Creative ...gratch/CSCI534/Lecture2021...CSCI 534(Affective Computing) –Lecture by Jonathan Gratch 3 Lecture 3 Emotion Theory (Part

CSCI 534(Affective Computing) – Lecture by Jonathan Gratch 1

Lecture 3Emotion Theory (Part 2)


“Hot-headed, irrational and swayed by

emotion – who’d want a human in control?”


Lecture 3Emotion Theory (Part 2)


Brief note on homework

▪ Lenient grading on HW1 (if you did it, got full credit)

▪ I’ll discuss results during next lecture on computer

models

▪ HW2 (pt 1) assigned after class. This is short

experiment to collect data needed in HW2 (pt 2)


Review: Introduced some key emotion theories

▪ Theory important– Makes predictions, proposes mechanisms, allows control

– Typically supported by substantial empirical research

▪ But no grand unified theory of emotion– Different theories explain different aspects of emotion

– Different theories make different claims▪ Emotion is discrete vs. emotion continuous

▪ Emotion an atom or vs. emotion a molecule or mixture

▪ Emotion follows from cognition vs. emotion precedes cognition

▪ But choice of theory has implications– E.g., for our labels in machine learning algorithm


Discrete or continuousRussell’s ‘80 circumplex model


Appraisal Models

Is emotion a cause or consequence of thought?

Event

Self-reported

Fear

Amygdala

activation

Increased

heartrate

Fight, Flight,

Freeze

Com

ponents

of E

motion

Construct

beliefs,

desires,

intentions

ThinkThink influence


James-Lange Perspective


Event

Amygdala

activation

Increased

heartrate

Fight, Flight,

Freeze

Com

ponents

of E

motion“Mindless”

Automatic Evaluation

influenceThink

Shapes

beliefs,

desires,

intentions

Label


Emotion components are tightly-

coupled and can be treated as a circuit

linking stimuli and response

Emotions are defined by loose

configuration of different components

Phoebe Ellsworth, Klaus Scherer, Lisa

Feldman Barrett

Compound

Molecule or Mixture“I feel disgusted”

Increased skin

conductance

Vomiting

Dis

gu

st

Cir

cu

it

Insula






Phoebe Ellsworth, Klaus Scherer, Lisa

Feldman Barrett

Mixture

Molecule or Mixture

Increased skin

conductance

Vomiting

I fe

eld

igu

ste

d

Insula


In class experiment (tried to induce disgust)


Tried to experience of emotion

1

2

3

4

5

6

7

8

9

Positivity Arousal Dominance1

2

3

4

5

6

7

8

9

Positivity Arousal Dominance

Discrete

emotions?

Continuous?

1st person:

Experienced video

3rd person:

Guessed feelings


Results

1

2

3

4

5

6

7

8

9

Positivity Arousal Dominance

Watchers


Automatic analysis (Facet)

21 seconds (Joy)

disgust



19 seconds (disgust)

25 seconds (surprise)



29 seconds (Joy)



12 seconds (confusion)

sadness


What’s going on here

Takeaway:

Emotional expression complicated


Outline

▪ Contrast top-down (appraisal) with bottom-up

(constructivist) theories

▪ In-class experiments on these themes

▪ Discuss dual process models– That emphasize disassociation between emotion and cognition

▪ Discuss more integrative models– That see emotion and cognition as two sides of larger process


Example: Thought precedes emotion


BREAKING: Biden’s Executive Order

Revokes Most Student Visas to fight COVID

National

By Tim Craig January 27, 2020 at 12:00 PM

WASHINGTON D.C. — Foreign students at US

universities had their future thrown into question

today by President Biden’s sweeping new executive

order revoking student visas from the Middle East,

and 30 Asian countries including China and India.

Immigration lawyers are warning students not to

leave the country because of the risk that they will be barred from re-entry.

Students already in the country may be targeted for deportation as early as

this March.


Co

nse

que

nce

s?

Expecte

d?

Co

ntr

ol?

Bla

me?

Self-reported

Fear

Amygdala

activation

Increased

heartrate

Fight, Flight,

Freeze

Com

ponents

of E

motion

Be in US

Need Visa

“Thoughtless”

Automatic

Evaluation

Appraisal

Con

scio

us

“Effort

ful”

thought

In a sense, emotion a byproduct of thinking


Another example

▪ Imagine the following situation

– You are a foreign student in another country. You discover a mistake

you made with your visa. If it is not fixed, you will be deported

– You spend $10,000 on an immigration lawyer to help you

– The lawyer knows he has to file paperwork by January 15

– He knows if he misses this deadline, you will be deported

– He’s busy and decides not to submit the paperwork

– Immigration agents come to your house and deport you

▪ What are you likely to feel?

▪ Who are you mad at?

– Yourself?

– Immigration agents?

– Lawyer?


Some appraisals require social inferences (theory of mind) – e.g. Weiner; Shaver

e.g., Causal attribution involved in Anger

NegativeConsequence

Attributional Inferences

Cause Foreseen

Unforeseen

Intended

Unintended

Voluntary

Coerced

No Blame No Blame No Blame No BlameIncreasing responsibility but not blame

Blame

or

Credit

Such reasoning underlies how people make sense of complex social

activities involving the attribution of blame and feelings of anger

Appraisal theory perspective on anger


Preview: computer appraisal models(Mao&Gratch2005;Oh et al 2007; Melissen)

Scenario 1

1. The VP British Petroleum discusses plans for new oil well

2. The VP states the program will likely increase profits

3. The VP states the program will likely harm the environment

4. The CEO orders the program to be started

5. The VP executes the new program

6. The environment is harmed



Scenario 1







VP: Cause

CEO: Cause + Intent + Voluntary → Blame



Scenario 1







VP: Cause + Intent



Scenario 1







VP: Cause + Intent + ¬Voluntary → ¬Blame



Scenario 1







VP: Cause + Intent + ¬Voluntary → ¬Blame

CEO: Cause + Intent + Voluntary → Blame


Scenario 1E1 The vice president of Beta Corporation goes to the chairman

of the board and requests, “Can we start a new program?”

E2 The vice president continues, “The new program will help us

increase profits,

E3 but according to our investigation report, it will harm to the

environment.”

E4 The chairman answers, “Start the program anyway.”

E5 The vice president executes the new program.

E6 However, the environment is harmed by the new program.

Appraise Representation

Intentional action

intend(x, p, t1) do’(p, x, A) t1<t3

(t2)(t1<t2<t3

intend(x, p, t2)) execute(x, A, t3)

Side effect

effect(A) intend(x, b, t1) by(b, A, e)

(t2)(t1<t2<t3 intend(x, b, t2))

t1<t3<t4 execute(x, A, t3)

occur(e, t4)

Coercion

coerce(y, x, p, t1) do’(p, x, A)

eeffect(A) t1<t2<t3

etc29(x, y, e, t2) coerce(y, x, e, t3)

BLAME = YES


Counterfactual Scenario 2E1 The vice president of Beta Corporation goes to the chairman

of the board and requests, “Can we start a new program?”

E2 The vice president continues, “The new program will help us

increase profits,

E3 AND according to our investigation report, it WON’T harm to the

environment.”

E4 The chairman answers, “Start the program.”

E5 The vice president executes the new program.

E6 However, the environment is harmed by the new program.


Appraisal theory perspective(emphasizes primacy of deliberative thought)

▪ Emotion is “goal relevant”

▪ It arises from how events impact goals

▪ The emotion prepares your body and mind to

address goal threats or opportunities

▪ The emotion is said to be “endogenous”– Meaning that it arises from thoughts about a task at hand

▪ Thinking determines emotion


Different example




Event

Amygdala

activation

Increased

heartrate

Fight, Flight,

Freeze

Com

ponents

of E

motion“Thoughtless”

Automatic

Evaluation

influenceThink

Shapes

beliefs,

desires,

intentions

Emotion determines thought

CSCI 534(Affective Computing) – Lecture by Jonathan Gratch

Yet another example (in class experiment)

▪ I’ll post Qualtrics survey

▪ You will listen to music

– May not play automatically on Macs

▪ Listen quietly for 2 minutes then complete survey

▪ Try to go with your first instincts (don’t use math)


This is the Education Building on USC

campus. Imagine that you have to walk to

the top floor using the stairwell

Don’t think carefully, don’t count how many

floors. Just from this image, guess how

many minutes it will take you to walk to the

top.

Have blank that will accep integers or real

numbers


What should have happened?

▪ Mozart induces happiness

▪ Happiness makes you energetic,

confident, (over) optimistic

▪ Mahler induces sadness

▪ Sadness makes you lethargic,

unconfident, realistic

▪ These impressions bias appraisals

– Particularly perceived control

Riener, Stefanucci, Proffitt & Gerald Clore (2011) An effect of mood on the perception of geographical

slant, Cognition and Emotion, 25:1, 174-182


Thinking



Event

Amygdala

activation

Increased

heartrate

Fight, Flight,

Freeze

Com

ponents

of E

motion“Mindless”

Automatic Evaluation

Shapes

beliefs,

desires,

intentions

influence

Make a

time

judgement


James-Lange perspective

▪ Characteristics of events automatically induce

emotional responses

▪ These responses can alter decision making

▪ The emotion is said to be “incidental” to what was

previously “in mind”


Stepping back

▪ Some work suggests primacy of thinking

– Thought shapes emotion

▪ In that thought is (somewhat) under our control,

suggests that we can control emotion

Emotion Cognition

“Automatic” appraisals of

the contents of thought

Lazarus, R. S. (1984). On the primacy of cognition. American Psychologist, 39(2), 124-129.


Stepping back



▪ Some work suggests primacy of affect

– Emotion shapes thought

Emotion Cognition

Perception

“Automatic” stimulus

evaluation

Zajonc, R. B. (1984). On the primacy of affect. American Psychologist, 39(2), 117-123


Stepping back





▪ Suggests emotion is unconscious and difficult to

control

Emotion Cognition


Stepping back





▪ Together, seem to emphasize two distinct processes

Emotion Cognition


Most emotion theories are dual-process theories

Passion,

visceral reward

The Allegory of the Chariot

Abstract goals,

Deliberation


System 1 System 2Integration

Behavior

Most emotion theories are dual-process theories

Together, these systems determine behavior


System 1 vs System 2: Kahneman

A useful fiction


Affective system

▪ fast

▪ unconscious

▪ reflexive

▪ myopic

▪ effortless

Analytic system

▪ slow

▪ conscious

▪ reflective

▪ forward-looking

▪ (but still prone to error:

heuristics may be analytic)

▪ self-regulatory

▪ effortful and exhaustible

Commonalities between classification schemes

CSCI 534(Affective Computing) – Lecture by Jonathan Gratch 47USC CSCI 534 Affective Computing © Jonathan Gratch

Quick illustration of automatic processing

If you are a

woman:

Raise your hand

when you find the

face

If you are a man:

Raise your hand

when you find the

gloves

I will show a bunch

of pictures

CSCI 534(Affective Computing) – Lecture by Jonathan Gratch 48USC CSCI 534 Affective Computing © Jonathan Gratch


Mesolimbic dopamine reward system

Frontalcortex

Parietalcortex

Affective vs. Analytic Cognition

mPFC

mOFC

vmPFC


▪ The brain makes decisions (e.g. constructs value)

by integrating signals from multiple systems

▪ These multiple systems process information in

qualitatively different ways and in some cases

differentially weight attributes of rewards (e.g.,

time delay)

Dual Process Theories: more recent perspective


▪ Passions vs Interests (Smith)

▪ Id vs Ego vs Superego (Freud)

▪ Automatic vs Controlled (Schneider & Shiffrin, 1977; Benhabib & Bisin, 2004)

▪ Hot vs Cold (Metcalfe and Mischel, 1979)

▪ Impulsive vs Deliberative (Frederick, 2002)

▪ Unconscious vs Conscious (Damasio, Bem)

▪ Effortless vs Effortfull (Baumeister)

▪ Doer vs Planner (Shefrin and Thaler, 1981)

▪ Visceral vs Abstract (Loewenstein & O’Donoghue 2006; Bernheim & Rangel, 2003)

▪ Mesolimbic dopamine vs PFC & parietal cortex (McClure et al, 2004)

▪ System 1 vs System 2 (Frederick and Kahneman, 2002)

Variations on this theme


A battle in your mind

Dual process theories often adopt the analogy

of two independent entities fighting for control

of your mind

Research then tries to identify which “horse” wins

• Developmental factors: children and teenagers are emotional

• Dispositional factors: some people are naturally emotional

• Cultural factors: some societies are naturally emotional

• Situational factors: some situations evoke emotional thinking

(e.g., when hungry, sleepy, rushed


An example

Visceral

reward:

pleasure

Abstract

goal:

dietIntegration

Behavior

Would you like a piece of chocolate?


Individual differences (simple test)


A battle in your mind

Dual process theories often adopt the analogy

of two independent entities fighting for control

of your mind

Research then tries to identify which “horse” wins

• Developmental factors: children and teenagers are emotional

• Dispositional factors: some people are naturally emotional

• Cultural factors: some societies are naturally emotional

• Situational factors: some situations evoke emotional thinking

(e.g., when hungry, sleepy, rushed

But are these really independent systems?

Is there always just one winner?


Buyer beware

▪ Psychological research often overemphasizes

independence of these systems

▪ This independence is exaggerated by experimental

designs that try to emphasize differences in thinking


Reductionism

▪ Psychological studies typically try to show importance

of a single mechanism

▪ Can overestimate importance of that mechanism and

underestimate contribution/interaction with other

mechanisms

IV →DV

Race

Gender

Degree

Sunny?

Hungry

Sleepy


Mechanically Separate emotion and cognition

▪ Ventral Medial/Orbital Prefrontal

Cortex damage

Phineas Gage


Mechanically Separate emotion and cogition

▪ Ventral Medial/Orbital Prefrontal

Cortex damage

– Show can “turn off” emotion and

“thinking” preserved

– And show that “thinking” “needs”

emotion▪ Severe impairments in judgment and decision-

making in real-life

– But keep in mind this sort of

“separation” never occurs in real world

Transcranial magnetic stimulation


Empirically separate Em & Cog “in the moment”(Clore, Schwarz)

▪ Emotions inform decisions

▪ But many experiments separate emotion from decision

– Induce an emotion:▪ Play happy/sad/angry music

▪ Read happy/sad/angry stories

– Make people perform an irrelevant task▪ Buy something

▪ Play ultimatum game

– Show logically irrelevant emotion biases decision making

▪ But how often is emotion irrelevant to the task?


▪ One CAN separate emotion and cognition

– Its seductive▪ Reflect longstanding theoretical and folk distinctions

▪ Consistent with some data

– But I’ll argue this data has limited ecological validity (e.g., see also

Gigerenzer)

▪ It is fun (and publishable) to show people are irrational

▪ But how often does this occur in real word

– But this leads to impoverish understanding of both▪ Cognition w/o emotion is a broken thing

Maybe dual processes are a cognitive illusion


Are these really separate systems?

▪ Much psychological work focuses on one system or

the other

▪ But some work tries to show how these systems

are more tightly linked


emotion AND cognition

▪ The majority of everyday “thinking” involves

– Acting in a dynamic and evolving world

– Juggling multiple goals and preferences

– Confronting opportunities and threats

▪ Emotion evolved hand and hand with cognition

– Two sides of the same system

▪ Attempts to separate them leads to anomalous behavior

▪ Yet that is what much of emotion psychology implicitly

or explicitly strives to do


Appraisal Models


Event

Self-reported

Fear

Amygdala

activation

Increased

heartrate

Fight, Flight,

Freeze

Com

ponents

of E

motion

Construct

beliefs,

desires,

intentions

ThinkThink influence

influence


Some theories show how these processes linked

Emotion CognitionIntegration

Behavior

Perception

Imagination

Perception and imagination

may use same neural circuits

(Damasio’s “somatic marker

hypothesis”)

Cognitive emotions strongest

when thoughts are vivid and

personal


EmotionAction

Tendencies“Affect”

PhysiologicalResponse

EnvironmentGoals/Beliefs/

Intentions

Another example (Lazarus, Smith, Gratch & Marsella)

Desirability

Expectedness

Controllability

Causal Attribution

Appraisal

Antecedents of Emotion


EmotionAction




Intentions

Appraisal theory explains antecedents of emotion

Desirability

Expectedness

Controllability

Causal Attribution

Appraisal

Antecedents of Emotion


EmotionAction




Intentions

Appraisal theory explains antecedents of emotion

Desirability

Expectedness

Controllability

Causal AttributionAntecedents of Emotion

Controlled thinking process

(e.g., planning, deliberation)

auto

matic


Emotion

CopingStrategy

ActionTendencies

“Affect”Physiological

Response


Intentions

“Coping” theory explains emotion’s consequence

Desirability

Expectedness

Controllability

Causal Attribution

Consequences of Emotion

Controlled thinking process

(e.g., planning, deliberation)

Automatic

thinking “bias”

(action tendency)

auto

matic


Emotion

CopingStrategy

ActionTendencies


Response

Withdraw /

Act on self

Emotion-Focused


Intentions

Coping

Approach /

act on world

Problem-Focused

Attempts to characterize How emotion shapes cognition

Consequences of Emotion

e.g., Call

immigration

lawyer

e.g., go home

and get a job


(act on world) (act on self)

Emotion

CopingStrategy

ActionTendencies


Response

Problem-Focused Emotion-Focused


Intentions

Resignation

Distancing

Wishful Thinking

Take action

Seek support

Coping

Attempts to characterize How emotion shapes cognition

These models emphasize that coping

responses are largely automatic


▪ Coping responses help explain common decision bias– Principle of rationality

– Desires (i.e., emotion) shouldn’t change beliefs (and vice versa)e.g., Just wanting something shouldn’t make it true

– Preferences fixed over time

Participate in an

Election

Lose

Win

p=.8

p=.2

Utility= 20

Intention(vote) EU=4


Wishful Thinking“we will win Georgia”

Participate in an

Election

Lose

Win

p=.8

p=.2

Utility= 20

Intention(play) Sad

HopeJoy

Distancing“it doesn’t matter

who wins”

Fear

Utility= 10p=.6

p=.4

Resignation “I’m not going to vote”

• Coping serves to “confound” beliefs and desires

▪ Emotion-biases on decision making (Loewenstein & Lerner, 2003)

▪ Cognitive dissonance (Festinger57)

▪ Motivated inference (Kunda87)

– Little attempt to computationally model (Marsella&Gratch; Dias)


Coping can impact many aspects of appraisal process

▪ E.g., James Gross emotion regulation theory


▪ E.g., James Gross emotion regulation theory

– Emotion arises from appraisal

– But emotion regulation seeks to modulate aspects of the appraisal process

– This theory relies on interdependence of appraisal and emotion over time

Coping can impact many aspects of appraisal process


People differ in automatic coping styles


Locus of control: self vs. other

▪ Internalizes blame, externalizes credit: “god’s plan”


Stability of control: changeable vs. constant

▪ Views situation as unchangeable across time


People differ in “coping style”


People differ in “coping style”


Cognition can also impact coping

▪ Coping styles tend to be automatic

▪ But people can learn new patterns of coping

▪ Cognitive behavior therapy tries to help people

relearn adaptive coping patterns


Some appraisal theories join both perspectives(Clore & Schwartz; Gratch and Marsella)

Endogenous

Emotion

Incidental

Emotion

Allow that prior emotions or incidental influences can shape appraisals


Coping shapes beliefs, desires and intentions

▪ Thought → Appraisal

▪ Appraisal → Emotion: – I’m afraid because I might lose

▪ Incidental influences (music) → Emotion

▪ Emotion → Coping– I don’t care about winning anyway

▪ Coping → Thought– I’m much happier now that I don’t care about wining

▪ This is a cycle


Integrated model also explains individual

difference in emotional responding

▪ Different individual goals– UCLA fan vs USC fan

▪ Different appraisal styles

▪ Different coping / regulation strategies


Appraisal theory

▪ For much of the rest of the class I’ll emphasize

appraisal theory (esp. work of Smith and Lazarus)– Emphasizes emotion as both an antecedent and consequence of

cognition▪ Provides detailed description of factors that elicit emotion (appraisal)

▪ Provides detailed description of how emotions can shape sequent cognition

▪ Thus can unify appraisal and constructivist approaches

– Relatively easy it translate into a computer program

– Serves as the theory underlying my own work on affective computing

– Can help explain several aspects of emotion▪ Why a given situation might produce a given emotion

▪ Why an emotion might influence subsequent decisions

▪ Why an expression might shape another’s decision


Summary

▪ Different theories of emotion– Emotion coherent/basic circuit vs. collection of loosely related systems

▪ Discussed dual process view of emotion– Emotion cause by thinking vs. automatic unconscious response

– Both perspectives seem active in many situations

▪ Introduced Lazarus’ appraisal model– Incorporates both deliberative and automatic processes into single

view


Some comments on projects

▪ Main source of your grade

▪ Project is intended to give you a deep and “hands on”

understanding of affective computing.

▪ Ideal group size is 4 (maybe 5) students.

▪ Projects can involve building affectively aware software or

proposing and piloting an empirical study involving humans

interacting with affective technology.

▪ List of pre-existing software tools available to students, and

summaries of prior student projects built with these tools can

be found

– http://people.ict.usc.edu/~gratch/CSCI534/Tools-and-projects-2020.pdf

http://people.ict.usc.edu/~gratch/CSCI534/Tools-and-projects-2020.pdf



▪ Expectations– not expecting/requiring projects to make novel scientific advance over

the state-of-the-art in affective computing research (of course, this

would be great). Projects that replicate prior findings in order to give

you a better grasp of a theory, algorithm, or phenomena are

fine/expected).

▪ Project Milestones– Feb 22: in class discussion of potential projects and teams

– Feb 25 (midnight): One paragraph tentative project proposal (list team)

– Mar 3: 5min in-class group project plan presentation

– April 26,28: 20min in class final presentation

– May 7 (11:59p): Final written report



▪ For mid-term presentations– Prepare between 5 minutes of material. Be prepared for heavy

questioning from me and the class.

– Should include group members

▪ Project proposal presentation should address:– What is your general area of interest (e.g., emotion recognition,

emotion modeling, emotion synthesis)?

– What is the problem you hope to address within this area?

– How do you plan to address this problem?

– How would you know if you succeeded?



▪ Final project should address the following points– Why this area is interesting (e.g., potential applications, open scientific

questions)?

– What are related approaches? are they inadequate? i.e., is your

proposal an advance over state-of-the art?

– What (if any) of the theoretical perspectives on emotion (introduced in

class) does your approach build on, question, test?

– If you are performing some empirical test of success▪ What are your hypotheses

▪ What are your independent, dependent, and control variables

▪ What are your results and are they statistically significant



▪ Final team presentations should be 15min.

▪ Writeups should be minimum of 5 pages – (12pt single-column, single space, including title, authors, figures and references).

▪ Both presentation and writeup should at least briefly address – Team-members and role played by each member.

– What is the problem you hope to address within this area?

– Why is this interesting (e.g., potential applications, open scientific questions)?

– What are related approaches? are they inadequate?

– What is the evidence you succeeded (e.g., theoretical or empirical arguments)

– What (if any) of the theoretical perspectives on emotion (introduced in class) does your

approach build on, question, test?

– If you are performing some empirical test of success

▪ What are your hypotheses?

▪ What are your measures

▪ Describe the population of subjects. How were they recruited? Is this between subjects? Within

subjects? Do subjects know the hypotheses? (hope not)

▪ What are your results and are they statistically significant

– Anything surprise you?


Speed Dating Exercise

▪ For rest of class, do couple rounds of speed date

▪ I’ll put you in breakout rooms

▪ Take turns introducing yourself– Who you are: I’m a senior in computer science…

– Why you took the class

– Tentative project ideas: I want to make a horror game…

▪ I’ll jump around rooms

▪ I’ll reassign groups after few minutes

Documents

Lecture 3 Emotion Theory (Part 2) - Institute for Creative ...gratch/CSCI534/Lecture2021...CSCI 534(Affective Computing) –Lecture by Jonathan Gratch 3 Lecture 3 Emotion Theory (Part