1 Lecture 17: Interaction, Work and Technology III: Affective HCI Jon Oberlander

1

Lecture 17:Interaction, Work and Technology III:

Affective HCI

Jon Oberlander

UG4: HCI Lecture 17 2

Introduction

Affect– Timescales, complexity

Affect in Computer mediated communication– Especially, detection

Affect in Human computer interaction– Uses and means– 12 recent papers in Affective HCI

The future of Affect


What is Affect?

Variable timescale of affect– Feelings

• Joyful: second, minutes

– Moods• Happy: hours, days

– Personalities/Temperaments• Outgoing: years, decades

Variable complexity of affect– Basic feelings:

• Happiness, surprise, fear, sadness, disgust, anger

– More complex ones:• Jealousy, hope, relief, pride, remorse … anxiety, frustration

– Or just:• Arousal versus valence


Personality: Five Factor Models

The Big Five factors of personality analysis (OCEAN)– Extraversion, Energy, Enthusiasm (I)

talkative, assertive, dominant quiet, reserved, shy, retiring

– Agreeableness, Altruism, Affection (II)sympathetic, kind, warm, helpful fault-finding, cold, unfriendly

– Conscientiousness, Control, Constraint (III)organized, thorough, efficient careless, disorderly, frivolous

– Neuroticism, Negative affectivity, Nervousness (IV)tense, anxious, moody, worrying stable, calm, contented

– Openness, Originality, Open-mindedness (V)imaginative, artistic, inventive commonplace, simple, shallow


Affect in computing: What is it good for?

But what can you actually do with affect in CMC and HCI? Picard 1997:

– Recognise people’s feelings, moods, temperaments– Express a particular feeling, mood or temperament– Have a feeling:

• caused by events in the world, or in yourself• causing—or at least biasing—further actions by you.


HCI diagnosis of HAL 9000 failure

According to Rosalind Picard, the problem wasn’t that HAL had emotions.

He could certainly recognise other people’s– “Look Dave, I can see you’re really upset about this”

He had his own– “I’m afraid, I’m afraid”

The problem was expressing them to others:– That level tone of voice betrayed nothing– And what about his facial expressions?


HCI diagnosis of HAL 9000 failure: A more expressive HAL?

Might have been more understandable to his crewmates And left us with a less dramatic film …


Affect in CMC: Why would it matter?

We’ve seen that CSCW deprives people of cuesto other’s states of mind

What is actually available in CSCW? Do people use it?


Affect in CMC: Personality Perception

Computer-mediated communication– Reduces available cues

Extraversion– High observability– Low evaluativeness

Neuroticism: more evaluative and less observable Previous findings:

– Relatively high agreement for Extraversion– Even at zero-acquaintance and in CMC environment


Perception: Methods

Participants– 30 experienced e-mail users– Current or recent university students– Naive raters of personality

Materials– Six texts representative for range of scores for Extraversion– Six texts representative for range of scores for Neuroticism– For a given dimension, other EPQ-R dimensions held constant

Procedure– Texts subjectively rated using exemplar descriptions

(Eysenck and Eysenck, 1991; Sneed, McCrae and Funder, 1998)– Ease of judgment rated– Participants also rated similarity of target


Perception: Results for Extraversion

Strong target-rater agreement– Aggregate correlation (McCrae and Costa, 1987):

rs(5)= .89, p=0.019– Also relatively high inter-rater agreement:

Mean rs = .48 (SD=0.17) High E judges rate High E targets as more similar

– Interaction effects of text author and rater personality– However, Mid E rated as more dissimilar than Low E– … All judges rated High E texts as more similar

High E texts viewed as easier to rate


Perception: Results for Neuroticism

Poor target-rater agreement– Aggregate correlation:

rs(5)= –.37, ns

– Somewhat better inter-rater agreement:Mean rs = .31 (SD=0.16)

No similarity rating effects No ease of rating effects


Perception: Discussion

Extraversion is highly salient (observable)– Even through minimal cues– Using exemplar description ratings– Confirms previous literature

But what about Neuroticism?– Apparently it is not consciously detected in language– Does this mean it is irrelevant to communication?– …and therefore irrelevant to language generation?


Affect in HCI: Why would it matter?

Reeves and Nass:– Computers Are Social Actors

eg: people are polite to computers Nass et al. (1995) had subjects use a text-only interface to solve

a problem with help from the computer.– Simple language variables were manipulated to provide Dominant

and Submissive system versions• Priority, hedges, confidence, name

They found similarity attraction effects– Preference extended to estimates of system efficiency, etc.– Cf. more recent results from Isbister and Nass (2000, 2001).– If a system can project a consistent—or even better, a convergent—

linguistic personality, this will enhance the user’s experience.


Affect in HCI: the medium term

In the neaar future, we are unlikely to have machines that Have emotions.

But, given the Reeves and Nass results, it may be worth working on the Recognition and Expression of emotions, moods and temperaments.

Why would this help?– Recognition: making personal agents sensitive to our feelings or

moods.• Boredom, inattention, stress• Content indexing (pain, fear, rage)• Deception, anxiety detection

– Expression: making computers appear to have feelings, moods or temperaments, even if they don’t

• More acceptable (?) companions• Better information transmission


The uses of recognition

Personal agents– Software on your mobile, PDA, desktop or dashtop

• It filters your email or vmail,• shops for bargains for you,• chooses mood music to calm you down or wake you up,• finds places of interest to visit,• finds news snippets,• solves problems• arranges meetings.

– To do this well, it spends a lot of time with you, learning what you do, and how you like to do it

But it would do much better if:– It knew when you were interruptible– It knew how well you liked its suggestions


Means of recognition

Emotion response, not mental workload … Physiological (contact):

– Galvanic Skin Response• How stressed you are

– Blood Volume Pressure– EEG

• Which areas of your brain are most active– ECG, ERP, pulse, respiration, pupil dilation …

Visual (remote):– Facial expression

• Muscle action units (cf. Ekman)– Posture, gait, gesture

Audio (remote):– Voice features:

• Volume, rate, pitch range, quality


The uses of expression

Software agents may be more or less confident in their recommendations

Presenting a lot of information via voice is a problem, since it takes time, and doesn’t really use the voice channel to the full


Means of expression

If an agent has an animated talking head, it should change its vocal and facial expression if it isn’t sure you’ll like its suggestion or if it’s unusual

A speech synthesiser should be ableto distinguish– News items you will or won’t enjoy hearing about– Email actually requiring an answer—as opposed to all that stuff you

just get cc’d on Non-human, but predictable, expression

might help too:– Culture drones have colour fields (Iain M. Banks)


J Scheirer, R Fernandez, J Klein, RW Picard – 2002

Frustrating the user on purpose: a step toward building an affective computer - Interacting with Computers, 2002

Timed visual sorting task, with mouse clicks– Eliciting frustration by adding unpredictable delays to some clicks

• Poor predictability, inconsistent honesty– Measuring:

• GSR (arousal -> sweat -> higher conductance); frustration & anxiety• BVP (anxiety -> cold feet -> light absorbtion in capillaries)

– Hidden markov models to learn sequence labels (frustration: 1/0)• 67% accuracy on test set (50% baseline)• Four click strategies (relative aggression?)

– Several HCI-affect guidelines, including• Eliciting affect may require deception, and violation of HCI guidelines

– More pattern recognition research needed; eliciting pleasure worth considering.


M Pantic, LJM Rothkrantz – 2003

Toward an affect-sensitive multimodal human-computer interaction - Proceedings of the IEEE, 2003

Major survey paper, covering physiology, visual (facial) and auditory (speech) modalities– Rapid behavioural signals communicate messages including:

affective states, emblems (wink), manipulators (scratch, put), illustrators (brow raise), regulators (nods)

– Recognise from 10ms audio, 40ms video– Notes face features (up to 98% acc), speech features (70-80% acc),

compared with (eg) 17% baseline– Notes drive toward bimodal recognition

Notes:– most work assumes clean, posed input,– ignores task and communicative context– Accepts “late integration”

Acceptance of affective HCI depends on user-friendliness and trustworthiness


FN Egger – 2001

Affective Design of E-Commerce User Interfaces: How to Maximise Perceived Trustworthiness - Proc … of CAHD2001: Conference on Affective Human Factors Design, 2001

All about trust in e-commerce (and hence, websites, mostly) Focus on initial trust

– Before (brand), during (user experience), after (customer service) Responsiveness, expensiveness

– Induce trust (cf. Fogg et al.)


F Rosis, C Pelachaud, I Poggi, V Carofiglio, BD … - 2003

From Greta's mind to her face: modelling the dynamics of affective states in a conversational … - … - International Journal of Human-Computer Studies, 2003

Affective Presentation Markup Language– To drive facial actions (given facial definitions) and voice tone

appropriate to affective status of conversation– Predicted from status of goals;

• cf. Ortony et al. 1988– Modelled with dynamic belief networks

Good survey of expressive agents Problems noted:

– Spurious variations– Overemotion– Emotion masking– Expression overlapping (and time stability)


F Rosis, C Pelachaud, I Poggi, V Carofiglio, BD … - 2003


C Lisetti, F Nasoz, C LeRouge, O Ozyer, K Alvarez – 2003

Developing multimodal intelligent affective interfaces for tele-home health care - International Journal of Human-Computer Studies, 2003

Doctors and nurses need to know how remote patients feel BodyMedia SenseWear wireless non-invasive wearable computer

for physiological recognition (GSR, temperature, movement) Haptek PeoplePutty Avatar (talking head) to elicit information,

reflect empathy 10 participants; 35 minutes of data with elicited emotion states

– K nearest neighbour; discriminant function analysis– Accuracies: 90% (sadness) 80% (anger, fear), 70% (frustration)


E Hudlicka – 2003

To feel or not to feel: The role of affect in human–computer interaction - International Journal of Human-Computer Studies, 2003

Introduction to special issue Draws together the disciplines:

– Neuroscience (bases), cognitive psychology (uses), AI(models)

Cites Picard as proponent, Hollnagel as sceptic– Notes the question: even if you can detect/express affect, does it

actually enhance HCI? Lots of useful references, and an orienting framework


T Partala, V Surakka – 2004

The effects of affective interventions in human–computer interaction - Interacting with Computers, 2004

Measure pupil size, in response to emotional auditory stimuli


D Johnson, J Wiles – 2003

Effective affective user interface design in games - Ergonomics, 2003


RW Picard – 2003

Affective computing: challenges - International Journal of Human-Computer Studies, 2003


RW Picard, SB Daily – 2005

Evaluating affective interactions: Alternatives to asking what users feel - CHI Workshop on Evaluating Affective Interfaces: Innovative …, 2005


K Hone – 2006

Empathic agents to reduce user frustration: The effects of varying agent characteristics - Interacting with Computers, 2006


ME Foster, J Oberlander - 2006

Mary Ellen Foster’s study explores types of variation in emphatic talking heads and finds that:– Variation is sometimes preferred to more simple model.


… a note of caution

Malesubjects


… a note of caution

FemaleSubjects

Cf. J. Hall1984


Do the (new) experiment

http://homepages.inf.ed.ac.uk/mef/head-experiments/


Affect in HCI: the longer term

Norman, in Emotional Design, points out:– Complex machines with emotions will be more understandable to us

than complex ones without them.– We predict and explain each other’s behaviour by reference to their

thoughts and feelings.– People who appear to have no feelings are very hard to predict or

explain.– So, having understandable emotions would make machines easier

to interact with


Articles

1. J Scheirer, R Fernandez, J Klein, RW Picard – 2002 - Frustrating the user on purpose:a step toward building an affective computer - Interacting with Computers

2. M Pantic, LJM Rothkrantz – 2003 - Toward an affect-sensitive multimodal human-computer interaction - Proceedings of the IEEE

3. FN Egger – 2001 - Affective Design of E-Commerce User Interfaces: How to Maximise Perceived Trustworthiness - Proc … of CAHD2001: Conference on Affective Human Factors Design

4. F Rosis, C Pelachaud, I Poggi, V Carofiglio, BD … - 2003 - From Greta's mind to her face: modelling the dynamics of affective states in a conversational … - … - International Journal of Human-Computer Studies

5. C Lisetti, F Nasoz, C LeRouge, O Ozyer, K Alvarez – 2003 - Developing multimodal intelligent affective interfaces for tele-home health care - International Journal of Human-Computer Studies

6. E Hudlicka – 2003 - To feel or not to feel: The role of affect in human–computer interaction - International Journal of Human-Computer Studies

7. T Partala, V Surakka – 2004 - The effects of affective interventions in human–computer interaction - Interacting with Computers

8. D Johnson, J Wiles – 2003 - Effective affective user interface design in games - Ergonomics9. RW Picard – 2003 - Affective computing: challenges - International Journal of Human-Computer

Studies10. RW Picard, SB Daily – 2005 - Evaluating affective interactions: Alternatives to asking what users

feel - CHI Workshop on Evaluating Affective Interfaces: Innovative …11. K Hone – 2006 - Empathic agents to reduce user frustration: The effects of varying agent

characteristics - Interacting with Computers12. ME Foster, J Oberlander - 2006 - Data-driven generation of emphatic facial displays. Proceedings

of the 11th European ACL


Further Reading

Byron Reeves and Cliff Nass “The Media Equation” Rosalind Picard “Affective Computing” Don Norman “Emotional Design” Cynthia Breazeal “Designing Sociable Robots” Alastair Gill, Jon Oberlander and Elizabeth Austin “Perception of

e-mail personality at zero acquaintance”

Documents

1 Lecture 17: Interaction, Work and Technology III: Affective HCI Jon Oberlander