Designing and Evaluating Life-like Agents as Social Actors

Helmut PrendingerDept. of Information and Communication Eng.

Graduate School of Information Science and Technology

University of Tokyo

helmut@miv.t.u-tokyo.ac.jphttp://www.miv.t.u-tokyo.ac.jp/~helmut/helmut.html

Short Bioeducation, experience Master’s in Logic (1994)

U. of Salzburg, Austria, Dept. of Logic and Philosophy of Science Dynamic modal logic (completeness, decidability) Non-degree studies in Psychology, Linguistics, Literature

Ph.D. in Artificial Intelligence (1998) U. of Salzburg, Dept. of Logic and Philosophy of Science and

Dept. of Computer Science; U. of California, Irvine Incomplete reasoning (deduction, hypothetical reasoning, EBL)

Post doctoral research U. of Tokyo, Ishizuka Lab JSPS Fellowship (4/1998-3/2000): Knowledge compilation,

hypothetical reasoning “Mirai Kaitaku” project (since 4/2000): Life-like characters,

affective communication with animated agents, markup languages for animated agents, emotion recognition

Social Computingmain objective and task

Social Computing aims to support the tendency of humans to interact with

computers as social actors.

Develop technology that reinforces human bias towards social interaction by appropriate feedback

in order to improve the communication betweenhumans and computational devices.

Social Computingrealization

Most naturally, social computing can be realized

by using life-like characters.

Life-like Characters at Worksample applications

Sales, DFKI

Tutoring, USC Knowledge Sharing, ATR

Presentation, U. of Tokyo

Entertainment, MIT

Life-like Charactersdesiderata

Life-like characters should be emphatic and engaging as tutors trustworthy as sales persona entertaining and consistent as actors stimulating as match-makers convincing as presenters (in short) … social actors [… and competent ]

Life-like characters should enable effective and natural communication with humans

Backgroundcomputers as social actors

Humans are biased to treat computers like real people

Psychological studies show that people tend to treat computers as social actors (like other humans)

Tendency to be nicer in “face-to-face” interactions, ...

Animated agents may support this tendency if they are designed as social actors

Ref.: B. Reeves and C. Nass, 1998. The Media Equation. Cambridge University Press, Cambridge.

Animated Agents as Social Actorsrequirements for life-likeness

Synthetic bodies Emotional facial

display Communicative

gestures Posture Affective voice

Embodiment

Features of Life-like Characters

Artificial Emotional Mind

Affect-based response Personality Response adjusted to

social context social role awareness

Adaptive behavior social intelligence

Outlinedesigning and evaluating life-like characters

The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting

Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game

Book project - character scripting languages and applications

SCREAM System ArchitectureSCRipting Emotion-based Agent Minds

Appraisal Modulethe cognitive structure of emotions

Evaluates external events according to their emotional significance for the agent

Outputs emotions joy, distress happy for, sorry for angry at resent, gloat … 22 in total

Ref.: A. Ortony, G. Clore, A. Collins, 1988. The Cognitive Structure of Emotions. Cambridge UniversityPress, Cambridge.

Social Filter Moduleemotion expression modulating factors

Ekman and Friesen’s facial “Display Rules” (’69)

Expression and intensity of emotions is governed by social and cultural norms

Brown and Levinson (’87) on linguistic style

Linguistic style is determined by social variables: power, distance, imposition of speech acts

Agent Modelcharacter profile, affect processing

Character Profile static and dynamic features

Static features personality traits, standards

Dynamic features goals, beliefs, attitudes

Attitudes (liking/disliking) are an important source of emotions toward other agents

an agent’s attitude decides whether it has a positive or negative emotion (toward another agent)

“happy for”– resent; “sorry for”– gloat an agent’s attitude changes as a result of communication

dependent on “affective interaction history”

Signed Summary Recordcomputing attitude from affective interaction history

joy (2)

distress (1)

distress (3)

angry at (2)

hope (2)

good mood(1)

gloat (1)

happy for (2)

winning emotionalstates

time

positive emotions

negativeemotions

joy (2)

hope (2)

good mood(1)

happy for (2)

distress (1)

distress (3)

angry at (2)

gloat (1)

+ Liking if positiveDisliking if negative

Attitudesummaryvalue

=

Ref.: A. Ortony, 1991. Value and emotion. In: W. Kessen, A. Ortony, and F. Craik (eds.), Memories,Thoughts, and emotions: Essays in the honor of George Mandler. Hillsdale, NJ: Erlbaum, 337-353.

inte

ract

ion

his

tory

<emotion,intensity> pairs

If a high-intensity emotion of opposite sign occurs – e.g., a liked interlocutor makes the agent very angry

Agent ignores “inconsistent” new information Agent updates summary value by giving greater weight to

“inconsistent” information (“primacy of recency”, Anderson ’65)

Updating Attitudeweighted update rule

disliking liking h-weight angry r-weight

3 = (3 0.25) (5 0.75)

Consequence for future interaction with interlocutor Momentary disliking: new value is active for current situation Essential disliking: new value replaces summary record

(Sitn) = (Sitn1) h + w

(Sitn) r

w: intensity of

(winning) emotion

, {+,}h/r: historical/recency

weight

Life-like Agentsmaking them act and speak

Realization of embodiment 2D animation sequences Synthetic affective speech

Technology Microsoft Agent package (installed client-side) JavaScript based interface in Internet Explorer

Microsoft Agent package Controls to trigger character actions Text-to-Speech (TTS) Engine Voice recognition

Multi-modal Presentation Markup Language (MPML) Easy-to-use XML-style authoring tool Interface with SCREAM system

Life-like Characters in Interactionsome demos

ComicsScenario

CasinoScenario

Life-like characters that change their

attitude during interaction

Animated comics actors engaging in developing social

relationships

BusinessScenario

Animated agents that storify tacit

corporate knowledge

Casino Scenariolife-like characters with changing attitude

Animated advisor (“Genie”) Emotion, personality Changes attitude dependent

on interaction history with user

Dealer (“James”), player (“Al”) Pre-scripted behavior

Implemented with MPML and SCREAM

Genie‘s Character Profile% Personality specificationpersonality_type(genie,agreeableness,3).personality_type(genie,extraversion,2).% Social variables specificationsocial_power(genie,user,0,_).social_distance(genie,user,1,_).% Goalswants(genie,user_wins_game,1,_).wants(genie,user_follows_advice,4,_).% Attitudeattitude(genie,user,likes,1,init).

User in the role of player of Black Jack game

Emotional Arcadvisor’s dominant emotions depending on attitude

sorry for (4)distress (4) gloat (5) sorry for (5) good mood (5)

ignores advice

pos. attitude

user looses

ignores advice

pos. attitude

user looses

ignores advice

neg. attitude

user looses

follows advice

pos. attitude

user looses

ignores advice

pos. attitude

user wins

Internal intensity values

Round 1 Round 2 Round 3 Round 4 Round 5

advisor has agreeable personality

advisor has agreeable personality, is socially slightly distant to user

sorry for (5)distress (1) gloat (2) sorry for (5) good mood (5)

Intensity values of expressed emotions

Implementation

Agent Scriptingsimple MPML script

<!--Example MPML script --><mpml>… <scene id=“introduction” agents=“james,al,spaceboy”> <seq> <speak agent=“james”>Do you guys want to play Black Jack?</speak> <speak agent=“al”>Sure.</speak> <speak agent=“spaceboy”>I will join too.</speak> <par> <speak agent=“al”>Ready? You got enough coupons?</speak> <act agent=“spaceboy” act=“applause”/> </par> </seq> </scene>…</mpml>

Mind-Body Interfaceinterface SCREAM MPML

<!--MPML script showing interface with SCREAM --><mpml>… <consult target=”[…].jamesApplet.askResponseComAct(‘james,’al’,’5’)”> <test value=“response25”> <act agent=“james” act=“pleased”/> <speak agent=“james”>I am so happy to hear that.</speak> </test> <test value=“response26”> <act agent=“james” act=“decline”/> <speak agent=“james”>We can talk about that another time.</speak> </test> … </consult>… </mpml>

Alternative Viewsmart characters vs. smart environments

“Sense-think-act” cycle Classical AI approach Internet softbots search for

information on the web, robots explore their environment

All the intelligence is agent-side

“Annotated” environments Shift from agent intelligence to

environment intelligence Semantic web, ubiquitous

computing, affordance theory Agents and environments can

be developed independently

“perceives”game state

infers “I am happy”

“acts”expresseshappiness

behaviorrepository

“tells”available behaviors

environment instructs agent “be happy now”

Outline revisiteddesigning and evaluating life-like characters

The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting

Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game

Book project - character scripting languages and applications

Affective Computingwhy should a computer recognize user emotions?

Human-human communication Based on efficient grounding mechanisms

including the ability to recognize interlocutors’ emotions (frustration, confusion,…)

Humans may react appropriately upon detection of an interlocutor’s emotion (clarification upon confusion)

Human-computer communication Computers typically lack ability to

recognize user emotions Ignoring users’ emotions causes users’

frustration Recognizing and responding to users’

(often) negative emotions may improve users’ interaction experience

Ref.: R. Picard, 1997. Affective Computing. The MIT Press.

Emotion Recognitionhow can computers recognize users’ emotions?

Stereotypes A typical visitor of a casino wants… (to win)

Communicative modalities Facial display (face recognition) Prosody (speech analysis) Linguistic style (NLU) Gestures (gesture recognition) Posture (posture recognition)

Physiological data Biosignals

Physiological Data AssessmentProComp+ unit

EMG: Electromyography EEG: Electroencephalography EKG: Electrocardiography BVP: Blood Volume Pressure GSR: Galvanic Skin Response Respiration Temperature

GSRBVP

sensors

Inferring Emotions from Biosignals Lang’s 2-dimensional emotion model

Lang’s two dimensions Valence - positive or negative

dimension of feeling Arousal - degree of intensity

of emotional response Biometric measures

Skin conductivity increases with arousal (Picard ’97)

Heart rate increases with negatively valenced emotions

Note introverts reach a higher level

of emotional arousal than extroverts

enraged

Valence

Arousal

excited

joyful

sad

relaxeddepressed

Ref.: Lang, P. 1995. The emotion probe: Studies of motivation and attention. American Psychologist 50(5):372–385.

some named emotions in thearousal-valence space

Experimental Studyeffects of a character-based interface

Aim of study Show that a character with affective expression may improve

users’ experience (= reduce frustration) of a simple quiz game Method

Biosignals to measure skin conductance and blood volume pressure (`objective’ assessment of user experience)

Questionnaire (users’ subjective assessment) Instruction

Addition/subtraction task (short-term memory load) Solve a series of 30 quizzes correctly and as fast as possible Frustration is deliberately caused by delay (in 6 out 30 quizzes)

Subjects 20 university students (all male Japanese, approx. 24 years old) JPY 1000.- for participation, JPY 5000.- for best score

Junichiro Mori -Experimenter

Analyser

Experimental Setup

Instructionmathematical quiz game

Add 5 numbers and subtract the i-th number (i < 5)

1 + 3 + 8 + 5 + 4 = [21]21] E.g.: subtract the 2nd number Result: 18

Select the correct answer by clicking the radio button next to the number

Then the character tells whether answer is correct

It is correct.(polite language)

timer

sometimes delayhere (6 – 14 sec.)

Two Versions of the Gameaffective vs. non-affective (independent variables)

Affective Version Non-Affective Version

Description

Character expresses happiness (sorriness) for correct (wrong) answer Character shows empathy (when delay occurs) Character expresses affect both verbal and nonverbal

Character does not show affective response Character ignores occurrence of delay

Hypotheses

Character may reduce user stress (SC) and decrease negative valence (heart rate)

Character has no significant effect on user emotion (SC, heart rate)

Character Responsesexamples of affective/non-affective feedback

I am sorry. It is wrong.(hyper-polite language)

Hanging shoulder gesture toexpress sorriness non-verbally

I am sorry for the delay.(polite language)

Character apologizes for thedelay

Non-affective feedback“Wrong.” No non-verbalemotion expression.

Non-affective feedbackCharacter ignores the occurrenceof delay.

Analyzing Physiological User Data

BVP

GSR

delaystarts

delayends

DELAYsegment

RESPONSEsegment

userresponse

agentresponse

BiographSoftware(ThoughtTechnologies)

BVP could not be takenreliably

Preliminary Findings9 subjects in each version (data of 2 subjects discarded)

Hypothesis (main): affective agent behavior reduces user frustration

Hypothesis (design): delay induces frustration in subjects All 18 subjects showed significant rise of SC in DELAY segment Corresponds to finding in behavioral psychology (if an individual is prohibited

from attaining a goal, the individual experiences primary frustration)

Preliminary evaluation suggests that an animated character expressing emotions and empathy may undo some of the user’s frustration.

DELAYsegment

mean values sf SC (BVP could not be taken reliably)

RESPONSEsegment

Non-affective version: mean = 0.05Affective version: mean = 0.2

t-test (assuming unequal variance)t(16)=2.57; p = .01

Agents Adapting to User Emotionassumes real-time recognition of user emotions

emotional state

user’straits

ti

bodily expressions

user’saction

ti+1

agent’s actions

sensors

emotional state

user’straits

bodily expressions

sensors

user modeluser model

DynamicDecisionNetwork(simplified)

learning learning

U

evidencenodes

evidencenode

QUESTION:Given user’s state at ti, which agent action will maximize agent’sexpected utility at ti+1, in terms of, e.g., user’slearning and emotion?

Dynamics of User Emotionsuser personality

bodily expressions

extraversion

skin conductivity

eyebrows position

agent’s action

pos valence

vision basedrecognizer

EMG

sensors

GSR

ti+1

ti

reproach

shame

joy

user’s emotional state at ti

user’s emotionalstate at ti+1

neg valence

highdown(frowning)

heart rate

BVP

high

provide help

do nothing

agreeableness

user goals

succeed bymyself

have fun

arousal

reproach shame joy

Ref.: Conati, C. 2002. Probabilistic assessment of user’s emotions in educational games. Applied Artificial Intelligence 16(7-8):555–575.

Outline revisiteddesigning and evaluating life-like characters

The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting

Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game

Book project - character scripting languages and applications

Book Projectcharacter scripting languages and applications

Wide dissemination of life-like character technology requires

standardized ways to represent the behavior of agents

Book will offer state-of-the-art on XML-based markup languages and tools

Scripting languages for face animation, body animation and gestures, emotion expression, synthetic speech, interaction with environment,…

Characters are already used in a wide variety of applications

Book contains some of the most successful character-based applications

Synopsis chapters on character design

H. Prendinger, M. Ishizuka (Eds.)Life-like Characters. Tools, Affective

Functions and ApplicationsSpringer Hardcover

(in preparation)

useful asStandard/Reference Book

State-of-the-Art in Life-like AgentsCourse Book

for HCI, HAI, multimedia, life-like agentapplications, scripting languages,…

Conclusion Social Computing

Human-computer interaction as social interaction Designing life-like characters as social actors

Believability-enhancing agent features Emotion, personality, social role awareness, attitude

change, familarity change Casino demo Future avenues – “smart” environments (character &

annotated environments) Evaluating life-like characters as social actors

Experimental study using user’s biosignals Life-like characters’ affective response may undo some

of the user’s negative feeling Future avenues – real-time adaptivity of agent

behavior to user’s emotion, decision-theoretic approach to agent behavior