1

Chap. 13Chap. 13Expressive Behaviors for Virtual WorldsExpressive Behaviors for Virtual Worlds

Stacy Marsella, Jonathan Gratch, and Jeff Rickel

November 22, 2004

2

OutlineOutline

• Introduction

• Steve• Jack and Steve• Carmen’s Bright IDEAS (CBI)

• Mission Rehearsal Exercise (MRE)

• Conclusion

3

IntroductionIntroduction• Goal of this chapter

– To create virtual humans that convey much information to humans while interacting with them in virtual worlds

– Information includes non-verbal behaviors concerning emotional state

• Requirements for virtual human design– Believable: they must provide human-like behaviors– Responsive: they must respond to the events surrounding them– Interpretable: the user must be able to interpret their response to

situation

• This chapter describes– The progress toward a model of outward manifestations of an

agent’s cognitive and emotional state– The review of three prior systems that influenced model of this

chapter– The model of this chapter is an integration of prior systems

4

ContributionContribution

• Previous systems provides impressive capabilities in its area of research focus, but they had some limitations

• Limitation of previous systems– Agents are modeled only for conversation between two of them– Collaboration of users and agents on one task is not allowed– Agent’s presence was partially limited to a 2D– Agent’s movement and relationships are limited

• Steve’s contribution– Only Steve can interleave task-related behaviors and face-to-face

dialogue with humans and virtual humans in dynamic virtual worlds

– Interleaving 은 성능을 높이기 위해 데이터가 서로 인접하지 않도록 배열하는 방법

Steve

5

The ArchitectureThe Architecture

Steve

• Steve consists of three modules: perception, cognition and motor control

SpeechRecognition

Graphics Simulator Speech Synthesis

Steve

Domain knowledge

General capabilities

Motor commands

Cognition

MotorControl Perception

Translate intomovements, speech

Broadcast toenvironment

Filter, assembleinto coherent view

Monitor events

Current state

Commands toenvironment

Eventnotifications

6

PrimitivesPrimitives

• The cognition module generates Steve’s communicative behavior by dynamically selecting next action from a list of primitives

• Primitives– Speak– Move to an object– Manipulate an object– Visually check an object– Point at an object– Give tutorial feedback– Offer turn– Listen to student– Wait for someone

Steve

– Acknowledge an utterance– Drop hands– Attend to Action

7

React to Interruption & Context (1)React to Interruption & Context (1)• To react to interruption

– In dynamic virtual worlds, to react to the unexpected events is important for task-oriented collaboration

– It is important to maintain coherence of an agent’s behavior

• Steve considers 2 separate but complementary types of context

1. Task context– Continually monitors the virtual world– Uses the task model to plan to complete the task – Uses variant of partial-order planning techniques

2. Dialogue context– Represents the state of interaction between a student and Steve– Uses focus stack: represents the hierarchy of tasks, subtasks, and

currently engaged actions– Maintains a record, Steve’s answer etc.

Steve

8

React to Interruption & Context (2)React to Interruption & Context (2)

• Based on the current task and dialogue contexts, Steve can choose next action

• Steve can choose an action of following three roles– Steve can respond to the student to know a student’s request– Steve can choose for himself how to advance the collaborative

dialogue– Steve can choose a turn-taking or grounding act that helps

regulate the dialogue without advancing the task

• In case several actions are appropriate at one moment, – Steve choose next action considering priorities– Actions with low priority will be performed later, not be deferred

• Summary– Steve does not include an emotion model, but– It serves a valuable foundation for MRE model

Steve

9

Jack and Steve vs. SteveJack and Steve vs. Steve• The Jack and Steve system was predicated on two basic claims:

– Plans and plan reasoning can mediate expressive behavior– Small biases in how plans are evaluated and generated can result

in large systematic differences in agent behavior

• Jack and Steve vs. Steve

Jack and Steve

Jack and Steve Steve

To support biases in plan evaluation, J-S include a richer plan representation

To translate small biases into large external variations, J-S focused on plan generation

Steve focused more on plan execution and repair

J-S explored how biases in the plan generation process could support a variety of non-collaborative interactions as well, but as the focus was exploring systematic differences in joint behavior, agents only interact with other computational agents and not with human users

Steve focused on collaborative interactions between agents and human users

10

Motivating ExampleMotivating Example

• This example shows two separate runs of system where the only difference is a change in the personality of Steve (rude / fair)

– Jack: I want to make some big money– Steve: I want to catch some waves

Jack and Steve

Mental state of each agent at specific point in the interaction

Mental state shows individual action, plan, emotion etc.

11

Plan-Based Social Appraisal (1)Plan-Based Social Appraisal (1)

• Jack and Steve supports expressive and flexible interactions by implementing social reasoning as a layer atop a general-purpose planning system

• Planning system provides – Domain-independent representations of world actions– General reasoning mechanisms that construct partial plans, repair

interactions among them, and oversee plan execution• The social layer

– Manages communication – Biases plan generation and execution according to social

context

Jack and Steve

General purpose planning system

Social reasoning layer

12

Plan-Based Social Appraisal (2)Plan-Based Social Appraisal (2)

• To support various social interactions, the social reasoning layer must provide a rich model of the social context

• The social situation is described in terms of many static and dynamic features

• Static features – include innate properties of the character– social role and small set of personality variables

• Dynamic features – Are derived from inference procedure that operate on the current

mental state of agent– Include - current communication obligation

- various relations between plans in memory (e.g. your plans threaten my plans) - a model of the emotional state of the agent

Jack and Steve

13

Plan-Based Social Appraisal (3)Plan-Based Social Appraisal (3)

• One novel aspects of this system is that the social layer alters the planning process fundamentally

• In terms of planning, the social layer can bias planning to be more or less considerate to the goals of other participants

• In terms of communication, agents can vary from bossy agents that try to tell others what to do to passive agents that avoid interactions or social conflicts

Jack and Steve

14

Social Context, Operator, RuleSocial Context, Operator, Rule

• Jack and Steve system maintains a rich representation of the social context

• The social context of Jack and Steve is divided into 3 categories of plan context, emotional context, and communicative context

• Social operators are actions that occur at the social level• Social operators are subdivided into meta-planning operators

and communicative operators.

• Distinct personalities are implemented via a set of social rules• Social rules execute sequences of social operators based on

appraised features of the social context• Jack and Steve system include 30 social rules

Jack and Steve

15

IntroductionIntroduction

• Carmen’s Bright IDEAS (CBI)– An agent-based system designed to realize an Interactive

Pedagogical Drama– The pedagogical goal is to help mothers of pediatric cancer

patients– The drama mirrors the mother’s own problems– It allows the learner to interactively influence how Carmen

copes with problems

• In CBI– to model the causes of emotions is important– agents needed effective ways to convey the impact of emotion

on both the agent’s dialogue and physical behavior

Carmen’s Bright IDEAS

16

The Drama of Carmen’s Bright IDEASThe Drama of Carmen’s Bright IDEAS

• CBI is an interactive pedagogical drama• Carmen discusses her problems with a clinical counselor, Gina• Gina suggests a problem solving technique called Bright IDEAS• Bright IDEAS means

– Bright: positive attitude– I: Identify a solvable problem– D: Develop possible solution– E: Evaluate your options– A: Act on your plan– S: See if it worked

• User, the human mother, interacts with the drama by making choice for Carmen

• Both Gina’s dialogue and the user’s choices influence the cognitive and emotional state of Carmen (in conflicting ways)

Carmen’s Bright IDEAS

17

Interaction ModelInteraction Model

Carmen’s Bright IDEAS

• In this interaction model, the user gets a vivid demonstration

• Learner influences Carmen by selecting thought balloons

18

CBI Agent ModelCBI Agent Model

Carmen’s Bright IDEAS

• IPD have 5 main components, but the discussion will focus on the on-screen character agents

• Each on-screen character is realized by an agent architecture that has modules for problem solving, dialogue, emotional appraisal, and behavior generation

19

Emotional ModelEmotional Model• The pedagogical goal was for the mothers to learn how to choose

and carry out the right coping strategy for a given situation

• Cognitive appraisal theory by Richard Lazarus influences CBI– This organizes human behavior around appraisal and coping– Appraisal leads to emotion by assessing the person-environment

relationship– Coping and appraisal interact and unfold over time, supporting the

temporal character of emotion evident in human behavior

• Coping is the process of dealing with emotion– Problem-focused coping– Emotion-focused coping

Carmen’s Bright IDEAS

20

RemarksRemarks

• The design of dialogue, emotions, and expressive behavior systems was driven by a need to convey deep inner conflicts and how those conflicts play out expressively

Carmen’s Bright IDEAS

21

IntroductionIntroduction• The Mission Rehearsal Exercise (MRE) system

– Brings ideas from each of the preceding systems– Creates a broader and more flexible array of expressive behaviors– The goal is to teach leadership skills in high-stakes social situation

• To model such dramatic and interactive scenario, the MRE – Combines Steve’s ability to flexibly interact with a human user– Augment it with the richer social and emotional behaviors of CBI

and Jack and Steve

• The MRE combines a variety of capabilities in service of realistic and natural collaboration with virtual humans

• Example

Mission Rehearsal Exercise

22

Cognition and EmotionsCognition and Emotions

• Our virtual humans must– Provide realistic expressive behaviors– Require cognitive machinery to recognize which behaviors are

appropriate in the course of an unscripted interaction with a human user

• The Steve system is selected as a starting point since it supports flexible face-to-face interactions with a human user

• Integrate plan-based appraisal model of the Jack and Steve system into the Steve system

• Integrate CBI’s coping model– Coping strategies were recast explicitly into procedures that

updated Steve’s task representations and reasoning processes

Mission Rehearsal Exercise

23

Cognition and EmotionsCognition and Emotions

Mission Rehearsal Exercise

Steve

Plan-based appraisal modelof Jack and Steve

Coping model of Carmen’s Bright IDEAS

Steve supports flexible face-to-face interaction with human users

This allows agents to negotiate over tasks and express emotions, i.e. task

model is extended in a number of ways to represent the socio-emotional

context

Integrating CBI’s coping model results in a tight integration between

appraisal, coping and task reasoning that closely follows the cognitive

appraisal theory of Richard Lazarus

+

+

24

Physical BehaviorPhysical Behavior• Virtual humans in the MRE attempt to manifest the rich

dynamics of this cognitive and emotional inner state through each character’s external behavior

• The key challenge is the range of behaviors that must be integrated: Each character’s body movements must reflect – Its awareness of events in the virtual world– Its physical actions– The number of non-verbal signals that accompany speech

during social interaction– Its emotional reactions

• Expressive physical behavior in the MRE agents integrates the task-related non-verbal behaviors of the Steve system and the coping behaviors of CBI

Mission Rehearsal Exercise

25

ConclusionConclusion• Steve serves as the foundation for the virtual humans in MRE

system as his ability to interleave task-related behaviors and face-to-face dialogue in dynamic virtual worlds

• The Jack and Steve system contributed a model of task-oriented emotional appraisal and a model of socially situated planning

• The CBI system contributed a complementary model of emotional appraisal focusing on social relationships

• MRE virtual humans integrate many of the ideas from these three prior systems