WP 6: Emotion in Interaction Catherine Pelachaud, U Paris 8 Plenary, 4-6 June 2007, Paris

WP 6: Emotion in Interaction

Catherine Pelachaud, U Paris 8

Plenary, 4-6 June 2007, Paris

humaine wp6 : Emotion in Interaction 2

WP 6: The area

Research theme: role of emotion in interaction.

Three domains of study

Perception domain: how certain aspects related to cognition may influence agent’s actions

Interaction domain: how to create relations between users and agents; how the agent can provide feedback

Generation domain: how to show expressive behaviours consistently and naturally across modalities


WP 6: the main teams

University of Paris8

DIST - University of Genova

EPFL - Lausanne

France-Telecom

ICCS - Athens

Limsi CNRS

OFAI - Wien

T-systems - Berlin

KTH - Stockholm

MIRALab - Geneva

DFKI - Saarburcken

University of Ausburg

University of Hertfordshire

University of Paris 8

University of Sheffield

Twente University

INESC-ID - Lisbon

TCD - Dublin

University of Bari

ISTC-CNR - Rome


PHIPS

Definition of an Affective Interactive Embodied Conversational Agent that encompasses the capabilities:

1. Cognitive Influences on Action

2. Creating Affective Awareness

3. Backchannel properties and architecture

4. Coordination of signs in multi modalities

5. Expressive behaviour and speech


Element 1:

Cognitive influence on actions

Agent perceptual attention (UP8)

Agents with real-time synthetic vision, attention and memory capabilities

Model of attention and emotion aspects related to facial expression and novelty relation (WP3 / WP6)

Evaluation study of the visual perception model

GPU-based visual attention speed-up for real-time perception model (WP6 / WP7)


Element 1:

Cognitive influence on actions: DEMO - UA

Reaction to Agent‘s expressions (UA)

Integration of tangible input device, speech recognition, emotional behavior control

Analysis of user‘s gaze behavior


Element 1: Visual attention in affective agents

Cooperation between UA and NII, Tokyo: Investigation of the relationship between visual attention and affect Combining bio sensor with eye tracking technology

Conduction of an empirical study under the leadership of Helmut Prendinger to investigate the potential benefits of attentive presentation agents

Prendinger, Bee, Nischt, 2006


Element 2:

Creating Affective Awareness

Investigation of the level of user’s engagement with one another and with an ECA in an emotionally rich context (UA, HU, UP8, DIST, ICCS, KTH)

create affective relationship with others humans / objects

study of user’s engagement– when initiating, maintaining, ending an interaction

– through music, emotion and movement

– detection and imitation: ability to replicate emotional state


Element 2:


Expressive control of music and visual media by full-body movement

Collaboration between InfoMus Lab-DIST (University of Genova) and KTH (Royal Institute of Technology, Stockholm)

Development of a system allowing users to express themselves through their full-body movement and to control in real-time the generation of an audio-visual feedback

System based on the integration of two different software platforms: EyesWeb (for movement analysis and visual feedback generation) and pDM (to synthesize in real-time expressive music performances)


Element 2:

Creating Affective Awareness: DEMO

•The real-time audio-visual feedback consists of

(i) the rendering of a music performance with different emotional characterisations by manipulating acoustic parameters

→ the dynamic variations of the motor cues control the dynamics of acoustic cues such as tempo, sound level, articulation

(ii) the rendering of the user's silhouette on a big screen in front of them coloured depending on the expressivity of their movement


Element 2:


Development of realtime continuous emotion recognition from the speech signal (UA)

Implementation of system to mirror the user‘s affective state by using:

the Greta agent (UA)

the empathic anthropomorphic robot (Collaboration between UA and Bielefeld University)


Element 3:

Backchannels

Three communication levels:

establishing and maintaining engagement (contact, perception, attention) (WP4)

comprehension (understand, interest)

reaction (believability, attitude, agreement)

Three different dimensions to characterise backchannel signals:

cognitive/reactive (signals done with/without explicit planning)

sincere/deceptive (sincerity/goal to deceive one’s reaction)

imitation/dictionary (signal of alignment, positive/negative signals)

Backchannel forms: verbal and nonverbal signals


Element 3:

Backchannels

InsightData collection and analysisTheory and models

Perceptual tests of affective bursts and facial expressions

Modeling and ImplementationRecognitionDecisionGeneration

Testing and evaluation

DFKI, UTwente, URoma, UParis8, ISTC-CNR


Element 3:

Backchannels: Dialogue Management

Integration of the various components of a dialogue system capable of non-verbal expressivity:

a visual renderer (Greta),

an audio renderer (MARY), and

a dummy dialogue system capable of generating non-verbal behaviour (Conversational Dialogue Engine / DFKI)

Using OpenAIR


Element 3:

Backchannels


Element 4:

Coordination of signs in multiple modalities

Models of coordination between modalities built from Automatic analysis of instructed/acted behaviors (ICCS)

Manual annotation of spontaneous behaviors (CNRS-LIMSI, UP8)

Perceptual studies:Comparison of the original video

with 4 animations: – basic emotion 1 (e.g. Anger)– basic emotion 2 (e.g. Despair)– multiple levels replay– facial blending replay (UP8)

40 subjects

No-audio

#3

#41

Audio

#3

#41


Element 4:

Coordination of signs in multiple modalities

Models of individual expressive behaviors in each modality: example of reaction movements (EPFL)

Semantic representations : find concepts and relationships among them

Morphological Descriptors: height, gender, age, etc.Individuality: personality, emotional state, cultural background, etc.Body: geometry, skeletal structureBehavior Controllers: inputs required for algorithm to work and output it produces.

Reaction behaviorInverse Kinematics


Element 5:

Expressivity

Expressive behaviour: analysis/synthesis of expressive behaviours (DIST, OFAI, UP8, ICCS)

Expressive speech synthesis: blending of emotions, control of voice quality in speech synthesis, copy synthesis of emotional speech (DFKI, FT, T-S)

Model of complex emotions (UP8)

Reliable features of sadness

Fake joy Sadness masked by joy

+ =+

Neutral expression

Joy Sadness Superposition ofSadness and Joy

=+

EmoTV


Element 5:

Expressivity

Effects of Expressivity parameters over head, facial expression and gesture

over different time span: gesture phase, whole gesture, whole sequence

behavior mimicry (ICCS-UP8)


Element 5:

Expressivity

GEMEP: Corpus of acted emotional performances created by WP3/ Geneva.Feature Extraction from Audio Channel (OFAI)

Phonetic segmentation intoPhonemesSyllables

Pitch Extraction

Features from Video Channel (OFAI, DIST, UP8)Face detectionSilhouettes & Bounding BoxesHand tracking

Manual annotation and replay (UP8)


Element 5:

Expressivity

Restitution of salient information in human-machine interactions (FT)

Prosodic copy (F0 + duration) of the focused part (words…) from dedicated corpus to neutral synthesized utterances

TD-PSOLA copy synthesis on the focused part

Use focused part as target in unit selection

Synergies with national project PAVOQUE on parameterisation of prosody and voice quality for expressivity in speech synthesis (DFKI)

Spectral interpolation using LSFVoice adaptation with HMM synthesis

Emofilt: emotional speech synthesis by prosody transformation (T-S)

Interface to DFKI‘s MARY TTSAvailable in 34 languages Meant as a pragmatic tool

Screenshot of Emofilt GUI

sad anger


Conclusion

Creation of affective ECA able to:Perceive, adapt, respond affectively to events, objects, people in real/virtual world

Create affective bonds

Provide affective feedback

Be multimodal and expressive

Documents

WP 6: Emotion in Interaction Catherine Pelachaud, U Paris 8 Plenary, 4-6 June 2007, Paris