ICat as a Companion Robot

ICat as a Companion Robot

Siska FitrianieDragos DatcuAlin ChituL.J.M Rothkrantz

{s.fitrianie, d.datcu, a.g.chitu, l.j.m.rothkrantz}@ewi.tudelft.nlhttp://mmi.tudelft.nl

Man-Machine-Interactionhttp://mmi.tudelft.nl

2

Human AIBO Interactionhttp://mmi.tudelft.nl/~aibo


3

Requirements and Challenges

• Requirements:– Develop a system architecture for a companion cat– Design and develop multimodal system to portray

emotions in iCat as a result of interaction with humans.– Design of a personality model adapted to iCat.

• Challenges:– Development of a complete set of an HCI system: input sensing, input

processor, input fusion, dialog management, output fission.– Use of iCat’s specific software that is new and still under development– Little is still known or standardized in the cognitive model for humans.


4

Requirements in other words …

9:00 AM

11:00 AM 14:00 AM 16:00 AM

iCat! What’s today whether?

iCat!, Let’s play tic tac

toe!

iCat .. Read me a story, please ..

8:00 AM

Get a life! I am still sleeping!

What was that?? Hey .. where is everybody??

I feel so lonely!!! I wish I have a companion …

Are you out of your mind? I am sleeping!!!

Today is Monday, 27 March 2006 .. It will be

bright sunshine but a bit cold …

Hurray!! Finally I have a friend!!

OK!!I do the

first move!

OK, ok! I’ll do it. Don’t

cry ..


5

Multimodal Human Computer Interaction

SpeechRecognition

Facial ExpressionRecognition

LanguageGenerator

EmotionAnalysis

Fusion Input &Interpretation

Dialog ActionManager

Fission +Emotion Generator

AnimationGenerator


6

Fusion Input(Feature Level)

• Audio-Video Fusion for Automatic Speech Recognition

Speech in waveform

Mel-FrequencyCepstral Coefficients

(MFCCs)and

Lip Geometry Estimation(LGEs)

5-State Left-Right

HMMs

Recognized text

39 mfcc +

11 lge

Why don’t you come home alone uncle Oliver?

Face detection/ tracking

Mouth detection/ tracking


7

Emotion Analysis:Emotion Extraction from Text (1)

VerbNet

AffectiveLexicon

Database

XML Ontology

text

UTTERANCE

CLAUSE

AGENT PREDICATE THEME ADVERBIAL

you treated me

wrong

EMOTION-OBJECT

CONJUNCTION

CONTRAST

But,

CLAUSE +

EmotionWords

Extraction

EmotionExtraction

fromSentence

Knowledgebase

Mapping toEmotion Type &Activation level

annotated text

41 emotiontypes & 3

activation levels

If (agent==pronoun)&&(theme==“myself”) and (emotion-object==adverbial)Then intention=“emotionally-directed”

If (intention==”emotionally-directed&& (pleasantness==”unpleasant”)Then emotion-type is “anger”

...

EmotionThermometers

<UTTERANCE> …. <CLAUSE></emotion emotion-type=neutral> <CONJUNCTION><CONTRAST>but</CONTRAST> </CONJUNCTION> <AGENT>you</AGENT> <PREDICATE>have treated</PREDICATE> <THEME>me</THEME> <ADVERBIAL> </emotion emotion-type=”anger” degree=“average”> <EMOTION-OBJECT pleasantness=”neg” degree=”average”> wrong </EMOTION-OBJECT> </ADVERBIAL> </CLAUSE> …</UTTERANCE>

Parsing treegeneration

Sentence types:(a) emotionally active toward an object(b) emotionally directed by an object(c) emotions that provoked by an object(d) emotions that experienced towards an object(e) appraisal toward an object


8

Emotion Analysis:Emotion Extraction from Text (2)

Active

Passive

PleasantUnpleasant

Angry, apparent, attentive, comic, cordial,courteous, covetous, curious, darling, dear,eager, evident, fierce, fine, full, funny, furious,good, grabby, grasping, great, hearty, hot,jovial, merry, mordacious, nice, palpable, plain,respected, terrific, troubled, unquestionable,warm, well

1

Affable, anxious, appreciated,bright, correct, cute, itchy, right,superb, sweet, treasured,ungratified, untroubled

2

Agreeable, clear,dead, hopeful,indignant, kind,obvious, petrified

3

Comfortablefamiliar, positive,prosperous

4Amazed, astonished, astounded,calm, considered, cozy, dazed,dramatic, dumbfounded, exhausted,fatigued, fit, flabbergasted, serene,soft speechless, stupefied

5

Accepted, approved,bad, concentrated,desperate, elated,fearful, isolated, joyful,joyous, mean, terrible,tired, uncertain

6

Affected, annoyed, aroused, close, confused,contemptible, crazy, choleric, defeated,deplorable, depressed, disabled, disappointed,disgusted, dispirited, disturbed, down, excited,frustrated, gentle, gloomy, horrible, humiliated,ill, impetuous , improper, intent, irascible,irritated, lamentable, obsessed, obstinate, sad,shamed, sick, stimulated, stirred, unhappy,unreasonable, upset, worried

7

Tasty, tender8 flabbergasted

amazedastonished

tired

passivecalm

serenefrustated

defeated

disappointed

unhappy

disgustedworried

disturbed

dispiritedill

gloomy

fearful

sad

aroused

exited

abhorrentdesperate

eager

annoyedirritated

heartyangry

fiercefurious

indignantcurious

active

elatedjoyful

anxious

covetousgrabby

2D space of Affective Lexicon database based on relative distance of words meaning from WordNet and Multidimensional Scaling


9

Emotion Analysis:Emotion Extraction from Facial Information

Viola & Jones features

Facial Characteristic Points

Face detection- Viola & Jones features- SVM classifier

Facial Characteristic Point (FCP) Detection- Viola & Jones features- SVM classifier

Facial expression recognition- FCP based model- SVM classifier

Face detection - Viola & Jones features - SVM classifier

Facial expression recognition - Viola & Jones features - SVM classifier


10

Emotion Analysis:Emotion Extraction from Speech Prosody

Filtering

Segmentation - Multi-frame based analysis

Feature extraction

Emotion recognition - Boosting techniques, - SVM classifier

ROC graph for classifier selection


11

Dialog Action Manager

Dynamic Context

Static Context

WorldModel

Intention/Belief

recognition

Linguistic/Semantic

Knowledge

DiscourseKnowledge

TaskKnowledge

User Model

PlanReasoning

DiscourseInterpretation

SemanticParsing

ResponsePlanning

FusedInput data

Response

ResponseContent

Assembly

DialogControl

InformationControl

Information

Task StateUser Mental

State

UserEmotion

State

DiscourseState


12

Fission Output

World Model

OUTPUT

Fission

-USER ADAPTATION-CONTEXT/SITUATION ADAPTATION-SCHEDULING TIMING-SYNCHRONIZING MODALITIES

LanguageGeneration

EmotionGenerator

SpeechFacial

expression-MODALITY

DAM

Dialog action

-INFORMATION EXTRACTION (*-DIALOG PROVIDER

AnimationGenerator

User Model


13

Fission Output:Language Generator

• Template-based (includes dialog act and emotion as parameters)

<aiml>

<dialog-act name="DIALOG-ACT"><topic name="TOPIC">

<user-group name="USER-GROUP"><appraisal name="APPRAISSAL">

<info name="INFO-TYPE">

<category><that>TOPIC-HISTORY</that><pattern>INPUT</pattern>

<template><random>

<li><sentence_lattice> ... </sentence_lattice></li>

... //other sentences

</random></template>

</category>

... //other categories

</info>

...//other categories with less available info

</appraisal>

... // other categories with different appraisal type

</user-group>

... //other categories for different user-group

</topic>

... //different topic

</dialog-act>

... // different dialog-act

</aiml>


14

Fission Output:Emotion Generator (nPME model) - 1

Personality

Standards,Principles, Goals

Moods,Emotions

Behaviour(+ new need)

Content-basedInformatione.g. location

Interpretation of the world (external)Need(Internal)


15

Fission Output:Emotion Generator (nPME model) - 2

Maslow Pyramid of Needs

OCEAN Model of Personality: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism

Mood: Level of valence and arousal.

Goals, Preferences, Standards (Rule-based)

41 Emotion Typesand

3 ActivationLevels


16

The Point

• Symmetrical I/O modalities of user-system interaction

• Modular and general, as a starting point of any CHIM

• Possible applications:supervising, entertainment, tutor or help desk robot

• Current project development milestone:Input Sensing and Emotion Analysis

Documents

ICat as a Companion Robot