An Adaptable Fuzzy Emotion Model for Emotion Recognition

8/3/2019 An Adaptable Fuzzy Emotion Model for Emotion Recognition

1/6

An Adaptable Fuzzy Emotion Model for Emotion Recognition

Natascha Esau Lisa Kleinjohann

C-LAB

Fuerstenallee 11

D-33102 Paderborn, Germany

e-mail: {nesau, lisa, bernd}@c-lab.de

Bernd Kleinjohann

Abstract

Existing emotion recognition applications usually dis-

tinguish between a small number of emotions. How-ever this set of so called basic emotions varies from

one application to another depending on their accord-

ing needs. In order to support such differing appli-

cation needs an adaptable emotion model based on

the fuzzy hypercube is presented. In addition to exist-

ing models it supports also the recognition of derived

emotions which are combinations of basic emotions.

We show the application of this model by a prosody

based fuzzy emotion recognition system.

Keywords: Fuzzy emotion model, fuzzy hypercube,

fuzzy emotion recognition, basic emotion.

1 Introduction

Emotions are an evident part of interactions between

human beeings. But also for interactions of humans

with computer systems emotions play a major role,

since humans can never entirely switch off their emo-

tions. During the last years interest in emotions in-

creased considerably in various domains of computer

based systems. Examples are robots or virtual agents

that show emotions or human-computer interfaces that

consider human emotions in their interaction capabil-

ities. In Japan an entire stream called KANSEI infor-

mation processing [10] deals with subjective human

feelings when interacting with IT systems. These few

examples already reveal two major tasks of emotion

processing in IT systems, the recognition of human

emotions and the (re)production of artificial emotions.

Whereas robots or virtual agents often show emotions

themselves, for many other IT systems the recognition

of human emotions and appropriate reactions suffice

to improve the systems performance or acceptance.

Imagine for instance a user who is angry, because the

IT system does not behave in the expected way or she

tried several times to accomplish a task without suc-cess. In such a situation it would be very helpful and

increase the system acceptance, if an IT system could

recognize this emotion and react accordingly. Another

example is speech recognition. According to investi-

gations at the MIT a doubling of the word error rate to

about 32% was observed when people talk in an angry

way [4]. In such a case an appropriate system reaction

would be to redirect the user to a human operator or

to give hints how she could decrease the error rate.

Depending on the intended application domain dif-

ferent emotions are relevant for emotion recogni-tion. According to the observation described above,

the speech recognition system Mercury distinguishes

only two classes of emotions: frustration and neu-

tral. For other applications like personal robots or en-

tertainment robots certainly the recognition of some

more emotions like happiness or sadness would be

interesting to react with according robot behavior.

Since also psychologists have not yet agreed upon a

set of basic emotions (see Section 2) it is not likely

to identify a set of emotions, that is appropriate for

all computer based emotion recognition (CBER) sys-tems. Therefore, in this paper we propose an emotion

model for emotion recognition that is easily adaptable

to the selected set of basic emotions for the CBER

problem (see Section 3). However humans do not only

feel some basic emotions in their pure form but also

some more complex or derived emotions [16]. An

example is for instance curiosity which is according

to experiments by Plutchik a combination of accep-

tance and surprise. Accounting for this observation

our emotion model does not only support basic emo-

EUSFLAT - LFA 2005

73


2/6

tions but also supports the representation of such de-

rived emotions or blends.

Furthermore different intensities or degrees of emo-

tions can be observed [16]. For modelling of different

degrees of membership to different classes in a classi-

fication system in many application domains, among

them also emotion recognition, fuzzy logic is a very

useful approach. Therefore we developed our adapt-

able emotion model using the fuzzy hypercube as ba-

sis (see Section 3). We show the applicability of our

approach by a system for emotion recognition from

prosody of natural speech in Section 4. Afterwards

we compare our approach with related work and give

a short conclusion.

2 Emotion Models in Psychology

Psychologists have tried to explain the nature of hu-

man emotions for decades or even centuries. Never-

theless no unique established emotion model exists.

However emotion is now usually seen as a dynamic

process that involves several modalities like motoric

expression, physiological arousal and subjective feel-

ing [13, 9]. For computer based emotion recogni-

tion (CBER), however models that help with the clas-

sification of emotions are more important. Among

these two major types of emotion models can be dis-tinguished (also mixtures of these types are found):

models that rely on basic emotions and emotion mod-

els that classify emotions according to different di-

mensions like valence, potency, arousal, intensity etc.

The first one has a major advantage for CBER, since it

considerably decreases recognition complexity due to

a small number of basic emotions to which the CBER

can be restricted. A well known earlier model of basic

emotions is the work of Plutchik [16]. He uses basic

emotions as a kind of building block for derived emo-

tions, so called secondary emotions. Plutchik evendistinguishes ternary emotions that are combinations

of secondary derived emotions. His model like many

others also describes the concept of emotion inten-

sity, that represents the strength by which an emo-

tion is felt. Although emotion models exist for sev-

eral decades, even now there is no general agreement

among psychologists how many basic emotions exist

and what they are. This is shown in table 1 which is

an excerpt from [15].

Due to the variety of basic emotions described in liter-

Table 1: Basic emotions distinguished by psycholo-

gists

Psychologist Basic Emotions

Plutchik Acceptance, anger, anticipaton, dis-

gust, joy, fear, sadness, surpriseEkman,

Friesen,

Ellsworth

Anger, disgust, fear, joy, sadness,

surprise

Frijda Desire, happiness, interest, surprise,

wonder, sorrow

Izard Anger, contempt, disgust, distress,

fear, guilt, interest, joy, shame, sur-

prise

James Fear, grief, love, rage

Mowrer Pain, pleasureOatley and

Johnson-

Laird

Anger, disgust, anxiety, happiness,

sadness

ature it seems reasonable to develop an emotion model

for emotion recognition that is easily adaptable to the

selected set of basic emotions for the CBER problem.

3 Fuzzy Emotion Model

As already stated, according to psychologists like

Plutchik humans do not only feel a single basic emo-

tion but have more complex emotional states, where

more than one basic emotion is involved with varying

strength or intensity. Therefore we propose a fuzzy

classification of emotional states using fuzzy hyper-

cubes [12]. Furthermore we assume that the inten-

sity of an emotion can be mapped to the interval [0, 1].First we define a fuzzy set corresponding to an emo-

tional state and then show how it is represented in a

fuzzy emotion hypercube.

Fuzzy set for emotional state. Let BE be a finite

base set ofn basic emotions e1, e2, . . . en and

{FEj : BE [0, 1], j = 1, 2, . . .} an infinite set offuzzy membership functions. Then each

FEj := {(ei, FEj(ei) | ei BE}, j = 1, 2, . . . de-fines a fuzzy set corresponding to one emotional state

Ej .

Fuzzy emotion hypercube. IfBE, FEj and FEjare defined as described above, we shall use the mem-

EUSFLAT - LFA 2005

74


3/6

bership vector

(FEj(e1), FEj (e2), . . . , FEj(en)) =: (FEj(ei))to denote a point in an n-dimensional hypercube.

Each axis of the hypercube corresponds to one basic

emotion ei. Thus a membership vector (FEj(ei))corresponds to one emotional state Ej and can be in-

terpreted psychologically as vector of emotion inten-

sities (Iei) := (Ie1 , Ie2 , . . . , I en).

The number of distinguished emotions depends on

the psychological theory or in the case of computer

based emotion recognition on the intended applica-

tion. If for instance the three basic emotions happi-

ness h, anger a and surprise s shall be distinguished,

a three dimensional unit cube as depicted in Figure 1

is needed for modelling emotional states.

(1,0,0)

(0,1,0)

(0,0,1)

Happiness

Anger

Surprise

E1

E2

(0,0,0)

Figure 1: Fuzzy unit cube for three emotions happi-

ness, suprise and anger

The corners in the unit cube describe dual mem-berships (0 or 1) for all emotions, vertices desribe

dual memberships for two emotions and the third one

varies from 0 to 1. For example, the point E1 =(1.0, 0.2, 0.3) corresponding to the fuzzy setFE1 = {(h, 1.0), (a, 0.2), (s, 0.3)} represents ahappy emotional state. The point E2 = (0.2, 1.0, 0.9)corresponding to

FE2 = {(h,0.2), (a, 1.0), (s, 0.9)} certainly repre-sents an emotional state for a derived emotion from

anger and suprise. The point (0, 0, 0) represents the

entirely neutral state where no emotion is present.

Anger

Surprise

Happin.

Surprise+

Happin.

Neutral

Surprise+

Anger

Anger+

Happin.

Surprise+ Anger+ Happin.

Figure 2: Subdivisions of unit cube representing basic

and derived emotions

Figure 2 shows how the unit cube could be further di-

vided in order to represent basic emotions and their

mixtures. In the subcubes denoted by a single emo-

tion the membership function of this emotion takes

values in the interval [0.5, 1.0] whereas the member-

ship values for the other emotions respectively theirintensities are below 0.5. Therfore it is reasonableto associate the subcube with this basic emotion. In

the subcubes denoted with a sum of emotions (e.g.

Surprise + Happiness) memberships of these emo-

tions are in the interval [0.5, 1.0] whereas the mem-bership of the third emotion is below 0.5. Hencea derived emotion from these two basic emotions

(e.g. surprise and happiness) is assumed. The sub-

cube where the membership values of all basic emo-

tions are between 0.5 and 1.0 is denoted by the sum

Surprise + Anger + Happiness.

If a general n-dimensional emotion hypercube is re-

garded certainly not all combinations of up to n emo-

tions make sense. However, whether a combination

is reasonable or not is certainly a psychological ques-

tion. If combinations that do not make sense are rec-

ognized by a CBER this could for instance indicate an

error.

4 Application

This section deals with the application of our adapt-

able emotion model for the fuzzy rule based emotion

recognition system PROSBER [2].

4.1 Overview of PROSBER

PROSBER recognizes emotions from the prosody of

natural speech. It takes single sentences as input and

classifies them into the emotion categories happiness,

sadness, anger and fear. Furthermore a neutral emo-

tional state is distinguished. PROSBER automaticallygenerates the fuzzy models for emotion recognition.

Accordingly two working modes are distinguished,

training and recognition, as depicted in Figure 3.

During the training the training samples with well-

known emotion values are used to create the fuzzy

models for the individual emotions. For that purpose

sequences of acoustic parameters like fundmental fre-

quency or jitter are extracted. PROSBER extracts

about twenty parameters that have shown their rele-

vance for emotion recognition in psychological stud-

EUSFLAT - LFA 2005

75


4/6

Featurecalculation

Parameterextraction

Preprocessing Membershipfunctionsgeneration

Featureselection

Fuzzyruleconstruc-tion

Fuzzyclassification

Featurecalculation

Parameterextraction

Preprocessing

Training

Recognition

peech

ignal

peechignal

Frames

Frames

Parametersequences

Featurevectors

Emotions of

training samples

Membership

functions

Fuzzy

rules

Emotion

Parameter

sequences

Featurevectors

Fuzzy model generation

Figure 3: Architecture of PROSBER

ies or in other speech based emotion recognition sys-

tems. The sequences of these acoustic parameters are

summarized by statistical analysis steps performed by

the feature calculation. The fuzzy model generation isbased on a fuzzy grid approach [11]. It performs the

following three steps on the training database. First

the membership functions for every feature are gener-

ated. Afterwards for each emotion up to six most sig-

nificant features are selected and then the fuzzy rule

system for each emotion is generated. These fuzzy

models are used in the emotion recognition process to

classify unknown audio data. A detailed description

of PROSBER can be found in [2].

4.2 Fuzzy Emotion Recognition in PROSBER

In order to describe the application of our emotion

model we shall now have a closer look at the fuzzy

classification. In principle it is structured as depicted

in Figure 4.

Emotion nintensity

Fuzzification DefuzzificationFuzzy Inference

Rule BaseEmotion n

Emotion n

Emotion 1

..

Emotion 1intensity

Featurevectors

Max

..Emotion

Figure 4: Principle structure of fuzzy classification

For each basic emotion ei, i = 1,...,n, a sepa-rate rule set is generated by an adapted fuzzy grid

method. Each rule takes the fuzzified features fj , j =1,...,K,K 6, as input and produces a fuzzy emo-tion value Iei as output. We represent each fea-

ture fj and emotion intensity Iei by five triangular

membership functions verylow, low,medium,high

and veryhigh as schematically depicted in Figure 5.

However, the actual start and end coordinates as well

as the maximum coordinates are generated automat-

ically during the training phase. This representationis simple enough to support real time emotion recog-

nition, yet allows to distinguish degrees to which a

feature or emotion is present in the current input sen-

tence. Furthermore, it is in line with psychologists

approaches who often use two up to ten levels for

characterizing psychological phenomena like emotion

intensities.

very

low low med. highvery

high

0

.5

0.25 0.5 0.75 1.0

.2

.8

1.0

Figure 5: Membership functions for features and emo-

tions

The rule set for the emotion ei is generated by a fuzzy

grid approach [11]. Since this approach uses only

the AND connector it generates 5K+1 rules of thefollowing form:

IFf1 IS verylow AND ... AND fK IS verylow THEN

Iei IS veryhigh

IFf1 IS verylow AND ... AND fK IS low THENIeiIS veryhigh

IFf1 IS verylow AND ... AND fK IS medium THEN

Iei IS medium

...

The number of rules could be reduced, if the OR con-

nector or rule pruning could be used. However, both

features are not yet supported by the fuzzy library we

use. For defuzzification of emotion values we use

the center of gravity (COG) method. By projecting

the COG to the x-axis we calculate the corresponding

emotion intensity. Hence a four dimensional vector

(Ih, Is, Ia, If) = (E(h), E(s), E(a), E(f))

containing the intensities of the four emotions hap-

piness h, sadness s, anger a, and fear f is generated.

This vector represents the membership values for each

EUSFLAT - LFA 2005

76


5/6

emotion and hence determines a point in the four di-

mensional emotion hypercube.

Presently PROSBER recognizes a single basic emo-

tion. In order to select this emotion we determine the

emotion erec {h,s,a,f} with maximum intensityIerec = max{Ih, Is, Ia, If}. If the maximum cannotbe determined unambiguously, since two or more in-

tensity values are maximal, that emotion is selected

which was recognized for the previous sentence. The

neutral emotional state is identified by a hypercube

part near to the origin as depicted in Figure 2. We

plan to extend PROSBER for recognition of combined

emotions as described above.

5 Related Work

Up to now a variety of emotion models have been

described in literature. They are mainly dedicated

to computer based emotion (re)production or sim-

ulation in different application domains. A broad

application domain are virtual agents, that show

(pseudo)emotional behavior in their communication

with humans [3, 8, 5, 7]. They rely on a dimen-

sional model of emotions based on the event-appraisal

emotion model of Ortony et al. [14]. They usu-

ally distinguish the dimensions pleasure, arousal and

dominance and try to maintain their dynamics over

time. The PETEEI system [7] for simulation of a

pets evolving emotional intelligence similarly to our

system uses fuzzy sets for emotion representation.

However this system is different from our approach

since it associates certain types of events with posi-

tive or negative feelings in order to react with accord-

ing emotions whereas our approach is dedicated to

emotion recognition from certain features (of speech,

facial expression etc.). Emotion recognition as in-

vestigated in our approach is to some extent covered

by Kismet [6]. However Kismet recognizes inten-

tions rather than emotions. The emotional system de-

veloped for AIBO and SDR [1] like our model uses

basic emotions. Since it is intended for production

of emotional behavior it uses the dimensions plea-

sure and arousal as mentioned above. But the dom-

inance dimension is substituted by a confidence di-

mension representing the certainty of recognized ex-

ternal stimuli. The models described above only deal

with single emotions and do not allow to represent

combinations or blends of emotions like our approach.

The Cathexis model [17] supports this feature and

is also adaptable to different sets of basic emotions

or emotion families as they are called there by sup-

porting the coexistence of several active so called

emotion proto-specialists representing different emo-

tion families. However Cathexis is also dedicated tothe (re)production of emotional behavior in synthetic

agents whereas our adaptable emotion model is in-

tended for emotion recognition.

6 Conclusion and Outlook

This paper presented an adaptable emotion model for

emotion recognition. It uses the concept of an n-

dimensional fuzzy hypercube to represent emotional

states made up of n basic emotions. In contrast to

other approaches this allows not only the represen-

tation and recognition of a fixed set of basic emo-

tions but also supports the handling of derived emo-

tions. We showed the application of this model us-

ing the fuzzy prosody based emotion recognition sys-

tem PROSBER. As a first step we proposed a divi-

sion of the unit hypercube in equally sized subcubes

to distinguish basic emotions and their combinations

or blends. An interesting point for further investiga-

tion is whether this subdivision corresponds to human

recognition. This could for instance be done using a

learning approach that automatically finds such sub-

divisions and compares them with human interpreta-

tions of corresponding emotional states.

References

[1] R. C. Arkin, M. Fujita, T. Takagi, and R. Hasegawa.

An ethological and emotional basis for human-robot

interaction. In Robotics and Autonomous Systems,

vol. 42, no. 3, pages 191201. Elsevier Science, 2003.

[2] A. Austermann, N. Esau, L. Kleinjohann, andB. Kleinjohann. Prosody based emotion recognition

for mexi. In Proceedings of IEEE/RSJ Int. Confer-

ence on Intelligent Robots and Systems (IROS 2005),

Edmonton, Alberta, Canada, August 2005.

[3] J. Bates. The role of emotion in believable agents.

Communication of the ACM, 37(7), pages 122125,

1992.

[4] A. Boozer. Characterization of Emotional Speech

in Human-Computer-Dialogues. M.Sc. Thesis. MIT

Press, 2003.

EUSFLAT - LFA 2005

77


6/6

[5] C. Breazeal. Affective interaction between humans

and robots. In Proc. of ECAL 01, pages 582591,

Prague, 2001.

[6] C. Breazeal and L. Aryananda. Recognition of affec-

tive communicative intent in robot-directed speech. In

Autonomous Robots 12, pages 83104. Kluwer Aca-demic Publishers, 2002.

[7] M. S. El-Nasr, J. Yen, and T. Ioerger. Flame - a

fuzzy logic adaptive model of emotions. In Automous

Agents and Multi-agent Systems 3, pages 219257,

2000.

[8] C. Elliot. The Affective Reasoner: A Process model of

emotions in a multi-agent system. Ph.D. Thesis. Insti-

tute for the Learning Sciences, Evanston, IL: North-

western University, 1992.

[9] N. H. Frijda. Neural Networks and Fuzzy Systems;

A Dynamical Systems Approach to Machine Intelli-gence. Cambridge University Press, 1986.

[10] S. Hashimoto. Kansei as the third target of informa-

tion processing and related topics. In Proceedings

of Intl. Workshop on Kansei Technology of Emotion,

pages 101104, 1997.

[11] H. Ishibuchi and T. Nakashima. A study on generat-

ing fuzzy classification rules using histogramms. In

Knowledge based Intelligent electronic Systems, Bd.

1. Prentice Hall, 1998.

[12] B. Kosko. Neural Networks and Fuzzy Systems; A Dy-

namical Systems Approach to Machine Intelligence.Prentice Hall, Englewood Cliffs, NJ, 1992.

[13] R. S. Lazarus. Emotion and Adaptation. Oxford Uni-

versity Press, 1991.

[14] A. Ortony, G. Clore, and A. Collins. The Cognitive

Structure of Emotions. Cambridge University Press,

1988.

[15] A. Ortony and W. Turner. Whats basic about basic

emotions? Psychological Review, pages 315331,

1990.

[16] R. Plutchik. The Emotions. University Press of Amer-

ica, Inc., revised edition, 1990.

[17] J. Velasquez. Modeling emotions and other motiva-

tions in synthetic agents. In Proceedings of the AAAI

Conference, pages 1015. Providence, RI, 1997.

EUSFLAT - LFA 2005

78

Documents

An Adaptable Fuzzy Emotion Model for Emotion Recognition