What a neural net needs to know about emotion words

R. COWIE, E. DOUGLAS-COWIE, B. APOLLONI, J. TAYLOR, A. ROMANO and W FELLENZ

School of Psychology, Queens University, Belfast, UKDepartment of Mathematics, Kings College, London, UK

r.cowie@qub.ac.uk john.g.taylor@kcl.ac.uk

Abstract: We describe an empirical approach to identifying the kind of task that an emotion recognitionsystem could usefully address. Three levels of information are elicited a basic emotion vocabulary, a basicrepresentation in evaluation-activation space of the meaning of each word, and a richer schemarepresentation. The results confirm that an approach of this kind is feasible.

Key-Words: - emotion recognition, semantics, neural network CSCC'99 Proceedings, Pages:5311-5316

1 IntroductionThis paper describes work being done as part of thePHYSTA project. PHYSTA is concerned with anincreasingly high profile problem in IT, which is todevelop artificial systems that are capable ofdetecting signs emotion in a human user, and ofreacting accordingly. PHYSTA will use hybridtechnology, involving neural net and symbolictechniques. One of the challenges facing the project is toidentify the kind of knowledge about emotion that isrelevant, and appropriate techniques for expressingit. The obvious approach is to select a few well-known emotion terms and to construct classificationsystems which try to assign these terms correctly torecords of human performance. However, that isunsatisfactory for a multitude of reasons.

1. The selection of terms defines the problem to besolved, and that is not something that should beleft to the experimenter's casual intuitions.

2. The selection criteria should take account of thestates that the system is likely to enounter - notjust of examples which are salient, and perhapstheoretically interesting, but rare.

3. Emotion terms are discrete, but emotional statesform a continuum. Discrete terms need to beembedded in continuous representations that atleast permit interpolation, and that ideally allowfor the kind of shaded judgement that peoplecan achieve (e.g. by qualifying emotion terms).

4. Real inputs often give partial information aboutan emotion - e.g. showing arousal withoutmaking it clear whether the person is happy orangry. An effective system should be able to usethat kind of information rather than being forced

to make a classification that goes far beyond theevidence.

5. Classifying emotion states is of no use in and ofitself. Use depends on associating the termswith a semantics which indicates what the useris likely to do, and what interventions might beappropriate.

6. Intuitively it seems quite likely that these issuesare linked - e.g. dealing with either incompleteinformation, or intermediate / compound states,involves assessing the underlying dimensions ofa person's state directly rather than via thecategories for which there are convenientemotion terms.

7. Last but not least, simple classification is notparticularly interesting intellectually. Using apractical IT problem to develop deeper ideasabout emotion, and emotion words, is.

Considerations like these led us to developtechniques for eliciting from human beings the kindof knowledge that neural net and hybrid systemsneed if they are to detect and respond to humanemotions in a way that is intellectually satifying andpractically useful. The core aims are first to providea rationale for selecting the emotion-related termsthat the system should be able to use, and second todefine the kind of representation that we want thesystem to produce in response either to an emotion-related word, or to a sample of emotionally colouredbehavior. The ideas that we have used are rooted in thepsychological and biological literature on emotion[1]. Our contribution has been to translate theseideas into a form that lends itself to IT applications,and particularly to training systems with a neural netcomponent. The result has three main elements

1.1 A Basic English Emotion VocabularyResearch on emotion is dogged by ad hoc selectionsof emotions to work with. There is no agreedbenchmark, in the form of a range of emotion termsthat a competent system should be able to apply.Without that, it is impossible to assess theperformance of emotion detection systems in ameaningful way. Investigators describe innumerabletests or variables that are claimed to be relevant tovarious specific distinctions with no reference to theimportance of the distinction, or the way the testwould function when other possibilities had to beconsidered. One approach to this problem has been explored bytheorists in biology and psychology since Descartes- attempting to identify a set of 'primary' emotions,the pure elements underlying the various compoundsthat tend to occur in everyday life. Despite severalcenturies of predominance, that approach has notproduced an agreed set of primaries [1]. We have developed a second approach, whichcomplements the traditional one and is moreimmediately relevant to the IT issue. It involvestrying to identify the main compounds (if such theyare) that actually occur in everyday life. We havedone that by trying to identify a relatively smallvocabulary of words that people regard as sufficientto describe most emotional states and events that arelikely to occur in everyday life. We call it a BasicEnglish Emotion Vocabulary - BEEV for short. As regards theory, our approach offers a way ofdefining the everyday 'compounds' (if that is whatthey are) that should be deriveable from a proposedset of elements - and without that, it is difficult tosee how variants of the traditional approach can beevaluated. As regards practice, the natural goal forIT is to develop a system that can use what humansagree is a basic emotion vocabulary.

1.2 A 2-D emotion spaceMany authors agree that emotions can be organisedroughly into a two-dimensional space whose axesare evaluation (i.e. how positive or negative theemotion is) and activation (i.e. the level of energy aperson experiencing the emotion is likely to display)[2]. That provides a useful basic continuum in whichto embed emotion words. We have developedtechniques that allow informants to assign co-ordinates in evaluation-activation space to bothwords and expressions of emotion (through the face,voice, or music). These techniques serve several functions. Theydefine a non-categorical targets that networks cannaturally be trained to emulate. They also provide a

very basic kind of semantics for emotion words - amachine that could reliably assess activation andevaluation levels from audio-visual images wouldhave at least some basis for making appropriateresponses. The techniques also allow people torecord aspects of their response to emotion-relateddisplays that are difficult to capture in categoricalterms. In particular, the 2-D space can be used torecord how emotion-related judgements changecontinuously over time - capturing an aspect ofhuman judgement that it would certainly be usefulfor a machine to emulate, but that is difficult torecord satisfactorily using categorical descriptions.

1.3 An emotion schemaEvaluation-activation space captures a goodproportion of distinctions between emotion-relatedterms, but there are many that it fails to capture. Forexample, fear and anger tend to be placed nearby inthe space. The important difference between theminvolves a different kind of dimension altogether.Higher order spaces are required to capture that kindof distinction. They reflect the fact that, as manyauthors have pointed out, emotion is closely linkedto the way the organism is disposed to act (e.g. fearinvolves a disposition to flee, whereas in anger thedisposition is to attack), and also to the way theorganism appraises the situation. Drawing on a range of psychological theories[2,3,4], we have constructed questions designed tocapture a wider range of distinctions in a systematicway. These express a simple but reasonably power-ful semantics for emotion terms, in a quantitativeformat that lends itself to implementation in neuralnets. The dimensions divide into three blocks. Questions in the first block deal with the broadkind of action that someone in a given emotion-related state would be likely to take - engage,withdraw, seek information. Questions in the secondask whether the emotion has an object - i.e. whethersaying that an individual is in a given emotion-related state implies that they are reacting to orthinking about a particular person or situation; and ifso, whether the relevant person or situation ispresent at the time, or located in the past, or thefuture, or in the individual's mind. Questions in thethird block ask about the broad characteristics of anysituation - present, past, future, or mental - that isdirectly relevant to the emotion. They deal first withthe individual's own perceived standing in thesituation - powerful or powerless, well-informed orlacking information, morally sound or not - and withrelevant characteristics of the situation or person human or not, powerful or not, appealing or not.

An important feature of the system is that it doesnot require a rating on every dimension. It is alwaysallows a question to be answered by saying that theword being considered does not provide informationabout that issue. The intended effect is to identifyfor each word a relatively compact range of featureson which it does carry information, and at the sametime to acknowledge that other features, which areimportant for other emotion words, are notparticularly relevant. The structure that we have outlined defines a formin which a partial but useable understanding ofemotions can be couched. The content has beenderived in the traditional psychological way, byexperiment. Subjects have been asked to give theirratings on the various dimensions. That approachensures that the information we provide is notsimply an expression of our own personal theorieson the subject. It also ensures that a net which istrained uing our data will use emotion terms in away that is broadly consistent with people who arehopefully a reasonably representative sample ofpotential users. The possibility is clearly open to usethe same elicitation techniques with other groups orin other languages if so that the system can beadapted for different users.

2 MethodWe have developed a program called BEEVer whichasks subjects to carry out the various ratingsassociated with the scheme outlined above. Thestudy has been carried out in two phases. Phase 1dealt with only the first two elements, identifying aBasic English Emotion Vocabulary (BEEV) andproviding ratings in the evaluation-activation space.It provided a basis for refining both those elementsand the choice of words from which the BEEV wasto be selected. Phase 2 presented modified versionsof those elements and the emotion schema. BEEVer uses an initial vocabulary of emotion-related words from which subjects select 16 thatthey regard as consituting an acceptable basicemotion vocabulary. For Phase 1, the initialvocabulary consisted of words that feature inpublished lists that are meant to summarise the maintypes of emotion, plus additional terms needed todescribe emotions that occur regularly in materialthat we have recorded for use as a database in thePHYSTA project. That produced an initialvocabulary of 45 terms. The initial vocabulary wasrevised for phase 2 by excluding terms that at mostone phase 1 subject included in his or her selectionof 16, and adding terms that subjects suggested

should have been present. That produced an initialvocabulary of 40 terms in stage 2. For the sake of readability, specific descriptions ofthe tasks are presented along with the results theyproduced. This kind of exercise depends on ensuring thatsubjects understand what they are being asked to do.Much of the effort in phase 1, and smaller pilotstudies for the schema, was devoted to that issue.The resulting procedure in phase 2 incorporated oralinstructions, particularly on the use of evaluation-activation space; a preliminary program using fourpractice terms and incorporating written instructionson the use of the schema; and a proviso that subjectswould be dropped if the experimenter was notsatisfied that they had understood the task. Eighteen subjects took part in each phase. Twophase 2 subjects were dropped because it was notcertain that they had understood the task.

3 Results3.1 A Basic English Emotion VocabularyThe elicitation of the basic vocabulary had twostages. The first two questions about each wordasked subjects to rate how common or rare the statewas, and how psychologically simple or complex.Those ratings were used to create a preliminarydivision into a set of 16 candidates for a basicvocabulary and a residue of less basic words.Candidates were entered on the basis of the lower ofthe two ratings, i.e. they were likely to be included ifthey were either common or psychologically simple.The second stage occurred at the end of the wholeexercise. At that stage, after subjects had rated allthe words in the initial vocabulary, they werepresented with a screen showing the 16 currentcandidates for a basic vocabulary on the left, and theresidue on the right. They were then allowed toswitch words from one side to the other until thewords on the left formed the best basic emotionvocabulary that they could construct. Figure 1 summarises the results. The words on theleft hand axis are the initial vocabulary used inphase 2. Their order is determined by the frequencywith which they were chosen as basic emotion termsin phase 1. That arrangement shows that there is agood deal of stability in the outcome despite quitelarge procedural differences between the two phases.The most frequently selected words in phase 1, from'disappointed' downwards, are also the 16 mostfrequently selected in the whole data set - with theexception of 'satisfied', which clearly ought to beincorporated into a balanced vocabulary.

It is not the aim of this paper to set out a definitiveBasic English Emotion Vocabulary - not leastbecause the data do not show a sharp cut-off.However, the results provide an empirical basis forassessing how defensible alternative lists are. Manylists in the literature, based on a priori judgements,quite clearly fare poorly by empirical criteria: theyomit terms that empirically appear important, andinclude others that very few subjects regard asparticularly useful. From an IT point of view, that isa non-trivial point. If research allows itself to beguided by that kind of list, and the intuitionsunderlying it, then it risks producing systems thatare expert in emotion judgements that are almostnever needed, and incapable of judgements that are. For practical purposes, it is useful to identify def-inite groups. It is worth distinguishing two - an innergroup of seven terms, chosen by a clear majority ofsubjects, consisting of happy, angry, sad, interested,pleased, relaxed, and worried; and an outer group often, chosen by half of the subjects or slightly more,consisting of affectionate, afraid, content, excited,bored, confident, amused, loving, disappointed, andsatisfied. Using the words in the inner group inparticular is a reasonable target for IT.

1.21.00.80.60.40.20.0

HAPPYANGRY

SADINTERESTED

AFFECTIONATEAFRAID

PLEASEDCONTENTEXCITED

RELAXEDBORED

CONFIDENTAMUSED

LOVINGWORRIED

DISAPPOINTEDANNOYED

IRRITATEDHURT

ANXIOUSSATISFIED

CALMJOYFUL

SYMPATHETICEMBARRASSED

NERVOUSRESENTFULSURPRISED

PROUDHOPEFUL

DISAPPROVINGDISGUSTED

JEALOUSPANICKY

DISAGREEABLERELIEVED

DESPAIRINGGUILTY

ASHAMEDSERENE

Fig. 1: Probability that each of 40 words will beincluded in a basic emotion vocabulary chosen bysubjects (based on a sample of 36 people).

3.2 Evaluation-activation spaceRatings were made using a representation associatedwith Plutchik, Russell and others, in which possibleemotions are arranged in a circle. Strong emotionslie at the periphery: an emotion-free state of alertneutrality lies at the centre. The vertical axis of thecircle represents activation level, the horizontal axisevaluation - positive emotions are on the right,negative on the left. Key emotion are arranged roundthe periphery to provide landmarks and help subjectsto orient themselves within the space. Subjects rateda word by clicking with a mouse at an appropriatepoint in the circle. They were allowed to revise theirinitial choice if they wanted to. Phase 1 data suggested that subjects had notunderstood the significance of distance from thecentre, and had chosen relatively peripheral pointsirrespective of emotion strength. Hence for phase 2,adjustments were made to the landmarks round theperiphery (ensuring that they all referred to extremeemotions, in line with their distance from the centre)and to the instructions (making explicit the meaningof distance from the centre). The resulting axes andlandmarks are shown in figure 2.

Fig. 2: Axes and landmark items of evaluation /activation space as presented to subjects.

Figure 3 shows mean positions for the frequentlyselected items - ratings for the inner group of sevenon the left, and for the outer group of ten on theright. The plots illustrate both the strength and theweakness of the evaluation-activation system.The strength of the representation is that it capturesa good deal of the information contained in basicemotion terms by way of a medium that is simple,

happy

-1

-.8

-.6

-.4

-.2

0

.2

.4

.6

.8

1

-1 -.8 -.6 -.4 -.2 0 .2 .4 .6 .8 1

worried

angry

sad

interested

pleased

relaxed

-1

-.8

-.6

-.4

-.2

0

.2

.4

.6

.8

1

-1 -.8 -.6 -.4 -.2 0 .2 .4 .6 .8 1

affectionate

afraid

content

confident

excited

amused

loving

disappointedsatisfied

bored

Fig.3 Mean ratings in evaluation/activation space. The horizontal axis is evaluation, the vertical axisactivation.

appealing

powerful

OTHER: human

justified

well-informed

SELF: powerful

appealing

powerful

OTHER: human

justified

well-informed

SELF: powerful

appealing

powerful

OTHER: human

justified

well-informed

SELF: powerful

appealing

powerful

OTHER: human

justified

well-informed

SELF: powerful

elsewhere

future

past

current

seek information

withdraw

engage

angry

afraid

worried

10-1 10-1

interested

excited

loving

AC

TIO

NO

BJE

CT

PR

ESEN

TPA

ST

FUTU

RE

MEN

TA

L

Fig. 4: Samples of profiles obtained from the schema element of BEEVer

meaningful, and convenient for both training andresponse. Also, people find it easy to place samplesof emotional behaviour in this framework, whichmakes it convenient for generating training samples. The weakness is that some discriminations thatmatter are not well drawn in this space. Oneexample has already been given - fear and angermore or less coincide in this space. Another isapparent on the positive side of the graph - happy,pleased, confident, amused, and affectionate are allessentially together. If they are to be discriminated,additional dimensions are needed. It is, of course,useful to have a this kind of indication that differentkinds of discrimination may be needed to manageanything beyond a minimal emotion vocabulary.

3.3 Schema representationsThe schema element is the medium that we haveused to provide more sophisticated discrimination.Figure 4 illustrates the kind of information that itprovides, taking as an illustration two clusters ofitems that are not effectively separable inevaluation/activation space. The left hand panel of the figure deals with termsthat lie relatively close together in the upper lefthand quadrant of evaluation-activation space. Theright hand panel deals with terms that lie in theupper right hand quadrant of evaluation-activationspace, and towards the upper end of it. There aregross differences between the profiles which reflectthat broad contrast, as one might expect. The toppanel shows that likely actions are biased towardswithdrawal in the first group, and engagement in thesecond. The lower panels, describing relevant situ-ations, show predominantly negative appraisals inthe first group, and predominantly positive apprai-sals in the second. More interesting, though, are thedifferences within the two clusters that evaluation-activation space is not well suited to capture. Anger can be considered as the simplest of theterms on the left. It does not indicate any particularcourse of action very strongly. It suggests reaction toa situation which is present, and whose main featureis that it contains something that is definitely notappealing. Fear is more distinctive. It is associatedwith a strong disinclination to engage, and an almostequally strong inclination to withdraw. It also carriesa distinctive appraisal of the present situation - thereis a powerful other involved. Worry carries aninclination to seek information, and an orientationtowards future events rather than current ones. Of the terms on the right, excited and interested arerelatively simple. The only distinctive feature of

excited is an orientation towards a future situationwith some - uspecified - appealing characteristic.Interest implies a disposition to seek information.Loving, in contrast, implies an object, present or inthe mind, which is human and appealing. The point of these summaries is not that they are inany way surprising. It is simply that the schemaappears to capture the obvious implications of theterms in a format that is straightforward, intuitive,and empirical. A system which registeredimplications like these could reasonably be said tohave a rough grasp of what the terms meant. That iswhy the schema provides a useful kind of input to asystem that is to learn how to use emotion terms.

4 ConclusionWe have described an empirical approach toidentifying the kind of task that an emotionrecognition system could usefully address. Theresults confirm that an approach of this kind isfeasible. We are currently engaged in extending it,in three main ways. First, we will obtain responsesfrom a much larger sample of English speakinginformants. Second, we will extend the approach toother European languages. Third, we are developingmethods of allowing subjects to rate examples ofemotional behaviour in terms of these dimensions. The approach lends itself to a particular style ofimplementation. It suggests that the domain ofemotion understanding can be represented as anetwork involving nodes of many kinds. Emotionterms are one kind of node, but only one. Alsoinvolved are highly compressed representations ofsituations, built round evaluations of the main agentsand forces; and of the actions that these situationsare likely to evoke. Various kinds of evidence maybe relevant to activating the nodes in the network,and activation in various combinations of nodes mayserve to activate high order representations, such asthe nodes associated with emotion terms.

References:[1].Review of existing techniques for humanemotion understanding Report for TMR Phystaproject Research contract FMRX CT97 0098(DG12-BDCN)[2]. Plutchik, R. The psychology and biology ofemotion. Harper Collins, New York, p.58, 1994[3] Lazarus, R.S. Emotion and adaptation. OxfordUniversity Press, New York, 1991[4] Oatley, K & Jenkins, J. UnderstandingEmotions. Blackwell, Oxford: 1996