Beräkningsmodell för moral och emotioner i EmoBN

Department of Science and Technology Institutionen för teknik och naturvetenskap Linköping University Linköpings universitet

gnipökrroN 47 106 nedewS ,gnipökrroN 47 106-ES

LiU-ITN-TEK-A-14/028-SE

Beräkningsmodell för moral ochemotioner i EmoBN

Henry Fröcklin

2014-08-28


Beräkningsmodell för moral ochemotioner i EmoBNExamensarbete utfört i Medieteknik

vid Tekniska högskolan vidLinköpings universitet

Henry Fröcklin

Handledare Pierangelo DellAcquaExaminator Pierangelo Dell'Acqua

Norrköping 2014-08-28

Upphovsrätt

Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare –under en längre tid från publiceringsdatum under förutsättning att inga extra-ordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner,skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat förickekommersiell forskning och för undervisning. Överföring av upphovsrättenvid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning avdokumentet kräver upphovsmannens medgivande. För att garantera äktheten,säkerheten och tillgängligheten finns det lösningar av teknisk och administrativart.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman iden omfattning som god sed kräver vid användning av dokumentet på ovanbeskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådanform eller i sådant sammanhang som är kränkande för upphovsmannens litteräraeller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press seförlagets hemsida http://www.ep.liu.se/

Copyright

The publishers will keep this document online on the Internet - or its possiblereplacement - for a considerable time from the date of publication barringexceptional circumstances.

The online availability of the document implies a permanent permission foranyone to read, to download, to print out single copies for your own use and touse it unchanged for any non-commercial research and educational purpose.Subsequent transfers of copyright cannot revoke this permission. All other usesof the document are conditional on the consent of the copyright owner. Thepublisher has taken technical and administrative measures to assure authenticity,security and accessibility.

According to intellectual property law the author has the right to bementioned when his/her work is accessed as described above and to be protectedagainst infringement.

For additional information about the Linköping University Electronic Pressand its procedures for publication and for assurance of document integrity,please refer to its WWW home page: http://www.ep.liu.se/

© Henry Fröcklin

Institutionen för teknik ochnaturvetenskap

Department of Science and Technology

Examensarbete

Design of a computational model for morality andemotions in EmoBN

Examensarbete utfört i Artificial General Intelligencevid Tekniska högskolan vid Linköpings universitet

av

Henry Fröcklin


Norrköping 2014

Department of Science and Technology Linköpings tekniska högskolaLinköpings universitet Linköpings universitet, Campus NorrköpingSE-601 74 Norrköping, Sweden 601 74 Norrköping

Design of a computational model for morality andemotions in EmoBN

Examensarbete utfört i Artificial General Intelligencevid Tekniska högskolan vid Linköpings universitet

av

Henry Fröcklin


Handledare: Pierangelo Dell’AcquaITN, Linköpings universitet

Linköpings universitet

Examinator: Pierangelo Dell’AcquaITN, Linköpings universitet

Norrköping, 28 augusti 2014

Avdelning, InstitutionDivision, Department

MITDepartment of Science and TechnologySE-601 74 Norrköping

DatumDate

2014-08-28

SpråkLanguage

� Svenska/Swedish

� Engelska/English

�

⊠

RapporttypReport category

� Licentiatavhandling

� Examensarbete

� C-uppsats

� D-uppsats

� Övrig rapport

�

⊠

URL för elektronisk version

http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-XXXXX

ISBN

—

ISRN


Serietitel och serienummerTitle of series, numbering

ISSN

—

TitelTitle

Design av en beräkningsmodell för moral och emotioner i EmoBN

Design of a computational model for morality and emotions in EmoBN

FörfattareAuthor

Henry Fröcklin

SammanfattningAbstract

Denna rapport presenterar en metod för att designa moraliskt beteende i ett scenario medeen. een är en iteration av emobn som är baserat på bn, ett system för aktions val med dy-namisk aktivering mellan moduler, mål orienterat och kapabelt till förutsägelse och planer-ing. Designen är baserad på nuvarande forskning från en prominent psykolog som Haidtoch använder Mikhails umg ramverk för kausal och medvetenhets validering. Även Rose-mans värderings modell och Haidts mft används för att fastställa moraliska emotioner i enmoralisk kontext. Designen är testad mot empiriska resultat av ett filosofiskt experimentkänt som Vagns problemet, ett välkänt moraliskt dilemma.

NyckelordKeywords artificial, general, intelligence, moral, decision making, appraisal, emotions, expected,

moral, bystander, footbridge, trolley problem, MFT, UMG, behaviour network, EmoBN, EEN

Sammanfattning

Denna rapport presenterar en metod för att designa moraliskt beteende i ett sce-nario med een. een är en iteration av emobn som är baserat på bn, ett systemför aktions val med dynamisk aktivering mellan moduler, mål orienterat och ka-pabelt till förutsägelse och planering. Designen är baserad på nuvarande forsk-ning från en prominent psykolog som Haidt och använder Mikhails umg ram-verk för kausal och medvetenhets validering. Även Rosemans värderings modelloch Haidts mft används för att fastställa moraliska emotioner i en moralisk kon-text. Designen är testad mot empiriska resultat av ett filosofiskt experiment käntsom Vagns problemet, ett välkänt moraliskt dilemma.

iii

Abstract

This master thesis presents an approach on how to design moral behaviour ina scenario with een. een is an iteration of emobn which is based on bn, anaction selection system with activation dynamics among modules, goal orientedand capable of prediction and planing. The design is based on current researchfrom prominent psychologist like Haidt and uses Mikhial’s umg framework forcausal and intentional validation. Also Roseman’s appraisal model and Haidt’smft is used for determining moral emotions in a moral context. The design istested against empirical results from philosophical experiment know as the trol-ley problem, a well known moral dilemma.

v

Acknowledgments

Foremost, I would like to thank to my advisor Pierangelo Dell’Acqua for the con-tinuous support during the thesis, for his patience, motivation, enthusiasm, andimmense knowledge.

Besides my advisor, I would like to thank my opponent, for his insightful com-ments and interesting questions.

Last but not the least, I would like to thankmy family: my parents Ingela Fröcklinand Victor Mendéz, for giving birth to me at the first place and supporting mespiritually throughout my life, to my Anna for undestanding and keeping mesane.

Norrköping, Maj 2014Henry Fröcklin

vii

Contents

List of Figures xi

List of Tables xii

Notation xiii

List of Definitions xv

1 Introduction 11.1 Problem description . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Report outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Background 32.1 Emotion theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Morality theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.3 Moral emotions theory . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3.1 Other-condemning family . . . . . . . . . . . . . . . . . . . 62.3.2 The Self-Conscious family . . . . . . . . . . . . . . . . . . . 72.3.3 The Other-Suffering Family . . . . . . . . . . . . . . . . . . 82.3.4 The Other-Praising Family . . . . . . . . . . . . . . . . . . . 8

2.4 Moral emotional decision making . . . . . . . . . . . . . . . . . . . 92.4.1 Ethics and morality . . . . . . . . . . . . . . . . . . . . . . . 92.4.2 Decision theory . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 emobn architecture 113.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Components of emobn . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2.1 States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.2.2 Behaviour Modules . . . . . . . . . . . . . . . . . . . . . . . 123.2.3 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2.4 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2.5 Emotions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2.6 Emotional Links . . . . . . . . . . . . . . . . . . . . . . . . . 13

ix

x Contents

3.3 Activation Spreading . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3.1 Network Parameters . . . . . . . . . . . . . . . . . . . . . . 143.3.2 Activation Spreading . . . . . . . . . . . . . . . . . . . . . . 15

3.4 Action Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4 Models 194.1 Background theory summary . . . . . . . . . . . . . . . . . . . . . 194.2 Moral Foundation Theory . . . . . . . . . . . . . . . . . . . . . . . 21

4.2.1 The six moral foundations . . . . . . . . . . . . . . . . . . . 224.3 The trolley problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.3.1 Bystander scenario . . . . . . . . . . . . . . . . . . . . . . . 254.3.2 Footbridge scenario . . . . . . . . . . . . . . . . . . . . . . . 25

4.4 Universal Moral Grammar . . . . . . . . . . . . . . . . . . . . . . . 26

5 The moral design 295.1 een, extension of the emobn . . . . . . . . . . . . . . . . . . . . . . 29

5.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 295.1.2 Components of een . . . . . . . . . . . . . . . . . . . . . . . 295.1.3 Graphical representation of . . . . . . . . . . . . . . . . . . 305.1.4 Transformation Γ . . . . . . . . . . . . . . . . . . . . . . . . 31

5.2 Implementing the trolley problem with een . . . . . . . . . . . . . 335.2.1 Expected moral emotions process . . . . . . . . . . . . . . . 335.2.2 Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

6 Testing 416.1 Aim of the tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416.2 Bystander (Test 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416.3 Footbridge (Test 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

7 Related work 437.1 Empathy in social agents . . . . . . . . . . . . . . . . . . . . . . . . 437.2 Modelling theory of mind . . . . . . . . . . . . . . . . . . . . . . . 447.3 lida . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

8 Discussion 478.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Bibliography 51

A Code 57A.1 Bystander code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57A.2 Footbridge code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

List of Figures

2.1 Lowenstein model of how emotions affect behaviour [20]. . . . . . 10

4.1 A diagram of the theories from the background chapter and howthe different concepts are linked according to our understanding. . 20

4.2 A diagram of the bystander case. . . . . . . . . . . . . . . . . . . . 254.3 A diagram of the footbridge case. . . . . . . . . . . . . . . . . . . . 254.4 umg intentional structure of the bystander dilemma [22]. . . . . . 274.5 umg intentional structure of the footbridge dilemma [22]. . . . . . 27

5.1 Graphical representation of a een component. . . . . . . . . . . . . 315.2 Roseman’s model of common emotions and how they are triggered

[30]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.3 A diagram showing activation flow of the bystander case. . . . . . 375.4 A diagram showing activation flow of the footbridge case. . . . . . 38

xi

List of Tables

3.1 Affective impact on the network parameters . . . . . . . . . . . . . 15

4.1 The five moral foundations [16] . . . . . . . . . . . . . . . . . . . . 23

5.1 Emotions from actions for the bystander case . . . . . . . . . . . . 355.2 Emotions from states for the bystander case. . . . . . . . . . . . . . 355.3 Emotions from actions for the footbridge case. . . . . . . . . . . . . 365.4 Emotions from states for the footbridge case. . . . . . . . . . . . . 36

6.1 Result of bystander test. . . . . . . . . . . . . . . . . . . . . . . . . 416.2 Result of footbridge test. . . . . . . . . . . . . . . . . . . . . . . . . 42

xii

Notation

Abbrevations

Abbrevation Meaning

agi Artificial General Intelligencetem Transient Episodic Memorymft Moral Foundation Theorylida Learning Intelligent Distribution Agentbn Behaviour Networkebn Extended Behaviour Networkemobn Emotional Behaviour Networkeen Expected Emotion Networkug Universal Grammarumg Universal Moral Grammarpomdp Partially Observable Markov Decision Processfmri Functional Magnetic Resonance Imagingnpc Non-Player Characteraicg Artificial Intelligence and Computer Graphicsbbc British Broadcasting Corporationama Actual Mechanical Advantage

xiii

List of Definitions

Morality from the Latin moralitas "manner, character, proper behavior", is thedifferentiation among intentions, decisions, and actions between those thatare good (or right) and bad (or wrong). A moral code is a system of morality(for example, according to a particular philosophy, religion, culture, etc.)and a moral is any one practice or teaching within a moral code. The ad-jective moral is synonymous with "good" or "right." Immorality is the activeopposition to morality (i.e. good or right), while amorality is variously de-fined as an unawareness of, indifference toward, or disbelief in any set ofmoral standards or principles.

Emotion is a complex psychophysiological experience of an individual’s state ofmind as interacting with biochemical (internal) and environmental (exter-nal) influences. In humans, emotion fundamentally involves "physiologicalarousal, expressive behaviors, and conscious experience." Emotion is asso-ciated with mood, temperament, personality, disposition, and motivation.Motivations direct and energize behavior, while emotions provide the affec-tive component to motivation, positive or negative.

Behaviour or behavior (see American and British spelling differences), refers tothe actions and mannerisms made by organisms, systems, or artificial enti-ties in conjunction with its environment, which includes the other systemsor organisms around as well as the physical environment. It is the responseof the system or organism to various stimuli or inputs, whether internal orexternal, conscious or subconscious, overt or covert, and voluntary or invol-untary.

xv

xvi 0 List of Definitions

Social Action in sociology, social action refers to an act which takes into accountthe actions and reactions of individuals (or ’agents’). According to MaxWeber, "an Action is ’social’ if the acting individual takes account of the be-havior of others and is thereby oriented in its course" (Secher 1962).

Ethic is sometimes known as philosophical ethics, ethical theory, moral theory,and moral philosophy, is a branch of philosophy that involves systematiz-ing, defending and recommending concepts of right and wrong conduct,often addressing disputes of moral diversity.

Judgment, a value judgment is a judgment of the rightness or wrongness ofsomething or someone, or of the usefulness of something or someone, basedon a comparison or other relativity. As a generalization, a value judgmentcan refer to a judgment based upon a particular set of values or on a par-ticular value system. A related meaning of value judgment is an expedientevaluation based upon limited information at hand, an evaluation under-taken because a decision must be made on short notice.

Empathy is the capacity to recognize emotions that are being experienced by an-other sentient or fictional being. One may need to have a certain amount ofempathy before being able to experience accurate sympathy or compassion.

Reciprocity in social and political philosophy, is the concept of reciprocity asin-kind positive or negative responses for the actions of others.

Faculties is an ability, skill, or power, often plural.

1Introduction

The research done in this thesis is primarily based on psychology articles fromprominent researchers such as Jonathan Haidt, Jesse Graham, Craig Joseph andSelin Kesebir. Another point of view comes from a professor in legal theory, JohnMikhail. Also to get a better understanding of the inner workings of the humanpsyche the field of biology and neuroscience has been taken into considerationduring the preliminary research. We have chosen the design primarily based onthe work from one particular psychologist, Jonathan Haidts, mainly because itwould otherwise be contradictory but also to remain consistent.

With the base of a common dilemma we have designed a model, ran tests andcompared our results with a psychology experiment. We used the well knowntrolley problem1 as scenario for validation. This scenario significantly reducedthe number of relevant emotions, since our model is not general, each emotionand what it does is hard coded into the system for this specific scenario, to havefull control and to narrow the scope as much as possible.

1See section 4.3 for explanation.

1

2 1 Introduction

1.1 Problem description

There is a great demand for more realistic models of human behaviour in be-haviour simulation and for npc. To further increase the believability of artificialagents, the decision making needs to incorporate morality as well. Recently, inthe aicg lab2 at LiU, an architecture for representing emotional influence in de-cision making, based on the research [20] of Lowenstein, has been implemented.The model is based on an extension of behaviour networks [18, 19]. At the cur-rent state of development, the system lacks morality. Thus, what it is needed, isto design a computational model of morality that will allow us to analyse spe-cific scenarios and dilemmas studied in research ethics and decision making inpsychology. The existing system emobn3, has been used for a public displayinstallation and as a interactive application for emotional behaviour in games.

Even the simplest model of artificial moral behaviour similar to humans is verycomplex, even so this is an attempt to evaluate if emobn can be used, with orwithout modifications. The aim is to design and investigate the possible imple-mentation of a computational model of morality with emobn architecture withfocus on aspects of behaviour.

1.2 Report outline

We start with introducing the background theory of emotions and morality inchapter 2. Then we move on to briefly explaining the emobn architecture inchapter 3, this chapter is very much based on the PhD thesis of Anja Johannsonand the work of Pierangelo Dell’Acqua. We proceed with chapter 4, where we de-scribe the relevant models from the background theory used in the design of theimplementation. Next in chapter 5 we go through the moral design. Chapter 6contains testing of the moral model. After that we present three related researchprojects and existing implementations with similar aims to this thesis in chapter7. The 8th and last chapter discusses the topic and future work.

2http://webstaff.itn.liu.se/ piede/aicglab/.3See chapter 3.

2Background

This chapter presents the theoretical background of the theories, upon which thisthesis is based on. These concepts have different hypothesis and this is a selectionbased on our intuitive acknowledgement with our problem statement in mind.

2.1 Emotion theory

It is evident that there is no onemodel of emotions1, Baumeister et al. summarizeswhat emotions are in one short sentence [1],

Put simply, the quick affective responses mainly indicate either goodor bad evaluations, which activate either the approach or avoidancesystems.

There are various hypothesis on what constitutes as an emotion, Wikipedia statesthat emotion could be the driving force of motivation, and therefore plays a ma-jor part in decision making. Humans needs motivation to act upon a goal. Thisis coupled with the statement that behaviour2 strives to change emotion, for thepurpose of becoming happier or reduce pain[1]. Another statement says that emo-tion does not directly cause certain behaviour [1], although the same article doesnot deny that emotions can have a direct effect on behaviour. Which coincideswith another statement that human beings function well when emotion directlystimulates cognition, and not so well when emotion directly stimulates behaviour[31]. Damasio also claims that emotion is feedback, that it comes after the rele-vant behaviour and is therefore too late to cause it [2], and another statement

1Subjective experiencing the arousal of the nervous system, see appendix2Actions made in an environment, see appendix

3

4 2 Background

suggest that emotion is the result, not the cause of behaviour [31].

Mostly emotions work toward the personal interests and the benefits that can bemade. A drastic and perhaps the most basic example would be the instinct tosurvive and procreate, a not so drastic instinct could be to increase ones statusor happiness. Living emotional organisms only have emotions about things thatmatter to them [2] according to Damasio. Shaver et al. writes that behaviour isaimed at producing change in emotion [31], since emotion can drive behaviourit indicates a relationship between behaviour and emotion, and thus makes emo-tion a prerequisite for moral behaviour as well.

Another paper writes about how emotion helps learning [1], this connects emo-tions with the social intuitionist model that Haidt writes about [8], these intu-itions are learned from every interaction in the environment and later appliedas intuitive gut feeling judgement, similar to a muscular reflex for the physicalworld but in this case as a mental reflexes. Also in neuroscience evidence of a con-nection between emotion and morality can be seen in an fmri image [6], whendealing with personal moral dilemmas the fmri image displays the same areasthat activate during emotional processing. This results in a strong connectionbetween moral information processing and emotional processes.

2.2 Morality theory

According to Jonathan Haidt morality3 is based upon what he calls moral founda-tions [12, 13, 16], these foundations are groups of moral values. The foundationsentail different capabilities of morality or faculties4. The two most important ofmorality are reciprocity5 and empathy6 [3]. Some older beliefs about moralityhas been that it can only be applied by humans and thus something not existingin the animal kingdom, Frans de Waal shows how monkeys act morally by notaccepting food until his friend also gets the same kind [3]. Historically moralityhas been interpreted as human reasoning, however this is not correct accordingto current research, since morality is also used in subconscious actions and judge-ments. On a cognitive level there are two kinds, moral intuition and moral rea-soning, according to Haidt [14]. The former involves a fast or intuitive reactionto something that happens, this is also something that Haidt writes about whendescribing the social intuitionst model [8]. However from analysing fmri scansHaidt et al. comes to the conclusion [6] that when trying to strip off the variousprocesses that the concept moral judgement7 is comprised off, there is not muchleft, they write in [6],

Thus, the interrelationships among these overlapping concepts is com-plicated, andmany relevant details remain unclear. What is becoming

3Intentions, decisions and actions which are good or bad, see appendix4Ability, skill or power, see appendix5Response actions, see appendix6Recognize emotions, see appendix7Evaluation of evidence to make a decision, see appendix

2.2 Morality theory 5

increasingly clear, however, is that there is no specifically moral partof the brain. Every brain region discussed in this article has also beenimplicated in non-moral processes.

There are two approaches tomorality one that depicts thatmorality is only learnedat childhood and the other suggests that it is been built into humans by evolution.According to Haidt, morality is both natural and learned [12]. Even though cul-tures around the globe are very different they still need to solve the same prob-lems(e.g. power, resources, care, conflicts). Different routines are formed but thebasic moral modules8 are the same across civilizations.

Evolution favours group selection9. morality is vital for cooperation to take placeand create positive results in a large group, which is necessary for competingand becoming victorious against other groups. But if there are selfish individ-uals in a cooperative group, cheaters or defectors, group selection can not takeplace. Evolution solves this by putting everyone in the same boat [11], this hashappened many times in nature, from bacteria to insects and humans. When hu-mans started to divide the work into groups they became much more effective,and became part of a greater cause, much of this behaviour is due to morality. AsJonathan Haidt said in one of is talks [11],

The most powerful force known on this planet is human cooperation- a force for construction and destruction

This means that humans have a very advanced understanding of reciprocity andthat we can apply it in our social interaction, for good and for bad. Haidt arguethat humans evolved to see sacredness all around us and to join into teams topreserve objects, people or ideas that benefit the group [11]. Most people wantto do something good and noble during their life time, overcome pettiness andbecome part of something larger, and that’s wheremorality comes in, to aid groupcooperation for making such desires possible.

Dispute often occurs between two parties, both with the belief that they are right,how that can happen can only indicate that they have different standards andvalues that they rely on. There will always be human differences in perceptionand interpretation of events. Just as we need a common understanding of wordsto communicate, we need a common understanding of behaviour to act morally.

morality hasn’t always been the same it has evolved with the advancement of ourprogress as a human civilization. For example, not long ago it was consideredimpermissible to steal medicine for a mortally ill relative from an apothecary thatwouldn’t give away medicine to a poor person who could not pay the price. Notlong ago this was generally accepted in society, compared to modern consensus,who proclaims that it is permissible to steal the medicine, since the value of lifeoverpasses greed and laws in this example. Today people think that the rightthing to do would be to acquire the medicine to save one person no matter thecost, specially if it is someone close.

8The most distinct are suffering, hierarchy, reciprocity and purity9As natural selection but between groups in evolutionary biology

6 2 Background

2.3 Moral emotions theory

A moral emotions is an emotion that is related to an action or event during be-haviour that affects the outcome of another individual or society in either a goodor bad manner, i.e. moral conduct, that emotion then becomes a moral emotionaccording to Haidt [9]. Haidt et al. describes this in [14],

Moral intuitions are about good and bad. Sometimes these affectivereactions are so strong and differentiated that they can be calledmoralemotions, such as disgust or gratitude, but usually they are more likethe subtle flashes of affect that drive evaluative priming effects.

There are two component features of an emotion that identifies a moral emotion,elicitors and action tendencies. The moral emotions are gathered into familiesby Haidt, the families are other-condemning, self-conscious, other-suffering andother-praising [9].

2.3.1 Other-condemning family

This family contains the negative emotions humans get from other individuals oractions that represent cheating, lies or faking the appearance of being reliable.

AngerAnger is an emotional response to a threat or provocation, it becomes the domi-nant emotion when a person takes action to stop that threat. It may be physicallycorrelated to such as increased heart rate, blood pressure[wiki].

Elicitors: Betrayals or unwarranted insults give rise to anger, goal blockage andfrustration will also trigger anger, it is associated with injustice[9].

Action tendencies: Revenge, either physical or emotional. To hurt, either throughattacking or humiliating the person who has acted in an unjust way[9].

DisgustThe response to something offensive, unpleasant or revolting, primarily to a sen-sation of taste but also by smell, touch or vision[wiki]. It helps to distinguishbetween groups of different status.

Elicitors: Mainly from violations of physical purity but also objects or ideas, forexample food and sex, and by moral depravity. It can also be applied in responseto hypocrisy, betrayal or cruelty[9].

Action tendencies: To avoid, expel, segregate or otherwise break off contact withthe offender[9].

ContemptIt is easily mistaken as disgust and sometimes as anger, considered a secondaryemotion it is a mix of anger and disgust. In contrast to anger and disgust, it is

2.3 Moral emotions theory 7

characterized by low arousal, bordering on indifference[wiki].

Elicitors: The emotion is triggered when experiencing violations of duty and hi-erarchy and also with disrespect[9].

Action tendencies: It doesn’t motivate withdrawal or attack, only a change inmentality, such as less respect and weakens other positive moral emotions[9].

2.3.2 The Self-Conscious family

These emotions are self triggered, used to correct one’s behaviour in groups soto avoid triggering anger, disgust or contempt of others. Shame and embarrass-ment are very similar, it mostly depends on culture if a human has developed adistinction between them.

EmbarrassmentEmbarrassment is related to hierarchy, and is thus considered a social emotion, itis the weakest of the self-conscious emotions.

Elicitors: When one’s social persona10 is threatened, a social convention rule isbreached by the self or events out of one’s control. Most often experienced whenone is around people of higher status[9].

Action tendencies: Hide, withdraw or disappear on both movement and speech,with the intention of reducing the punishment. With a lesser urge than shame[9].

ShameShame is more aligned to towards moral norms and the violations of them. It ismore cognitively oriented than embarrassment.

Elicitors: It triggers when the self is under the belief that someone else knowsabout the self caused violation, from failure to live up to morality, aesthetics orcompetence[9].

Action tendencies: Hide, withdraw or disappear on both movement and speech,with the intention of reducing the punishment. With a larger urge than embar-rassment [9].

GuiltThis emotion is often confused with shame since it is also in relation to hierarchy,but aligned more towards relationship and focuses on the behaviour rather thanthe self image.

Elicitors: It can trigger when one’s self has the belief that one has caused harm,loss or distress to a another person. The emotion is strengthened by the closeness

10The image an individual presents to the world.

8 2 Background

of the relationship[9].

Action tendencies: It motivates one to help one’s victim or otherwise to makeup for one’s transgression to restore or improve the relationship[9].

2.3.3 The Other-Suffering Family

This is one of the more basic family of emotions that develops early, even within the first year of a child, and they are also found in chimpanzee. They arecharacterized by being moved by or understanding others suffering.

CompassionIt is also know as empathy and sympathy.

Elicitors: When perceiving suffering or sorrow in another person, and can butnot a always, bring the desire to help. It can be felt for a complete stranger andcan be very strongest towards a person with a close relationship[9].

Action tendencies: It creates a desire to help, comfort or reduce the sufferingin other[9].

2.3.4 The Other-Praising Family

The family of emotions that is associated with good actions and positive moralperceptions belong to this category. They work differently than the negative emo-tions, usually there is no procedure or action to correct wrong behaviour, ratherthey open perception to novel ideas, relationships or possibilities. These emotionencourages people to improve themselves to better comply with future events.Greatly motivates good behaviour.

GratitudeEncouraging others to repay the benefactors.

Elicitor: Is the feeling that triggers when someone has done a good deed to one’sself, the more costly and unexpected the greater will the feeling to repay be[9]

Action tendencies: Attempts to return a similar favour, and as a motivator formorally good behaviour[9].

AweSimply awe is a combination of surprise and fear, but it is also likely that thisemotion is a very complex emotion comprised of many more components such asreverence, wonder, admiration, respect, genius, great beauty, might, etc.

Elicitor: Awe triggers by beauty and exemplar actions, and also by kindness andcharity[9].

Action tendencies: Makes peoples stop, admire and open their hearts andminds[9].

2.4 Moral emotional decision making 9

ElevationThis emotion has the opposite composition of disgust.

Elicitor: Acts or actors of charity, kindness, or loyalty and self-sacrifice are triggers[9].

Action tendencies: It motivates people to help others and to become a betterperson[9].

2.4 Moral emotional decision making

2.4.1 Ethics and morality

Morality can sometimes interchangeable be used with the word ethics, howeverthere is a distinct difference between them. While both entail the essence of rightand wrong behaviour, morality centres around the self image and is subjectiveand intuitive to the actor, it can also change depending on that persons beliefs.Where ethics is a code of conduct imposed by i.e. a group of people or culture[4].Meaning that ethics depict objectively how different persons should act in a givensituation and environment. Both concepts are important in understanding theessence of how morality affect behaviour. This is how Haidt describes how hismoral modules of morality affects our behaviour[13],

These modules generally have as one of their outputs the emotion ofcompassion: the individual is motivated to act so as to relieve suffer-ing or otherwise protect the child.

Therefore a moral person would be someone that is making good decisions ac-cording to the ethics or in other words society’s values along the lines of thedefinition of a good person. Two approaches are suggested in the literature forprocessing moral behaviour, top down and bottom up.

Top down

In moral context this approach consists of moral theories derived from Conse-quentialism, which is a set of rules. These rules set boundaries for the algorithmsdriving the behaviour[34]. This approach encounters difficulty with computa-tional load since there is a need for vast amount of human knowledge requiredto process such concepts. It is also difficult to set the initial conditions for thisapproach. The top down approach is a symbolic way of implementing morality.

Bottom up

This approach represents routines used for action selection, values and normsformed by experience, it considers goals or standards that may or may not bebased on moral theory, if they are, it is to declare the task for the process of ac-tion selection. In contrast to the top down approach the bottom up approach

10 2 Background

depends on the learning ability and the sensors of a system [34]. As such this ap-proach encounters technological constraints. It could mimic the subjective partof morality and it is consider dynamic and flexible.

2.4.2 Decision theory

A true moral behaviour process will arguable need both ends of the top downand bottom up approaches to fully cover the different aspects of moral behaviour.Most of human behaviour is composed by a rapid intuitive response mechanisms.Loewensteins model of decision making describes how emotions can affect thecognitive process [20] and create such mechanism, the same mechanism thatHaidt calls intuitive social behaviour which are conducted in a moral context.A figure of Lowensteins process can be viewed in the figure 2.1. This emotionalprocess is modelled in emobn created by Johansson[17] based on ebn. Mostof human behaviour is composed by these rapid intuitive response mechanism,this behaviour caused by intense emotions can even be against the subjects inter-ests. This could explain how emotions can override reason and how decisionscan seem irrational during strong emotional episodes. Since with less informa-tion about the world that persons perception is more narrow than with a weakeremotional influence.

Figure 2.1: Lowenstein model of how emotions affect behaviour [20].

3EMOBN architecture

This section is based upon Johansson’s Ph.D thesis [17] and the article [18].

3.1 Introduction

The behaviour network approach was first introduced by Maes [21] in 1989 asan energy-driven approach to decision making. A behaviour network consists ofgoals, states, behaviour modules and parameters. The decision making mecha-nism is based on the notion of activation. At each cycle of the behaviour networkactivation is spread from the goals of the network to behaviour modules which inturn spread activation to other behaviour modules. The activation of a behaviourmodule can be seen as the utility of the behaviour. The more activation, the moredesirable it would be to perform that behaviour.

In 2009, Johansson and Dell’Acqua [18] extended the behaviour networks withemotions to create a general affective decision making model. They called thenew model Emotional Behaviour Network (emobn). The parameters of the net-work were made subject to the emotional state of the agents. The authors intro-duced the notion of influences to let emotional states directly affect the activationof behaviour modules without being preconditions. They also let emotions affectthe probabilities of the effects to mimic the pessimistic vs. optimistic judgementof humans.

The different components of emobn, the activation spreading and the action se-lection mechanism are described in detail below.

11

12 3 emobn architecture

3.2 Components of EMOBN

A emobn is represented by a tuple 〈S ,B, G,R, E ,L〉 of six components: a set ofstates S , a set of behaviour modules B, a set of goals G, a set of resources R, a setof emotions E , and a set of emotional links L.

3.2.1 States

A state represents a belief the agent has about some aspect of the environment(and itself). We write v(s) to indicate the value of a state s ∈ S . We let 0 ≤ v(s) ≤ 1.Given a state a, we write a to indicate its opposite.

States are graphically represented as rectangles.

3.2.2 Behaviour Modules

A behaviour module represents an action that can be performed by the agent. Abehaviour module k ∈ B consists of a set of preconditions Prec(k), a set of effectsEffects(k), and a set of emotional links eLinks(k).

Preconditions. Preconditions are represented by states, that is Prec(k) ⊆ S . Abehaviour module has none, one or possibly several preconditions. They must betrue for the behaviour to be executable. When the preconditions take continuousvalues, the executability value of k is obtained by multiplying the values of thepreconditions. (See step (2) in Fig.3.4.)

Effects. A behaviour module k has one or several effects representing the conse-quences of the behaviour. Effects are represented by states Effects(k) ⊆ S . Eacheffect s of k is coupled with a probability pk(s), with 0 ≤ pk(s) ≤ 1.

Emotions affect how optimistic and pessimistic we are, and also how much riskwe are willing to take. Johansson [17] suggested that emotions should affect theprobability of the effects. To do this, each effect must know the perceived “good-ness” of the effect. If the agent is optimistic, the probability for a negative effectshould decrease while the probability for a positive effect should increase. Theperceived goodness is called benevolence and has value +1 if it is positive, −1 ifit is negative, or 0 if it is neutral. The benevolence is specified for each effect ofthe behaviour module. Using this value, the new emotional probability p̂k(s) ofan effect s of k is defined as:

p̂k(s) = pk(s) × (1 + benevolence

× (posE − negE) × π)

where π is a constant used for determining to what extent emotions should alterthe perceived probability, and posE and negE are the average values of emotionsthat affect risk-taking positively and negatively, respectively.

For example, happiness and anger make humans subjectively increase probabili-ties of favorable events occurring, while at the same time lowering the probabil-ities of negative events happening. In this case, the formula above changes theprobability to a higher value if anger is high and benevolence is positive.

3.2 Components of emobn 13

3.2.3 Goals

A goal specifies what an agent wants to accomplish. Every goal has one or moreconditions, represented by states, that need to be fulfilled in order for the goal tobe successful.

A goal has a static importance x, with 0 ≤ x ≤ 1, and a dynamic importance y thatis a value linked to states and emotions. This enables the goal to have a varyingimportance depending on the value of that state. The importance ig of a goalg ∈ G is defined as:

ig = f (x, y)

where f is any continuous triangular norm. In our framework we use multiplica-tion.

Goals are depicted as diamonds.

3.2.4 Resources

A resource represents a necessary means to execute a behaviour module. Eachresource is coupled with the number of available units and the number of boundunits (units of this resource that are currently being used by behaviours).

3.2.5 Emotions

E is a set of emotions. For each emotion e ∈ E we write v(e) to indicate its value,and we let 0 ≤ v(e) ≤ 1.

3.2.6 Emotional Links

Emotions should be able to affect individual behaviour modules directly. For in-stance, people might be more inclined to dance if they are happy than if they aresad, regardless of the effect of dancing. However, being happy is not a require-ment for dancing and should therefore not be a precondition.

Emotional links allow us to couple emotions to behaviourmodules. An emotionallink 〈e, k, z〉 ∈ L expresses a connection between an emotion e and a behaviourmodule k with a given strength z (with −1 ≤ z ≤ 1). z determines the extent towhich e affects the selection of k for execution. We let eLinks(k) be the set of allemotional links to a behaviour module k. eLinks(k) contains the tuple 〈e, z〉 forevery emotional link 〈e, k, z〉 ∈ L.

Given a behaviour module k, the emotional influence n(k) on k is defined as:

n(k) =∑

〈e,z〉∈eLinks(k)

(v(e) × z)

where v(e) is the value of the emotion e and z its strength.


3.3 Activation Spreading

Every behaviour module receives activation from each goal in the network. Theactivation is spread from the goal to the behaviour modules. In turn, the be-haviour modules spread activation internally from module to module. The totalactivation for each behaviour module is calculated and used to select which be-haviour to execute.

3.3.1 Network Parameters

The activation spreading in is controlled by the following parameters.

γ : activation influence. It determines how much activation is spread throughpositive links1. The activation influence implicitly determines the amountof planning the agent is capable of.

δ: inhibition influence. It determines how much activation is spread throughnegative links. The inhibition influence implicitly determines the amountof planning the agent is capable of.

β: inertia. It determines how much the last activation affects the current one.Having a high inertia value will suppress erratic, indecisive behaviour. How-ever, having a high inertia will decrease reaction time.

θ: activation threshold. It determines the threshold that the execution valuemust exceed for the behaviour to be executed.

The parameters affect the activation spreading in different ways. The inertia pa-rameter β affects how easily the agent switches behaviour. A low value of β willresult in a more flip-flop behaviour, where the agent has a tendency to switchbehaviours more often and more easily. The parameters δ and γ both affect howfar in the future the agent can plan. Using low values for these parameters willgive much less activation to behaviour modules that are further away from goals.

The parameters γ , δ and β of the behaviour network have to lie within the inter-val [0, 1] for the system to be stable. One must take this into account when lettingemotions influence the parameters. Setting the initial values of the parametersis fairly straightforward using trial-and-error methods. It has furthermore beenproven that a behaviour network is goal converging no matter the parameter val-ues if it is dead-end free and terminating [24].

In , the network parameters are effected by the emotional state of the agent. Dif-ferent emotions affect different parameters. For example, the emotions used in[18] are given in Table 3.1.

The overall influence of the emotions in Table 3.1 is controlled by an emotionalimpact parameter ε. Any parameter P ∈ {γ, δ, β} has a corresponding emotional

1Positive links occur when the effect of a behaviour module is the same as the precondition ofanother behaviour module. Likewise, negative links signify a link where the effect of a behaviourmodule is the opposite of the precondition of another behaviour module.

3.3 Activation Spreading 15

version P̂ defined as:

P̂ = P × (1 + (posE − negE) × ε)

where posE is the average value for the emotions with positive impact for thatparameter P and negE is the average value for the emotions with negative impact.

3.3.2 Activation Spreading

Each cycle time t, activation propagates from the goals to the behaviour modules.There are four ways by which a behaviour module can receive activation. Below,we let k be a behaviour module, g a goal with importance ig and t the cycle time.

Case 1. k receives positive activation A from g at t if there is an effect s of k thatis one of the conditions of g .

A = γ × ig × p̂k(s)

Case 2. k receives negative activation B from g at t if there is an effect s of kthat is the opposite to one of the conditions of g .

B = −δ × ig × p̂k(s)

Case 3. Consider all behaviour modules j having a precondition sj that is oneof the effects of k. The amount of activation C that k receives from g at t via allj’s is:

C = γ ×∑

for everyj∈Bsuch that

∃sj∈S(sj∈Prec(j)∧sj∈Effects(k))

(σ(at−1jg ) × p̂k(sj ) × (1 − v(sj ))

where

σ(x) =1

1 + eλ(µ−x)

Parameter Positive Negativeγ sadness fear

fatigue hungeranger

δ sadness fearfatigue hunger

angerβ anger happiness

sadnessfear

Table 3.1: Affective impact on the network parameters


σ(x) is a Sigmoid function used here to filter the previous activation value forthat particular goal and module. The parameters λ and µ are constants used tomodify the shape of the Sigmoid curve.

The equation above states that the less fulfilled a precondition of a module is, themore activation will be sent to modules that fulfil that precondition.

Note that using at−1jg in the definition of activation implies that each module muststore the activation received by each goal during the previous cycle time t − 1.

Case 4. Consider all behaviour modules j having a precondition that is the op-posite of one of the effects of k. The amount of activation D that k receives fromg at t via all j’s is:

D = −δ ×∑

for everyj∈B such that∃sj∈S(sj∈Prec(j)∧sj∈Effects(k)))

(σ(at−1jg ) × p̂k(sj ) × (1 − v(sj )))

−δ ×∑

for everyj∈B such that∃sj∈S((sj∈Prec(j)∧sj∈Effects(k))

(σ(at−1jg ) × p̂k(sj ) × (1 − v(sj )))

Total activation. The activation atkg given to the behaviour module k by the goalg at cycle time t is set to the activation that has the highest absolute value:

atkg = absmax(A, B, C, D)

This implies that only the strongest path from each goal to a behaviour moduleis used. Combining activations from the different paths is not allowed.

The total activation atk for a behaviour module k is the sum of the activationsgiven to the module from all goals in the network plus the total activation calcu-lated at the previous cycle multiplied by the inertia parameter β.

atk = β × at−1k +∑

for every g∈G

atkg

3.4 Action Selection

The action selection mechanism for emobn is specified by the following proce-dure. Let t be the cycle time.

3.4 Action Selection 17

1. Calculate the total activation atk for every k ∈ B.

atk = β × at−1k +∑

for every g∈G

atkg

2. Calculate the executability etk for every k ∈ B.

etk =∏

s∈Prec(k)

v(s)

3. Calculate the execution value htk for every k ∈ B.

htk = atk × etk × (1 + n(k))

4. Sort all behaviour modules by their execution values, largest valuefirst.

5. For each behaviour module k in the sorted list, check that:

(i) the execution value htk exceeds θ, the required activation thresh-old.

(ii) there are enough unbound resources required to execute k.

If both conditions (i-ii) are met, bind the resources and choose kfor execution.

6. Unbind all resources, increase the cycle time t by one unit.

Figure 3.4 Action selection mechanism

Remark 1. The executability at step (2) is calculated by multiplying the value ofthe preconditions of k. In general, any triangular norm over the preconditions ofk can be used. The executability is a measure of how likely it is that a behaviourcan execute successfully.

Remark 2. At step (3) since at−1k and not ht−1k is used in the calculation of atk , emo-tional influences will only affect the module locally. The change in the executionvalue is not spread to other modules in the next cycle.

Remark 3. The action selectionmechanism presented above differs from the orig-inal one [18] for what concerns the handling of the activation threshold θ. Theoriginal definition contains a threshold decay ∆θ that determines how much acti-vation threshold should be lowered if no behaviour module could be selected forexecution. Here, the activation threshold is fixed, and cannot be lowered. Onlya change in the values of the emotional influences can trigger a behaviour thatcouldn’t be triggered earlier.

4Models

We begin this chapter with a summary of the theories in the background chapterand move on to explain the relevant models for the moral design, these modelsand theories are used as tools for the design.

4.1 Background theory summary

From the presented background theory emotions are triggered by sensory inputand/or thoughts, to drive the agents actions towards beneficial outcomes for theself. They can be controlled by conscious thought, which will lead to either in-creased or decreased strength of the emotion. Conscious thought processes cre-ates and modifies knowledge and stores this in memory as values, that are alwaysavailable for access during interaction with the environment. This extraction ofthe values with out consciousness is the concept of intuitions. Behaviour that ispurely driven by strong emotions could result in uncontrolled and/or irrationalbehaviour. The moral emotions are a combination of emotions and personal val-ues that bring positive outcomes for others, and in the best case also for the self.The figure 4.1 is a conceptual diagram of how emotions and reasoning can belinked, created from our understanding of the theory researched for this thesis.

19

20

4Mod

elsFigure 4.1: A diagram of the theories from the background chapter and how the different concepts are linked accordingto our understanding.

4.2 Moral Foundation Theory 21

4.2 Moral Foundation Theory

Haidt describes mft as evolutionary prepared basic routines linked to appraisalin specific social behaviours [13]. mft is an attempt to categorize similaritiesacross different cultures with respect to morality. It has been coined intuitiveethics by Haidt. He is the co-founder of the moral foundations. He argues thatmorality is not based around one core value butmany. To better explain themoralfoundations Jonathan Haidt uses an analogy with taste. Humans have differentreceptors for different basic tastes, he argues that morality could be describedin a similar way. In mft care, fairness, liberty, loyalty, authority and sanctityare the receptors of different basic moral values that exist in every culture andsome even in the animal kingdom. It is also an organized way of displaying ameasurement of the moral concerns among individuals, groups or cultures. mftis standardized and can be used to quantify morality in a systematic way [12, 13,16]. mft is based on earlier work from Fiske, Shweder and Hogan. Haidt’s viewon morality is that it is both inherent and learned during childhood. Haidt hasfound evidence for the existence of inherent morality as he writes [13],

It is therefore implausible thatmammals learn entirely through domain-general learning mechanisms how to recognize suffering or distress intheir offspring. Rather, many mammals have innate harm-detectionmodules that were shaped by evolution to be responsive to the properdomain of signs of suffering in their own offspring.

The two most basic and best understood foundations are care and fairness. Ac-cording to Frans de Waal these two moral foundations are also found in the ani-mal kingdom, which exists in monkeys and elephants according to Frans de Wall[3]. We will focus on one foundation, care/harm, perhaps the most basic founda-tion in this thesis for our design/implementation.

Virtues are the positive traits that are morally good and a person behaving invirtuous way is considered a moral being. The inverse concepts are vices, theycould be considered bad behaviour that are counterproductive towards morallygood behaviours.

22 4 Models

4.2.1 The six moral foundations

Haidt et al. have established [16] six moral foundations that morality consists of,here is a short description of the six pillars.

• Care/Harm is the virtue of kindness, gentleness, and nurturance, the oppo-site is violence.

• Fairness/Cheating is the pillar that contains justice, rights, and autonomy.

• Liberty/Oppression which contain solidarity, to oppose or take down an op-pressor.

• Loyalty/Betrayal can be summed up in the saying "‘one for all, all for one"’,virtues of patriotism and self-sacrifice for the group describes this pillarwell.

• Authority/Subversion contains virtues of leadership and followership, thisis true for traditions.

• Sanctity/Degradation entails living in an elevated, less carnal, more nobleway, for example by not living up to certain moral values you desecrate yourself.

Five of the moral foundations that are fully mapped out can be viewed in thetable 4.1. The liberty foundation has recently begun to be evaluated in Haidt’slatest paper and therefore is not fully mapped out.

4.2Moral

Fou

ndation

Theory

23

care/ harm fairness/ cheat-ing

loyalty/ be-trayal

authority/ sub-version

sanctity/ degra-dation

Adaptive chal-lenge

Protect andcare for chil-dren

Reap benefitsof two-waypartnership

Form cohesivecoalitions

Forge benefi-cial relation-ships withinhierarchies

Avoid com-municablediseases

Proper domain Suffering, dis-tress, or threatto one’s kin

Cheating, coop-eration, decep-tion

Threat or chal-lenge to group

Signs of domi-nance and sub-mission

Waste prod-ucts, diseasedpeople

Actual domain Baby seals,cute cartooncharacters

Martial fidelity,broken vend-ing machines

Sports teams,nations

Bosses, re-spected profes-sionals

Taboo ideas,racism, deviantsexuality

Characteristicemotions

Compassionfor victim,anger forperpetrator

Anger, grati-tude, guilt

Group pride,rage at traitors,belongingness

Respect, fear Disgust

virtues Caring, kind-ness

Fairness, jus-tice, honesty,trustworthi-ness

Loyalty, patrio-tism, self sacri-fice

Obedience, def-erence

Temperance,chastity, piety,cleanliness

vices Cruelty Dishonesty Treason, cow-ardice

Disobedience,uppitiness

Lust, intemper-ance

Table 4.1: The five moral foundations [16]

24 4 Models

4.3 The trolley problem

We have chosen a well known dilemma called the trolley problem [25]. Thisis a philosophical thought experiment with various cases. The different caseshave seemingly small differences to the composition of the dilemma, while thebehaviour to execute remains the same, but the permissibility changes from eachcase. In the study of the trolley problem compassion is the primary emotion andtherefore we can narrow down the mft foundations to just the one concerningcare and harm.

The scenario plays out at a track where an accident is witnessed in first person.A runaway trolley is heading towards five people that will be hit by the trolleyand die, this is for certain. There is no way around this outcome according to thesetup of the scenario. Thus this is the belief of the observer. However the observerhave the ability to, also with 100% certainty, direct the trolley away from the fivepersons, but by doing so the observers action will for certain kill one other person.There more cases which have seemingly small differences to the composition ofthe dilemma. While the outcomes remains the same in all cases the action toexecute changes, and so the permissibility changes from each case.

Since the trolley problem is a well know dilemma both in the public and in cog-nitive psychology, we consider it a good reference point. These cases have beenstudied by linguists, philosophers and psychologists. The two most polar cases,which can be seen in the graphs of [22], are the bystander and the footbridgecases. With a small rearrangement of the dilemma the action to take suddenlybecomes completely impermissible in contrast to the previous arrangement. Thesimplicity of the dilemma and with differentiating outcomes makes this a suitedtest bed for establishing the validity of our design. Since the trolley problemis a well know dilemma both in the public and in experimental psychology, weconsider it to be a good reference point.

4.3 The trolley problem 25

4.3.1 Bystander scenario

In the bystander case you are standing on the side of the track with a lever thatcan switch the track so the run away trolley diverts into one person instead of thefive, which the trolley was heading towards. A diagram of the bystander case isgiven in figure 4.2.

Trolley

?

Figure 4.2: A diagram of the bystander case.

4.3.2 Footbridge scenario

For the footbridge you are standing on a bridge next to another human that youdo not know, however you do know for certain that his weight will stop the trolleyheading towards the five persons on the track, which is depicted in the figure 4.3.This other person on the bridge has not given consent to being pushed down onthe track.

Trolley

?

Figure 4.3: A diagram of the footbridge case.

26 4 Models

4.4 Universal Moral Grammar

umg is a parallel perspective to ug1 in the respect that it is true to the sameconcepts and models used to figure out what the elements of language are andhow language work. But in the case of umg, the goal is to formulate the buildingblocks of morality and how it works. Mikhail proposes a framework for umg [22]that can aid in understand and model moral behaviour.

According to Mikhail umg might be innate [22]. He claims that we have innatemoral principles, that they are the foundations used to develop our moral valuesupon. Such that morality would indicate a innate base of right and wrong in chil-dren that are not taught by parents, society or cultures, but like a inborn trait thatfunctions as a rule system guiding the process of building up a persons morality.In never encountered situations they guide our behaviours, the so called gut feel-ings. Based on that we will always have some preferred behaviour instinctively,and this is also what Haidt suggests with his social intuitionist model. With thesedifferent perspectives they strengthen the theory of a innate moral foundation,and this gives ground to why certain feelings of right and wrong can not be ex-plained by the elicitor [8, 9, 10, 12, 13, 16, 23].

An example in language for how grammar is innate is clearly made by a quotefrom Chomsky,

Colourless green ideas sleep furiously

It is a nonsemantical and nonsensical sentence, but its well formed English from asyntax point of view. Some how English native speakers feel that a sentence withthe format: adjective adjective noun verb adverb, sounds good. When people areasked in experiments, they often can not explain why they have that opinion, yetthey still have a notion that it is acceptable. For example a reverse order of thesame sentence is not a well formed English sentence, syntax wise, and some howfeels wrong.

A intentional structure diagram with umg for the bystander case can be viewedin the figure 4.4. A similar intentional structure diagramwithumg can be viewedin figure 4.5 for the footbridge case.

With umg we can verify causality, intentions and morality of a given state or be-haviour. This is used as a validation layer for designing the states and behaviourswith the een and how they are sequentially linked together.

1A theory that proposes innate properties for language in humans.

4.4 Universal Moral Grammar 27

Figure 4.4: umg intentional structure of the bystander dilemma [22].

Figure 4.5: umg intentional structure of the footbridge dilemma [22].

5The moral design

The extension of the emobn and the process of how our design is implementedare explained in this chapter.

5.1 EEN, extension of the EMOBN

The addition of moral emotions in the decision making created a need for newcomponents in emobn, we added expected emotion to the existing architectureand called it een short for Expected Emotion Network.

5.1.1 Introduction

In emobn all goals should be affective [17]. As the author points out, psycholog-ical theories suggest that humans make decisions that try to maximize positiveemotions while minimizing negative emotions (for more information, see Anja’sphd Section 2.3.2). The ultimate reason behind many ordinary behaviours is toincrease positive emotions, such as happiness, and likewise minimizing negativeemotions, such as fear or sadness.

While many of our choices are not consciously made to improve our mood, sub-consciously the prediction of the future consequences for our emotional state isa key factor in decision making.

5.1.2 Components of EEN

A een is represented by a tuple 〈S ,B,R, E ,L,X〉 of six components: a set of statesS , a set of behaviour modules B, a set of resources R, a set of emotions E , a set ofemotional links L, and a set of expected emotions X . We assume that S containsboth states and their opposite, that is, k ∈ S if and only if k ∈ S .

29

30 5 The moral design

Emotions, states, and resources and behaviour modules are defined similarly asin emobn.

Expected Emotions

An expected emotion t ∈ X has a valence represented as Val(t) and an importancerepresented as Imp(t).

An expected emotion is induced by states and behaviours. Given t ∈ X , we writeInd(t) and Rel(t) to indicate the set of states and behaviours that induce, resp.release, the value of t.

Ind : X → (S ∪ B)

Rel : X → (S ∪ B)

Given an expected emotion t and a state or behaviour s, we indicate with qt(s) thestrength by which s induces or releases t, with 0 < qt(s) ≤ 1.

Expected emotions are depicted as rectangles with a triangle on the left side.

5.1.3 Graphical representation of

Let 〈E , S ,R,B,X〉 be a een. A graphical representation of a een network is builtin two phases: in the first phase p-arcs are added to the network and in the sec-ond phase n-arcs.

Phase 1

• For every t ∈ X , add a p-arc from any element in Ind(t) to t.

• For every k ∈ B, add a p-arc

- from every precondition of k to k, and

- from k to every effect of k.

• For every 〈e, k, z〉 ∈ L with z ≥ 0, add a p-arc from e to k.

Phase 2

• For every t ∈ X , add a n-arc from any element in Rel(t) to t.

• If there is a p-arc from a state s to a behaviour module k, add an-arc from s to k.

• If there is a p-arc from a state s to a behaviour module k, add an-arc from s to k.

• For every 〈e, k, z〉 ∈ L with z < 0, add a n-arc from e to k.

Below we use a plain arrows to represent positive arcs (p-arc), and dash arrowsto represent negative arcs (n-arc).

5.1 een, extension of the emobn 31

5.1 Example

Figure 5.1: Graphical representation of a een component.

5.1.4 Transformation Γ

In this sectionwe introduce a transformation, called Γ, that maps een into emobn.Γ maps an een 〈E , S ,R,B,X〉 into a emobn 〈E ′ , S ′ ,R′ , G′ ,B ′〉 as follows.


Transformation Γ

• E ′ = E

• S ′ contains all the states in S (that is, S ⊆ S ′).

• R′ = R

• B ′ contains all the behaviour modules in B that neither induce norrelease any expected emotion.

• For every t ∈ X do:

- Add two new states a and a to S ′ both with value 0.5.

- Add a new goal g to G′ . g has static importance Imp(t) and dynamicimportance 0.

- If Val(t) > 0, the condition of g is a, otherwise a.

- For every state s ∈ Ind(t) add a new behaviour module j to B ′ . Theprecondition of j is s and its effect is a with benevolence = 0 and pj (a) =qt(s).

- For every state s ∈ Rel(t) add a new behaviour module j to B ′ . Theprecondition of j is s and its effect is a with benevolence = 0 and pj (a) =qt(s).

- For every behaviour k ∈ Ind(t) add a new behaviour module j to B ′ .j has the same preconditions and effects as k plus the effect a withbenevolence = 0 and pj (a) = qt(k).

- For every behaviour k ∈ Rel(t) add a new behaviour module j to B ′ .j has the same preconditions and effects as k plus the effect a withbenevolence = 0 and pj (a) = qt(k).

5.2 Implementing the trolley problem with een 33

5.2 Implementing the trolley problem with EEN

To computationally create test runs with the moral design we use the programinglanguage prolog for the een engine. Prolog uses formal logic. It is the first andstill the most popular of its type and widely used for artificial intelligence devel-opment and computational linguistics. The reason why prolog is popular derivesfrom the fact that it is possible to formalize problems in a declarative, abstractway.

5.2.1 Expected moral emotions process

The figure 5.2 shows Roseman’s model of appraisal of the most basic emotionsand the criteria required for them to be elicited. In Petter Grundströms thesis [7]there is a short and concise description of the appraisal steps Roseman proposesfor how emotions are triggered. This is used to determine which emotion a stateor behaviour elicites.

Once the emotions are established for a given state or behaviour we can decidewhich emotion is in a moral context with Haidt’s categorization of moral emo-tions into different families. If it belongs to one of the moral emotional familythen i needs to be weighted with the apropriate foundation score. A foundationscore of 0 means that this foundation is completly rejected and will inflict nochange on the value of the emotion. Where as 5 means that one strongly endorsethis foundation and this will increase the value of the emotion to its maximum.

After the moral context has been established for a given state or behaviour, theweighting of themft foundations is applied. If the state or behaviour also affectsother agents utility an evaluation of the utility in the scenario is needed. Theutility is often objects or resources in relation to the agent. In the scenario forthe trolley problem the objects affecting the states are humans, they receive auniform distribution value so that the sum of each object’s fraction is one. Thestates and behaviours that depend on some utility are then weighted with therespected amount of each belonging resource or object.

It is important to note that in our scenario both cases have two separate time linesthat are compared against each others, competing for activation energy.

To implement the scenario a diagram with emotions is specified, validated withumg and then scripted with prolog. The expected emotions and their initialvalues for the bystander case are derived from tables 5.1 and 5.2 for actions andstates respectively Figure 5.3 shows how the behaviours and states are connected,and also how activation is flowing. Regarding the footbridge table the figure 5.3and 5.4 contains the expected emotions and figure 5.4 shows the diagram of theactivation flow.


Figure 5.2: Roseman’s model of common emotions and how they are trig-gered [30].

5.2Im

plem

entin

gthetrolley

prob

lemwitheen

35

Actions Switch not thrown Switch thrownAgency circumstance selfProbability certain certainSituation consistent consistentMotivation aversive aversivePower/Control weak strongResulting emotion Relief Pride

Moral emotion no noPolarity + +Initial value 0.3 0.7

Table 5.1: Emotions from actions for the bystander case

States Trolley moving Save(1) Kill(5) Trolley turned Save(5) Kill(1)Agency circumstance circumstance circumstance self self selfProbability certain certain certain certain certain certainSituation consistent consistent inconsistent consistent consistent inconsistentMotivation aversive appetitive aversive aversive appetitive aversivePower/Control strong weak weak strong weak weakResulting emotion Relief Joy Disgust Pride Pride Guilt

Moral emotion no no yes no no yesPolarity + + - + + -Initial Value 0.3 0.3 0.3 0.7 0.7 0.7

Table 5.2: Emotions from states for the bystander case.

36

5Themoral

design

Actions Man not pushed Man pushedAgency circumstance selfProbabilty certain certainSituation consistent inconsistentMotivation aversive aversivePower/Control weak strongResulting emotion Relief Regret

Moral emotion no noPolarity + -Initial value 0.3 0.7

Table 5.3: Emotions from actions for the footbridge case.

States Trolley moving Save(1) Kill(5) Trolley hit man Save(5) Kill(1)Agency circumstance circumstance circumstance self self selfProbability certain certain certain certain certain certainSituation consistent consistent inconsistent inconsistent consistent inconsistentMotivation aversive appetitive aversive aversive appetitive aversivePower/Control strong weak weak strong weak lowResulting emotion Relief Joy Disgust Regret Pride Guilt

Moral emotion no no yes no no yesPolariy + + - - + -Initial Value 0.3 0.3 0.3 0.7 0.7 0.7

Table 5.4: Emotions from states for the footbridge case.

5.2Im

plem

entin

gthetrolley

prob

lemwitheen

37

save (5)

kill (1)

trolley turns

kill (5)

save (1)

trolley moving

do not

kill

switch not

thrown

pride

guilt

pride

pride

releif

relief

joy

disgust

switch

thrown

Figure 5.3: A diagram showing activation flow of the bystander case.

38

5Themoral

design

save (5)

kill (1)

trolley hit man

kill (5)

save (1)

trolley moving

do not

kill

man not

pushed

pride

guilt

regret

regret

releif

relief

joy

disgust

man

pused

Figure 5.4: A diagram showing activation flow of the footbridge case.

5.2 Implementing the trolley problem with een 39

5.2.2 Weighting

This section explanation how morality and utility is taken into account in thedesign for the states and behaviours.

Morality

Taking morality into account using the relevant mft foundation, in our scenariothe foundation for care, on the states and behaviours with moral context, weapply a weight defined as:

X ′ = X + (1 − X )ρ

ρm

where ρ is the the relevant foundation score. Since the score ranges from 0-5 wewill therefore normalize the foundation score with ρm = 5, so that the resultingrange remains between 0− 1. And X is the current expected emotional value, theparenthesis (1 − X ) will let the weighting be scaled depending on the current Xvalue, if the X value is low a greater increase in emotion will occur, and if the Xvalue is high a smaller increase will occur, this is again to ensure that the resultstays with in 0-1 range.

Utility

The utility weighting is similarly defined as:

X ′ = X + (1 − X )U

Um

In this context utility is referred to the humans gained or lost in a state or be-haviour. U stands for the amount of resources or objects gained or lost for aparticular state, and Um is the total number for a certain object or resource in thescenario that can be gained or lost. For this trolley problem it is 0 when a statehas no humans affected by it, 1 and 5 for 1 human and 5 humans respectivelyand 6 is the max number humans in the scenario.

For mft weighted expected emotions with utility connected to the event a morecomplex weighting is applied. Since the utility weighting is applied after themftweighting and to keep the result in the range between 0 − 1 the total weightingapplied is defined as:

X ′ = X + (1 − X )ρ

ρm+ (1 − (X + (1 − X )

ρ

ρm))

U

Um


5.2 Example

A current expected emotion of 0.32 and a mft foundation score of 4.2 where 1person is affected the calculation would look like this: 0.32 + (1 − 0.32) ∗ 4.2/5 +(1 − (0.32 + (1 − 0.32) ∗ 4.2/5)) ∗ 1/6 and the new expected emotional value wouldbe approximately 0.91.

6Testing

Two tests have been performed with the moral design and our results are com-pared to the psychological empirical studies. The full prolog script for both casesis referenecd in appendix A.

6.1 Aim of the tests

Two test were done to establish accuracy towards the empirical test results and tocompare different mft pillar scores and how they affect the result of our design.

The activation is marked after initialization and one cycle through the een. Allemotional activation is set 0.5.

6.2 Bystander (Test 1)

One cycle of computations for the bystander case with different MFT care score1.0, 2.5 and 5.0 can be seen in table 6.1. A low value value means rejection of thisfoundation and a high value means full endorsement.

mft care foundation score 1.0 2.5 5.0Switch turned 0.5797 0.4758 0.3255Switch not turned 0.4612 0.4505 0.4331Difference 0.1185 0.0253 -0.1076

Table 6.1: Result of bystander test.

The small difference between the actions presents how equally desired/undesired

41

42 6 Testing

the two actions are. In this test there is even a shift in behaviour when the mftscore rises towards max. About 77% [33] of the 4000 people in a bbc internetpoll consider it permissible to turn the switch and 90% [15], in an other onlinetest performed by Marc Hauser thought it was permissible as well. This is thesame result as our design indicates at mid to lower foundation score. Howeverour results shows that with a high inclination towards caring, the act of causingsomeone harm, would be impermissible. One could speculate that a high supportfor caring hinders a person to perform an action that will cause an other personsdeath even if the there is an action that logically has greater benefits.

6.3 Footbridge (Test 2)

Likewise for the footbridge case one cycle of computations are executed withdifferent MFT care score 1.0, 2.5 and 5.0 can be seen in table 6.2.

mft care foundation score 1.0 2.5 5.0Man not pushed 0.4612 0.4505 0.4331Man pushed 0.0558 -0.0481 -0.1984Difference 0.4054 0.4986 0.6315

Table 6.2: Result of footbridge test.

With different care score in the footbridge the action not to push the man receivesthe exact same activation as the action to not turn the switch in test 1, which isexpected, since this is the same series of events. The action to push the man downon the track receives a very low activation depicting the undesirability of this ac-tion. Since the difference is increasing with higher caring score since the morea person endorse that pillar the action to push the man becomes even less de-sired. And indeed that is also the case for the philosophical thought experiment,where the action to push themman received almost 27% [33] permissibility, totalnumber of votes casted for the footbridge case is over 20000 in the bbc poll.

These results obtain with our design are aligned towards the empirical results inthe experiments from psychology studies, this indicates that the design is moreor less accurate for these cases.

7Related work

There are numerous ways to implement social interactions and human cognition,this chapter contains three of the more interesting projects with similar agendas.Even though their goals are aligned towards similar problem descriptions eachproject has entirely different approaches. The first following the psychologicaltheory, while the second wants to apply a framework of general algorithms andthe last adopts full-fledged biological processes to mimicking the human mindin decision making.

7.1 Empathy in social agents

Ana Paiva and her group are studying, testing and implementing social agentsthat behave in a natural way, their focus is on simulating empathy [27]. Empathyis the process between two persons, where one person, in this case called observer,is perceiving the feelings of a second persons, called called target. According toPaiva [26], the empathic process consists of five key constructs, (1) the observer,(2) the subject, (3) the event cue, (4) the emotion and (5) the situation/context.Where they claim that the context is extremely important. Also that identify-ing with others is required for empathy to arise, and with a stronger/closer rela-tionship the empathic emotions increases. A definition for an empathic agent isstated in [26]:

Def: “Empathic agents” are agents that respond emotionally to situ-ations that are more congruent with the user’s or another agent’s sit-uation, or as agents, that by their design, lead users to emotionallyrespond to the situation that is more congruent with the agent’s situa-tion than the user’s one.

43

44 7 Related work

The same article [26] summarizes three distinct cases where the above constructsare been studied, (a) a game called FearNot! used for teaching children what ef-fects bulling can have on a person (b) emotional iCat whom can react and respondin an emotional way with verbal comments and facial expressions and (c) a pro-cess model used in computational interaction and simulations between artificialagents.

7.2 Modelling theory of mind

The work of Pynadath et al. has an interesting approach, since they are tryingto model general algorithms to induce complex behaviour. They have created amulti-agent framework called Thespian [32], which is built on PsychSim, whichis based on pomdp (Partially Observable Markov Decision Process). Where theagents are goal oriented. Usually computational implementations of emotionshave incorporated appraisal theory, instead this projects agenda desires to de-sign decision-theoretic algorithms to demonstrate human physiological and so-cial phenomena. The PsychSim framework has proven to model (a) wishfulthinking, (b) influence factors, (c) childhood aggression and (d) emotion [28].

As an example for their implementation they use the well know story of little redriding hood as scenario and context, where the four characters, the wolf, littlered riding hood, granny and the hunter interact. Each character acquires (a) astate, (b) dynamics, (c) goals, (d) beliefs(theory of mind), (e) policy and (f) socialrelationship [32].

These components are then used in the decision making process, where beliefsare actively updated by observations and expectations derived from the agentsown decision making. These beliefs are built up by mental models about theworld and characters in it. A look-ahead process evaluates the effect of each actionon the self, other affected agents and their actions effect in turn recursively withrespect to the agent’s goals, until a set boundary, much like how bn works.

7.3 LIDA

To my knowledge lida [5] is the most advanced cognitive model for simulatingvarious human mental faculties, their decision implementation is based on thesame theory as the behaviour module in this paper, specifically the action selec-tion.

lida stands for Learning Intelligent Distribution Agent, and is a working modelof cognition for complex environments. It has been designed as a computationalarchitecture from cognitive science and neuroscience perspectives. lida is de-rived from the system based on global work-space theory, who is actively "living"and working for the US Navy performing human tasks. The difference betweenthe two systems are three learning mechanism, perceptual learning, episodiclearning and procedural learning. In total there are six mechanisms which lida

7.3 lida 45

is comprised of, and the action selection is based on the same theory as to that ofthis paper, namely Patty Maes’ behaviour network.

1 Perception system makes sense of input information from the environmentand from the self-knowledge-base formulating connections, merges and changesfor recognition and thus understanding. They do this with symbol systems anda semantic network coupled with the memory.

2 Work-space is acting as a buffer for temporary structures and templates to bewritten and read. A part of the work-space is an interface for the episodicmemory.

3 Episodic memory is the memory with temporal and contextual association,that decays(tem). This mimics the human memory system by using patternsto evaluate where the information is located and to retrieve large amount ofinformation. A factual information storage is also modelled in conjunctionwith tem, these two models constructs the short and long term consciouslyrecallable memory.

4 Conscious mechanism implements four components. Small pieces of codethat run independent of each other and are activated under set conditions. Thecode-let with the highest activation is chosen by a so called spotlight routine,another routine broadcasts information from the work-space to all the code-lets. And a coalitions manager that keeps track of the code-lets activation level.

5 Procedural memory can be analogous to muscle memory, it to consists of anetwork like the perception system, however the procedural memory networkcontains schemes with an action, context and a result. The network is orderedso primitive schemes with no context or result resides at the edge.

6 Action selection is evaluated in a directed graph of behaviours, which are in-stantiated action schemes with it’s relations. There are three sources that gen-erate activation for the graph, pre-existing activation stored in the behaviours,from the environment through the perception, and from feelings and emotions.To be select-able the behaviour must be executable and have activation over athreshold, then the behaviour with the highest activation is selected.

These six mechanisms are are then linked together and a cyclic process runs ontop of them. The process consists of nine steps to be executed for each cycle,which are, (1) perceive (2) perception to pre-conscious buffer (3) local associa-tions (4) compete for consciousness (5) conscious broadcast (6) recruitment ofresources (7) setting goal context hierarchy (8) action chosen or action taken.

As the group states in their article [29], not only is their model useful for design-ing control systems for autonomous agents, but lida is also useful for generatingtestable conjectures about how the humanmind works, using a systems approachto cognition, including a cognitive cycle and a system of systems. lida can helpout in testing and verifying new theories of complex human behaviour.

The lida project is researching and trying to implement a framework of an ama

46 7 Related work

to make moral decision. It is meant to be a practical solution to a practical prob-lem, by taking in as much relevant information as possible to behave morally ina certain amount of time [34]. They are focusing on six areas, (1) how new val-ues are learned, (2) automatization, (3) deliberation, (4) the end of a deliberation,(5) imagination and (6) monitor successful decisions. Since there is no knownpart of the brain that is dedicated for morality, that we know of, the aim of theconceptual moral model of the lida project is to with the implemented mecha-nisms described in this section investigate above mentioned areas with moralityin mind.

8Discussion

8.1 Conclusions

From various articles resonates that emotions are the reflex system for social in-teraction with other entities, much like humans motoric reflexes for the physicalenvironment. The emotions arises without the need for cognitive thought, andwhen they reach a certain level influences your decisionmaking drastically whichin turn changes the behaviour, often undetected by the person that triggered theemotion [21]. This is why it is difficult to detect and synthesize these processes,however with better technology this can be achieved.

These theories that claim inheritance in human have been witnessed at a per-sonal level when observing a 10 month child. The child definitely displays anunderstanding of the situation and the concept of care, the primary foundationof morality. For example when someone is sad the child also becomes sad, thisis a display of empathy. When the child has done something that is commonlyknow to adults as bad behaviour, the child displays reluctant cautiousness as in, Iknow this is not right but I want to do this anyway, perhaps out of curiosity, sinceit’s a novel experience for them and it needs to be investigated by the child toobtain a better understanding. And in such cases when the child acts upon unde-sired behaviour the child displays shame and guilt from that wrongful act. Thesewrongful acts are overridden by the child’s desire of achieving a goal, the goalbeing as mentioned above to understand its environment. Which gives validityfor the inheritance of morality according to the research in this thesis.

Yes, it is possible to implement morality with emobn and the extension een. Ac-cording to the test results and comparison with the empirical results from thepsychology studies the design holds for these two cases of the trolley problem.

47

48 8 Discussion

We see problems in the thought experiment that could cause the activation result-ing values not to be as polar as the values from the psychology studies are. It isnot clear if the intention to create as similar cases as possible is what is intendedin the trolley problem. The different cases are not so similar that you can focus onjust one aspect of the dilemma, this gives rise to undesired variables that can af-fect the outcome, perhaps not inflict critical changes, but non the less create gapsand questions of uncertainty in the process of establishing the inner workings ofthe concept morality.

This design is an implicit top-down approach in the sense that there are no ruleset that bounds the process, however mft consists of a persons evaluated ethicsin the form of quantitative values, they form a constraint on the behaviour. Sincethe een is goal driven and emotional influenced the design also implements thebottom up approach. This design is by no means a fully defined and completecomputational model for simulating morality, it definitely needs more testingand development.

Both umg and Roseman’s appraisal model provide information on how the be-haviours and states will be weighed and what emotions are elicited. By lookingat the intentional structure diagram for the bystander in figure 4.4, it reveals thatthe observer is not in close proximity to the one person getting killed, this doesnot give a negative influence towards the goal. The figure 4.4 also shows that thestate where the behaviour to turn the trolley causes one person to get killed is aside effect and there is no physical contact with that person. This differ in thefootbridge case, where contact is taking place to achieve the same state, viewedin figure 4.5, thus battery is committed by the observer, and is no longer a side ef-fect, the outcome of the situation becomes a stronger reality in the footbridge caseand this will elicited a stronger negative emotional feeling. In contrast to the by-stander case where throwing the lever elicits a weaker and positive emotion sincebattery is caused by the trolley which is considered a side effect. However neitheris connected to morality according to Haidt’s moral emotional family grouping.

With a deeper understanding of morality also comes the ability to create agi-agents who can interact and understand the complex behaviour of humans, weatherit being in a game or in the real world, the underling processes are the same. Per-haps this thesis will spark some new perspectives to this complex problem.

8.2 Future work

Some general thoughts on what is needed for continued work on een develop-ment:

• change input at run time

• automate een generation

• access to knowledge representation or memory

• graphical feedback to better monitor the processes at run time

8.2 Future work 49

• debugger system for the moral activation

• vast amount of testing with new and more complex scenarios

To further establish if an outcome or action is in a moral context, we could makeuse of Haidt’s dictionary of moral words. This dictionary would be a referencedlook up table for comparing objects, actions and events and classify them if theyare used in a moral context.

Bibliography

[1] R. F. Baumeister, K. D. Vohs, C. N. DeWall, and L. Zhang. How EmotionShapes Behavior: Feedback, Anticipation, and Reflection, Rather Than Di-rect Causation, pages 167–203. 2007. Cited on pages 3 and 4.

[2] A. Damasio. Descartes’ Error: Emotion, Reason and the Human Brain. 1994.Cited on pages 3 and 4.

[3] F. de Waal, 2011 (accessed 25 August, 2014).http://www.ted.com/talks/frans_de_waal_do_animals_have_morals.Cited on pages 4 and 21.

[4] Diffen. Ethics vs. morals. (accessed 25 August, 2014).http://www.diffen.com/difference/Ethics_vs_Morals. Cited on page9.

[5] S. Franklin and J. F. G. Patterson. The lida architecture: Addingnew modes of learning to an intelligent, autonomous, software agent.Integrated Design and Process Technology, IDPT-2006, June 2006.http://ccrg.cs.memphis.edu/assets/papers/zo-1010-lida-060403.pdf. Citedon page 44.

[6] J. Greene and J. Haidt. How (and where) does moral judgment work? Trendsin cognitive sciences, 6(12):517–523, 2002. Cited on page 4.

[7] P. Grundström. Design and implementation of an appraisal module for vir-tual characters. diploma thesis, Linköping University, April 2012. Cited onpage 33.

[8] J. Haidt. The emotional dog and its rational tail: a social intuitionist ap-proach to moral judgment. Psychological review, 108(4):814, 2001. Citedon pages 4 and 26.

[9] J. Haidt. The moral emotions. Handbook of affective sciences, pages 852–870, 2003. Cited on pages 6, 7, 8, 9, and 26.

[10] J. Haidt. The new sience of morality. July 2010 (accessed 25 August,

51

52 Bibliography

2014). http://www.edge.org/conversation/a-new-science-of-morality-part-1. Cited on page 26.

[11] J. Haidt, 2012 (accessed 25 August, 2014).http://www.ted.com/talks/jonathan_haidt_humanity_s_stairway_to_self_transcendence.Cited on page 5.

[12] J. Haidt and C. Joseph. Intuitive ethics: How innately prepared intuitionsgenerate culturally variable virtues. Daedalus, 133(4):55–66, 2004. Citedon pages 4, 5, 21, and 26.

[13] J. Haidt, C. Joseph, et al. The moral mind: How five sets of innate intuitionsguide the development of many culture-specific virtues, and perhaps evenmodules. The innate mind, 3:367–391, 2008. Cited on pages 4, 9, 21, and 26.

[14] J. Haidt and S. Kesebir. Morality, chapter 22, pages 797–832. 2010. Citedon pages 4 and 6.

[15] M. D. Hauser. Moral minds: How nature designed our universal sense ofright and wrong. Ecco New York, 2006. Cited on page 42.

[16] G. Jesse, H. Jonathan, K. Sena, M. Matt, I. Ravi, W. S. P., and D. P.H. Moral foundations theory: The pragmatic validity of moral plural-ism. Advances in Experimental Social Psychology, Forthcoming, 2012.http://ssrn.com/abstract=2184440. Cited on pages xii, 4, 21, 22, 23, and 26.

[17] A. Johansson. Affective Decision Making in Artificial Intelligence : MakingVirtual Characters With High Believability. PhD thesis, Linköping Univer-sityLinköping University, Media and Information Technology, The Instituteof Technology, 2012. Cited on pages 10, 11, 12, and 29.

[18] A. Johansson and P. Dell’Acqua. Affective States in Behavior Networks. Intel-ligent Computer Graphics, Vol. 240, Studies in Computational Intelligence,chapter 2, pp. 19-39. Springer Berlin / Heidelberg, 2009, 2009. Cited onpages 2, 11, 14, and 17.

[19] P. Lankoski and A. Johansson. AI Design for Believable Characters viaGameplay Design Patterns, chapter 2, pages 15–33. 2011. Cited on page2.

[20] G. Loewenstein and J. S. Lerner. The Role of Affect in Decision Making.Oxford University Press: New York, 2003. Cited on pages xi, 2, and 10.

[21] P. Maes. How to do the right thing. Connection Science Journal, 1(3), 1989.Cited on pages 11 and 47.

[22] J. Mikhail. Universal moral grammar: Theory, evidence and the future.Trends in Cognitive Sciences, 11(4):143–152, 2007. Cited on pages xi, 24,26, and 27.

[23] J. Mikhail. Moral Cognition and Computational Theory, pages 81–92. Cam-

Bibliography 53

bridge, Mass.,: MIT Press, 2008. http://ssrn.com/abstract=1029511. Citedon page 26.

[24] B. Nebel and Y. Babovich-Lierler. When are behavior networks well-behaved? In Proceedings of 16th European Conference on Artificial In-telligence (ECAI), 2004. Cited on page 14.

[25] M. Otsuka. Double effect, triple effect and the trolley problem: Squaring thecircle in looping cases. Utilitas, 20(01):92–110, 2008. Cited on page 24.

[26] A. Paiva, C. M. Chiu, S.-S. Hung, J.-J. Tsay, N. Pelechano, B. Spanlang,A. Beacco, T. Jang, R. B. i Ribera, J. Bae, et al. Empathy in social agents. Inter-national Journal of Virtual Reality, 10(1), 2011. Cited on pages 43 and 44.

[27] A. Paiva, J. Dias, D. Sobral, R. Aylett, S. Woods, C. Zoll, and L. Hall. Caringfor agents and agents that care: Building empathic relations with syntheticagents. In AAMAS’2004 - International Conference on Autonomous Agentsand Multi-agent Systems. ACM Press, July 2004. Cited on page 43.

[28] D. V. Pynadath, M. Si, and S. C. Marsella. Modeling theory of mind andcognitive appraisal with decision-theoretic agents. In Appraisal, pages 1–30. Apr. 2011. http://ict.usc.edu/pubs/ModelingCited on page 44.

[29] U. Ramamurthy, B. Baars, S. K. . D’Mello, and S. Franklin. Lida: A workingmodel of cognition. In D. Fum, F. Del Missier, and A. Stocco, editors, Pro-ceedings of the 7th International Conference on Cognitive Modeling, pages244–249, 2006. Cited on page 45.

[30] I. J. Roseman. Appraisal determinants of emotions: Constructing a moreaccurate and comprehensive theory. Cognition amp; Emotion, 10(3):241–278, 1996. http://dx.doi.org/10.1080/026999396380240. Cited on pages xiand 34.

[31] . B. D. Shaver P. R., Hazan C. Love as attachment: The integration of threebehavioral systems, pages 68–99. Yale University Press, 1988. Cited onpages 3 and 4.

[32] M. Si, S. C. Marsella, and D. V. Pynadath. Modeling appraisal in theory ofmind reasoning. Autonomous Agents and Multi-Agent Systems, 20(1):14–31, 2010. http://dx.doi.org/10.1007/s10458-009-9093-x. Cited on page 44.

[33] D. Sokol. What if..., 2006 (accessed August 25, 2014).http://news.bbc.co.uk/1/hi/magazine/4954856.stm. Cited on page42.

[34] W. Wallach, S. Franklin, and C. Allen. A conceptual and computationalmodel of moral decision making in human and artificial agents. Topics inCognitive Science, 2:454–485, 2010. Cited on pages 9, 10, and 46.

Appendix

ACode

A.1 Bystander code

%%%%%%%%%%%%%% init.States(outcome)

state(save(5), 0.0).

state(kill(1), 0.0).



state(trolley_moving, 0.0).

state(trolley_turned, 0.0).

state(decision_made, 0.0).

%%%%%%%%%%%%%% init.Expected emotions

%energy sent into the decision model

expectedE(eagernessE, ’+’, 0.5).

expectedE(prideE, ’+’, 0.7).

expectedE(joyE, ’+’, 0.3).

expectedE(reliefE, ’+’, 0.3).

%%%%%%%%%%%%%% init.Expected Moral emotion

expectedE(disgustE, ’+’, 0.3).

expectedE(guiltE, ’+’, 0.7).

%%%%%%%%%%%%%% init.Goals integrated)

57

58 A Code

%%%%%%%%%%%%%% init.Behaviors

behavior(not_turned_switch,

[true],

[ (kill(5), 1, 0), (save(1), 1, 0),

(trolley_moving, 1, 0), (decision_made, 1, 0)],

[ ],

[ ]

).

behavior(turned_switch,

[true],

[ (save(5), 1, 0), (kill(1), 1, 0),

(trolley_turned, 1, 0), (decision_made, 1, 0)],

[ ],

[ ]

).

%%%%%%%%%%%% Weighting states(outcomes)

induce(decision_made, eagernessE, 0.5).

%weighting formula

%X + (1-X)P/Pm + (1-(X+(1-X)P/Pm))U/Um

%%%------------not thrown branch

%care = 5.0, 0.999

%care = 2.5, 0.958

%care = 1.0, 0.933

release(kill(5), disgustE, 0.958).

% 0.5 + (1-0.5)1/6

induce(save(1), joyE, 0.583).

induce(trolley_moving, reliefE, 0.5).

induce(not_turned_switch, reliefE, 0.5).

%%%------------thrown branch

% 0.5 + (1-0.5)1.0/5.0

induce(save(5), prideE, 0.916).

%care = 5.0, 0.999

%care = 2.5, 0.792

%care = 1.0, 0.666

release(kill(1), guiltE, 0.792).

induce(trolley_turned, prideE, 0.5).

induce(turn_switch, prideE, 0.5).

%%%%%%%%%%%% init.Emotions

A.1 Bystander code 59

%%%------------positiv emotions

emotion(pride, 0.0).

emotion(joy, 0.0).

emotion(relief, 0.0).

%%%------------negative emotions

emotion(disgust, 0.0).

emotion(guilt, 0.0).

pos_emo(PosEmo) :-

emotion(pride, PA),

emotion(joy, PB),

emotion(relief, PC),

PosEmo is (PA+PB+PC)/3,

!.

neg_emo(NegEmo) :-

emotion(disgust, NA),

emotion(guilt, NB),

NegEmo is (NA+NB)/2,

!.

%%%------------needed to load

items([]).

60 A Code

A.2 Footbridge code

%%%%%%%%%%%%%% init.States(outcome)





state(trolley_moving, 0.0).

state(trolley_turned, 0.0).

state(decision_made, 0.0).

%%%%%%%%%%%%%% init.Expected emotions

%energy sent into the decision model

expectedE(eagernessE, ’+’, 0.5).

expectedE(prideE, ’+’, 0.7).

expectedE(joyE, ’+’, 0.3).

expectedE(reliefE, ’+’, 0.3).

%%%%%%%%%%%%%% init.Expected Moral emotion

expectedE(disgustE, ’+’, 0.3).

expectedE(regretE, ’+’, 0.7).

expectedE(guiltE, ’+’, 0.7).

%%%%%%%%%%%%%% init.Goals integrated)

%%%%%%%%%%%%%% init.Behaviors

behavior(not_turned_switch,

[true],

[ (kill(5), 1, 0), (save(1), 1, 0),

(trolley_moving, 1, 0), (decision_made, 1, 0)],

[ ],

[ ]

).

behavior(turned_switch,

[true],

[ (save(5), 1, 0), (kill(1), 1, 0),

(trolley_turned, 1, 0), (decision_made, 1, 0)],

[ ],

[ ]

).

A.2 Footbridge code 61

%%%%%%%%%%%% Weighting states(outcomes)

induce(decision_made, eagernessE, 0.5).

%weighting formula

%X + (1-X)P/Pm + (1-(X+(1-X)P/Pm))U/Um

%%%------------not thrown branch

%care = 5.0, 0.999

%care = 2.5, 0.958

%care = 1.0, 0.933

release(kill(5), disgustE, 0.958).

% 0.5 + (1-0.5)1/6

induce(save(1), joyE, 0.583).

induce(trolley_moving, reliefE, 0.5).

induce(not_turned_switch, reliefE, 0.5).

%%%------------thrown branch

% 0.5 + (1-0.5)1.0/5.0

induce(save(5), prideE, 0.916).

%care = 5.0, 0.999

%care = 2.5, 0.792

%care = 1.0, 0.666

release(kill(1), guiltE, 0.792).

release(trolley_turned, regretE, 0.5).

release(turn_switch, regretE, 0.5).

%%%%%%%%%%%% init.Emotions

%%%------------positiv emotions

emotion(pride, 0.0).

emotion(joy, 0.0).

emotion(relief, 0.0).

%%%------------negative emotions

emotion(disgust, 0.0).

emotion(guilt, 0.0).

pos_emo(PosEmo) :-

emotion(pride, PA),

emotion(joy, PB),

emotion(relief, PC),

PosEmo is (PA+PB+PC)/3,

!.

62 A Code

neg_emo(NegEmo) :-

emotion(disgust, NA),

emotion(guilt, NB),

NegEmo is (NA+NB)/2,

!.

%%%------------needed to load

items([]).

Upphovsrätt

Detta dokument hålls tillgängligt på Internet — eller dess framtida ersättare —under 25 år från publiceringsdatum under förutsättning att inga extraordinäraomständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner,skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för icke-kommersiell forskning och för undervisning. Överföring av upphovsrätten viden senare tidpunkt kan inte upphäva detta tillstånd. All annan användning avdokumentet kräver upphovsmannens medgivande. För att garantera äktheten,säkerheten och tillgängligheten finns det lösningar av teknisk och administrativart.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsmani den omfattning som god sed kräver vid användning av dokumentet på ovanbeskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådanform eller i sådant sammanhang som är kränkande för upphovsmannens litteräraeller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press se förla-gets hemsida http://www.ep.liu.se/

Copyright

The publishers will keep this document online on the Internet — or its possi-ble replacement — for a period of 25 years from the date of publication barringexceptional circumstances.

The online availability of the document implies a permanent permission foranyone to read, to download, to print out single copies for his/her own use andto use it unchanged for any non-commercial research and educational purpose.Subsequent transfers of copyright cannot revoke this permission. All other usesof the document are conditional on the consent of the copyright owner. Thepublisher has taken technical and administrative measures to assure authenticity,security and accessibility.

According to intellectual property law the author has the right to be men-tioned when his/her work is accessed as described above and to be protectedagainst infringement.

For additional information about the Linköping University Electronic Pressand its procedures for publication and for assurance of document integrity, pleaserefer to its www home page: http://www.ep.liu.se/

© Henry Fröcklin

Documents

Beräkningsmodell för moral och emotioner i EmoBN