Emergence of Collective Behavior and AI: Intention ... · Evolution of Cooperative Behavior as AI Research Evolution of Cooperation Cognitive Modeling Mind-reading, Intention, Belief,

Introduction Intention Commitment Apology Conclusions

Emergence of Collective Behavior and AI:Intention Recognition, Commitment and Apology

The Anh HanArtificial Intelligence Lab, Vrije Universiteit Brussel

Guest Lecture, Current Trends in Artificial Intelligence,VUB/ULB

April 25, 2014

1 / 69


Outline

1. Introduction• Explaining the evolution of cooperation (state-of-the-art and

methods)• Evolution of cooperative behavior in AI research

2. Evolution of Cooperation• Intention recognition• Commitment (pairwise and group interactions)• Apology

2 / 69


What is Cooperation?

• pay a cost for someone else to receive a benefit.

• Cooperation is widespread: insects, hunter-gatherer societies,team work, international relationships, etc.

3 / 69


Paradox of cooperation

If natural selection is based on competition, how can it lead to cooperation?

• Natural selection: only the fittestsurvive

• Everyone wants to increase theirfitness.

• No one wants to pay the cost,happily accepting benefits.

• Dilemma: everyone would bebetter off cooperating with eachother (benefit > cost).

4 / 69


Interdisciplinary research

• Understanding the evolution of cooperation remains afundamental challenge, for scientists from fields likeevolutionary biology, physics, economics, mathematics,computer science & AI1, etc.

• Many fields ... same math!!

• Mathematical Framework: Evolutionary Game Theory &Agent-based Simulations

• Metaphors: Prisoner’s dilemma, Public Goods Game,Ultimatum game, etc.

1IJCAI, AAMAS, CEC, GECCO, ALIFE, COGNITION5 / 69


Interdisciplinary research

• Understanding the evolution of cooperation remains afundamental challenge, for scientists from fields likeevolutionary biology, physics, economics, mathematics,computer science & AI1, etc.

• Many fields ... same math!!

• Mathematical Framework: Evolutionary Game Theory &Agent-based Simulations

• Metaphors: Prisoner’s dilemma, Public Goods Game,Ultimatum game, etc.

1IJCAI, AAMAS, CEC, GECCO, ALIFE, COGNITION6 / 69


Evolutionary Game Theory

payoff → fitness → social success

Individuals with higher fitness will reproduce more

genetic evolution

cultural/social evolution

OR their behavior will be imitated more often

7 / 69


Imitation and Reproduction Processes

• Players with lowest fitness imitate or are replaced by the oneswith highest fitness in the population.

• Imitation/replacement probability: player B imitates playerA’s strategy with probability

PB→A =[1 + e−β(fA−fB)

]−1

• fA and fB are the payoffs (fitnesses) of A and B;• β is called imitation strength, which controls how strong the

payoffs play a role in the imitation decision;

8 / 69


Imitation and Reproduction Processes

• Players with lowest fitness imitate or are replaced by the oneswith highest fitness in the population.

• Imitation/replacement probability: player B imitates playerA’s strategy with probability

PB→A =[1 + e−β(fA−fB)

]−1

• fA and fB are the payoffs (fitnesses) of A and B;• β is called imitation strength, which controls how strong the

payoffs play a role in the imitation decision;

9 / 69


Evolutionary Game Theory

payoff → fitness → social success

Natural selection leads to the destruction ofcooperation (tragedy of commons).

10 / 69


Mechanisms of cooperation a

• kin selection: the more individuals arerelated, the more cooperation is feasible

• direct reciprocity: repeated interactions,memory

• indirect reciprocity: reputation basedstrategies

• group selection, structured population,punishment, etc.

aM. Nowak. Five rules for the evolution of cooperation. Science, 2006

11 / 69


Mechanisms of cooperation a

• kin selection: the more individuals arerelated, the more cooperation is feasible

• direct reciprocity: repeated interactions,memory

• indirect reciprocity: reputation basedstrategies

• group selection, structured population,punishment, etc.

aM. Nowak. Five rules for the evolution of cooperation. Science, 2006

12 / 69


Where is cognition?

• Black-or-white, simple strategies.

• Humans (and other species) arecognitively skillful: mind reading,intention, learning, negotiation,commitment, anticipation, etc.

• What are the roles of cognition inthe evolution of cooperation?

• Little attention here!!!

13 / 69


Where is cognition?





14 / 69


Where is cognition?





15 / 69


Where is cognition?





16 / 69


Evolution of Cooperative Behavior as AI Research

Evolution of Cooperation

Cognitive ModelingMind-reading, Intention,

Belief, Reasoning, Learning, Prediction

Multi-agent SystemsCollaborative, Self-

organized, Decentralized systems; Markets design

Robotics Swam intelligence, Self-organization,

Coordination

Network ScienceSocial networks,

Cooperation in network, Spatial systems

17 / 69


Intention reading promotes cooperation

• AI cognitive modeling combinedwith Evolutionary Game Theorymodeling.

• Recognizing intention helpschoosing cooperative partners andavoiding free-riders and exploiters.

• Recognizing intentions enablesstrategies with shorter memories.

18 / 69


Intention recognition research in AI

Set of Intentions

Plan Library Plan CorpusAction Theory

A1 A2 A3 ....... Model-Bayesian KB-Markov KB

....

Sequence of Observed Actions

Recognized Intentions

With Rankings (Probabilistic) Without Rankings (Non-probabilistic)

Observed Agent

19 / 69


Intention recognition promotes emergence of cooperationin repeated interactions

Intention recognizers can effectively detect (using BayesianNetwork inference)

1. ‘good’ and ‘bad’ 2

2. ‘unconditionally good’ , ‘conditionally good’, and ‘bad’ 3 4

Intention recognizers quickly learn to cooperate with eachother.

2Han, Pereira, Santos. Intention recognition promotes emergence of cooperation. Adaptive Behavior, 2011

3Han, Pereira, Santos. Corpus-based intention recognition in cooperation dilemmas. Artificial Life, 2012

4Han, Pereira, Santos. The role of intention recognition in evolution of cooperative behavior. IJCAI, 2011

20 / 69










21 / 69










22 / 69


Why arrange commitments?

• Sometimes difficult to predict others’ behavior orrecognize their intentions with sufficient confidence.

• Commitment proposal can help clarify intentions of others.• contracts, marriage, apartment rental, etc.

23 / 69


Why arrange commitments?

• Sometimes difficult to predict others’ behavior orrecognize their intentions with sufficient confidence.

• Commitment proposal can help clarify intentions of others.• contracts, marriage, apartment rental, etc.

24 / 69


Prisoner’s Dilemma (PD)

✓ C D

C R, R S, TD T, S P, P

◆• C: cooperate; D: Defect

• T > R > P > S

• T = temptation to defect

• R = reward

• P = punishment

• S = sucker’s payoff

• D is always the best individual’schoice but everyone plays C is thebest outcome for the population.

25 / 69


Prisoner’s Dilemma (PD)

✓ C D

C R, R S, TD T, S P, P

◆• C: cooperate; D: Defect

• T > R > P > S

• T = temptation to defect

• R = reward

• P = punishment

• S = sucker’s payoff

• D is always the best individual’schoice but everyone plays C is thebest outcome for the population.

26 / 69


Commitment strategy (COMP)

When meeting a co-player (to play PD)

• proposes its co-player to commit tocooperate.

• if accepted,• Pay the cost ε of setting up the

commitment.• Cooperate in the game and receive

payoff.• If opponent defects, get

compensation δ.

• if not accepted• Don’t play the game, hence 0 reward.

27 / 69


What are the strategies of the co-players

• Cooperator (C): always acceptscommitment.

• Defector (D): never acceptscommitment.

• Fake committer (FAKE): acceptscommitment, yet defects in game.

• Commitment free-rider (FREE): accepts commitment andcooperates, yet defects when receiving no commitmentproposal.

There are other potential strategies yet they are all domi-nated by one of these.

28 / 69


What are the strategies of the co-players

• Cooperator (C): always acceptscommitment.

• Defector (D): never acceptscommitment.

• Fake committer (FAKE): acceptscommitment, yet defects in game.

• Commitment free-rider (FREE): accepts commitment andcooperates, yet defects when receiving no commitmentproposal.

There are other potential strategies yet they are all domi-nated by one of these.

29 / 69


Commitment strategy

Payoff matrix for the five strategies

COMP C D FAKE FREE

COMP R − ε/2 R − ε 0 S + δ − ε R − εC R R S S SD 0 T P P PFAKE T − δ T P P PFREE R T P P P

.

30 / 69


Viability of COMP

• Risk-dominance: indicates whether a strategy isviable with respect to the other strategies.

• Consider any two strategies A and B, A is said to berisk dominant against B when

πA,A + πA,B > πB,A + πB,B

where πX ,Y is the (average) payoff of strategist Xwhen interacting with strategist Y.

31 / 69


Viability of COMP

For which cost ε and compensation δ COMP is viable againstD, FAKE and FREE

1. ε < min {2(R − P), 2(R − P)/3}2. δ > (T − R + P − S)/2 + 3ε/4.

Simplifying: T = b,R = b − c ,P = 0 and S = −c

1. ε < 2(b − c)/3: The cost should be smaller than the benefitof cooperation.

2. δ > c + 3ε/4: The compensation should be sufficiently largewith respect to the cost of cooperation and of commitment.

32 / 69


Viability of COMP

For which cost ε and compensation δ COMP is viable againstD, FAKE and FREE

1. ε < min {2(R − P), 2(R − P)/3}2. δ > (T − R + P − S)/2 + 3ε/4.

Simplifying: T = b,R = b − c ,P = 0 and S = −c

1. ε < 2(b − c)/3: The cost should be smaller than the benefitof cooperation.

2. δ > c + 3ε/4: The compensation should be sufficiently largewith respect to the cost of cooperation and of commitment.

33 / 69


Viability of COMP

COMP69%

C5%

D13%

FAKE3%

2.0ρN

9.5ρN

9.5ρN

24.1ρN

FREE10%

9.5ρN

5.6ρN

2.4ρN

Figure: Stationary distributions for a viable solution:δ = 4, ε = 0.25, b = 2, c = 1.

34 / 69


Viability of COMP

10

δ

ÚÚÚÚÚÚÚÚÚ

Ú

Ú

Ú

ÚÚÚÚÚÚÚÚÚ Ú Ú Ú Ú Ú Ú Ú Ú Ú

Ê Ê Ê Ê ÊÊ Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê

‡ ‡‡ ‡

‡ ‡‡ ‡

‡ ‡‡ ‡ ‡

‡ ‡ ‡ ‡ ‡‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡

‡ ‡ ‡ ‡

Ï Ï Ï Ï Ï Ï ÏÏ Ï Ï Ï Ï Ï Ï Ï Ï Ï

Ï Ï Ï Ï Ï Ï Ï Ï Ï Ï Ï Ï ÏÙ ÙÙ Ù

ÙÙÙÙÙÙÙÙÙÙÙÙÙÙÙÙ Ù

Ù ÙÙ Ù Ù Ù Ù Ù

Ù

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.40.0

0.2

0.4

0.6

0.8 b

Freq

uenc

y

a

COMP C FAKED FREE

COMS CS FAKS FRES

0.84

0.650.18

0.39

0.12

0.21

0.51

0.6

0.75

0.36

0.03

1

0

2

3

4

5

6

0.0 0.2 0.4 0.0

0.2

0.4

0.6

0.8

0.0 0.2 0.4 1.2

ε ε

35 / 69


Sharing the cost?

dC 0.78

0.73

0.56

0.63

0.42

0.28

0.14

0.0 0.5 1.0 1.5 0.0 0.2 0.4 0.6 0.8

1

0

2

3

4

5

6

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

2.0

c

ε ε

Freq

uenc

y

δ

1. ε < 2(b − c)/3→ ε < 2(b − c): higher costs allowed

2. δ > c + 3ε/4→ δ > c + ε/4: smaller compensations possible

36 / 69


Conclusions: commitments 5 6

• When the cost of commitment is justified with respect to thebenefit of the cooperation and the compensation is sufficientlyhigh, cooperation can thrive.

• Sharing costs leads to even better friends; cooperation for awider range of parameter values

5Han, Pereira, Santos. Emergence of commitment and cooperation. AAMAS, 2012.

6Han, Pereira, Santos, and Lenaerts. Good agreements make good friends. Nature Scientific Reports, 2013.

37 / 69


Why apologize?

• Sincere apology can resolve conflicts and misunderstandings.

• Involving external parties (e.g. courts) may cost all sides of aconflict significantly more.

38 / 69


Why to apologize not?

• Sincere, acceptable apology may be too costly (e.g. losingface, reputation).

• No sufficient incentive to apologize (e.g. commitment-freeinteractions).

39 / 69


Questions to be asked about apology

• We build computational models to study whether (and how)explicit apology can lead to the emergence of cooperation:

• Would apology need to be sincere (to function properly)?

• May apology lead to high levels of cooperation by itself?

• Abundance and sincerity of apology in committedrelationships vs. in commitment-free relationships?

40 / 69


Apology in repeated games under noise

• Typically (sincere) apology occurs in repeated interactionsettings, to ensure future long-term cooperation.

• Apology is usually modeled implicitly as one or morecooperative acts after a wrongful defection due to noise.

• ’I hit you once, you are allowed to hit me back’.

TFT (tit-‐for-‐tat): C C *D C D ...TFT (tit-‐for-‐tat): C C C D C ...

Noise

41 / 69


Apology in repeated games under noise

• Typically (sincere) apology occurs in repeated interactionsettings, to ensure future long-term cooperation.

• Apology is usually modeled implicitly as one or morecooperative acts after a wrongful defection due to noise.

• ’I hit you once, you are allowed to hit me back’.

TFT (tit-‐for-‐tat): C C *D C D ...TFT (tit-‐for-‐tat): C C C D C ...

Noise

42 / 69


Explicit Apology

• It is more natural to explicitly apologize to resolve conflictbefore the next interaction occurs.

Apologizer: C C *D C C ...Apologizer: C C C C C ...

I am so sorry

Noise

43 / 69


Models: Apology with and without Commitments

• COMA: propose commitments and apologize when makingmistake.

• If the co-player commits, interacts; otherwise, no interaction.• When the co-player commits, but defects and does not

apologize, ends the relationship.

• AP: pure apologizers, do not arrange commitments.

• Defecting strategies (always defect when playing)• AllD (pure defectors): don’t commit when being asked to.• FAKA (fake apologizers): commit, but don’t apologize.• FAKC (fake committers): commit, and apologize.

44 / 69








45 / 69








46 / 69


Key parameters

• Apology cost (γ): Apologizer compensates co-player after adefection (the higher the more sincere).

• Arrangement cost (ε): Commitment proposer has to pay anarrangement cost.

• Compensation cost (δ): Committed defaulting player has tocompensate the non-defaulting one.

47 / 69


Key parameters

• Apology cost (γ): Apologizer compensates co-player after adefection (the higher the more sincere).

• Arrangement cost (ε): Commitment proposer has to pay anarrangement cost.

• Compensation cost (δ): Committed defaulting player has tocompensate the non-defaulting one.

48 / 69


Apology is more frequent in committed relationships

0.83

0.81

0.750.62

0.3

0.54

0.46

0.33

0.160.040.1

Apology With Commitment (COMA)

b/c

apology cost/sincerity level (ɣ)

1

2

3

4

5

6

7

1

2

3

4

5

6

7

bene

fit-to

-cos

t rat

io (b

/c)

Apology Without Commitment (AP)

1

0

49 / 69


Apology needs to be sincere to function properly

0.83

0.81

0.750.62

0.3

0.54

0.46

0.33

0.160.040.1

Apology With Commitment (COMA)

b/c

apology cost/sincerity level (ɣ)

1

2

3

4

5

6

7

1

2

3

4

5

6

7

bene

fit-to

-cos

t rat

io (b

/c)

Apology Without Commitment (AP)

1

0

50 / 69


Commitments bring about sincerity

COMA vs. AP

Ú

Ú

Ú Ú

Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú

Ê

Ê Ê Ê

Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê

2 4 6 8 101

2

3

4

5

6

7

Fraction Ú

ÚÚ Ú Ú

Ú Ú Ú Ú Ú Ú Ú Ú ÚÚ Ú Ú Ú Ú

ÊÊÊÊÊÊÊ ÊÊ Ê Ê

Ê Ê Ê ÊÊ Ê Ê Ê

2 4 6 8 100.0

0.2

0.4

0.6

0.8

optim

al a

polo

gy c

ost

b/c

COMA

AP

b/c

1

2

3

4

5

6

7

2 4 6 8 10

51 / 69


Apology supported by commitment (COMA) prevails

ÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏ

ÁÁÁÁÁÁÁ

ÁÁÁÁÁÁÁÁÁÁÁÁÁ

Á

·············

········

Û Û Û Û ÛÛ Û Û Û Û Û Û

Û Û Û Û Û ÛÛ Û ÛÚ Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú

‡

‡‡‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡

ÙÙÙ Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù

ÊÊÊÊÊ ÊÊ Ê Ê

Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê Ê

0.00 0.05 0.10 0.15 0.200.0

0.1

0.2

0.3

0.4

0.5

ÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏ

ÁÁ ÁÁ ÁÁ ÁÁ ÁÁ Á Á

Á Á Á ÁÁ Á Á Á Á

·· ·· ·· ·· ·· · ·

· · · ·· · · · ·

ÛÛ ÛÛ ÛÛ ÛÛ ÛÛ Û Û

Û Û Û ÛÛ Û Û Û Û

Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú Ú

‡

‡‡‡‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡

Ù ÙÙÙÙ Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù Ù

0.00 0.05 0.10 0.15 0.200.0

0.1

0.2

0.3

0.4

0.5

Fraction

noise level (𝞪)

COMA AllC AllD FAKA

TFTFAKC

WSLSGTFT

COMA wins Without COMA Defectors win

52 / 69


Viability of COMA

• They can avoid being exploited by AllD, winning when

ε <2(1− 2α)

1− ω(b − c)

• They can get rid of fake committers FAKC, winning when

γ >3(1− ω)

4(1− 2α)ε+ c

• They can get rid of fake apologizers FAKA, winning when

δ > a2γ + a1ε+ a0

• Parameters: c - cost of cooperation; b - benefit cooperation;α - noise level; ω: probability that next round occurs.

53 / 69


Conclusions: apology and commitment 7

• Apology needs to be sincere to function properly, whether in acommitted or commitment-free relationship.

• Apology supported by commitment promotes high levels ofcooperation.

• Commitments bring about sincerity in apology.

7Han, Pereira, Santos, Lenaerts. Why is it so hard to apologize? IJCAI’2013.

54 / 69







55 / 69







56 / 69


Conclusions

• Intention recognition promotes cooperation in repeatedgames.

• Enable to choose cooperative partners and avoid free-riders.• Dealing well with noise, hence high level of cooperation.

• Arranging commitments provides incentive to cooperateeven in one-shot interactions

• Compensation must be sufficiently high compared to costs.• Shared cost is even better.

• Apology promotes cooperation if sincere• More efficient with commitment.• Commitment brings about sincerity.

57 / 69


Conclusions






58 / 69


Conclusions






59 / 69


Potential directions

• Other cognitive abilities?

• Any wisdom?

• AI modeling & build EGT models → Agent-based simulations.

60 / 69




• Any wisdom?


61 / 69




• Any wisdom?


62 / 69


Potential directions – Master Theses??

AI Cognitive Modeling Techniques Agent-based simulation & EGT

Theory-of-mind Preference & Belief Formation

Anticipation & prediction Reasoning (abduction, reduction, ...)

63 / 69


Cooperators

AI lab, Computer science department, Vrije Universiteit Brussel, Belgium

Tom Lenaerts Luis Martinez

Jean-Sebastian

David Catteeuw

Machine Learning Group, Université Libre de Bruxelles

Ioannis Zisis

Luis Moniz Pereira, AI Centrer, Universidade Novade Lisboa, Portugal

Francisco C. Santos INESC-ID and Instituto Superior Técnico, TU Lisbon, Portugal

64 / 69


Thank you!

QUESTIONS?

65 / 69


Replicator Equation

• Populations are infinite;

• There is a fraction xC of Cs;

• hence, xD = 1− xC is the faction of Ds.

• Populations are well-mixed: everybody is equally likely tointeract with everybody else.

• Evolution dynamics is determined by replicator dynamics:strategies’ evolution follow the gradient of natural selectiondetermined by relative fitness.

66 / 69


Replicator Equations

Replicator equations for two strategies (C and D){

˙xC = xC

(fC (−→x )− φ(−→x )

)

˙xD = xD

(fD(−→x )− φ(−→x )

)

Those strategies whose fitness f (reproductive success) exceeds theaverage fitness φ of the population will increase in frequency; thosethat don’t will decline.

Replicator equations for n strategies

xi = xi

(fC (−→x )− φ(−→x )

), φ(−→x ) =

n∑

j=1

xj fj(−→x )

where xi is the fraction of xi in the population; −→x = (x1, ..., xn);fi (−→x ) is the fitness of strategy i ; φ(−→x ) is the average population

fitness.

67 / 69


Future directions – Master Theses??

1. How mind-reading can help resolve the dilemmas ofcooperation and coordination between self-interested agents?

2. What kind of preferences is more relevant in a given context,e.g. preferring to interact with a greedy/unfair family memberor a generous/fair stranger?

3. How do adaptive aspirations in choosing partners influence thedynamics of cooperation and competition?

4. How to infer relevant patterns of behavior from largebehavioral experimental data where human participantsinteract with each other on a network?

68 / 69


Han, T. A., Pereira, L. M., and Santos, F. C. 2011a.Intention recognition promotes the emergence of cooperation.Adaptive Behavior, 19(3):264–279.

Han, T. A., Pereira, L. M., and Santos, F. C. 2011b.The role of intention recognition in the evolution ofcooperative behavior.In Walsh, T., editor, Proceedings of the 22nd internationaljoint conference on Artificial intelligence (IJCAI’2011), pages1684–1689. AAAI.

Han, T. A., Pereira, L. M., and Santos, F. C. 2012.Corpus-based intention recognition in cooperation dilemmas.Artificial Life journal, 18(4):365–383.

69 / 69

Documents

Emergence of Collective Behavior and AI: Intention ... · Evolution of Cooperative Behavior as AI Research Evolution of Cooperation Cognitive Modeling Mind-reading, Intention, Belief,