Krishnendu Chatterjee1 Graph Games with Reachabillity Objectives: Mixing Chess, Soccer and Poker Krishnendu Chatterjee 5 th Workshop on Reachability Problems,

Krishnendu Chatterjee 1

Graph Games with Reachabillity Objectives: Mixing Chess, Soccer and Poker

Krishnendu Chatterjee

5th Workshop on Reachability Problems,

Genova, Sept 30, 2011


Games on Graphs

Games on graphs.

History Zermelo’s theorem about Chess in 1913 From every configuration

Either player 1 can enforce a win. Or player 2 can enforce a win. Or both players can enforce a draw.


Chess: Games on Graph

Chess is a game on graph. Configuration graph.


Graphs vs. Games

Two interacting players in games: Player 1 (Box) vs Player 2 (Diamond).


Game Graph


Game Graphs

A game graph G= ((S,E), (S1, S2)) Player 1 states (or vertices) S1 and similarly player 2

states S2, and (S1, S2) partitions S.

E is the set of edges. E(s) out-going edges from s, and assume E(s) non-

empty for all s.

Game played by moving tokens: when player 1 state, then player 1 chooses the out-going edge, and if player 2 state, player 2 chooses the outgoing edge.


Game Example


Game Example


Game Example


Strategies

Strategies are recipe how to move tokens or how to extend plays. Formally, given a history of play (or finite sequence of states), it chooses a probability distribution over out-going edges. ¾: S* S1 D(S).

¼: S* S2 ! D(S).


Strategies Strategies are recipe how to move tokens or how to extend plays. Formally, given a

history of play (or finite sequence of states), it chooses a probability distribution over out-going edges.

¾: S* S1 ! D(S).

History dependent and randomized.

History independent: depends only current state (memoryless or positional). ¾: S1 ! D(S)

Deterministic: no randomization (pure strategies). ¾: S* S1 ! S

Deterministic and memoryless: no memory and no randomization (pure and memoryless and is the simplest class).

¾: S1 ! S

Same notations for player 2 strategies ¼.


Objectives

Objectives are subsets of infinite paths, i.e., Ã µ S!.

Reachability: there is a set of good vertices (example check-mate) and goal is to reach them. Formally, for a set T if vertices or states, the objective is the set of paths that visit the target T at least once.


Applications: Verification and Control of Systems

Verification and control of systems

Environment

Controller

M satisfies property (Ã)

E

C


Applications: Verification and Control of Systems

Verification and control of systems

Question: does there exists a controller that against all environment ensures the property.

M satisfies property (Ã)EC || ||


-synthesis [Church, Ramadge/Wonham, Pnueli/Rosner]

-model checking of open systems

-receptiveness [Dill, Abadi/Lamport]

-semantics of interaction [Abramsky]

-non-emptiness of tree automata [Rabin, Gurevich/ Harrington]

-behavioral type systems and interface automata [deAlfaro/ Henzinger]

-model-based testing [Gurevich/Veanes et al.]

-etc.

Game Models Applications


Reachability Games

Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X.

X

T


Reachability Games


X

T


Reachability Games


X

T


Reachability Games


Fix-point

X

T


Reachability Games

Winning set for a partition: Determinacy Player 1 wins: then no matter what player 2 does,

certainly reach the target. Player 2 wins: then no matter what player 1 does, the

target is never reached.

Memoryless winning strategies.

Can be computed in linear time [Beeri 81, Immerman 81].


Chess Theorem

Zermelo’s Theorem

Win1Win2

Both draw


Game Graphs Till Now

Game graphs we have seen till now

Many rounds (possibly infinite).

Turn-based.


Simultaneous Games

Theory of rational behavior as game theory von Neumann- Morgenstern games Matrix zero-sum games

R P S

R (0,0) (-1,1) (1,-1)

P (1,-1) (0,0) (-1,1)

S (-1,1) (1,-1) (0,0)


Simultaneous Games

Theory of rational behavior as game theory von Neumann- Morgenstern games Matrix zero-sum games

R P S

R (0,0) (-1,1) (1,-1)

P (1,-1) (0,0) (-1,1)

S (-1,1) (1,-1) (0,0)


Simultaneous Games

Example: Prisoners dilemma. Another example.

R L C

R (1,-1) (-1,1) (-1,1)

L (-1,1) (1,-1) (-1,1)

C (-1,1) (-1,1) (1,-1)


Simultaneous Games

Example: Prisoners dilemma. Another example.

R L C

R (1,-1) (-1,1) (-1,1)

L (-1,1) (1,-1) (-1,1)

C (-1,1) (-1,1) (1,-1)


Simultaneous Games

Another example: Penalty shoot-out (Soccer)

R L C

R (1,-1) (-1,1) (-1,1)

L (-1,1) (1,-1) (-1,1)

C (-1,1) (-1,1) (1,-1)


Chess Vs. Soccer (Penalty)

Chess: Turn-based Possibly infinite rounds

Theory of simultaneous games (Soccer) Concurrent One-shot (one-round)

Mix chess and soccer Concurrent games on graphs


Mixing Chess and Soccer: Concurrent Graph Games


Concurrent Game Graphs

A concurrent game graph is a tuple G =(S,M,¡1,¡2,±)

• S is a finite set of states.

• M is a finite set of moves or actions.

• ¡i: S ! 2M n ; is an action assignment function that assigns the non-empty set ¡i(s) of actions to player i at s, where i 2 {1,2}.

• ±: S £ M £ M ! S, is a transition function that given a state and actions of both players gives the next state.


An Example: Snow-ball Game

s Rrun, waithide, throw

hide, wait

run, throw[Everett 57]

Run

Hide

Throw Wait


New Solution Concepts

Sure winning for turn-based.

New solution concepts

Almost-sure winning.

Limit-sure winning.


Almost-sure Winning Example

s Rhead, headtail, tail

head, tailtail, head

Almost-sure winning strategy: say head and tail with probability ½.Randomization is necessary.


Concurrent reachability games: limit-sure


hide, wait

run, throw[Everett 57]

Move Probabilityrun qhide 1-q (q>0)

Win at s with probability1-q, for all q > 0.

Run

Hide

Throw Wait


Concurrent reachability games: limit-sure


hide, wait

run, throw

Run

Hide

Throw Wait

[Everett 57]

Move Probabilityrun qhide 1-q (q>0)

Win at s with probability1-q, for all q > 0.

w = 0 1 1

Player 1 cannot achieve w(s) = 1, only w(s) = 1-q for all q > 0.


Results for Concurrent Reachability Games

Sure winning: Deterministic memoryless sufficient. Linear time.

Almost-sure winning: Randomization is necessary. Randomized memoryless is sufficient. Quadratic time algorithm.

Limit-sure winning: Randomization is necessary. Randomized memoryless is sufficient. Quadratic time algorithm.

Results from [dAHK98, CdAH06, CdAH09]


Games Till Now

Turn-based graph games

Concurrent graph games Applications: again verification and synthesis with

synchronous interaction.

Both these games are perfect-information games. Players know the precise state of the game.

The game of Poker: players play but do not know the perfect state of the game.


Summary: Theory of Graph Games

Winning Mode/ Game Graphs

Sure Almost-sure Limit-sure

Turn-based Games (CHESS)

Linear time (PTIME-complete)

Linear-time (PTIME-complete)

Linear-time(PTIME-complete)

Concurrent Games (CHESS+ SOCCER)


Quadratic time (PTIME-complete)

Quadratic time(PTIME-complete)

Partial-information Games(CHESS + SOCCER+ POKER)


Mixing Chess and Poker: Partial-information Graph Games


Why Partial-information

Perfect-information: controller knows everything about the system. This is often unrealistic in the design of reactive

systems because • systems have internal state not visible to controller (private

variables)• noisy sensors entail uncertainties on the state of the game

Partial-observationHidden variables = imperfect information.

Sensor uncertainty = imperfect information.


Partial-information Games

A PIG G =(L, A, , O) is as follows L is a finite set of locations (or states). A is a finite set of input letters (or actions). µ L £ A £ L non-deterministic transition relation that

for a state and an action gives the possible next states.

O is the set of observations and is a partition of the state space. The observation represents what is observable.

Perfect-information: O={{l} | l 2 L}.


PIG: Example

a,b

a

b a

b


New Solution Concepts

Sure winning: winning with certainty (in perfect information setting determinacy).

Almost-sure winning: win with probability 1.

Limit-sure winning: win with probability arbitrary close to 1.

We will illustrate the solution concepts with card games.


Card Game 1

Step 1: Player 2 selects a card from the deck of 52 cards and moves it from the deck (player 1 does not know the card).

Step 2: Step 2 a: Player 2 shuffles the deck. Step 2 b: Player 1 selects a card and view it. Step 2 c: Player 1 makes a guess of the secret card or

goes back to Step 2 a.

Player 1 wins if the guess is correct.


Card Game 1

Player 1 can win with probability 1: goes back to Step 2 a until all 51 cards are seen.

Player 1 cannot win with certainty: there are cases (though with probability 0) such that all cards are not seen. Then player 1 either never makes a guess or makes a wrong guess with positive probability.


Card Game 2

Step 1: Player 2 selects a new card from an exactly same deck and puts is in the deck of 52 cards (player 1 does not know the new card). So the deck has 53 cards with one duplicate.

Step 2: Step 2 a: Player 2 shuffles the deck. Step 2 b: Player 1 selects a card and view it. Step 2 c: Player 1 makes a guess of the secret duplicate

card or goes back to Step 2 a.

Player 1 wins if the guess is correct.


Card Game 2

Player 1 can win with probability arbitrary close to 1: goes back to Step 2 a for a long time and then choose the card with highest frequency.

Player 1 cannot win probability 1, there is a tiny chance that not the duplicate card has the highest frequency, but can win with probability arbitrary close to 1, (i.e., for all ² >0, player 1 can win with probability 1- ², in other words the limit is 1).


Sure winning for Reachability

Result from [Reif 79]

Memory is required.

Exponential memory required.

Subset construction: what subsets of states player 1 can be. Reduction to exponential size turn-based games.

EXPTIME-complete.


Almost-sure winning for Reachability

Result from [CDHR 06, CHDR 07]

Standard subset construction fails: as it captures only sure winning, and not same as almost-sure winning.

More involved subset construction is required.

EXPTIME-complete.














EXPTIME-complete EXPTIME-complete


Limit-sure winning for Reachability

Limit-sure winning for reachability is undecidable [GO 10, CH 10].

Reduction from the Post-correspondence problem (PCP).


Mixing Chess, Soccer and Poker

Partial-information concurrent games

Concurrency can be obtained for free (polynomial reduction) for partial-information games.

So all the results for partial-information turn-based games also hold for partial-information concurrent games.














EXPTIME-complete EXPTIME-complete Undecidable.


Strategy Complexity


Classes of strategies

rand. action-invisible

pure

rand. action-visible Classification

according to the power of strategies


Classes of strategies

rand. action-invisible

pure

rand. action-visible Classification

according to the power of strategies

Poly-time reduction from decision problem of rand. act.-vis. to rand. act.-inv.


Known results

Almost-sureplayer 1 partialplayer 2 perfect

player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis.

exponential (belief) [CDHR’06]

memoryless[BGG’09]

exponential (belief) [BGG’09]

rand. act.-inv.

exponential (belief)

[CDHR’06(remark), GS’09]

exponential (belief) [GS’09]

pure ? ? ?

Reachability - Memory requirement (for player 1)

Positiveplayer 1 partialplayer 2 perfect

player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis. memoryless memoryless memoryless

rand. act.-inv. memoryless memoryless memoryless

pure ? ? ?


Beliefs

• Belief is sufficient.

• Randomized action invisible or visible almost same.

• The general case memory is similar (or in some cases exponential blow up) as compared to the one-sided case.

Three prevalent beliefs:


Pure Strategies


Proofs• Doubts.

Belief


Pure Strategies


Proofs• Doubts

Lesson: Doubt your belief and believe in your doubts!!! See the unexpected.

Belief


New results


player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis.




rand. act.-inv.

exponential (belief)

[CDHR’06(remark), GS’09]


pure ? ? ?



player 1 perfect

player 2 partial

2-sidedboth partial



pure ? ? ?


New results


player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis.




rand. act.-inv.

exponential (more than belief)


pure exponential (more than belief) ? ?



player 1 perfect

player 2 partial

2-sidedboth partial





New results


player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis.




rand. act.-inv.






player 1 perfect

player 2 partial

2-sidedboth partial





Pure Strategies: Player 1 Perfect, Player 2 Partial (positive)

Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless

Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless

Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless

Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential







Pl1 Perfect, Pl 2 Partial: Stochastic, Pure.

Add probabilityRestrict to pure

Pl 2 less informedPl 1 more informed, Pl 2 less informed







Pl1 Perfect, Pl 2 Partial: Stochastic, Pure. Non-elementary complete

Add probabilityRestrict to pure

Pl 2 less informedPl 1 more informed, Pl 2 less informed


New results


player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis.




rand. act.-inv.



pure exponential (more than belief)

non-elementarycomplete

?



player 1 perfect

player 2 partial

2-sidedboth partial





?


New results


player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis.




rand. act.-inv.





?



player 1 perfect

player 2 partial

2-sidedboth partial





?

Player 1 wins from more states, but needs more memory !


New results


player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis.




rand. act.-inv.





finite (at least non-elementary)



player 1 perfect

player 2 partial

2-sidedboth partial







Player 1 perfect, player 2 partial

• Win from more places.

• Winning strategy is very hard to implement.

Information is useful, but ignorance is bliss !!!

More information:


Reductions for equivalence

Equivalence of the decision problems for almost-sure reachwith pure strategies and rand. act.-inv. strategies• Reduction of rand. act.-inv. to pure choice of a subset of actions (support of prob. dist.)

• Reduction of pure to rand. act.-inv. (holds for almost-sure only)

It follows that the memory requirements for pure hold for rand. act.-inv. as well !


New results


player 1 perfect

player 2 partial

2-sidedboth partial

rand. act.-vis.




rand. act.-inv.








player 1 perfect

player 2 partial

2-sidedboth partial







Beliefs


• Randomized action invisible or visible almost same.

• The general case memory is similar (or in some cases exponential blow up) as compared to the one-sided case.

Three prevalent beliefs:

Belief Fails!

[CD11] Chatterjee, Doyen. Partial-Observation Stochastic Games: How to Win when Belief Fails. CoRR abs/1107.2141, July 2011.


The Message

Play Chess; Play Soccer;

But stay away from Poker !!!


Conclusion Theory of graph games

Turn-based, concurrent, and partial-information games. Different solution concepts and different complexity. Several algorithmic questions open.

Partial information games

Problem with clear practical motivation.

Challenging to establish the right frontier of complexity.

Important generalization of perfect-information games.

Unfortunately, undecidable and also high complexity.

Current research: identifying decidable and more efficient sub-classes.


Collaborators

Luca de Alfaro

Laurent Doyen

Thomas A. Henzinger

Jean-Francois Raskin


Thank you !

Questions ?

The end

Documents

Krishnendu Chatterjee1 Graph Games with Reachabillity Objectives: Mixing Chess, Soccer and Poker Krishnendu Chatterjee 5 th Workshop on Reachability Problems,