Look ahead cec2011

THE IMPORTANCE OF A LOOK-AHEAD DEPTH TO

EVOLUTIONARY CHECKERS

Belal Al-Khateeb Graham Kendall bxk@cs.nott.ac.uk gxk@cs.nott.ac.uk

School of Computer Science (ASAP Group)

University of Nottingham

Outline

-Introduction

- Checkers

- Samuel’s Checkers Program

- Previous Work

- Experimental Setup

- Results and Discussion

- Conclusions

2

Checkers3

Opening Board of Checkers

Checkers4

Black Forced to make Jump

move

Checkers5

Black Gets King

Samuel’s Checkers Program

- 1959, Arthur Samuel started to look at

Checkers

- The determination of weights through

self-play

- 39 Features

- Included look-ahead via mini-max (Alpha-

Beta)

- Defeated Robert Nealy

6

o Samuels’s program defeated Robert

Nealy, although the victory is surrounded

in controversy

oDid he lose the game or did Samuel win?

How good was Samuel’s

player?7

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : Just about to make move 16

How good was Samuel’s

player?8

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program)

Forced Jump

How good was Samuel’s

player?9

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program)

How good was Samuel’s

player?10

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program)Strong

(Try to

keep)

Trapped

Only

advance to

Square 28

How good was Samuel’s

player?11

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program)

How good was Samuel’s

player?12

What Move

would you

make?

20

21

22

26

32

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program)

How good was Samuel’s

player?13

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program)

How good was Samuel’s

player?14

o This was a

very poor

move

o It allowed

Samuel to

retain his

“Triangle of

Oreo

o AND.. By

moving his

checker

from 19 to

24 it

guaranteed

Samuel a

King

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23K

How good was Samuel’s

player?15

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23K

How good was Samuel’s

player?16

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23K

16-12 then 5-1, Chinook said

would be a draw

How good was Samuel’s

player?17

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23K

How good was Samuel’s

player?18

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23K

How good was Samuel’s

player?19

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23K

How good was Samuel’s

player?20

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23K

How good was Samuel’s

player?21

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23K

How good was Samuel’s

player?22

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

This

checker is

lost

How good was Samuel’s

player?23

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

How good was Samuel’s

player?24

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

This checker

could run (to

10) but..

K

How good was Samuel’s

player?25

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

K

How good was Samuel’s

player?26

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

K

How good was Samuel’s

player?27

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

Forced

Jump

K

How good was Samuel’s

player?28

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

K

How good was Samuel’s

player?29

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

K

How good was Samuel’s

player?30

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 24

25 2627

28

29 30 31 32

White (Nealey)

Red (Samuel’s Program) : After Move 25

23

K

Victory

How good was Samuel’s

player?31

Two Mistakes by Nealy

Allowing Samuel to get a King

Playing a move that led to defeat

when there was a draw available

How good was Samuel’s

player?32

o The next year a six match rematch was

won by Nealy 5-1.

o Three years later (1966) the two world

championship challengers (Walter

Hellman and Derek Oldbury) played four

games each against Samuel’s program.

They won every game.

How good was Samuel’s

player?33

Blondie24

- Produced by Fogel and Chellapilla in 1999-

2000

- Neural network as an evaluation function.

- Values for input nodes

Red (Black) – positive

White – negative

Empty – zero

- Piece differential

- Subsections (sub-boards)

34

Blondie2435

Blondie24’s EANN Architecture

Blondie24

- Initial population of 30 neural networks

(players).

- Each neural network plays 5 games (as red)

against 5 randomly chosen players:-

+1 for a win

0 for a draw

-2 for a loss

-Best 15 players retained, the other 15 players

eliminated.

-Copy the best 15 players (replacing the worst

15) and mutate the weights.

36

Blondie24

- Repeat the process for 840 generations and

the best player after these generations is

retained.

- Played 165 games at zone.com.

- Rating: 2045.85 at that time

- In top 500 of over 120,000 players on

zone.com at that time.

- Better than 99.61% of registered players on

zone.com

End Product

37

Blondie24

- There has been a lot of discussion about the

importance of the look-ahead depth

- It is believed to be important and many

people state it, but we wanted to investigate

- Fogel, in his work on evolving Blondie24 said

that “At four ply, there really isn’t any “deep”

search beyond what a novice could do with a

paper and pencil if he or she wanted to”.

38

Blondie2439

-Generating four ply depth using a paper and

pencil:

- Not an easy task for novices.

- Time consuming.

- It might be done at some subconscious

level, where pruning is taking place.

- Has not been reported in the scientific

literature.

Previous Work40

-Many researchers have shown the importance

of the look-ahead depth for computer games.

-None of them was related to checkers.

-Most of the findings are related to chess

- Increasing the depth level will produce

superior chess players.

Previous Work41

- Runarsson and Jonsson for Othello:

- Better playing strategies are found when

using TD learning when ε–greedy is

applied with a lower look-ahead search

depth and a deeper look-ahead search

during game play.

Experimental Setup

- For the purpose of investigating our

hypothesis an evolutionary checkers

player, was implemented.

- Our implementation has the same

structure and architecture that Fogel

utilised in Blondie24.

- Four players were evolved.

C1 is evolved using one ply depth.

C2 is evolved using two ply depth.

C3 is evolved using three ply depth.

C4 is evolved using four ply depth.

42

Experimental Setup

-Our previous efforts to enhance Blondie24

introduced a round robin tournament. Al-Khateeb, B and Kendall, G Introducing a round robin tournament into Blondie24. CIG 2009: 112-116, 2009

- We also use this player, Blondie24-RR

(evolved using four ply) to investigate the

importance of the look-ahead depth.

- Three players were evolved (in addition to

Blondie24-RR.

- Blondie24-RR1Ply is evolved using one

ply.

- Blondie24-RR2Ply is evolved using two

ply.

43

Experimental Setup

- C1, C2, C3 and C4 played against each other

by using the idea of a two-move ballot and

each player allowed to search to a depth of 6-

ply.

- The games were played until either one side

wins or a draw is declared after 100 moves

for each player.

- The same procedure was also used to play

Blondie24-RR1Ply, Blondie24-

RR2Ply, Blondie24-RR3Ply, Blondie24-RR.

44

Results and Discussion45

C1 C2 C3 C4 Σ wins

C1- 28 17 13 58

C233 - 24 19 76

C345 31 - 27 103

C459 40 35 - 134

C1

C2 C3

C4 Σ draws

C1- 25 24 14 63

C225 - 31 27 83

C324 31 - 26 91

C414 27 26 - 67

Number of wins (for the row

player) out of 258 games.

Number of draws (for the row

player) out of 258 games.

Results and Discussion46

Mean SD Class

C1

C2

1188.94 28.94 E

1206.24 27.62 D

C1

C3

1146.58 27.40 E

1266.18 26.14 D

C1

C4

1264.11 27.21 D

1474.99 26.14 C

C2

C3

1179.47 26.85 E

1205.10 25.60 D

C2

C4

1114.61 27.17 E

1200.21 25.88 D

C3

C4

1176.02 28.26 E

1205.26 26.98 D

C2 C3 C4

C1Red Lost Lost Lost

White Drawn Lost Lost

C2Red - Lost Lost

White - Drawn Lost

C3Red - Lost

White - Lost

Standard rating formula for all players after

5000 different orderings of the 86 games

played.

Wins/Loses for C1, C2, C3 and C4 when not

using two-move ballot.

Results and Discussion47

1pl

y

2ply 3ply 4ply Σ wins

1ply - 28 20 14 62

2ply 32 - 29 21 82

3ply 42 34 - 27 103

4ply 57 46 39 - 142

1ply 2ply 3ply 4ply Σ draws

1ply - 26 24 15 65

2ply 26 - 23 19 68

3ply 24 23 - 20 67

4ply 15 19 20 - 54

Number of wins (for the row

player) out of 258 games for the

round robin players.

Number of draws (for the row

player) out of 258 games for the

round robin players.

Results and Discussion48

Mean SD Class

1Pl

y

2Pl

y

1187.79 28.86 E

1200.74 27.55 D

1Pl

y

3Pl

y

1160.17 28.15 E

1252.67 26.84 D

1Pl

y

4Pl

y

1256.00 27.71 D

1450.51 26.58 C

2Pl

y

3Pl

y

1194.62 29.30 E

1212.04 27.98 D

2Pl 1335.38 28.72 D

2Ply 3Ply 4Ply

1Ply Red Lost Lost Lost

White Lost Lost Lost

2Ply Red - Lost Lost

White - Lost Lost

3Ply Red - Lost

White - Lost

Standard rating formula for all players after

5000 different orderings of the 86 games

played.

Wins/Loses for 1Ply, 2Ply, 3Ply and 4Ply

when not using two-move ballot.

Conclusions

- Many evolutionary checkers players

produced, using different depths of ply

during learning.

- Better value functions would be learned when

training with deeper look-ahead search.

49

Conclusions

- Increasing of the ply depth will increase the

computational cost of evolving evolutionary

checkers. In our experiments as all the

experiments were run for the same amount of

time (19 days).

- The results suggest that starting with a depth

of four ply is the best value function to start

with during learning phase for checkers. That

is, train at four ply and then play at the

highest ply possible.

50

References

51

1- Samuel, A. L., Some studies in machine learning using the game of checkers 1959,1967.

2- Fogel D. B., Blondie24 Playing at the Edge of AI, United States of America Academic Press, 2002.

3- Chellapilla K. and Fogel, D. B., Anaconda defeats hoyle 6-0: A case study competing an evolved

checkers program against commercially available software 2000.

4- Fogel D. B. and Chellapilla K., Verifying anaconda's expert rating by competing against Chinook:

experiments in co-evolving a neural checkers player.

5- Chellapilla K. and Fogel D.B., Evolution, Neural Networks, Games, and Intelligence,” 1999.

6- Chellapilla K. and Fogel D. B., Evolving an expert checkers playing program without using human

expertise, 2001.

7- Chellapilla K. and Fogel D. B., Evolving neural networks to play checkers without relying on

expert knowledge.1999.

8- Runarsson, T.P. and Jonsson, E.O, Effect of look-ahead search depth in learning position

evaluation functions for Othello using ε–greedy exploration, 2007.

Questions/Discussions

Thank You

52