Upload
belal-al-khateeb
View
101
Download
4
Tags:
Embed Size (px)
Citation preview
THE IMPORTANCE OF A LOOK-AHEAD DEPTH TO
EVOLUTIONARY CHECKERS
Belal Al-Khateeb Graham Kendall [email protected] [email protected]
School of Computer Science (ASAP Group)
University of Nottingham
Outline
-Introduction
- Checkers
- Samuel’s Checkers Program
- Previous Work
- Experimental Setup
- Results and Discussion
- Conclusions
2
Checkers3
Opening Board of Checkers
Checkers4
Black Forced to make Jump
move
Checkers5
Black Gets King
Samuel’s Checkers Program
- 1959, Arthur Samuel started to look at
Checkers
- The determination of weights through
self-play
- 39 Features
- Included look-ahead via mini-max (Alpha-
Beta)
- Defeated Robert Nealy
6
o Samuels’s program defeated Robert
Nealy, although the victory is surrounded
in controversy
oDid he lose the game or did Samuel win?
How good was Samuel’s
player?7
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : Just about to make move 16
How good was Samuel’s
player?8
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
Forced Jump
How good was Samuel’s
player?9
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
How good was Samuel’s
player?10
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program)Strong
(Try to
keep)
Trapped
Only
advance to
Square 28
How good was Samuel’s
player?11
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
How good was Samuel’s
player?12
What Move
would you
make?
20
21
22
26
32
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
How good was Samuel’s
player?13
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
How good was Samuel’s
player?14
o This was a
very poor
move
o It allowed
Samuel to
retain his
“Triangle of
Oreo
o AND.. By
moving his
checker
from 19 to
24 it
guaranteed
Samuel a
King
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
How good was Samuel’s
player?15
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
How good was Samuel’s
player?16
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
16-12 then 5-1, Chinook said
would be a draw
How good was Samuel’s
player?17
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
How good was Samuel’s
player?18
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
How good was Samuel’s
player?19
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
How good was Samuel’s
player?20
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
How good was Samuel’s
player?21
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
How good was Samuel’s
player?22
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
This
checker is
lost
How good was Samuel’s
player?23
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
How good was Samuel’s
player?24
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
This checker
could run (to
10) but..
K
How good was Samuel’s
player?25
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
K
How good was Samuel’s
player?26
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
K
How good was Samuel’s
player?27
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
Forced
Jump
K
How good was Samuel’s
player?28
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
K
How good was Samuel’s
player?29
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
K
How good was Samuel’s
player?30
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 2627
28
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
Victory
How good was Samuel’s
player?31
Two Mistakes by Nealy
Allowing Samuel to get a King
Playing a move that led to defeat
when there was a draw available
How good was Samuel’s
player?32
o The next year a six match rematch was
won by Nealy 5-1.
o Three years later (1966) the two world
championship challengers (Walter
Hellman and Derek Oldbury) played four
games each against Samuel’s program.
They won every game.
How good was Samuel’s
player?33
Blondie24
- Produced by Fogel and Chellapilla in 1999-
2000
- Neural network as an evaluation function.
- Values for input nodes
Red (Black) – positive
White – negative
Empty – zero
- Piece differential
- Subsections (sub-boards)
34
Blondie2435
Blondie24’s EANN Architecture
Blondie24
- Initial population of 30 neural networks
(players).
- Each neural network plays 5 games (as red)
against 5 randomly chosen players:-
+1 for a win
0 for a draw
-2 for a loss
-Best 15 players retained, the other 15 players
eliminated.
-Copy the best 15 players (replacing the worst
15) and mutate the weights.
36
Blondie24
- Repeat the process for 840 generations and
the best player after these generations is
retained.
- Played 165 games at zone.com.
- Rating: 2045.85 at that time
- In top 500 of over 120,000 players on
zone.com at that time.
- Better than 99.61% of registered players on
zone.com
End Product
37
Blondie24
- There has been a lot of discussion about the
importance of the look-ahead depth
- It is believed to be important and many
people state it, but we wanted to investigate
- Fogel, in his work on evolving Blondie24 said
that “At four ply, there really isn’t any “deep”
search beyond what a novice could do with a
paper and pencil if he or she wanted to”.
38
Blondie2439
-Generating four ply depth using a paper and
pencil:
- Not an easy task for novices.
- Time consuming.
- It might be done at some subconscious
level, where pruning is taking place.
- Has not been reported in the scientific
literature.
Previous Work40
-Many researchers have shown the importance
of the look-ahead depth for computer games.
-None of them was related to checkers.
-Most of the findings are related to chess
- Increasing the depth level will produce
superior chess players.
Previous Work41
- Runarsson and Jonsson for Othello:
- Better playing strategies are found when
using TD learning when ε–greedy is
applied with a lower look-ahead search
depth and a deeper look-ahead search
during game play.
Experimental Setup
- For the purpose of investigating our
hypothesis an evolutionary checkers
player, was implemented.
- Our implementation has the same
structure and architecture that Fogel
utilised in Blondie24.
- Four players were evolved.
C1 is evolved using one ply depth.
C2 is evolved using two ply depth.
C3 is evolved using three ply depth.
C4 is evolved using four ply depth.
42
Experimental Setup
-Our previous efforts to enhance Blondie24
introduced a round robin tournament. Al-Khateeb, B and Kendall, G Introducing a round robin tournament into Blondie24. CIG 2009: 112-116, 2009
- We also use this player, Blondie24-RR
(evolved using four ply) to investigate the
importance of the look-ahead depth.
- Three players were evolved (in addition to
Blondie24-RR.
- Blondie24-RR1Ply is evolved using one
ply.
- Blondie24-RR2Ply is evolved using two
ply.
43
Experimental Setup
- C1, C2, C3 and C4 played against each other
by using the idea of a two-move ballot and
each player allowed to search to a depth of 6-
ply.
- The games were played until either one side
wins or a draw is declared after 100 moves
for each player.
- The same procedure was also used to play
Blondie24-RR1Ply, Blondie24-
RR2Ply, Blondie24-RR3Ply, Blondie24-RR.
44
Results and Discussion45
C1 C2 C3 C4 Σ wins
C1- 28 17 13 58
C233 - 24 19 76
C345 31 - 27 103
C459 40 35 - 134
C1
C2 C3
C4 Σ draws
C1- 25 24 14 63
C225 - 31 27 83
C324 31 - 26 91
C414 27 26 - 67
Number of wins (for the row
player) out of 258 games.
Number of draws (for the row
player) out of 258 games.
Results and Discussion46
Mean SD Class
C1
C2
1188.94 28.94 E
1206.24 27.62 D
C1
C3
1146.58 27.40 E
1266.18 26.14 D
C1
C4
1264.11 27.21 D
1474.99 26.14 C
C2
C3
1179.47 26.85 E
1205.10 25.60 D
C2
C4
1114.61 27.17 E
1200.21 25.88 D
C3
C4
1176.02 28.26 E
1205.26 26.98 D
C2 C3 C4
C1Red Lost Lost Lost
White Drawn Lost Lost
C2Red - Lost Lost
White - Drawn Lost
C3Red - Lost
White - Lost
Standard rating formula for all players after
5000 different orderings of the 86 games
played.
Wins/Loses for C1, C2, C3 and C4 when not
using two-move ballot.
Results and Discussion47
1pl
y
2ply 3ply 4ply Σ wins
1ply - 28 20 14 62
2ply 32 - 29 21 82
3ply 42 34 - 27 103
4ply 57 46 39 - 142
1ply 2ply 3ply 4ply Σ draws
1ply - 26 24 15 65
2ply 26 - 23 19 68
3ply 24 23 - 20 67
4ply 15 19 20 - 54
Number of wins (for the row
player) out of 258 games for the
round robin players.
Number of draws (for the row
player) out of 258 games for the
round robin players.
Results and Discussion48
Mean SD Class
1Pl
y
2Pl
y
1187.79 28.86 E
1200.74 27.55 D
1Pl
y
3Pl
y
1160.17 28.15 E
1252.67 26.84 D
1Pl
y
4Pl
y
1256.00 27.71 D
1450.51 26.58 C
2Pl
y
3Pl
y
1194.62 29.30 E
1212.04 27.98 D
2Pl 1335.38 28.72 D
2Ply 3Ply 4Ply
1Ply Red Lost Lost Lost
White Lost Lost Lost
2Ply Red - Lost Lost
White - Lost Lost
3Ply Red - Lost
White - Lost
Standard rating formula for all players after
5000 different orderings of the 86 games
played.
Wins/Loses for 1Ply, 2Ply, 3Ply and 4Ply
when not using two-move ballot.
Conclusions
- Many evolutionary checkers players
produced, using different depths of ply
during learning.
- Better value functions would be learned when
training with deeper look-ahead search.
49
Conclusions
- Increasing of the ply depth will increase the
computational cost of evolving evolutionary
checkers. In our experiments as all the
experiments were run for the same amount of
time (19 days).
- The results suggest that starting with a depth
of four ply is the best value function to start
with during learning phase for checkers. That
is, train at four ply and then play at the
highest ply possible.
50
References
51
1- Samuel, A. L., Some studies in machine learning using the game of checkers 1959,1967.
2- Fogel D. B., Blondie24 Playing at the Edge of AI, United States of America Academic Press, 2002.
3- Chellapilla K. and Fogel, D. B., Anaconda defeats hoyle 6-0: A case study competing an evolved
checkers program against commercially available software 2000.
4- Fogel D. B. and Chellapilla K., Verifying anaconda's expert rating by competing against Chinook:
experiments in co-evolving a neural checkers player.
5- Chellapilla K. and Fogel D.B., Evolution, Neural Networks, Games, and Intelligence,” 1999.
6- Chellapilla K. and Fogel D. B., Evolving an expert checkers playing program without using human
expertise, 2001.
7- Chellapilla K. and Fogel D. B., Evolving neural networks to play checkers without relying on
expert knowledge.1999.
8- Runarsson, T.P. and Jonsson, E.O, Effect of look-ahead search depth in learning position
evaluation functions for Othello using ε–greedy exploration, 2007.
Questions/Discussions
Thank You
52