Parallel Monte-Carlo Tree Search with Simulation Servers
HIDEKI KATO†‡ and IKUO TAKEUCHI†
† The University of Tokyo‡ Fixstars Corporation
November 7th, 2008
Contents
Computer Go
Monte-Carlo Tree Search
Parallel Monte-Carlo Tree Search
Client-Server Approach
Experiments and Discussion
Conclusion and Future Work
Computer Go• The game of Go
– Task par excellence for AI (H. Berliner 1978)– Most challenging; largest search space
• 19 x 19 10171, 9 x 9 1038 cf. Chess 1050
– Minimax tree search and a static evaluation function with domain knowledge was used so far without major success
• The Monte-Carlo Go revolution– MoGo beat an 8-dan professional player on 9 x 9– Crazy Stone beat a 4-dan professional player with 8 st
ones handicap
Monte-Carlo Tree Search (MCTS)
Descend tree from root to leaf
Update values ofthe moves
Repeat until time-up
Play most visited move in root
Add a node
Simulatea game
Parallel MCTS (PMCTS)
Lock
Search tree (shared)
Thread 1
• Symmetrical multi-thread (SMT) PMCTS– Commonly used straightforward implementation– MCTS threads share a search tree
Thread 3
Thread 2
Thread 4
Related Work• S. Gelly et al. introduced SMT PMCTS for share
d-memory SMP systems (2006)• T. Cazenave et al. proposed and evaluated three P
MCTS algorithms on a 16 Intel Pentium-4 MPI cluster (2007)
• G. Chaslot et al. evaluated root, leaf and tree parallelization on 2 x 8 core IBM Power5 (2008)
• S. Gelly et al. proposed SMT PMCTS for MPI clusters of shared-memory SMP nodes (2008)
Problems
• Number of processors–Shared tree PMCTS can run only on shared-me
mory systems; currently up to 16 or 32 processors
–PMCTS algorithms for clusters of computers connected through networks is necessary
–Longer communication time decreases performance like other parallel applications
–Increasing the threads increases the overhead of the locks to share search tree
MoGo’s Solution• Combine fine and coarse grain PMCTS
– For MPI clusters with shared-memory SMP nodes (S. Gelly et al. 2008)
– Runs SMT PMCTS on each node– Periodically exchanges and merges values in the tree– Excellent performance
• MoGoTitan beat an 8-dan Korean Professional Go player with 9 stones handicap (2008)
Huygens super computer at SARA in Amsterdam, the Netherlands
25 out of 104 SMP nodes were usedEach node consists of 16 dual core Power6
processors at 4.7 GHz
MoGo’s Solution (cont’d)• Disadvantages
– Expensive• High speed network interfaces such as InfiniBand are very
expensive (so are the clusters)
– Lack of flexibility• MPI does not allow to add or remove computers on the fly• MPI requires special setup; must be pre-configured
• Applicable to non-MPI clusters on moderate speed networks?– Nobody tried yet
Client-Server Approach• Recent success of grid computing
– Folding@home achieved one petaflop with major benefits by 41,145 Sony Playstation 3 consoles all over the world (2007)
– Less expensive massive parallel approach– Applicable to PMCTS?
• Basic idea– Separate tree search part and simulation part– Broadcast positions to be simulated using UDP/IP– Don’t wait the end of slow simulations
Client-Server Approach (cont’d)• Client-server PMCTS
– A client searches tree and send a position; a server simulates a game from the position and sends back the result
– Runs on a cluster of loosely-coupled computers– Servers can run on small memory computers even if
the tree is going to be huge– No special set-up for servers; just a small application– Longer communication time due to moderate speed
networks– Performance? Scales well?
Client-Server PMCTS
Descend tree from root to leaf
Update values of the moves
Repeat until time-up
Select most visited move in root
Add a node
Broadcast the position
Receive a result(no wait)
Repeat forever
Send the result
Simulatea game
Repeat forever
Receive positions
Send the result
Receive positions
Simulatea game
Server 1 Server 2
Client
Search tree
Loop
Experimental System
CPU: Q9550/3GHz (400 x 7.5)OS: Ubuntu Linux 8.04M/B: ASUS P5K-VM (G33)RAM:PC3200 4GiBNIC: Intel EXP9300PT (PCI-Ex x1)
CPU: Q9550/3GHz (400 x 7.5)OS: Ubuntu Linux 8.04M/B: DFI LP JR P45-T2RS (P45)RAM:PC3200 4GiBNIC: Intel EXP9300PT (PCI-Ex x1)RTT: 151±22 s @ 1 kB
CPU: Q6600/3GHz (333 x 9)OS: Ubuntu Linux 8.04M/B: ASUS P5K-VM (G33)RAM:PC3200 4GiBNIC: Intel EXP9300PT (PCI-Ex x1)RTT: 154±20 s @ 1 kB
CPU: Q6600/3GHz (333 x 9)OS: Ubuntu Linux 8.04M/B: ASUS P5WDG2-WS Pro (975X)RAM:PC3200 4GiBNIC: Intel EXP9300GT (PCI)RTT: 159±22 s @ 1 kB
PC1 (1 client and 3 servers) PC2 (4 servers)
PC3 (4 servers) PC4 (4 servers)
Allied Telesis GS908XLSwitching delay: 2.2 ms @ 64 byte
Switch
Experiments• A tree searcher or a simulator exclusively uses a core• One core or other on the client computer is used for a
tree searcher or a simulator thread, respectively• The simulators on the server computers run as
individual processes • All results are ELO ratings against GNU Go 3.7.11
level 0
9 x 9 13 x 13Games 2,000 500Time per move (s) 0.005 to 0.64 0.05 to 6.4
Simulation servers 1 to 15 1 to 15
How to evaluate the results?• Simulations per second?
– Commonly used for shared memory SMP systems but not a good measure for clusters
– The benefits of simulations are not the same• Use equivalent-strength speed-up
– The ratio of time-per-move settings that give the same strength at different number of simulators
– “Equivalent speed-up” for short• Number of simulators or cores
– The number of simulators is used to evaluate scalability while the number of all cores is used to evaluate performance
Equivalent Speed-up
Time per move (s)
84211/21/4
1/8
1/16
ELO
rati
ng
4 core (13 x 13)
16 core (13 x 13)
-300
-200
-100
0
100
200
300
4 core (9 x 9)
16 core (9 x 9)
Performance (4 core vs. 16 core)
y = 0.0001x + 1.5969
0
1
2
3
4
5
-600 -500 -400 -300 -200 -100 0 100 200
ELO rating
Equiv
ale
nt sp
eed-u
p
9 x 9
13 x 13
Scalability
9 x 9 (0.08 s/move)
13 x 13 (0.4 s/move)
1 2 3 155 7 114 8
Number of simulators
ELO
rati
ng
y = 86.864x - 281.43
R2 = 0.9794
-300
-200
-100
0
100
Conclusion and Future Work• Client-server parallel Monte-Carlo tree search
– Runs on a cluster of loosely coupled computers– Small memory computers such as game consoles can
be used for simulation servers– Allows servers to connect or disconnect on-the-fly– Reduced communication by broadcasting– No overhead to share search tree– Scales well on 13 x 13 with 15 simulators
• Future work– Multiple clients for single or multiple users