Embed Size (px)
DESCRIPTION
Trabajo Ya Publicado presentado en CoSECiVi'14. Resumen: Flocking strategies are sets of behavior rules for the interaction of agents that allow to devise controllers with reduced complexity that generate emerging behavior. In this paper, we present an application of genetic algorithms and flocking strategies to control the Ghost Team in the game Ms. Pac-Man. In particular, we define flocking strategies for the Ghost Team and optimize them for robustness with respect to the stochastic elements of the game and effectiveness against different possible opponents by means of genetic algorithm. The performance of the methodology proposed is tested and compared with that of other standard controllers. The results show that flocking strategies are capable of modeling complex behaviors and produce effective and challenging agents.
Citation preview
Evolving Evil: Optimizing Flocking Strategies through Genetic Algorithms for the Ghost Team
in the Game of Ms. Pac-Man
F. Liberatore (University Rey Juan Carlos)A. Mora (University of Granada)P. Castillo (University of Granada)J.J. Merelo (University of Granada)
Outline
- The Game of Ms. Pac-Man.- State of the Art.- Flocking.- Flocking Strategies for the Ghost Team.- A Genetic Algorithm for Flocking Strategies.- Experiments.- Conclusions.
What is this about?
In this work we propose:- A Controller for the Ghost Team in the game of Ms. Pac-Man …- … based on Flocking Strategies (more about this in the following) …- … optimized by a Genetic Algorithm.
The game of Ms. Pac-Man [1/2]
The game of Ms. Pac-Man:- Released on 1981.- Chosen for the Ms. Pac-Man vs Ghosts competition.- Participants can submit controllers for Ms.Pac-Man and the Ghosts.- The objective of Ms. Pac-Man is to maximize the final score.- The objective of the Ghosts is to minimize it.
The game of Ms. Pac-Man [2/2]
Movement restrictions for the Ghosts:- A ghost can never stop.- A ghost can choose its direction only at a junction.- A ghost cannot turn back on itself.- At every tick of the game all the ghosts obligatorily reverse their direction according to a small random probability.
State of the Art [1/2]Most contributions proposed controllers for Ms. Pac-Man:- Genetic Programming (Alhejali and Lucas 2010, 2011, Brandstetter and Ahmadi 2012),- Genetic Algorithms for parameters optimization (Thawonmas 2010),- Evolved Neural Networks (Burrow and Lucas 2009, Keunhyun and Sung-Bae 2010),- Ant Colonies (Martin et al. 2010),- Monte Carlo Tree Search (Samothrakis et al. 2011, Ikehata and Ito 2011, Alhejali and Lucas 2013),- Reinforcement Learning (Bom et al. 2013).
State of the Art [2/2]
Controllers for the Ghosts:- Monte Carlo Tree Search (Nguyen and Thawonmas 2011, 2013),- Influence Maps (Svensson and Johansson 2012),- Neural Networks (Jia-Yue et al. 2011),- Genetic Algorithm + Rules (Gagne and Congdon 2012),
- Competitive Co-Evolution of Ms. Pac-Man and Ghosts controllers (Cardona et al. 2013).
Flocking [1/2]
Swarm Intelligence technique:- Coordinated intelligence that arises from the collective behavior of decentralized, self-organized systems.
Flocking:- Developed to mimic lifelike behaviors of groups of beings.- Flocking systems consist of a population of simple agents (or boids) interacting locally with one another depending on the distance between them.
Flocking [2/2]- The agents follow very simple steering behaviors:
· Separation: makes the agent steer away from close flock mates.· Alignment: makes the agent steer toward the average heading of
the flock.· Cohesion: makes the agent steer toward the average position of
distant flock mates.- The interactions between such agents lead to the emergence of "intelligent" global behavior
Flocking Strategies [1/4]
- Generalization of the concept of Flocking.- The direction taken by a ghost at a junction depends on:
· State:- Normal,- Vulnerable,- Flashing.
· Position WRT that of the other actors:- Ms. Pac-Man,- Powerpills,- Other Ghosts (depending on their state).
Flocking Strategies [2/4]
- Each Ghost is surrounded by concentric rings called Neighborhoods.- The neighborhoods can have different radius and size.- Each neighborhood defines the strength of the interaction between the ghost and the other actors:
· Negative value → Separation· Positive value → Cohesion
Flocking Strategies [3/4]Formally, a Flocking Strategy is represented by a tuple (N, δ, Α):
- N, number of neighborhoods, indexed by n=1,…,N.- δ∈ℝN
>0, vector of N real positive numbers. Each element δ
n is
associated to one neighborhood and represents its maximum radius.- Α∈Μ, three-dimensional matrix with real entries. Each element α
s,a,n is the magnitude of the steering force on the current ghost in
state s, resulting from the interaction with the actor a, falling into neighborhood n.
Flocking Strategies [4/4]
- First, all the interactions with the other actors are calculated as vectors.- Then, all the vectors are summed to find the total interaction vector.- Finally, the direction taken by the ghost (UP, DOWN, LEFT, RIGHT) depends on the component of the total interaction vector having maximum absolute value.
A Genetic Algorithm for Flocking Strategies [1/3]
Flocking Strategies could be designed manually by an expert.We propose a standard Genetic Algorithm
A Genetic Algorithm for Flocking Strategies [2/3]
The Genetic Algorithm has been applied for the automatic definition of effective Ghosts agents:- Each individual is a Flocking Strategy, represented as a tuple (N, δ, Α). - Random initial population.- Roulette wheel selection.- Cross-over: two parents generate one child by random recombination of their chromosomes.
A Genetic Algorithm for Flocking Strategies [3/3]
- Fitness function:· The objective is to create effective Ghosts that perform well against
different Ms. Pac-Man strategies. · Each individual is tested against two Ms. Pac-Man agents:
StarterPacMan (SPM) and NearestPillPacMan (NPPM).· To ensure robustness in the face of random events, the game is
simulated 30 times for each Ms. Pac-Man agent and 95% confidence intervals are calculated.
· Fitness is given by:FITNESS = 1/CI+
SPM + 1/CI+
NPPM
Where CI+ represents the upper limit of the 95% confidence interval.
Experiments [1/2]
Comparison of the performance of the Ghosts controllers obtained with different values of the parameter N (number of Neighborhoods)
N=1 N=2 N=3 N=4 N=5
Best FITNESS-1
783.38 726.84 815.66 766.96 720.20
Avg. FITNESS-1
871.30 ±
55.45861.57 ±
70.13876.31 ±
65.06905.86 ±
62.66863.17 ±
72.15
Avg. CPU time (s)
1373 ±
150.661484.3 ±
122.011561 ±
193.941562.60 ±
109.901473.00 ±
74.02
Experiments [2/2]
Performances of the controllers included in the competition framework.
Controller Aggressive Ghosts
Legacy Legacy 2 The Reckoning
Random Ghosts
Starter Ghosts
FITNESS-1 1893.13 2210.9 1429.20 4200.70 1603.49
Conclusions [1/2]
- Sample controller (link)- Flocking Strategies: sets of behavior rules that determine the next move of an agent as a force resulting from the interaction of the agents in the game.- Genetic Algorithm presented to design optimized strategies.- The fitness function evaluates each individual by pitting it against two Ms. Pac-Man controllers 30 times, so as to avoid noise in the function.
Conclusions [2/2]
- Future lines of research:· Include in the fitness function the best Ms. Pac-Man controllers
that took part to the competition.· Compare the Flocking Strategy-based controller with the best
Ghosts controllers that took part to the competition.· Explore optimization methods other than Genetic Algorithms.· Change the fitness function structure to reduce the
computation requirements.
