FSM-Based Agents for Playing Super Mario Game

Evolutionary FSM-Based Evolutionary FSM-Based Agents for PlayingAgents for PlayingSuper Mario GameSuper Mario Game

R.M. Hidalgo-Bermúdez, M.S. Rodríguez-Domingo, A.M. MoraR.M. Hidalgo-Bermúdez, M.S. Rodríguez-Domingo, A.M. MoraP. García-Sánchez, J.J. Merelo, Antonio J. Fernández-LeivaP. García-Sánchez, J.J. Merelo, Antonio J. Fernández-Leiva

LION 2013

INDEXINDEX1. INTRODUCTION1. INTRODUCTION2. PRELIMINARY CONCEPTS AND BACKGROUND2. PRELIMINARY CONCEPTS AND BACKGROUND3. MARIO AI: COMPETITION AND ENVIRONMENT3. MARIO AI: COMPETITION AND ENVIRONMENT4. EVOLUTIONARY FSM-BASED AGENT4. EVOLUTIONARY FSM-BASED AGENT5. EXPERIMENTS AND RESULTS5. EXPERIMENTS AND RESULTS6. CONCLUSIONS AND FUTURE WORK6. CONCLUSIONS AND FUTURE WORK

INTRODUCTIONINTRODUCTION Why did we choose Mario Bros?Why did we choose Mario Bros?

This This game is worldwide known game is worldwide known and remains and remains very popular nowadays.very popular nowadays.There is an There is an open source clone open source clone of the game of the game named Infinite Mario Bros.named Infinite Mario Bros.

INTRODUCTIONINTRODUCTION

The main goal of the game The main goal of the game is to is to go across go across lateral platforming levelslateral platforming levels, trying to avoid , trying to avoid different enemies and obstacles by using different enemies and obstacles by using some useful items.some useful items.We have developed an We have developed an autonomous agent autonomous agent using different approachesusing different approaches based on Finite based on Finite State MachinesState Machines (FSMs) combined (FSMs) combined with with Genetic AlgorithmsGenetic Algorithms (GAs) techniques. (GAs) techniques.

PRELIMINARY CONCEPTS AND PRELIMINARY CONCEPTS AND BACKGROUNDBACKGROUND

GAs are based on the model of darwinistic GAs are based on the model of darwinistic evolution. A evolution. A standard GA’s standard GA’s procedureprocedure is: is:

A A Finite State Machine (Finite State Machine (FSMFSM) is a computational model ) is a computational model which represents a set of which represents a set of states and connections states and connections between thembetween them. . It is represented as a directed graph, where each node It is represented as a directed graph, where each node corresponds to a state.corresponds to a state.

PRELIMINARY CONCEPTS AND PRELIMINARY CONCEPTS AND BACKGROUNDBACKGROUND

Infinite Mario BrosInfinite Mario Bros is an open source modification of is an open source modification of Super Mario Bros. It Super Mario Bros. It includes wide includes wide support for support for implementing autonomous agents to control Marioimplementing autonomous agents to control Mario character using AI techniques.character using AI techniques.

The Agent follows the rules of The Agent follows the rules of Mario AI ChampionshipMario AI Championship. . The tracks are:The tracks are: GameplayGameplay Turing Test Turing Test LearningLearning Level Generation Level Generation

MARIO AI: COMPETITION AND MARIO AI: COMPETITION AND ENVIRONMENTENVIRONMENT

The The possible actionspossible actions of the Agent are: of the Agent are: Go RightGo Right Go Left Go Left JumpJump Down Down Shoot/RunShoot/Run

Mario can be in three Mario can be in three different modesdifferent modes:: Fire Fire Big Big Small Small


Game dataGame data is presented in the form of is presented in the form of two two 19x19 matrices19x19 matrices. Each . Each cellcell of each matrix of each matrix holds a holds a valuevalue, which , which represents a type of represents a type of obstacle or enemyobstacle or enemy, respectively, respectively


ObstaclesObstacles and useful and useful itemsitems::


Enemies:Enemies:


EVOLUTIONARY FSM-BASED AGENT EVOLUTIONARY FSM-BASED AGENT

Possible Possible statesstates

Chromosome Chromosome (modelling a FSM)(modelling a FSM)EVOLUTIONARY FSM-BASED AGENT EVOLUTIONARY FSM-BASED AGENT

Two approaches:Two approaches:

Mono-seedMono-seedAll the individuals are tested in the same levelAll the individuals are tested in the same levelThe length of the level and the time limit to complete it The length of the level and the time limit to complete it grow grow with each generationwith each generation

Multi-seedMulti-seedEvery individual is tested in 30 random levelsEvery individual is tested in 30 random levelsThe fitness is computed considering the results of all the The fitness is computed considering the results of all the playsplays


There areThere are two types of fitnesstwo types of fitness:: Generic fitnessGeneric fitness Hierarchical fitnessHierarchical fitness

The The selection mechanism selection mechanism considers the best individual considers the best individual and a percentage of the best ones, selected by and a percentage of the best ones, selected by tournament according to their fitness.tournament according to their fitness.

CrossoverCrossover is performed considering the best individual of is performed considering the best individual of the present generation as one of the parents, and one of the present generation as one of the parents, and one of the individuals with positive fitness as the other parent.the individuals with positive fitness as the other parent.

In the In the mutation operatormutation operator, , various genes in each of these various genes in each of these individuals are randomly selected to be mutatedindividuals are randomly selected to be mutated


EXPERIMENTS AND RESULTSEXPERIMENTS AND RESULTS Mono-seed approach

EXPERIMENTS AND RESULTSEXPERIMENTS AND RESULTS Mono-seed approach

EXPERIMENTS AND RESULTSEXPERIMENTS AND RESULTS

EXPERIMENTS AND RESULTSEXPERIMENTS AND RESULTS

EXPERIMENTS AND RESULTSEXPERIMENTS AND RESULTS Multi-seed approach

Difficulty level 1 (completed): Difficulty level 1 (completed): http://www.youtube.com/watch?v=6Pj6dZCE070

Difficulty level 2 (completed): Difficulty level 2 (completed): http://www.youtube.com/watch?v=gtfuY-L0WDA

Difficulty level 3 (completed): Difficulty level 3 (completed): http://www.youtube.com/watch?v=qQVQ43sWwYY

Difficulty level 12 (stacked): Difficulty level 12 (stacked): http://www.youtube.com/watch?v=zNGfBApX7sk

SOME OF OUR APPROACHESSOME OF OUR APPROACHES

http://www.youtube.com/watch?v=6Pj6dZCE070

http://www.youtube.com/watch?v=gtfuY-L0WDA

http://www.youtube.com/watch?v=qQVQ43sWwYY

http://www.youtube.com/watch?v=zNGfBApX7sk

An An agentagent has been has been implemented using Finite State Machineimplemented using Finite State Machine (FSM) (FSM) models, models, improvedimproved by means of by means of Genetic AlgorithmsGenetic Algorithms (GA) and (GA) and considering different evaluation schemes: considering different evaluation schemes: mono-seedmono-seed and a and a multi-multi-seedseed evaluation evaluation approaches.approaches.

Mono-seedMono-seed can yield can yield excellent agents for the level where they were excellent agents for the level where they were ‘trained’ (evolved)‘trained’ (evolved), having a quite bad behaviour in a different level., having a quite bad behaviour in a different level.Multi-seedMulti-seed takes much more computational time and has higher takes much more computational time and has higher resource requirements, but the resource requirements, but the agentsagents it yields it yields are very good are very good playing in any level of the considered difficultyplaying in any level of the considered difficulty (in the evolution). (in the evolution).

All these agents play much better than an expert human player and All these agents play much better than an expert human player and can complete the levels in a time impossible to get for the human.can complete the levels in a time impossible to get for the human.

CONCLUSIONS AND FUTURE WORKCONCLUSIONS AND FUTURE WORK

Thanks!

Technology

FSM-Based Agents for Playing Super Mario Game