Jorge Munoz, German Gutierrez, Araceli Sanchis

Presented by: Itay Presented by: Itay BittanBittan

1. Introduction 2. Related work 3. TORCS competition 4. Controllers 5. Controller learning by imitation 6. Results 7. Conclusions 8. Future works

Initial approach to create a controller for TORCS by learning how another controller or humans play the game.

The data obtained from 3 controllers.One human player and two controllers:

The winner of the WCCI 2008 Simulated Car Racing

Hand coded controller that performs a complete lap in all tracks.

First, each kind of controller is imitated separately, then a mix of data is used to create new controllers.

The imitation is performed by means of training a feed forward neural network with the data, using the backpropagation algorithm for learning.

ANN - Artificial Neural Networks

Human players realize they are not playing vs. another human – and finds a way to beat the NPC (non-player character).

NPC sometimes cheats for winning humans. Another option is to play in Internet against

other human players

With a lot of cheats or playing versus experienced human make you lose in every game – boring!

Create opponents as intelligent as a human player.

The AI must be able to adapt its behavior depending on the opponent (play in the same level of the human)

In this way the AI will provide a better entertainment for the player!

Realistic game where human plays versus one or more NPC.

There is option to compare the results with other researchers.

Allows to analyze behaviors that take place in a short period of time.

In all the experiments the controller created is a feed-forward ANN (Artificial Neural Networks) that was trained with data generated by the controllers.

The learning algorithm for the ANN was backpropagation.

1. Introduction 2. Related work 3. TORCS competition 4. Controllers 5. Controller learning by imitation 6. Results 7. Conclusions 8. Future works

Wide area of researching is to create computational intelligence in games with ANN.

NEAT – NeuroEvolution of Augmenting Topologies

NEAT is an effective method (algorithm) to create ANN - it alters both the weighting parameters and structures of networks.

It starts with a small population of random ANN (with only input & output layers) that evolves to the problem.

An approaches to adapt the AI of the game to the player

Examples:Rapidly adaptive game AI – method that

applies continuously small adaptations to the AI based on observations and evaluation of the user actions.

Dynamic Scripting – based on a set of rules that are used for the game, whose weights to select one or another rule are modified through a machine learning algorithm.

One researcher clone the behavior of RoboCup player using case base reasoning (solving new problems based on the solutions of similar past problems)

RoboCup – simulated league of soccer.

Other researcher program robosoccer agents by modelling human behaviors with successful results.

RoboCup – simulated league of soccer.

1. Introduction 2. Related work 3. TORCS competition 4. Controllers 5. Controller learning by imitation 6. Results 7. Conclusions 8. Future works

Very realistic simulator that has a sophisticated physic engine that takes into account many aspects of the racing such as fuel consumption, collisions or traction.

Provides a lot of tracks, cars with different feature

TORCS is open software and that allows the researchers to make modifications to the game and adapt it to their requirements.

Info. Provided:• The lap (current lap

time, best lap time, distance raced, race position)

• The car status (damage, fuel, actual gear, speed, lateral speed and R.P.M)

• Distanse between the car and the track edges

• More..

Action sent:• Acceleration level• Brake level• Gear• Steering of the wheel

1. Introduction 2. Related work 3. TORCS competition 4. Controllers 5. Controller learning by imitation 6. Results 7. Conclusions 8. Future works

In the experiments we use the data obtained from three different controllers:

Human player The winner of the WCCI2008 competition Hand coded controller

The information that the human gets from watching the game monitor is much richer.

He drove the car trying to go through the middle of the road, with soft accelerations and brakes, braking much before the next curve started, without fast and sharp turns.

The human tried to drive the car as programmed controller would do, but with the mistakes that human makes.

Created my Matt Simmerson. This controller was the winner of the WCCI 2008 simulated car racing competition.

As INPUTs of the ANN created by NEAT he selected: the current speed, the angle to track axis, the track position with respect to left and right edges, the current gear selection, the four wheels spin sensors, the current R.P.M and 19 track sensors.

All these inputs were scaled to the range [0,1].

The OUTPUTs of the ANN were:The power (accelerate and brake), the gear

change and the steering.The two first are range [0,1] and the last one

is in range [-1,1].The fitness function used to evaluate the ANN

in NEAT took into account the distance raced, the R.P.M, the maximum speed reached and a value to measure of how much the car stayed on the track.

The idea of creating another controller was due to the human controller sometimes make mistakes and the Simmerson’s controller does not perform one complete lap in all the tracks and sometimes gets out from the track.

Thus, the requirements are:• To have same outputs for same inputs

(without mistakes) – deterministic• To perform a lap without getting out of the

track

To calculate the values for the acceleration and the brake, we calculate the speed the car should have (estimated speed).

Where sum_semsors is the sum of the three front sensors (which give the distance between the car and the edge of the track).

Alpha and beta are predefined parameters.With this value (estimated speed) we

calculated the diffrence:

The acceleration and brake values are proportional to the absolute value of the difference of the actual speed and the estimated speed:

Where again there are adjustment parameters.

The steering value calculation:First, we check if the car is in straight or in a

curve. We suppose that the car is in a straight when any of the 3 front sensors has the max value, and in a curve otherwise.

Straight equation:

Curve equation:

Finally the gear is calculated by:

Where lambda is an adjustment parameter and speed is the current speed of the car.

(for the gear the controller does not allow to change the gear twice in less than a second)

1. Introduction 2. Related work 3. TORCS competition 4. Controllers 5. Controller learning by imitation

6. Results 7. Conclusions 8. Future works

For the goal of learning the behavior of the controller we have used an ANN.

First, we obtain the data from the controller we want to imitate, then the ANN is trained with the backpropagation algorithm and finally the new controller is tested in the tracks.

Inputs:• Current speed of the

car• Angle of the car with

the axis• The current gear• The lateral speed of

the car• The R.P.M• The 4 spins of the

wheels.• 19 sensors – distance

between the car and the adges of the track

Outputs:• The accelerate / brake

value• The gear• The steering

For all the experiments the ANN has 3 hidden layers of 28 neurons each one, were trained during 1000 cycles and the learning rate starts in 0.9 and finished in 0.0001.

The data was taken from 17 roads tracks (next slide) – but only if the controller complete almost 3 laps.

Hidden Layer

Inputlayer

Outputlayer

That have been obtained per each controller:

1. Introduction 2. Related work 3. TORCS competition 4. Controllers 5. Controller learning by imitation 6. Results 7. Conclusions 8. Future works

For the “controllers learning by imitation” we used the data of all tracks to train the ANN of the controller and then test it in each track.

The time obtained by the controllers described before each track

The results of the learnt controllers for all the 3-controllers.

A controller created with the data of human, Simmersons and handcoded controllers was not created because they did not get good result with mixed configurations, as shown in the last 3 tables.

1. Introduction 2. Related work 3. TORCS competition 4. Controllers 5. Controller learning by imitation 6. Results 7. Conclusions 8. Future works

It is very complicated to learn the human behavior in a video game:

• The human makes different actions in same situations.

• He does not make all actions in the proper way.

• He makes a lot of mistakes that have to solve.

This sort of data makes completely impossible that an ANN could learn something useful.

The gear problem: the gear change has not been learned, despite it is probably the easiest output to learn. Maybe it is because of the high amount of data and due to the gear is also an input of the ANN.

The mixed data from two types of controllers – the result show that those controllers do not work.

The controller learned has mixed features, but non of these is learned properly, so the car goes out of the track easily and the simulation ends.

1. Introduction 2. Related work 3. TORCS competition 4. Controllers 5. Controller learning by imitation 6. Results 7. Conclusions 8. Future works

Pre-process of the data before use it to train the neural network. Two ideas:

To decrease the amount of data / remove duplicates.

Data pattern with same input and different output must be removed.

Train the controllers with some data of one controller.

For example, the data of the straights of one controller that performs the straights good and the data of the curves of another controller that make well the turns.

If we want to imitate the human behavior:We have to increase the information used to

train the ANN. There is a lack of information because the human player senses more information from the domain that the other controllers use. The human also remember and improve his/her behavior in each lap.

The human also not perform in the same action under the same circumstances.

The ANN:There is no context information, the

controller does not remember its past actions and cannot take decision with that information.

Maybe recurrent neural networks need to be used and maybe we have to try different learning algorithms.