Upload
ashley-allison
View
225
Download
0
Tags:
Embed Size (px)
Citation preview
Machine Learning in Computer Games
Machine Learning in Computer Games
Marc Ponsen
11/29/04
2
Game AI: The Last Frontier
“Progress in graphics and sound has slowed in recent years… Now, more than ever, good game play is at the forefront and AI is one of the most critical components (Rabin 2004).”
“… Interactive computer games provide a rich environment for incremental research on human-level AI… Populating games with realistic, human-level characters will lead to fun, challenging games with great game play (Laird 2000).”
3
Why Learning of Game AI?
The process of learning in games generally implies the adaptation of behavior for opponent players in order to improve performance
Self-correction Automatically fixing exploits
Creativity Responding intelligently to new situations
Scalability Better entertainment for strong players Better entertainment for weak players
4
Online – during gameplayAdapt to player tacticsAvoid repetition of mistakesRequirements: computationally cheap, effective,
robust, fast learning (Spronck 2004) Offline - before the game is released
Devise new tacticsDiscover exploits
Offline vs. Online Learning
5
Machine Learning Techniques
6
Overview
Reinforcement Learning (RL) Introduction to RLRL application: Dynamic ScriptingDemo NeverWinter Nights ©
Evolutionary Algorithms (EA) Introduction to EAEA application: EA in Stratagus/WargusDemo Wargus
7
Introduction to Reinforcement Learning Techniques Framework based on punishments and rewards Maximize the frequency of rewards Minimize the frequency of punishments RL is popular research area because:
RL can solve wide variety of complex problems RL will find close to optimal solution RL learns as it interacts with the environment
8
How to provide Punishments & Rewards Weight adaptation function based on fitness
score(s) Lookup tables Neural networks
9
When to provide Punishments & Rewards Reward when AI achieves objective or the opponent
finds itself in a state where it can’t achieve its objective
Reward when AI does something to increase the chance of achieving objective (guided rewards)
Punish when opponent has reached objective Punish when AI does something to decrease the
chance of achieving objective (guided negative rewards)
Using both overall and guided (negative) rewards, speeds up the learning process
10
RL application: Dynamic Scripting
Dynamic Scripting (DS) is an online learning technique inspired by RL
Original implementation of DS (Spronck 2004) in the Computer RolePlaying Game NeverWinter Nights ©
11
Dynamic Scripting
Rulebase A
Rulebase B
Script A
Script B
Combat
generate
generate script
script
scripted
scripted
control
control
human control
human control
weight updates
team controlled by human player
team controlled by computer
A
B
12
Dynamic Scripting and Requirements
Computationally Cheap - Script generation and weight updates once per encounter
Effective - Rules are manually designed Robust - Reward/penalty system Fast Learning – Experiments showed that DS
is able to adapt fast to an unchanging tactic
13
NeverWinter Nights © Demo
14
Dynamic Scripting in Wargus
Different rulebases for different game states State transition on constructing a building that
allows new units or new research
15
Rules in Rulebases
12 Build rules 9 Research rules 4 Economy rules 25 Combat rules
AiNeed(AiBarracks)
AiResearch(AiUpgradeArmor1)
AiNeed(AiWorker)
AiForce(1, {AiSoldier, 9})AiWaitForce(1)AiAttackWithForce(1)
16
Script Generation
State 1Rulebase
State nRulebase
StartRule1.1
BuildRule 1.x
Rulen.1
State 20Rulebase
Rule20.20
AttackLoop
Rule20.19
17
Tactics Two `balanced’ tactics
Small Balanced Land Attack (SBLA)Large Balanced Land Attack (LBLA)
Two `rush’ tacticsSoldier Rush (SR)Knight Rush (KR)
18
Dynamic Scripting Test
Dynamic player (using dynamic scripting) plays 100 consecutive games against static player
Randomisation Turning Point (RTP): First game that dynamic player outperforms static player with 90% probability according to a randomisation test (Cohen, 1995)
19
Dynamic Scripting RTP Results
Tactic Tests Low High Avg. Med. >100 Won
SBLA 31 18 99 50 39 0 59.3
LBLA 21 19 79 49 47 0 60.2
SR 10 10 1.2
KR 10 10 2.3
Dynamic Scripting WORKS! It adapts efficiently against SBLA and LBLADynamic Scripting is unable to adapt to the optimized tactics SR and KR
20
Improve Domain-Knowledge Reliability
Successful adaptive game AI is based invariably on domain knowledge of the game it is used in
Step 1: Use EA to evolve new tactics to deal with problematic static opponents
Step 2: Translate evolved tactics to improved domain knowledge (rulebase revision)
21
Introduction to Evolutionary AlgorithmsIdea: Biological analogy on how populations of species evolve over
generations
Step 1: start with a population (each member is a candidate solution)
…
Step 2: Create the next generation by considering evolutionary operations on the population from the previous generation (genetic operators such as mutation or crossover) and a fitness function (only the more fit get to contribute to the next generation)
…
Continue the process until a certain condition is reached
22
Chromosome
Start State 1 State 2 End
State marker Rule x.1 Rule x.2 Rule x.n
State m
Chromosome
State
Rule ID Parameter 1 Parameter p Rule Parameter 2
Start S C1 2 5 def B S E 8 R 15 B 4 3 S
State number x
1 3 4
Rule 1.1 Rule 1.2 Rule 3.1 Rule 3.2 Rule 3.3
State 1 State 3
AiForce(2, {AiSoldier, 5})AiForceRole(“defend”)
23
Genetic Operators State Crossover Rule Replace Mutation Rule Biased Mutation Randomization
Start State 1 State 2 End Parent 1 State 6 State 8 State 12 State 13 State 16 State 19 State 20
Start State 1 State 2 End Child State 6 State 8 State 12 State 13 State 16 State 19 State 20
Start State 1 State 3 End Parent 2 State 4 State 8 State 12 State 13 State 14 State 17 State 20
24
Wargus Demo
25
Evolution Results
Tactic Tests Low High Avg.
SR 10 0.73 0.85 0.78
KR 10 0.71 0.84 0.75
EA was successful in rapidly discovering tactics able to defeat both rush tactics
26
Analysis of Counter-Tactics
Counter `soldier rush’ with (enhanced) `soldier rush’
Against `knight’s rush’Expand territory when defendableQuickly develop strong unitsUse catapults
27
Rulebase Improvement
New rulesAnti-SR: create strong soldiers and attackAnti-KR: if a `knights’ state can be reached, do so
and start building knightsDefendable expansion: build defensive force then
expand Changes
Adjustment of parameters of several combat rules
28
Improved Rulebase Results
Tactic Tests Low High Avg. Med. >100 Won
SBLA 11 10 34 19 14 0 72.5
LBLA 11 10 61 24 26 0 66.4
SR 10 10 27.5
KR 10 10 10.1
18 99 50 39 59.3
19 79 49 47 60.2
1.2
2.3
Enhanced performance against ALL tactics!
29
Conclusions
Dynamic scripting can be successfully implemented in (commercial) CRPG & RTS games
Offline evolutionary learning can design counter-tactics for RTS games
Evolved counter-tactics can be used to improve domain knowledge used by adaptive game AI
30
References
Spronck, P., I. Sprinkhuizen-Kuyper, and E. Postma. 2004a. “Online Adaptation of Game Opponent AI with Dynamic Scripting.”
Sweetser, P. (2002). Current AI in Games: A review Manslow. “Learning and Adaptation” & “Using reinforcement
learning to Solve AI Control Problems” Laird, J.E. 2000. “Human-Level AI's Killer Application:
Computer Game AI.” Ponsen, M. Spronck. P. 2004 “Improving Adaptive AI with
Evolutionary Learning”