Basic AI Techniques for NPC Behaviours: FSTNfor NPC Behaviours: FSTN

Finite-State Transition Networks

B

13

ab

Action

B

13

ab

Action

Team Buddies (SCEE)™

A

C

D2

3

1

dState

Percept

Transition

ActionA

C

D2

3

1

dState

Percept

Transition

Action

Introduction

• Behaviours characterise the possible actions that NPCs or automatic players will perform in reaction to, e.g.:

– their physical environment

(e.g. Lemmings)

– human player’s location or real-time actions (e.g.

Doom, Tekken, Air Combat, etc.)

– current phase of the game

(e.g. RISK)

– Etc.

“Intelligent” Behaviours

• We here consider behaviours to be restricted to “intelligent behaviours” (i.e. AI-based)

• Computation of a trajectory in a racing game would not fall under that concept not fall under that concept --- only if based on kinematics.

• … but decisions to undertake/attack other cars/vessels do!

Behaviours and Gameplay

• Complex behaviours implement more intelligent opponents.

• They can also define a kind of “personality” for NPCs or opponents.

• The more you formalise behaviours, the more you make some gameplay concepts explicit.

Finite-State Systems

• They describe devices that can take discrete states.

• They can represent compiled plans.compiled plans.

• Two main concepts:

–SSSStates–TTTTransitions

FSTN

A

B

D

13

A

C

D2

3

1

State

Percept

Transition

(adapted from A. Whittaker)

FSTN (2)

A

B

D

1

2

3

ab

ActionA

C

D2

3

1

dState

Percept

Transition

Action

(adapted from A. Whittaker)

FSTN (3)

B3

iii

iii

Register

A

C

D

1

2

3

3

1

ab

dState

Percept

Transition

Action

(adapted from A. Whittaker)

Example: Quake™

• The artificial opponents go through a cycle of:

– being idle,

– attacking,

– reacting to attack,

– etc.

• Their behaviour is based on the definition of:

– the cycle of states and,

– the transitions between such states.

The “Original” Quake™ FSA

Quake FSA (comments)

attackattackattackattacksees enemy

kills enemy

gets killed

gets hurt

Compute LOS Compute damage

idleidleidleidle deaddeaddeaddead

kills enemy

loses interest loses sight

of enemy

gets hurt

Compute LOSCompute, e.g. time limit

retaliate

Search procedure

Change

appearancesearchsearchsearchsearch hurthurthurthurt

• For a given FSA, you can fine-tune behaviour:– You can vary the time limit for losing interest to make

characters more persistent.

– You can alter line of sight.

Quake FSA (Quake FSA (Quake FSA (Quake FSA (commentscommentscommentscomments))))

– You can have them escape if hurt, rather than retaliate.

• A good formalisation gives you the basis to make consistent changes and/or improvements.

Prefer FSA if:

• Small set of clearly identified states.

• Meaningful transitions.

• Clear pre-conditions (action representation).

• A priori definition, with little need for • A priori definition, with little need for change or experimentation.

• Variant: cascaded FSA.

FSA: Representing Transitions

FSA: from description to interpretation

• FSA are generally described for parsing

sequences of tokens.

• When they describe (compiled) plans or behaviours, you have to compute the next behaviours, you have to compute the next

transition, rather than accepting a token and checking compatible transitions.

Developing FSTN Systems

• The development of FSTN systems comprises several steps:

– Collecting the data.

– Formalising FSTN.

– Choosing an implementation tool/method.

– Testing.

– Maintenance.

Maintenance

• It is often difficult to manage a large large large large number of FSTN.

?

• This is a problem when adding new FSTN:

FSTN are a procedural formalism (and not declarative) and, as such, rather difficult to maintain

?? FSTN:

• integrating• testing

FSTN ExampleFSTN Example

Queuing Agents (1)

The cheeky agent

The violated agent

An available position

The ‘cheeky’ agent will

need to move to this

position before they

can advance in the

queue

Figure 2.2

How an agent jumps the queue

Queuing Agents (2)

Introduce “conflict”Introduce “conflict”

Baseline Behaviour

“Double” Skip

Successful Skip

Cheating Fails …

Finite-State Transition Networks

Finite State Machines (FSMs) Finite State Machines (FSMs) - Implementation Issues -

FSMs

• are a simple and efficient method to implement many game features.

• may be diagrammed using a standard diagram • may be diagrammed using a standard diagram format called a directed graph, which is easy to read and understand, even for non-programmers (this facilitates discussions with the Design team).

FSMs (2)

• describe under which events/conditions a current state is to be replaced by another

– for example, switching from an attack mode to an

escape mode if the NPC is hit.

• is mostly a design concept

– i.e. the game has no general FSM interpreter, but the

FSMs are usually created using individual entities

(scripts, or else).

Ghosts in PacMan

Ghosts’ behaviour in PacMan

• All ghosts have the same Evade state,

• Each ghost has its own Chase state,

– the actions of which are implemented differently for

each ghost.each ghost.

• The input of the player eating one of the power pills is the condition for the transition from Chase to Evade.

• The input of a timer running down is the condition for the transition from Evade to Chase.

Quake-like Bots

Bots’ behaviour

• Bots include states such as

– FindArmor, FindHealth, SeekCover, and RunAway.

• Even the weapons in Quake implement their own • Even the weapons in Quake implement their own mini finite state machines.

– E.g. a rocket may implement states such as

Move, TouchObject, and Die.

FIFA 2002

Football players in FIFA 2002

• Players’ behaviour can be defined via states such as:

– Strike, Dribble, ChaseBall, and MarkPlayer.

• In addition, the teams themselves are often implemented as FSMs and can have states such as:

– KickOff, Defend, or WalkOutOnField.

World of Warcraft

RTS

• NPCs in RTSs (real-time strategy games) such as World of Warcraft also make use of finite state machines.

• They have states such as:

– MoveToPosition, Patrol, and FollowPath.

Example for RTS genre

Example for FPS genre

State Transition Tables

• A better mechanism for organizing states and affecting state transitions.

• Definition: a table of conditions and the states those conditions lead to.those conditions lead to.

State Transition Tables (2)

• table can be queried by an agent at regular intervals,

• enabling it to make any necessary state transitions based on the stimulus it receives from the game environment. environment.

• Each state can be modeled as a separate object or function existing external to the agent, providing a clean and flexible architecture.

• One that is much less prone to confusion than embedding all transitions in an if-then/switch

Embedded Rules

• An alternative approach is to embed the rules for the state

transitions within the states themselves.

• States are encapsulated as objects and contain the logic

required to facilitate state transitions.required to facilitate state transitions.

• Each state is a self-contained unit and not reliant on any

external logic to decide whether or not it should allow itself

to be swapped for an alternative.

Bot class

• Imagine a Bot class that has member variables for attributes such as health, anger, stamina, etc., and an interface allowing a client to query and adjust those values. those values.

• A Bot can be given the functionality of a finite state machine by adding a pointer to an instance of a derived object of the State class, and a method permitting a client to change the instance the pointer is pointing to.

Process

• When the Update method of an Agent is called, it in turn calls the Execute method of the current state.

• The current state may then use the Bot interface • The current state may then use the Bot interface to query its owner, to adjust its owner’s attributes, or to effect a state transition.

• In other words, how a Bot behaves when updated can be made completely dependent on the logic in its current state.

References

• Programming Game AI by Example by Mat Buckland ISBN 1-55622-078-2

• Finite State Machine Tutorial• Finite State Machine TutorialBy Nathaniel Meyer http://www.generation5.org/content/2003/FSM_Tutorial.asp

… Even Better

• Develop a visual programming interface when FSTN are “drawn”

• Generates code corresponding to this FSTN • Generates code corresponding to this FSTN

• Communication between NPC/Bot’s states and those produced by the FSTN code

Quake™: from FSTN to RuleQuake™: from FSTN to RuleQuake™: from FSTN to RuleQuake™: from FSTN to Rule----based based based based SystemsSystemsSystemsSystemsSystemsSystemsSystemsSystems

(introducing SOAR-Quakebot by John Laird)

Quake III NPC AI

Most material from [van Waveren and Rothkrantz, 2001]

Quake’s FSTN

Layered Architecture

Team leader AI

AI NetworkMisc. AI Commands

Fuzzy Character Goals ChatsNavigation

Area Awareness System Basic Actions

Layered Architecture

• 1st layer: I/O layer for the bot (basic actions are the output)

• 2nd layer: “subconscious” intelligence, “fuzzy logic” for goal selection

• 3rd layer: mixture of production rules and “AI • 3rd layer: mixture of production rules and “AI network” (FSTN). High-level reasoning: navigation and fighting

• 4th layer: for team leaders (co-ordination knowledge)

Game “AI Network”

Seek

Long-Term Goal

Seek

Short-Term Goal

Seek

Activate Enemy

Stand Respawn

Battle

Short-Term GoalBattle RetreatBattle ChaseBattle Fight

Integration in the Game Engine

Game

Server

Bot AI (layers 3-4)

Bot AI (layers 1-2)

networking

Client

Client Game

Renderer

Client code

provided the IO

functionality for

Human players

3D image

networking

Area Awareness System

• Provides Information about the current state of the world

• Includes: navigation, routing, other entities …

• Based on a 3D representation of the world• Based on a 3D representation of the world

• 3D bounded hulls (= areas)

• Everything the bot senses goes through the AAS

Area Awareness System

• The map is subdivided into convex hulls using Binary Space Partitioning (BSP)

• Areas specific property: the navigation • Areas specific property: the navigation complexity for travelling between any two reachable points in the area is minimal

• It is further required to compute reachability

“Terrain Reasoning”

• Predicting out-of-sight threats

• Dynamic situations (seeking cover, concealed paths)paths)

• Waypoint resolution

• Terrain pre-processing

Rule-based decisions

IF the bot is fighting

AND the bot is low on health

AND the bot does not have a powerful weaponAND the bot does not have a powerful weapon

THEN retreat from the fight

Rule-based decisions

IF the bot is fighting

AND the bot is not fit enough to fight

THEN retreat from the fightTHEN retreat from the fight

(but in Quake III, these are essentially implemented in several

procedures, not taking advantage of rule modularity)

“Fuzzy” Coefficients

• Inspired from Fuzzy Logic

• Fuzzy relations are used to express the relation between the bot’s current state and possible desires/goals

• Example: based on which weapon the bot is • Example: based on which weapon the bot is holding, and how much ammo the bot has for the weapon, the bot can attach a fuzzy value to its desire for more ammo

• Also used in conjunction with rules

Bot Characters

• 25 characteristics: –name, gender, aggression, alertness,

jumper, walker, attack skills, aim jumper, walker, attack skills, aim

accuracy, weapon weights, item weights,

chats, etc.

Results

• Quake III Arena bot

• Soar-Quakebots• Soar-Quakebots

• Performance vs. human-like ability

• Build an AI Engine around the Soar AI architecture– Soar/Quake II

– Soar/Descent 3

– Soar/Half-life

Soar/Games Project

Quake II

• Build autonomous, unscripted AIs

Interface

DLL

AI Engine

(Soar)

Knowledge

FilesActions

Sensor Data

© J. Laird

Soar

• AI engine to support multi-method problem solving

– Applied to wide variety of tasks and methods

• Proposed unified theory for modeling human cognition

– Models a wide variety of human behavior: language, HCI, ...

• Combines reactive and goal-directed symbolic processing• Combines reactive and goal-directed symbolic processing

• Supports very large bodies of knowledge (>100,000 rules)

• Optimized implementation in ANSI C

• In the public domain

© J. Laird

Soar Quakebot

• Goal: Human-like behavior– Fun and challenging to play against

– Not super-human

– Currently plays as well as good player

• Completely implemented in Soar– > 750 rules– > 750 rules

– Spread across operators for exploring, wandering, chasing, …

• Sensing similar to human

• Runs on separate PC from game– Uses ~5-10% of 400 MHz Win-95/98/NT machine

© J. Laird

Execution Flow of an AI Engine

Sense

Think

G

A

M

Simulated Perception

Self: orientation, position, health

Senses: vision, hearing, smell, taste, touch

Augmented Senses: radar, night-vision, …

> 80 sensors in Quakebot

Obstacle

Field of Vision

Think

Act

M

E

Simulated Actions

Movement, weapons’ use, communication

~20 actions in Quakebot

Obstacle

© J. Laird

Execution Flow of an AI Engine

Sense

G

A

Finite-state machines

Subsumption

Think

Act

A

M

E

?Neural nets

Rule-based systems

C code

Planning systems

© J. Laird

Rule-based Systems

• Program = set of if-then rules, not sequential code.

– Use whatever knowledge that is relevant to current

situation

Rule MemoryTactics, Movement Knowledge, ...

Working Memory

sensing, elaborations, persistent memories, goals, actions …

[ ]

[ ] [ ][ ]

[ ]

[ ]

[ ]

[ ]

Match Changes

[ ] [ ][ ]

[ ] [ ][ ]

© J. Laird

Rules

• Conditions contain variables and other first-order tests– If sense a weapon and do not have that weapon,

pickup the weapon.

(p rule1

(sense weapon ?weapon1)

(self weapon <> ?weapon1)

����

(action pickup ?weapon1))

• A single rule can have multiple instantiations– A single condition can match multiple working memory

elements

• Compilers convert rules into discrimination networks

© J. Laird

Which Rule Should Fire?

• Matching finds which rules can fire in current situation.

Rule

Memory

Working [ ] [ ] [ ] [ ]

[ ] [ ]

[ ] [ ]

[ ] [ ]

Rule Instantiations

?Conflict resolution

Match

• Conflict resolution picks which rules actually fire.

– Based only on syntactic features of rules and data.

– Must engineer rules so correct rule is selected each time.

Working

Memory

[ ] [ ] [ ] [ ]

[ ] [ ] [ ] [ ]

[ ] [ ]

[ ] [ ]

© J. Laird

Weaknesses in Rules

• Conflict resolution based on syntactic features – Not knowledge of the task

• Rules combine three types of knowledge about actions– What is possible– What is possible

– What should be done (based on conflict resolution)

– How to do it

Leads to duplication if have multiplicity in one of the types

• No hierarchical organization of knowledge and goals

© J. Laird

Soar’s Approach

• Organize knowledge as operators:

– Primitive & abstract actions

– Turn, move, climb, goto-door, get-item, wander, attack, chase

– Provides trace of behavior comparable to human decisions

• Use rules to propose, select, and apply operators

– Rules fire in parallel - act as an associative memory– Rules fire in parallel - act as an associative memory

– Provides fine-grain, situation-dependent behavior

– Conditional selection and execution determined dynamically

© J. Laird

Execution Flow of Soar Engine

Elaboration

Propose Operators

Sense

G

A

Soar

Evaluate Proposed Operators

Select One Operator

Perform Operator Actions

Think

Act

A

M

E

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Soar Quakebot Top Operators

WanderAttack ExploreRetreat Record

Enemy

Remove

Enemy

Chase

© J. Laird

Soar Quakebot Top Operators

WanderAttack ExploreRetreat Record

Enemy

Remove

Enemy

If enemy visible and

my health is > very-low-health-value (20%) and

his weapon is not much better than mine

THEN propose attack

Chase

THEN propose attack

If enemy visible and

my health is < very-low-health-value (20%) or

his weapon is much better than mine

THEN propose retreat

If no enemy visible or recorded and explored level

THEN propose wander

If no enemy visible or recorded and not explored level

THEN propose explore

© J. Laird

Soar Quakebot Top Operators

WanderAttack ExploreRetreat Record

Enemy

Remove

Enemy

If enemy just became not visible

and no enemy visible or recorded

THEN propose record-enemy

Chase

If hear enemy and no enemy visible or recorded

If record-enemy selected

THEN record enemy’s last position and time + 20 seconds.

If hear enemy and no enemy visible or recorded

THEN propose record-enemy

If time > recorded enemy’s saved time

THEN propose remove-enemy

If remove-enemy is selected

THEN remove recorded enemy structure

If there is a recorded enemy structure

THEN propose chase

© J. L

aird

Example Rules

IF I see a weapon that is much better than any I have

THEN I need that weapon

Attack

Get-item Face-enemy Side-step Approach Shoot

IF attacking an enemy and

there is an object I need and that

object is closer than attack-get-item-range

THEN propose get-item for that object

IF my health is less than low-health-value (40%)

THEN I need any health object

IF two get-item operators are proposed

THEN prefer selecting the one for the closer object

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Get item

Go through

door

Get item in room

Detect item

missing

Face item Move

to item

Stop

moving

Exit

room

Go to

door missingto itemmoving

to item

room door

Align

with door

Face

door

Move

to door

Record

at door

Shoot Slide to

door

Stop move

to door

Stop slide

to door

© J. Laird

Example Tactics

• Collect-Powerups– Pick up best weapons from spawn locations

– Remember when missing items will respawn

– Use shortest paths to get objects

– Get health/armor if low on health/armor

– Pickup up other good weapons/ammo if close by to deny enemy

• Attack • Attack – Use circle-strafe

– Move to best distance for current weapon

• Chase enemy based on sound of running

• Ambush in corner that can’t be seen by enemy

• Hunt at nearest spawn room after killing enemy

Mapping

• Need information on location of walls, doors, etc.– Many tactics require this information

• Built up through exploration of a level– Similar to a robot exploring with range sensors

– Saved and reused when return to level

• Represented internally as a graph structure– Currently restricted to 2D rectangles

© J. Laird

© J

. L

aird