Minimax Pathology

Minimax PathologyMinimax Pathology

Mitja Luštrek 1, Ivan Bratko 2 and Matjaž Gams 1

1 Jožef Stefan Institute, Department of Intelligent Systems2 University of Ljubljana, Faculty of Computer and Information

Science

2005-11-17

Mitja Luštrek

Plan of the talkPlan of the talk

What is the minimax pathology Past work on the pathology A real-valued minimax model Why is minimax not pathological Why is minimax beneficial

2005-11-17

Mitja Luštrek


2005-11-17

Mitja Luštrek

What is the minimax pathologyWhat is the minimax pathology Conventional wisdom:

the deeper one searches a game tree, the better he plays;

no shortage of practical confirmation.

Theoretical analyses: minimaxing amplifies

the error of the heuristic evaluation function;

therefore the deeper one searches, the worse he plays;

Pathology!

2005-11-17

Mitja Luštrek

The pathology illustratedThe pathology illustrated

Current position

Game treeFinal values(true)

2005-11-17

Mitja Luštrek


Static heuristicvalues (with error)

Final values(true)

Current position

2005-11-17

Mitja Luštrek


Backed-up heuristicvalues (should be moretrustworthy, but have larger error instead!)

Minimax

Current position

Static heuristicvalues (with error)

Final values(true)

2005-11-17

Mitja Luštrek


Static heuristic values(with smaller error)

Current position

Final values(true)

2005-11-17

Mitja Luštrek


2005-11-17

Mitja Luštrek

The discoveryThe discovery First discovered by Nau [1979]. A year later discovered independently by Beal [1980].

Beal’s minimax model:1. uniform branching factor;2. position values are losses or wins;3. the proportion of losses for the side to move is constant;4. position values within a level are independent of each

other;5. the error is the probability of mistaking a loss for a win

or vice versa and is independent of the level of a position.

None of the assumptions look terribly unrealistic, yet the pathology is there.

2005-11-17

Mitja Luštrek

Attempts at an explanationAttempts at an explanation Researchers tried to find a flaw in Beal’s model by attacking

its assumptions.

1. Uniform branching factor: geometrically distributed branching factor prevents the

pathology [Michon, 1983]; in chess endgames asymmetrical branching factor causes

the pathology [Sadikov, 2005].2. Node values are losses or wins:

multiple values do not help [Bratko & Gams, 1982; Pearl, 1983];

multiple values used in a game, which is pathological [Nau, 1982, 1983];

multiple/real values used to construct a realistic model, which is not pathological [Scheucher & Kaindl, 1998; Luštrek, 2004].

2005-11-17

Mitja Luštrek

Attempts at an explanationAttempts at an explanation

3. The proportion of losses for the side to move is constant: in models where it is applicable, it was agreed to be

necessary [Beal, 1982; Bratko & Gams, 1982; Nau, 1982, 1983].

4. Node values within a level are independent of each other: nearby positions are similar and thus have similar

values; most researchers agreed that this is the answer or at

least a part of it [Beal, 1982; Bratko & Gams, 1982; Pearl, 1983; Nau, 1982, 1983; Schrüfer, 1986; Scheucher & Kaindl, 1998; Luštrek, 2004].

2005-11-17

Mitja Luštrek

Attempts at an explanationAttempts at an explanation5. The error is independent of the level of a position:

varying error cannot account for the absence of the pathology [Pearl, 1983];

used varying error in a game and it did not help [Nau, 1982, 1983];

varying error is a part of the answer (with the other part being node-value dependence) [Scheucher & Kaindl, 1998].

Despite some disagreement, node-value dependence seems to be the most widely supported explanation.

But is it really necessary? Is there no simpler, more fundamental explanation?We believe there is!

2005-11-17

Mitja Luštrek


2005-11-17

Mitja Luštrek

Why multiple/real values?Why multiple/real values? Necessary in games where the final outcome is multivalued

(Othello, tarok). Used by humans and game-playing programs.

Seem unnecessary in games where the outcome is a loss, a win or perhaps a draw (chess, checkers).

But: in a losing position against a fallible and unknown

opponent, the outcome is uncertain; in a winning position, a perfect two-valued evaluation

function will not lose, but it may never win, either. Multiple values are required to model uncertainty and to

maintain a direction of play towards an eventual win.

2005-11-17

Mitja Luštrek

A real-valued minimax modelA real-valued minimax model Aims to be a real-valued version of Beal’s model:

1. uniform branching factor;2. position values are real numbers;3. if the real values are converted to losses and wins, the

proportion of losses for the side to move is constant;4. position values within a level are independent of each

other;5. the error is normally distributed noise and is

independent of the level of a position.

The crucial difference is the assumption 5.

2005-11-17

Mitja Luštrek

Assumption 5Assumption 5 Two-value error:

Real-value error:

+-

Loss Win

0.31 0.74

2005-11-17

Mitja Luštrek

Assumption 5Assumption 5

Beal’s assumption 5:

Static P (loss ↔ win) constant with the depth of search.

Our assumption 5:

The magnitude of static real-value noise constant with the depth of search.

P (loss ↔ win) Real-value noise

Depth Depth

Note: static = applied at the lowest level of search.

2005-11-17

Mitja Luštrek

Building of a game treeBuilding of a game tree

2005-11-17

Mitja Luštrek


True values distributed uniformly in [0, 1]

2005-11-17

Mitja Luštrek


True values backed up

2005-11-17

Mitja Luštrek



2005-11-17

Mitja Luštrek



2005-11-17

Mitja Luštrek



2005-11-17

Mitja Luštrek


Search to this depth

2005-11-17

Mitja Luštrek


Heuristic values = true values +normally distributed noise

2005-11-17

Mitja Luštrek


Heuristic values backed up

2005-11-17

Mitja Luštrek



2005-11-17

Mitja Luštrek



2005-11-17

Mitja Luštrek

What we do with our modelWhat we do with our model Monte Carlo experiments:

generate 10,000 sets of true values; generate 10 sets of heuristic values per set of true

values per depth of search. Measure the error at the root:

real-value error = the average difference between the true value and the heuristic value;

two-value error = the frequency of mistaking a loss for a win or vice versa.

Compare the error at the root when searching to different depths.

2005-11-17

Mitja Luštrek

Conversion of real values to losses and Conversion of real values to losses and winswins To measure two-value error, real values must be converted to

losses and wins.

Value above a threshold means win, below the threshold loss. At the leaves:

the proportion of losses for the side to move = cb (because it must be the same at all levels);

real values distributed uniformly in [0, 1]; therefore threshold = cb.

At higher levels: minimaxing on real values is equivalent to minimaxing on

two values; therefore also threshold = cb.

2005-11-17

Mitja Luštrek

Conversion of real values to losses and Conversion of real values to losses and winswins

Real values Two values

2005-11-17

Mitja Luštrek



Minimaxing

2005-11-17

Mitja Luštrek



Minimaxing

2005-11-17

Mitja Luštrek



Apply threshold

2005-11-17

Mitja Luštrek



2005-11-17

Mitja Luštrek



2005-11-17

Mitja Luštrek



Applythreshold

2005-11-17

Mitja Luštrek



Minimaxing

2005-11-17

Mitja Luštrek



Minimaxing

2005-11-17

Mitja Luštrek



2005-11-17

Mitja Luštrek


2005-11-17

Mitja Luštrek

Error at the root / constant static real-value Error at the root / constant static real-value errorerror Plotted: real-value and two-value error at the root. Static real-value error: normally distributed noise with

standard deviation 0.1.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 1 2 3 4 5 6 7 8 9 10

Depth of search

Err

or

at t

he

roo

t

Real-value

Two-value

2005-11-17

Mitja Luštrek

Static two-value error / constant static real-Static two-value error / constant static real-value errorvalue error Plotted: static two-value error. Static real-value error: normally distributed noise with

standard deviation 0.1.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 1 2 3 4 5 6 7 8 9 10

Depth of search

Sta

tic

two

-val

ue

erro

r

2005-11-17

Mitja Luštrek

Static real-value error / constant static two-Static real-value error / constant static two-value errorvalue error Plotted: static real-value error. Static two-value error: 0.1.

0

0.02

0.04

0.06

0.08

0.1

0.12

0 1 2 3 4 5 6 7 8 9 10

Depth of search

Sta

tic

re

al-

va

lue

err

or

2005-11-17

Mitja Luštrek

Error at the root / constant static two-value Error at the root / constant static two-value errorerror Plotted: two-value error at the root in our real-value model

and in Beal’s model. Two-value error at the lowest level of search: 0.1.

After a small tweak of Beal’s model, we get a perfect match.

0

0,05

0,1

0,15

0,2

0,25

0,3

0,35

0,4

0,45

0 1 2 3 4 5 6 7 8 9 10

Depth of search

Tw

o-v

alu

e e

rro

r a

t th

e r

oo

t

Beal's model

Real-value model

2005-11-17

Mitja Luštrek

Conclusions from the graphsConclusions from the graphs Static real-value is constant:

static two-value error decreases with the depth of search;

no pathology. Static two-value error is constant:

static real-value error increases with the depth of search;

pathology.

Which static error should be constant?

2005-11-17

Mitja Luštrek

Should realShould real-- or two or two--value value static error static error be be constant?constant? Already explained why real values are necessary. Real-value error most naturally represent the fallibility of

the heuristic evaluation function.

Game playing programs do not use two-valued evaluation functions, but if they did: they would more often make a mistake in uncertain

positions close to the threshold; they would rarely make a mistake in certain positions far

from the threshold.

2005-11-17

Mitja Luštrek

Should realShould real-- or two or two--value value static error static error be be constant?constant?

2005-11-17

Mitja Luštrek

Two-value error larger at higher levelsTwo-value error larger at higher levels Some simplifications:

branching factor = 2; node values in [0, 1]; consider only one type of error: wins mistaken for losses; consider two levels at a time to avoid even/odd level

differences.

X ... true real value of a nodeF (x) = P (X < x) ... distribution function of the true real valuee ... real-value errorX – e ... heuristic real valuet ... threshold

Two-value error:P (X > t X – e < t) = P (t < X < t + e) = F (t + e) – F (t)

2005-11-17

Mitja Luštrek

Two-value error larger at higher levelsTwo-value error larger at higher levels We need to show that two-value error at higher levels is

larger than at lower levels: Fi – 2 (t + e) – Fi – 2 (t) > Fi (t + e) – Fi (t)

the difference in F between the points t + e and t is larger at higher levels, which means that F is steeper at higher levels.

Example: uniform distribution at the leaves, depth = 10: F10 (x) = x

F8 (x) = 4 x 2 – 4 x 3 + x 4

2005-11-17

Mitja Luštrek

Two-value error larger at higher levelsTwo-value error larger at higher levels

F8 (x) steeper than F10 (x) between x = a and x = b.

2005-11-17

Mitja Luštrek

Two-value error larger at higher levelsTwo-value error larger at higher levels

In general, two-value error at higher levels is larger than at lower levels if:F (a) = 0.1624 < F (t) < 0.7304 = F (b)

Why can we expect this condition to be true: F (t) = the proportion of losses, which is constant; a constant proportion of losses is achieved by each

player having just enough advantage after his move so that the opponent can balance it out after his move;

when one’s advantage is too large, it gets even larger at each successive level;

therefore F (t) can be expected not to be very large or very small.

2005-11-17

Mitja Luštrek

Two-value error Two-value error sufficientlysufficiently larger at higher larger at higher levelslevels We have shown that two-value error at higher levels is

larger than at lower levels. Is it larger enough?

Baseline: when searching to the maximum depth, we compute

two-value error for all levels: pi ... the error at level i;

when searching to depth d, two-value error at depth d = pd .

Two-value error at level i is larger than the baseline when:

4

)(7)(124)(6)()(

2tFtFtFetFtF iii

ii

2005-11-17

Mitja Luštrek

Two-value error Two-value error sufficientlysufficiently larger at higher larger at higher levelslevels

Pathology only when F (t + e) close to 1: F (t) not expected to be close to 1; therefore this means a very large error.

2005-11-17

Mitja Luštrek

Minimax is not pathologicalMinimax is not pathological Real-valued evaluations are necessary for successful game-

playing. If static real-value error is constant, static two-value error is

larger when searching to smaller depths. This happens because at smaller depths, node values are

closer to the threshold separating losses from wins. The pathology is thus eliminated.

Our explanation for the lack of pathology is a necessary consequence of a real-value minimax model and requires no additional assumptions.

Minimax is known to be beneficial, not only non-pathological.We still do not know where the benefit comes from.

2005-11-17

Mitja Luštrek


2005-11-17

Mitja Luštrek

PPreamblereamble Will only consider real values. Must compare searches to

different depths of the same tree: the shape of the tree does

not matter; what matters is that each

minimaxing step reduces the error;

therefore more steps mean smaller error.

Some simplifications: constant difference

between the values of sibling nodes;

branching factor = 2.

2005-11-17

Mitja Luštrek

One minimaxing stepOne minimaxing step

2005-11-17

Mitja Luštrek

Ten minimaxing stepsTen minimaxing steps

2005-11-17

Mitja Luštrek

ConclusionConclusion Theoretical analyses in the past have shown that minimax

is pathological. The explanations that followed have introduced

unnecessary complications. If the error is modeled in the way a real-valued model

suggests, the pathology disappears. Real values also lend themselves well to an explanation of

why is minimax beneficial.

An issues not yet completely resolved:if multiple discrete values are used instead of real values, which is what game-playing programs do, some of what was said is not quite true.

Thank you.Thank you.

Questions?Questions?

Documents

Minimax Pathology