Use of available data to probe methods for decision under

uncertainty

Raphael T. Haftka (haftka@ufl.edu)

Department of Mechanical and Aerospace Engineering

University of Florida

Gainesville, Florida 32611

With: Raluca Rosca (UF), Efstratios Nikolaidis (University of Toledo)

Outline

• Why validating design methods is trickier than validating analysis models

• Domino tower data base• Invented design/decision problem based on

domino tower data• Performance of probabilistic and possibilistic

models and effect of inflation of variance with little data

• Conclusions

Validation of methods for design under uncertainty

• When data is missing, it may not be possible to validate a method. Must compare against other methods

• Choice of method may depend on amount of information

• I give simple instructions how to get to my home to people who do not know Gainesville. I tell locals what is the shortest route.

Gigerenzer’s Approach• Use existing data bases to compare methods for

making binary decisions (choose A or B)• Invent decision problem that can be based on data• Give decision maker part of data base to formulate

decision rule• Test performance of decision rule on entire data• Showed advantage of simple decision-making

methods• Gigerenzer and Todd (1999), Simple Heuristics

That Make Us Smart, Oxford University Press

Our objectives• Extend approach to non-binary decisions,

involving choice of variable (s)

• Focus on use of existing data sets for probing weaknesses rather than strengths– Experiments are more effective for disproving

rather than for proving

• Demonstrate lessons learned from validation

Domino tower data base

• Dominoes stacked until tower collapses

• One set of 50 towers built by Rosca

• Another set of 90 towers built by 16 students in a competition

Competition and Rosca’s histograms

• Analytical model of tower collapse fit best to Gamma distribution

• Data passed chi-test also for normal distribution

0 10 20 30 40 50 60 700

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

0 10 20 30 40 50 600

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Guaranteed performance decision

• We use data to test decisions on level of guaranteed performance

• Company gains advantage over competitors by guaranteeing performance

• How to select guaranteed level?

• High target risks failure, low target risks loss of customers

Decision invented for domino data

• Rosca guarantees the height of the tower she will build

• She wins if she fulfills her guarantee and if competitor cannot build a tower higher by a handicap than her guarantee

• High guarantee risks failure to fulfill• Low guarantee risks failure due to competitor’s

tower exceeding guarantee plus handicap

Examples

• Handicap of 5, guarantee of 30 tiles

• Competitor wins by building 35, no matter what Rosca builds

• Rosca loses by building 29 even if competitor builds only 20

• Rosca wins by building 32 even if competitor builds 36

Test procedure

• Give decision maker (Rosca) five data points for both her and competitor results

• Decision maker fits distributions to each set of five points

• Guarantee selected on basis of distribution• Repeat 80 times for random quintets• Performance based on percentage of

success for all possible pairs (50x90)

Decision methods compared

• Probability based on two distributions

• Possibility based on triangular membership function spanning data

• Probability and possibility with inflated variance to compensate for scarce data

Optimum guarantees – all data

Probabilistic optimum (gamma)

Probabilistic optimum (normal)

Possibilistic optimum (triangular)

Handicap

guarantee Likelihood of success

guarantee Likelihood of success

guarantee Likelihood of success

2 32 0.307 33 0.318 33 0.318 5 31 0.411 32 0.433 32 0.433 8 29 0.563 30 0.536 31 0.513 11 27 0.640 29 0.615 29 0.615 15 26 0.724 27 0.726 28 0.737

Optimum guarantees are marked in bold. The optimum bid for a handicap of 11 is 28, corresponding to a likelihood of success of 0.6533.

Effect of inflation

Rosca Competitor Probabilistic optimum (normal fit)

Possibilistic optimum (triangular fit)

inflation factor for nhand=5

guarantee Likelihood of success

guarantee Likelihood of success

0 0 32 0.433 32 0.433 0 15 29 0.399 33 0.402 15 0 34 0.358 31 0.411 15 15 32 0.433 32 0.433

All data case; The probabilistic guarantee decreases when competitor inflation factor increases and increases when Rosca’s inflation factor increases while the possibilistic guarantee exhibits the opposite trend.

Five data points

Likelihood of success

for probabilistic optimum (normal fit)

Likelihood of success for possibilistic

optimum (triangular fit)

sample size=5

nhand Mean (of 80 runs) Mean (of 80 runs)

2 0.282 0.290 5 0.392 0.392 8 0.503 0.492

11 0.599 0.588 15 0.707 0.694

Conclusions from experiments

• Comparable performance of probability and possibility may indicate need to explain and improve probabilistic performance

• Adverse effect of inflation indicates need for theoretical study of this effect

• Probability downgraded importance of inflated mode of uncertainty and possibility emphasized it. This we only later derived analytically.

• Experiment surprised us and led us to useful insight into difference between the methods

Concluding remarks• We extended Gigerenzer’s method for using

existing data bases to test methods for decisions under uncertainty– Optimum value instead of binary decision– Repeated random partitioning of data

• Demonstrated that experiments can yield insights into decision making methods– Possible weaknesses in standard probabilistic approach

and inflation– Unreported difference between probability and

possibility