Decision Making with Incomplete Information: Measures, Mediators, and Decision Support Marc C. Canellas Advisor: Dr. Karen M. Feigh Ph.D. Thesis Proposal,

1

Decision Making with Incomplete Information:Measures, Mediators, and Decision Support

Marc C. CanellasAdvisor: Dr. Karen M. Feigh

Ph.D. Thesis Proposal, School of Aerospace Engineering

Dr. Amy Pritchett, School of Aerospace Engineering, Georgia TechDr. Brian German, School of Aerospace Engineering, Georgia TechDr. Stephen Cross, School of Industrial and Systems Engineering, Georgia TechDr. Juan Rogers, School of Public Policy, Georgia Tech

Sept. 3, 2015

• What is heuristic decision making?• Decision algorithms that only use a subset of “necessary” information

• Why is heuristic decision making important?• Heuristics are used by people in time-stressed, high stakes environments (e.g. aviation,

medical, emergency response, and military domains)• Can established decision support research help support heuristic decision makers?• Previous decision support research has almost exclusively focused on supporting

normative decision making strategies which do not apply to heuristic decision making environments.

• What recent methods for heuristic decision support have been developed?• Structuring the decision making process and environment

How should systems be designed to support heuristic decision makers?

Sept. 3, 2015

• What is an unstudied area of heuristic decision support with potential?• Heuristic decision support for structuring information (information acquisition)

2

3

Issues with normative decision making

Sept. 3, 20151Orasanu and Connolly, 1993; 2Green and Mehr, 1997;3Tversky and Kahneman, 1974; 4Katsikopoulos and Fasolo,

2006; 5Elwyn et al., 2001; 6This result is the output of the naturalistic decision making and fast-and-frugal heuristics research programs

Descriptive problem: • People often do not make decisions by gathering and processing all

information (due to time, cost, experience, etc.)1

• Necessary information often not available2

• People often cannot provide reliable assessments of probabilities, attributes weights, and value3

• Lack of transparency and understanding of underlying methods4,5

Normative decision making is not a good basis for building decision support tools for these environments4.

• People are often very well-adapted to their environment and use simple rules that enable them to perform just-as-good if not, better than normative methods6

4

Normative vs. Heuristic

Sept. 3, 20151Nelson, 2005; 2Nelson, 2008; 3Meder and Nelson, 2012; 4Katsikopoulos and Fasolo, 2006; 5Todd,

Gigerenzer, and the ABC Research Group, 2012

Normative Heuristic Benefits of Heuristic Decision Making

Decision Making

Strategies

Gather and process all information about the

environment

Simple rules that process information deemed “necessary”

• Faster to perform4

• Easier to communicate4

• More accurate and robust in many environments5Information

AcquisitionQuantify the value of each

potential piece of information1,2,3 ?

Heuristic information acquisition methods have the potential tobenefit heuristic decision making performance.

Simple rules for determining which

information to acquire

5Sept. 3, 2015

Objectives and MethodDevelop methods of heuristic information acquisition

6

Research Objectives and Method

Sept. 3, 2015

1. Define measures of incomplete information

2. Develop methods of heuristic information acquisition

3: Examine the effect of distributions of incomplete information

4: Compare heuristic information acquisition methods to normative methods

5: Examine the effect of distributions of incomplete information on human decision making

6: Implement the heuristic information acquisition methods in a decision support tool

Simulation: Artificial Environment

Simulation: Empirical Environment



Human-Subjects Study Human-Subjects Study

Q: What measures define distributions of incomplete information and affect decision making accuracy?

Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?

Q2: What aspects of the environment affect when the methods work?

Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?

Q: Do the heuristic information R&G methods perform similarly to established normative information acquisition methods?

Q: What are the effects of distributions of incomplete information on human decision making?

Q: How can the heuristic information R&G methods be implemented in a decision support tool?

Complete1,2,3 Complete4 In-progress Future work Future work Future Work

7Sept. 3, 2015

Task 0: Construct a Decision Making Simulation Engine Enables comprehensive study of all combinations of environments and incomplete information

8

Example Decision Task: UAV Path Selection

Sept. 3, 2015

Three attributes of the paths: time, danger, and control.

Preferences• Minimum time is preferred• Low danger is preferred to high danger• Autopilot control is preferred to manual control

Decision Task

BASE

TARGET

B30 min

Autopilot Option Time to Target Danger Control

A 10 High Manual

B 30 Low Autopilot

A10 minManual

9

Decision Making Simulation Engine

Sept. 3, 2015

Decision Task Decision Making Strategies

Option Time to Target Danger Control

A 10 High ManualB 30 Low Autopilot

Incomplete information

Environment


A K ? KB ? K K


A 10 ? ManualB ? Low Autopilot

ChoiceEnvironment


Decision Task

The state of the world

Missing information about theoptions’ attributes

Option-attribute informationpresented to the decision maker

Process information to choose a course of action

10

Weighted Additive (WADD)1 Equal Weighting (EW)1 Tallying2 Take-Two3,4 Take-the-Best (TTB)5

Linear model with weights

Linear model without weights

Linear modelcounting positive

attributesTwo-attribute

reasoning by rankingOne-attribute

reasoning by ranking

Multi-attribute decision making6;Novice decision

makers7

When weights cannot be determined or

agreed upon8

Criminal sentencing decisions9;

Medical decision making10

People often search for a second,

confirming attribute3,4

Consumer choice11; Expert decision

makers7

Decision Making Strategies

Sept. 3, 2015Operationalizations: 1Payne et al., 1990; 2Gigerenzer and Gaissmaier, 2011; 3Karelaia, 2006; 4Dieckmann and Rieskamp, 2007; 5Gigerenzer and Goldstein, 1996; 6Park, 2004; 7Garcia-Retamero and Dhami, 2009; 8Dawes, 1979; 9von Helversen

and Rieskamp, 2009; 10Kattah et al., 2009; 11Kohli and Jedidi

Analytic:gather as much information as

possible to use in a linear model

Heuristic:Use only a subset of “necessary”

information

11

Decision Making Simulation Engine

Sept. 3, 2015

Decision Task Decision Making Strategies


A 10 High ManualB 30 Low Autopilot


Environment


A K ? KB ? K K


A 10 ? ManualB ? Low Autopilot

ChoiceEnvironment


Decision Task

The state of the world

Missing information about theoptions’ attributes

Option-attribute informationpresented to the decision maker

Process information to choose a course of action

WADD

EW

Tallying

Take Two

TTB

12Sept. 3, 2015

Task 1. Define measures of incomplete information

13

Task 1: Identify important measures of distributions of incomplete information

Sept. 3, 2015

Define important measures of distributions of incomplete information.

Identify potential methods for decision support based on incomplete information.

14

Current measures of incomplete information are not sufficient

Sept. 3, 2015

Option Time to Target Danger ControlA ? ? ?B 30 Low Autopilot

Option Time to Target Danger ControlA 10 ? ?B 30 Low ?

Equivalent total information

3 pieces(50%)

Total information is not sufficient to understand the full effects of incomplete information.

1 2

15

Measuring distributions of incomplete information

Sept. 3, 20151Martignon and Hoffrage, 2002; 2Garcia-Retamero and Rieskamp, 2008; 3Canellas et al., 2014; 4Canellas

et al., 2015; 5Canellas and Feigh, 2014

Measure Definition MeasuresTotal Information (TI)1,2 Count (percent) of attribute scores

known- The amount of information known to the

decision makerInformation Imbalance (II)3,4 Difference in total information for

each option.- Approximates the number of known-

unknown attribute comparisons typically used by heuristics

- Situations where one option is well-known to the decision maker

Complete Attribute Pairs (CAP)5 Count (percent) of attributes in which attribute scores are known for both options.

- The number of known-known attribute comparisons

16

Measuring distributions of incomplete information

Sept. 3, 2015


A ? ? ?B 30 Low Autopilot


A 10 ? ?B 30 Low ?

Total Information 3 (50%) Know half the information

Information Imbalance 3 (100%)

Know all the information about

one option and nothing about the

other

Complete Attribute Pairs 0 (0%)

No attributes have information for both

options.

Total Information 3 (50%) Know half the information

Information Imbalance 1 (33%)Know slightly more

information about one option than the other

Complete Attribute Pairs 1 (33%)

For one attribute, there is information about

both options.

17

Task 1. Decision Making Simulation Engine (artificial environment)

Sept. 3, 2015 Simulation set-up used in: Canellas et al., 2014; Canellas and Feigh, 2014; Canellas et al., 2015

Decision Task Decision Making StrategiesOption A1 A2 A3 A4

O1 40 20 80 40O2 40 60 40 60


EnvironmentChoice

Environment (48)

Incomplete Information (28)

Decision Task (88: 16 million)

Artificial: randomly generated attribute scores and weights.

Option A1 A2 A3 A4

O1 ? ? K K

O2 K K K K

Option A1 A2 A3 A4

O1 ? ? 80 40

O2 40 60 40 60

Accuracy is determined by a linear model that is not representative and biased toward WADD.

WADD

EW

Tallying

TTB

Two analytic strategies and two heuristic strategies.

Accuracy: percent of decision tasks that the decision making strategy selects the correct option.


Choice

18

Task 1. Effect of incomplete information on decision making accuracy

Sept. 3, 2015

0% 25% 50% 75% 100%50%

60%

70%

80%

90%

100%

Information Imbalance

0% 20% 40% 60% 80% 100%50%

60%

70%

80%

90%

100%

Total Information

Accuracy

0% 25% 50% 75% 100%50%

60%

70%

80%

90%

100%

Complete Attribute Pairs

Decision Support Recommendation:

Provide decision makers with more information.


For heuristic decision makers, provide equal information about both options.For analytic decision makers, there is no effect.


Provide decision makers with attribute information for both options.

19

4 883 84 822 79 77 771 75 72 72 720 64 65 67 68

1 2 3 4 5 6 7 8

4 1003 88 912 78 81 851 71 73 77 810 62 68 73 78

1 2 3 4 5 6 7 8

Tradeoff: Total Information – Complete Attribute Pairs

Sept. 3, 2015

Total Information

Complete Attribute

Pairs

Complete Attribute

Pairs

Take-the-Best (TTB)Weighted-Additive (WADD)

Total InformationCells are empty when that combination cannot occur in a 2-option, 4-attribute task.

• Increasing total information:• Always increases accuracy

• Increasing complete attribute pairs while keeping total information constant:• Does not increase accuracy

• Increasing total information:• Decreases accuracy (if CAP ≥ 1)

• Increasing complete attribute pairs while keeping total information constant:• Increases accuracy

20

4 883 84 822 79 77 771 75 72 72 720 64 65 67 68

1 2 3 4 5 6 7 8

Task 1: Identify important measures of distributions of incomplete information

Sept. 3, 2015

Define important measures of distributions of incomplete information.

Within levels of total information, changing information imbalance or complete attribute pairs has a statistically significant effect on heuristic decision making performance. 0% 25% 50% 75% 100%

50%

60%

70%

80%

90%

100%

Information Imbalance Identify potential methods for decision support based on incomplete information.

Identified trade-offs between total information and the two measures of incomplete information (information imbalance and complete attribute pairs)

Complete Attribute

Pairs

Take-the-Best (TTB)

Total Information

21


Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015;












1. Added new metrics: information imbalance and complete attribute pairs.

2. Identified trade-offs with total information.







Complete1,2,3 Complete4 In-progress Future work Future work Future Work

22Sept. 3, 2015

Task 2. Develop methods of heuristic information acquisition

23


Sept. 3, 2015



24

Choice


Task 2. Decision Making Simulation Engine (empirical environments)

Sept. 3, 2015 Environments from 1Czerlinski et al., 1999, used in numerous simulations (e.g. 2Hogarth and Karelaia, 2006; and 3Katsikopoulos, 2013); 4Hogarth and Karelaia, 2007

Decision Task (36 million)

Incomplete Information

Environment

Artificial: randomly generated attribute scores and weights.

Accuracy is determined by a linear model that is not representative.

WADD

EW

Tallying

TTB

Two analytic strategies and two heuristic strategies.

Empirical: 15 standard benchmarking simulation environments that represent “real-world” non-linear environments1,2,3

Measure environmental parameters4 that have been shown to affect decision making accuracy.

Measure: total information, information imbalance, and complete attribute pairs

Examine 3, 4, and 5 attribute decision tasks.

Take Two

Two analytic strategies and three heuristic strategies.

25

4 Rules of Information Restriction and Guidance

Sept. 3, 2015

Restriction(Remove 1 piece of information)

Guidance(Add 1 piece of information)

Complete Attribute Pairs Maintain the same number of complete

attribute pairsIncrease the number of complete

attribute pairs

Information Imbalance Reduce information imbalance Reduce information imbalance

26

86.6 A1 A2 A3 A4O1 K K KO2 K K

89.674.6 83.267.9 77.6

Example: Reduce information while maintaining the number of complete attribute pairs

Sept. 3, 2015

86.6 A1 A2 A3 A4

O1 K K K ?

O2 K K ? ?

86.6 A1 A2 A3 A4

O1 -12.0 -3.4 +3.0 ?

O2 -18.7 -9.0 ? ?

Removing one piece of information in a way that keeps the same number of complete attribute pairs can increase accuracy.

Total information: 5Complete attribute pairs: 2

Total information: 4Complete attribute pairs: or 21

Total information: 4Complete attribute pairs: or 21

Initial decision task with incomplete information (initial accuracy: 86.6%)

Resulting accuracy of TTB(heuristic)

Change in accuracy of TTB(heuristic)

2

27Sept. 3, 2015

If the decision maker is using Tallying, each instance of removing information via II-R increases the decision maker’s accuracy by 2.5%.

Results Summary: Reduce information while maintaining the number of complete attribute pairs

WADD EW Tallying Take Two TTB

AverageAccuracyChange

Complete Attribute Pairs – Restriction

73.0 A1 A2 A3 A4

O1 K K +3.0 ?

O2 K K ? ?

Total information: 4Complete attribute pairs: 2

Change in accuracy of TTB(heuristic)

Analytic Heuristic

28Sept. 3, 2015

StrategyComplete

Attribute Pairs- Guidance

Complete Attribute Pairs

- Restriction

InformationImbalance- Guidance

InformationImbalance

- RestrictionWADD +3.9 +3.2 +1.4 -0.5

EW +4.1 +3.3 +2.0 -0.1

Tallying +4.6 +2.0 +9.8 +2.5

Take Two +5.0 +2.1 +7.3 +1.4

TTB +4.8 +2.7 +6.3 +1.4

If the decision maker is using Tallying, each instance of removing information via II-R increases the decision maker’s accuracy by 2.5%.

Task 2 Summary: Average accuracy change of all 4 information restriction and guidance methods

By reducing information or adding information in specific ways, the accuracy of decision making strategies can be increased – heuristic decision making strategies in particular.

Each value is average accuracy change that results from using the rule once when applicable(averaged across all 15 environments and most of the 36 million runs)

Analytic

Heuristic

29


Sept. 3, 2015

Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?• For heuristic strategies, the average accuracy change is generally positive when

using the methods.• For analytic strategies, the average accuracy change generally positive for the

complete attribute pairs methods.

Q2: What aspects of the environment affect when the methods work?The methods enable decision making strategies to achieve close to their full information accuracy. Therefore, in environments where the strategies perform well, the information restriction and guidance methods perform well.

30Sept. 3, 2015

Current and Future Work

31


Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015; 4Canellas and Feigh, 2015












1. Defined information imbalance and complete attribute pairs.


1. The methods work generally for heuristic strategies and the CAP-methods work for analytic strategies.

2. Work best in environments conducive to decision strategies performance





Complete1,2,3 Complete4 In-progress In-progress Future work Future Work

32

Task 3: Examine the effect of distributions of incomplete information

Sept. 3, 2015

Using the more comprehensive simulation environment developed for Task 2:Q1. Do the effects of complete attribute pairs and information imbalance generalize

to empirical simulation environments?Q2. Do any of the environmental parameters mediate the effect of incomplete

information?

0% 25% 50% 75% 100%50%

60%

70%

80%

Information Imbalance0% 25% 50% 75% 100%

50%

60%

70%

80%

Information Imbalance

Accuracy

Artificial Environment Empirical Environment

33

Task 4: Compare normative and heuristic information acquisition methods

Sept. 3, 2015 1Meder and Nelson, 2012; 2Canellas and Feigh, 2015; Table contains hypothetical results

NormativeInformation Acquisition

HeuristicInformation Acquisition

Quantify the value of each potential piece of

information1

Adapt the normative methods to my decision task, then use the simulation engine to study:

Q1. How often and in what cases do the methods agree or disagree as to what information should be acquired or not?

Q2. How do the methods differentially affect performance of decision making strategies?

Simple rules for determining which

information to acquire

Heuristic information

restriction and guidance methods2

A1 A2 A3

O1 K K ?O2 K ? ?

?→K

?→K

WADD TTB

Normative +2% +3%

Heuristic +5% +9%

34























35

Task 5: Examine human decision making with incomplete information

Sept. 3, 2015 1Rieskamp and Hoffrage, 2008; 2Monti et al., 2012

Simulated Decision Making Strategies Choice

Environment

Incomplete Information

Decision Task with Incomplete

Information

Human Decision Maker

Q. How do distributions of incomplete information on decision making accuracy extend to human subjects?

Method: • Present decision tasks with incomplete information

• Use varying time pressure to incentivize the use of heuristic or analytic decision making strategies1

• Design decision tasks with incomplete information such that decision strategies can be identified (process- and outcome-oriented methods)1,2


A 10 ? ?

B 30 Low ?

36

Task 6: Implement the heuristic information acquisition methods in a decision support tool

Sept. 3, 2015

BASE

TARGET

B30 min

Autopilot

A10 minManual

Objective: Identify methods for implementing heuristic acquisition methods

Method: • Instead of tables of decision tasks being

presented to decision makers (Task 5), use context-relevant displays of information

• The planned task domain is a planning tool for controlling and operating a swarm of UAVs

37

Research Method






















38Sept. 3, 2015

Conclusions

39

Motivation

• Heuristics are often used when there is time pressure, high information acquisition costs, information overload, or ill-structured environments.• A majority of previous decision

support research does not apply to heuristic decision making• There is potential for heuristic

information acquisition to support decision making performance

Sept. 3, 2015

40

Contributions

Sept. 3, 2015


2. Develop methods of heuristic information

restriction and guidance (R&G)

3: Examine the effect of distributions of

incomplete information

4: Compare heuristic information R&G

methods to normative methods

5: Examine the effect of distributions of

incomplete information on human decision

making

6: Implement the heuristic information

R&G methods in a decision support tool

A new understanding of heuristic decision making with incomplete information

Completed In-Progress

How to implement this knowledge in a decision support tool.

Future

41Sept. 3, 2015

Acknowledgements• Dr. Karen Feigh, Dr. Zarrin Chua, and Rachel Haga for contributing to the research• Members of the German Research Group and the Cognitive Engineering Center

for their guidance and support• The NSF Graduate Research Fellowship Program which funds my research• The Office of Naval Research which funds the larger project which motivated this

research

Documents

Decision Making with Incomplete Information: Measures, Mediators, and Decision Support Marc C. Canellas Advisor: Dr. Karen M. Feigh Ph.D. Thesis Proposal,