Computational Exploration in Long Term Policy Analysis for Complex

Social and Organizational Systems  

Steven Bankes

2

Many Policy Issues Are Framed by Two Related Questions

• What are the possibly significant, long-term consequences of alternative near-term actions?

– Nuclear waste storage

– Major infrastructure

– Constitutional changes

– Education reform

– Research and development

– Just about anything related to children

3

And...

• What near-term actions are most likely to achieve desired long-term objectives?

– Climate change

– Combating Terrorism

– Biodiversity

– National security

– Outer space

– Most long-term societal goals

4

But We Often Stumble When We Apply Formal Analysis to Such Decisions

Gross national product (billionsof 1958 dollars)

Energy use (1015 Btu per year)

2,200

0

2,000

1,800

1,600

1,400

1,200

1,000

800

600

400

200

0

1970

19201929

19401950

1960

1910

1973

1930

19001890

20 40 60 80 100 120 140 160 180

2000 Actual

1990

19801977

Historical trend continued

1975 conservation scenarios

5

Our Tools Have Difficulty When Prediction is No Longer Feasible

Gross national product (billionsof 1958 dollars)

Energy use (1015 Btu per year)

2,200

0

2,000

1,800

1,600

1,400

1,200

1,000

800

600

400

200

0

Historical trend continued

1970

19201929

19401950

1960

1910

1973

1930

19001890

20 40 60 80 100 120 140 160 180

1975 conservation scenarios2000 actual

1990

19801977

6

All Efforts to Apply “Foresight” to Policy Planning Face Similar Challenges

• Tendency to recapitulate conventional wisdom

• “Tyranny of the present”

• Illusion of control

• Implementing “foresight-ful” results in existing institutions

• Defining next steps (“so what?”)

• Need for prediction to inform prescription

7

• Traditional analytic methods: characterize uncertainties prior to assessing alternative decisions

– Assumes scientists provide probabilistic characterization of uncertainties so decisionmakers can rank utility of policy options

• In practice, decisionmakers often rely on a choice of strategy, not on additional information, to reduce uncertainty

– Robust Adaptive Planning formalizes this approach: characterize deep uncertainties systematically as vulnerabilities of robust strategies

• Requires scientists to describe the strengths and weaknesses of specific, alternative policy options

Characterization of Uncertainty has Strong Ties to Decisionmaking Process

Predict Act

Policy Vulnerabilities

8

We Derive Four Key Principles for Informing Decisions Under Deep Uncertainty

•Consider ensembles of large numbers of scenarios, treat models as scenario generators

•Seek robust, rather than optimal, strategies, which satisfice across a broad range of plausible scenarios and values

•Employ adaptive strategies, which evolve over time in response to new information to achieve robustness

•Use data and models to support iterative reasoning, helping users to characterize uncertainties by their implications for alternative strategies

Illustration: The Challenge of Sustainable Development

in the 21st Century

http://www.rand.org/publications/MR/MR1626

10

We Chose the “Wonderland” Model as Scenario Generator

Economy Environment

Demographics

LongevityOutput per capita Annual pollution

Valuing scenarios

Carrying capacity

Carrying capacity

Output per capita

Population

Population

Future adaptive response Carrying capacityPolicy response

11

Scenario Generation: Explore over Four Classes of Factors

Uncertainties Levers

Economic Parameters (N&S) 14 parameters Demographic Parameters 3 parameters

Environment Parameters (N&S) 6 parameters Future Generations (N&S) 6 parameters

Measures Relationships

5 parameters 14 state equations

Near term innovation policyYear for full implementation of innovation policyMinimal level of decline in carrying capacity for triggerNear term emissions intensity milestoneYear emissions intensity milestone must be met...

12

Comparative 21st Century Trajectories of the Global Scenario Group

Population (billions)

Grossworld

product($trillions)

1990

105

20

250Great Transition Conventional

Worlds

Barbarization

Fortress world

Breakdown

Policy reformMarket

Eco-communalism

New sustainabilityparadigm

13

Illustration: The Challenge of Sustainable Development in the 21st Century

Landscape of plausible futures helps illuminate key challenges to ensuring strong economic growth and a healthy environment over the

course of the 21st century.

Economic growth rate

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

India since 1960

U.S. 1890-1930

U.S. since 1950

U.S. in 20th century

China since 1960

Brazil since 1980

Russia since 1993

Conventional World scenario

Barbarization scenario

Great Transition scenario

Decoupling rate

14

Look for Robust Strategies

XLandscape of

plausible futures

Alternative strategies

Ensemble of scenarios

Robuststrategies

15

Strategies Should Be RobustAcross Multiple Measures of “Goodness”

– Use measures inspired by UN’s Human Development Index (HDI)

• Discounted, average rate of improvement in GDP/capita, longevity, and environmental quality (but no education level) time series

• Four different weightingsN$: North GDP/capita and longevity

W$: Global GDP/capita and longevity

NG: North GDP/capita, longevity, and environmental quality

WG: Global GDP/capita, longevity, and environmental quality

16

Compare “Fixed” Near-Term Strategies Across Scenarios

Near Term

Choose policies

Assume near-term policy continues until changed by future generations

Future decision-makers recognize

and correct our mistakes

Future

17

Speeding Decoupling Performs Well in Many Futures Using North HDI Measure

Slight speed-upSlight speed-up

1.0 2.0 3.0 4.0–1.0

0

5.0

0

N$ W$

NG WG

1.0

3.0

4.0

2.0

Conventionalworld scenario

U.S. in 19thcentury

U.S. since 1950

U.S. in 20thcentury

Economic growth rate

Decoupling Rate

No regretMild

A lotOverwhelming

18

But Often Fails for Global Green Measure

1.0 2.0 3.0 4.0–1.0

0

1.0

5.0

3.0

4.0

2.0

0

N$ W$

NG WG

Conventionalworld scenario

Economic growth rate

Decoupling rate

No regretMild

A lotOverwhelming

Slight speed-upSlight speed-up

19

Exploration DemonstratesNo “Fixed” Strategy Is Robust

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Conventionalworld

scenario

No regretMild

A lotOverwhelming

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Conventionalworld

scenario

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Conventionalworld

scenario

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Conventionalworld

scenario

Economic growth rate

Decouplingrate

N$ W$

NG WG

Stay the CourseStay the Course Crash EffortCrash Effort

20

Design and Examine Additional Strategies

XLandscape of

plausible futures

Alternative strategies

Ensemble of scenarios

Robuststrategies

21

Compare Adaptive “Milestone” Strategies Across Scenarios

Near Term

Choose Milestones in North and South

Future decision-makers recognize

and correct our mistakes

Future

Adjust policies over time to achieve milestones

Assume near-term policy continues until changed by future generations

22

“No Increase in Emissions Intensity” GoalPerforms Well Over Many Futures and Values

No increaseNo increase

Economic growth rate (%)

Dec

ou

plin

g r

ate

(%)

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

N$ W$

NG WG

Economic growth rate (%)1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

No regretMild

A lotOverwhelming

N$ W$

NG WG

+

Worst Case

U.S. in 19thcentury

U.S. since 1950

U.S. in 20thcentury

U.S. in 19thcentury

U.S. since 1950

U.S. in 20thcentury

23

Landscape ofplausible futures

Alternative strategiesX

Ensemble of scenarios

Robuststrategies

Look for Breaking Scenarios

24

“No Increase” Strategy Can Fail Catastrophically

No increaseNo increase

N$ W$

NG WG

Cost to speed decoupling, South

0.1 0.2 0.3 0.4

0

1.0

3.0

4.0

2.0

No regretMild

A lotOverwhelming

Dec

ou

plin

g r

ate

(%)

–1.0

5.0

0 0.5 0.6 0.7 0.8 0.9 1.0

WorstCase

+

NominalCase

X10-2

25

Landscape ofplausible futures

Alternative strategiesX

Ensemble of scenarios

Robuststrategies

Design and Examine Additional Strategies

26

Start with Milestone, but Evaluate Progress and Modify If Necessary (Safety Valve)

NODoes the carrying capacity change?

Choose policies to maximize

utility

Determine best policy to meet milestone

Select near-term milestone

YES

Is milestone achievable with

current approach?

Relax milestone

Present Future

YES

NO

Implement policy

27

“Safety Valve” Strategy Appears Highly Robust

Safety valveSafety valve

Economic growth rate (%)

Dec

ou

plin

g r

ate

(%)

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

N$ W$

NG WG

Economic growth rate (%)1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

No regretMild

A lotOverwhelming

N$ W$

NG WG

+

WorstCase

U.S. in 19thcentury

U.S. since 1950

U.S. in 20thcentury

U.S. in 19thcentury

U.S. since 1950

U.S. in 20thcentury

28

The Advisory Panel Suggested Several Potentially Stressing Surprises

– Rapid technological advance that eliminates emissions

– Plague that decimates population for twenty years

– Future generations whose values (utility) are completely disconnected from concern about the environment

29

“Safety Valve” Strategy Is Still Robust, Even with Surprises

–1.0

0

1.0

5.0

3.0

4.0

2.0

No surprise

1.0 2.0 3.0 4.00–1.0

0

1.0

5.0

3.0

4.0

2.0

Population surprise

–1.0

0

1.0

5.0

3.0

4.0

2.0

Technological surprise

1.0 2.0 3.0 4.00–1.0

0

1.0

5.0

3.0

4.0

2.0

Value surprise

N$ W$

NG WG

Economic growth rate

Rate of change in emissions intensity

30

How Can We Better Understand the Nature of the Failure Modes

No increaseNo increase

N$ W$

NG WG

Cost to speed decoupling, South

0.1 0.2 0.3 0.4

0

1.0

3.0

4.0

2.0

No regretMild

A lotOverwhelming

Dec

ou

plin

g r

ate

(%)

–1.0

5.0

0 0.5 0.6 0.7 0.8 0.9 1.0

WorstCase

+

NominalCase

X10-2

31

Massive Scenario Generation Followed By Data Mining

Insights, summaries, robust policy recommendations

32

Statistical Analysis of Scenario Ensembles Characterizes Strategies’ Vulnerabilities

Lempert, Popper, and Bankes, 2003: Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis, RAND, MR-1626-CR

• Initial scan suggests most robust strategy is “Safety Valve” with stringent near-term emissions intensity milestones & cost thresholds

• “Data-mining” method reduces 41-dimensions of uncertainties to an easy-to-interpret “Low Global Decoupling” scenario where strategy performs poorly

-0.03 -0.00288 0.03

0.0004 0.00812 0.04

-0.01 0.0139 0.05

North's Innovation Rate

Difference in InnovationRate bet. the N. and S.

North's Economic

Growth Rate

1950-99 (U.S.)

1960-99 (India)

1963-99 (Brazil)

1978-99 (China)

1993-99 (Russia)

1890-1930 &

1890-1930 (U.S.) 1950-99 (U.S.)

33

0.000

0.005

0.010

0.015

0.020

0.025

0.01 0.10 1.00 10.00 100.00

Relative Odds of a Low Global Decoupling Future

Detailed Analysis Suggests Best Safety Valve Design

1:100

1:10 1:1 10:1 100:1

SV01-0.5%-0.2%

SV02-1%-1.5%

SV01-1%-1.5%

SV02-1%-1.5%

SV02-0.5%-0.2%

SV01-1%-1.5%RobustRegions}

SVab-x%y%a = N milestoneb = S milestonex% = N cost thresholdy% = S cost threshold

34

RDM Analysis Helps Policymakers Focus on a Small Number of Key Tradeoffs

Assessment of strategies over two computer-generated scenarios

0.000 0.002 0.004 0.006 0.008 0.010 0.012

Regret in SV01.005.002 Satisficing Futures

0.00

0.02

0.04

0.06

0.08R

egre

t in

Lo

w G

lob

al D

eco

up

lin

g F

utu

res

SV02.010.015

Safety valve strategyMilestone strategy

M12

SV01.010.015

SV01.005.002

M12

SV02.005.015

M22

M0XM13

Regret in SV01.005.002 “Satisficing” Futures

Regret in low-global- decoupling futures

35

Repeated Application Can Reveal Multiple Regions

• Soft Landing (3 params)– Low population– High natural conservation– High efficiency costs

• Rapid Growth (2 params)– High population– Low natural conservation

Most relevant scenarios to the choice of policy!

5%

7%

9%

11%

13%

15%

17%

19%

21%

23%

25%

75% 85% 95% 105% 115% 125%Population [% of DOF forecast growth]

Nat

ura

lly

Occ

uri

ng

Co

nse

rvat

ion

Rapid Growth Scenario

Soft Landing Scenario

(and Efficiency MC slope > 1.04)

36

Influence Diagrams Capturing Belief Can Serve As Scenario Generator

Reading the next-use paper reveals an implicit structure:

Blue = Situational Attributes

Green = U.S. Policies

Yellow = Intermediate Variables

Orange = Longer-term Effects

Red = Aggregate Metric

37

Ten Thousand Random Cases

38

Data Mined Cluster of Bad Cases

39

Filter Discovered By Data Mining

40

Filter Dimensions Suggest New Axes

41

Picture Clarified By Filter

42

Influence Diagram For An S&T Planning Problem

43

Generate Massive Numbers of Possible Futures

44

Discover Traditional Positive Scenarios

45

As Well As Competing Options

46

Some Representative RAP/CARs Applications

• Government– Weapons acquisition– Counter-terror strategies– Pre-conflict management, anticipation and shaping– Higher education planning– Human development strategies– Social security solvency– Infectious disease – HIV/AIDS, Bird Flu– Science and technology policy planning

• Private– R&D investment planning– Product and process planning and design– Market foresight planning

47

Rigor in Exploratory Modeling

Computational machinery (models -> scenario generators) are “lab equipment”

They can be verified and calibrated, but never validated

Valid arguments require

claiming ensemble construction implies properties true for all members of ensemble should be true of the world as well

claiming to establish properties of the ensemble based on observed computational experiments

Validate Arguments, not models

48

Computer Modeling to Help Shape the Future

• Abandon the aspiration of predicting the future– Aggressively tabulate all the sources of uncertainty– To create a diverse “challenge set” of plausible

futures

• Combine human and machine creativity to discover near-term options that have low regret across all challenges

– For any plan, search for its failure modes

Co-evolve solutions and challenges to discover shaping strategies with maximal adaptive capacity

49

Backup Slides

50

Influence Diagram For S&T Planning

51

And Also Reveal Dependencies