Exploratory Modelling and Analysis

Jan KwakkelErik Pruyt

1

an approach for model-based foresight under deep uncertainty

Exploratory Modeling and Analysis (EMA)

“Exploratory modeling is using computational experiments to assist in reasoning about systems where there is significant (or deep) uncertainty” (Bankes, 1993)”*

•EMA was developed at the RAND Corporation •EMA represents a new way of thinking about the use of computer models to support policy making•Traditional modeling consists of consolidating known facts about a system that are then used as a surrogate for the system when confronted with uncertainties about details or mechanisms, modelers use educated guesses (resulting in best estimate predictive models).

2

*BANKES, S. 1993. Exploratory Modeling for Policy Analysis, Operations Research, 43 (3), p. 435-449.

Predictive Modeling vs. Exploratory Modeling

Exploratory Modeling• Model is used as a hypothesis

generator (“what if . . .”)• Take into account external

(scenario) uncertainty, structural (model) uncertainty, and uncertainty about valuation of outcomes

• The objective is to reason about system behavior: under which circumstances would a policy succeed or fail?

• Uses rapid assessment models, because the uncertainties may swamp model results

Predictive Modeling • Model is used to predict

• Take into account (external) uncertainty; deal with internal uncertainty using educated guesses• The objective is to predict system

behavior and whether a policy will succeed or fail

• Aims at detailed models that capture the state of the art

3

EMA Approach

• Specify policy problem• Analyze the uncertainties associated with the problem• Develop one or more fast and simple models consistent with

the available information and knowledge that allow for exploring the specified uncertainties

• Explore the behavior of these models across the ranges of the uncertainties

• Assess the implications of the exploration for policy• Iteratively modify plan in light of revealed weaknesses until

a satisfying plan emerges

4

Explore the parameter space

• Exploration versus directed search

• Exploration (sampling techniques)• Factorial methods• Monte Carlo sampling• Latin Hypercube sampling

• Directed search (optimization techniques)• Conjugant Gradient Optimization• Genetic Algorithms• Simulated Annealing• Etc.

5

Mineral Scarcity Problem

• Many crucial high-volume minerals are expected to become exhausted in the coming decades

• The disparity between the expected exponential growth of metal demand and the expected limited growth of metal supply may result in temporary and/or chronic scarcity; and

• Strategic and/or speculative behavior of countries that have a quasi-monopoly on the extraction of (rare earth) metals may seriously hinder the transition of modern societies towards more sustainable ones.

• The asynchronous dynamics of supply and demand, aggravated by reinforcing behaviors and knock-on effects, is a breeding ground for acute and/or chronic crises

• What kinds of dynamics can happen?

6

System Dynamics Model

7

PRUYT, E. 2010. Scarcity of Minerals and Metals: A Generic Exploratory System Dynamics Model. In: MOON, T. H. (ed.) The 28th International Conference of the System Dynamics Society. Seoul, Korea.

Uncertainties to be explored

• Parametric variations• Lifetime of mines • Lifetime of recycling facilities• Initial values for most stock variables• Price elasticity and desired profit margins

• Order of time delays• Building time of mines, recycling capacity

• Non linear relations captured in table functions• Learning effect• Impact shortages on price• Substitution behaviour

• In total, 27 uncertainties are jointly explored8

Results for 100 LHS runs

9

Results

Number of runs

Number of behavior patters

1000 3715000 121410000 204215000 274220000 338625000 389430000 454735000 497640000 551145000 597250000 6404

• Behavioral clustering of time series

• Each run is specified as a concatenation of atomic behavior patters

10

Airport Planning Problem• Schiphol Airport is a

environmentally constrained airport

• It loses demand to other airport• Low cost and charter to

regional airports• Long haul and transfer to

other hubs in Europe• How can the airport invest to

remain competitive, despite a wide variety of uncertainties?

11

A Fast and Simple Model for Calculating Airport Performance• A variety of tools are readily available for aspects of airport

performance (e.g. noise, emissions, capacity) • Uncertainty about the airport system itself is low• The Fast and Simple Component integrates

– FAA capacity tool (FCM)– FAA emissions tool (EDMS)– FAA noise tool (AEM)– NATS external safety methodology

• Outcomes:– Ratio capacity to demand, latent demand, size of noise

contour, average casualty expectancy, emissions

Uncertainties to be explored

• The main uncertainties faced by airports come from external forces

• We developed generators for key external forces:– Engine technology (exponential and logistic performance

increase)– Air Traffic Management technology (exponential and

logistic performance increase)– Population (logistic growth, logistic growth followed by

logistic decline)– Aviation Demand (exponential, logistic, decline)– Composition of fleet (logistic change, linear change)– Weather (parametric uncertainty)

• Together, these uncertainties result in 48 structurally different scenario generators, each of which can generate an infinite range of quantitatively different scenarios

Results

• Basic plan• Try to limit movements to 510.000 in 2020• From 2015 move up to 70.000 movements to regional

airport• In 2020, a new runway should become operational

• What is the bandwidth of outcomes for this plan given the uncertainties?

• Approach:• Conjugant gradient optimization across all the uncertainties• Multiple different initializations for each optimization to

handle local vs. global optima• Time required: roughly a week of computer simulations on

a normal desktop PC

14

Performance Bandwidth of Basic Plan

• The basic plan has a very wide bandwidth

• Plan unsuccessful in guiding the future development of the airport• Overinvestment in runways• Unnecessary moving of

operations to regional airports• How to improve the plan?

• Introduce flexibility• Specify the conditions under

which pre-specified actions are taken

• E.g. build runway only if there is a certain level of demand or certain deterioration in capacity do to wind

15

Basic plan

Noise 13 – 64 km2

Emissions 2,1 – 19,6 ton CO

External Safety

0,9 – 2,7 ACE

Ratio capacity versus demand

0,3 – 2,5

Performance Bandwidth of Adaptive Plan

• The adaptive plan significantly reduces the bandwidth of outcomes on the shown indicators across the same uncertainties

• How big is the difference in performance between the two plans?• Are there regions were the initial static plan is still better?

16

Initial Static Plan Adaptive Plan

Noise 13 – 64 km2 10 – 47 km2

Emissions 2,1 – 19,6 ton CO 1,9 – 10,3 ton CO

External Safety

0,9 – 2,7 ACE 1,1 – 2,3 ACE

Ratio capacity versus demand

0,3 – 2,5 0, 9 – 1,1

Results

17

Results

18

Static plan performs better

Results

19

Modify the adaptive plan to deal with these regions

Concluding remarks

• Deep uncertainty has to be addressed explicitly in any long-term decision making problem

• EMA offers a useful technique that allows the utilization of models to explore the implications of the uncertainties

• EMA can be used to develop dynamic adaptive strategies capable of coping with the multiplicity of plausible futures

• Research is needed• Visualizing and analyzing results of exploration• Communication of results to clients• Efficient techniques for both directed searches and open

exploration

20

21

DEEP UNCERTAINTY

22

What is uncertainty?

“Any departure from the unachievable ideal of complete determinism. Uncertainty is not simply a lack of knowledge, since an increase in knowledge might lead to an increase of knowledge about things we don’t know and thus increase uncertainty (Walker

et al. 2003)”* •Sources of uncertainty can be specified according to their nature, location and level•Nature: character of the uncertainty

• Limited knowledge, inherent variability, ambiguity•Location: which aspect of the system or model are we uncertain about

• Inputs, model structure, outputs, valuation of outputs•Level: degree of uncertainty

• Ranges from complete certainty to absolute ignorance23

*WALKER, W. E., HARREMOËS, J., ROTMANS, J. P., VAN DER SLUIJS, J. P., VAN ASSELT, M. B. A., JANSSEN, P. H. M. & KRAYER VON KRAUSS, M. P. 2003. Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support. Integrated Assessment, 4, p. 5-17.

Deep Uncertainty

• Situation where the relevant actors do not know or cannot agree on• (Aspects of) how the system works• How likely or plausible various (paths to) future states are• How to value the various outcomes of interest

• In almost all long-term decision making problems, deep uncertainty is encountered• e.g. the climate change debate

• Few techniques are readily available for offering decision support• Decision making should be based on robustness instead of

optimality• Robustness is to be achieved in part through adaptiveness

• Only take near term actions that overall have desirable consequences

• Prepare actions to be taken in light of how the future enfolds24

Treating Uncertainty in Model-Based Decision Support• A wide variety of methods and techniques are available for

dealing with low levels of uncertainty in the context of model-based decision support• Sensitivity analysis, Monte-Carlo simulation, Multi-Criteria

Decision Analysis, etc.• Deep uncertainty is more problematic in model-based

decision support• e.g. what about disagreement between experts about a

functional relationship in a model?• Conclusion: Limited capabilities for dealing with deep

uncertainty in the context of model-based decision support

25

EXPLORATION DETAILS

26

Exploration

• Generic approach• Specify the ranges for the parameters• Choose a sampling strategy• Specify the number of samples

• Useful for• Open exploration of what kind of outcomes are possible• Open exploration of what kinds of behavior can occur

• Most frequently employed strategy in EMA• Easy to execute, but big risk of information overload

27

Choosing a distribution

28

Choosing a distribution

29

DIRECTED SEARCH

30

Directed search

• Generic approach:• Conclusions are derived from a model for a specific set of

parameter values• For each parameter a range of possible values is specified• Identify under which combinations of parameter values the

conclusions are invalidated• Directed search is useful for

• Identifying the worst possible performance of a policy option• Identifying the maximum difference in performance

between several policy options• Identifying the conditions under which model behavior

changes (so called tipping points)

31

IDENTIFICATION OF PLAUSIBLE TRANSITION PATHWAYS FOR THE FUTURE DUTCH ELECTRICITY GENERATION SYSTEM 32

Case details

• Problem• Many electricity companies have to replace a large part of

their generation capacity in the coming 20 years• Will this enable a transition towards sustainable generation?

• ElecTrans Model• Agent based model of the Dutch system• Covers generator companies, network companies, and users

• Uncertainties• Operational costs of options• Investment costs• Planning horizon• Desired Return on investment• Various demand developments

33

Results

34

EMA AND SA

35

Sensitivity Analysis vs. EMA

Sensitivity analysis (SA) is the study of how the variation in the output of a mathematical model can be apportioned, qualitatively or quantitatively, to different sources of variation in the input of

the model.*•EMA is interested in exploring the behavior of a modeled system across a wide variety of uncertainties to

• determine modes of behavior• support the development of adaptive robust strategies• provide insight into the combinatorial effects of the

uncertainties•EMA and SA have a different purpose•EMA not only interested in model inputs, but also structural or model paradigmatic variations•EMA is directly related to supporting policy development

36

*Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D. Saisana, M., and Tarantola, S., 2008, Global Sensitivity Analysis. The Primer, John Wiley & Sons.