Is robustness really robust? how different definitions of robustness impact decision-making under climate change

Is robustness really robust? How different definitions of robustness impact decision-making under climate change

M. Giuliani and A. CastellettiNRM Polimi

EWRI2016 NRM

What is robustness?

“the insensitivity of system design to errors, random or

otherwise, in the estimates of those parameters

affecting design choice”

(Matalas and Fiering, 1977)

EWRI2016 NRM

Why searching robustness?

“robust strategies perform well across a wide range of

deeply uncertain scenarios”

(Lempert, 2002)

EWRI2016 NRM

Why searching robustness?

“robust strategies perform well across a wide range of

deeply uncertain scenarios”

SPM

Summary for Policymakers

21

Figure SPM.7 | CMIP5 multi-model simulated time series from 1950 to 2100 for (a) change in global annual mean surface temperature relative to 1986–2005, (b) Northern Hemisphere September sea ice extent (5-year running mean), and (c) global mean ocean surface pH. Time series of projections and a measure of uncertainty (shading) are shown for scenarios RCP2.6 (blue) and RCP8.5 (red). Black (grey shading) is the modelled historical evolution using historical reconstructed forcings. The mean and associated uncertainties averaged over 2081−2100 are given for all RCP scenarios as colored verti-cal bars. The numbers of CMIP5 models used to calculate the multi-model mean is indicated. For sea ice extent (b), the projected mean and uncertainty (minimum-maximum range) of the subset of models that most closely reproduce the climatological mean state and 1979 to 2012 trend of the Arctic sea ice is given (number of models given in brackets). For completeness, the CMIP5 multi-model mean is also indicated with dotted lines. The dashed line represents nearly ice-free conditions (i.e., when sea ice extent is less than 106 km2 for at least five consecutive years). For further technical details see the Technical Summary Supplementary Material {Figures 6.28, 12.5, and 12.28–12.31; Figures TS.15, TS.17, and TS.20}

6.0

4.0

2.0

−2.0

0.0

(o C)

4232

39

historicalRCP2.6RCP8.5

Global average surface temperature change(a)

RC

P2.6

R

CP4

.5

RC

P6.0

RC

P8.5

Mean over2081–2100

1950 2000 2050 2100

Northern Hemisphere September sea ice extent(b)

RC

P2.6

R

CP4

.5

RC

P6.0

R

CP8

.5

1950 2000 2050 2100

10.0

8.0

6.0

4.0

2.0

0.0

(106 k

m2 )

29 (3)

37 (5)

39 (5)

1950 2000 2050 2100

8.2

8.0

7.8

7.6

(pH

uni

t)

12

9

10

Global ocean surface pH(c)

RC

P2.6

R

CP4

.5

RC

P6.0

R

CP8

.5

Year

EWRI2016 NRM

Robustness in the literature: first works

RISK, AMBIGUITY, AND THE SAVAGE AXIOMS*

By DANIEL ELLSBERG

I. Are there uncertainties that are not risks? 643. II. Uncertainties that

are not risks, 647.- JII. Why are some uncertainties not risks? - 656.

I. ARE THERE UNCERTAINTIES THAT ARE NOT RISKS?

There has always been a good deal of skepticism about the

behavioral significance of Frank Knight's distinction between "meas- urable uncertainty" or "risk," which may be represented by numeri-

cal probabilities, and "unmeasurable uncertainty" which cannot.

Knight maintained that the latter "uncertainty" prevailed - and hence that numerical probabilities were inapplicable - in situations when the decision-maker was ignorant of the statistical frequencies of events relevant to his decision; or when a priori calculations were

impossible; or when the relevant events were in some sense unique;

or when an important, once-and-for-all decision was concerned.' Yet the feeling has persisted that, even in these situations, people

tend to behave "as though" they assigned numerical probabilities, or "degrees of belief," to the events impinging on their actions. How- ever, it is hard either to confirm or to deny such a proposition in the absence of precisely-defined procedures for measuring these alleged "degrees of belief."

What might it mean operationally, in terms of refutable predic- tions about observable phenomena, to say that someone behaves "as if" he assigned quantitative likelihoods to events: or to say that he does not? An intuitive answer may emerge if we consider an example proposed by Shackle, who takes an extreme form of the Knightian

* Research for this paper was done as a member of the Society of Fellows, Harvard University, 1957. It was delivered in essentially its present form, except for Section III, at the December meetings of the Econometric Society, St. Louis, 1960. In the recent revision of Section III, I have been particularly stim- ulated by discussions with A. Madansky, T. Schelling, L. Shapley and S. Winter.

1. F. H. Knight, Risk, Uncertainty and Profit (Boston: Houghton Mifflin, 1921). But see Arrow's comment: "In brief, Knight's uncertainties seem to have surprisingly many of the properties of ordinary probabilities, and it is not clear how much is gained by the distinction. . . Actually, his uncertainties produce about the same reactions in individuals as other writers ascribe to risks." K. J. Arrow, "Alternative Apprbaches to the Theory of Choice in Risk-taking Situa- tions," Econometrica, Vol. 19 (Oct. 1951), pp. 417, 426.

643

This content downloaded from 131.175.28.131 on Tue, 03 May 2016 09:15:29 UTCAll use subject to http://about.jstor.org/terms

Oxford University Press [1961].

“willingness of decision makers to sacrifice expected performance to improve robustness to

uncertainty” (Maas et al. 1962)

EWRI2016 NRM

Robustness in the literature

This content downloaded from 131.175.28.198 on Fri, 05 Jun 2015 11:51:14 UTCAll use subject to JSTOR Terms and Conditions

A Comparison of Robustness Metrics forScheduling DAGs on Heterogeneous Systems

Louis-Claude Canon and Emmanuel JeannotLORIA, INRIA, Nancy University, CNRS

Campus Scientique – BP 23954506 Vandoeuvre-les-Nancy Cedex, France

{louis-claude.canon,emmanuel.jeannot}@loria.fr

Abstract— A schedule is said robust if it is able to absorb somedegree of uncertainty in tasks duration while maintaining a stablesolution. This intuitive notion of robustness has led to a lot ofdifferent interpretations and metrics. However, no comparison ofthese different metrics have ever been preformed. In this paper,we perform an experimental study of these different metrics andshow how they are correlated to each other in the case of taskscheduling, with dependencies between tasks.

I. INTRODUCTION

Research in scheduling has gathered a lot of differentsolutions depending on the pursued objective. For instance,if the objective function to minimize is the makespan (thetotal execution time of the application) different heuristics havebeen proposed in the literature such as HEFT [17], CPOP [17],hybrid remapper [11], BIL [12], hybrid method [13] orGDL [16]. However, there are a lot of other possible objectivesthan minimizing the makespan. Among these objectives therobustness has recently received a lot of attention [1], [3],[5], [7], [14], [15]. A schedule is said robust if it is able toabsorb some degree of uncertainty in the task duration whilemaintaining a stable solution. Thus, it is important to note thatthe robustness alone is not a metric but it gives an idea of thestability of the solution with regards to another performancemetric such as schedule length, load balance of an application,queue waiting time of batch scheduler, etc. The reason whyrobustness is becoming an important objective is the recentfocus on large systems that can be dynamic and whereuncertainty in terms of workload or resource usage can be veryimportant. Moreover, a brief look at the literature shows thatdespite the fact that robustness is a very intuitive notion thereis no consensus on a single metric. Conversely, almost eachpaper uses its own metric depending on the studied problemand the general context of the work. Furthermore, there doesnot exist a comparison between these different metrics, henceit is not possible to decide which metric to use when designinga heuristic.

In this paper we focus on comparing different metricsof robustness in the context of scheduling task graph onheterogeneous systems: we model an application as a set oftasks having precedence constraints and a task as a set ofstatements. The performance metric we use is the makespan(the completion time of the application) and therefore, welook at the robustness of the makespan when tasks may have

variations in their duration. Moreover, we try to see to whichextend optimizing the makespan can help in optimizing therobustness. In other words, we try to answer the followingquestion: are short schedules more robust that long ones?In this work we also test some makespan-centric schedulingheuristics of the literature (BIL, HEFT, Hyb.BMCT) and seeon different scenarios how they perform in terms of robustness.

Therefore, the contribution of this paper is the following:we provide a comprehensive study of different robustnessmetrics in the case of task graph scheduling. We study howthey are correlated to each other and whether robustnessand makespan are conflicting objectives or not. Finally, wecompare the robustness of three different makespan-centricscheduling heuristics.

The remaining of the paper is organized as follows. InSection II we present the problem and the notations used inthis paper. Several works dealing with robustness are detailedin Section III. The robustness metrics we use are described inSection IV. In Section V we present the experimental setup weused for testing and comparing the different metrics. Resultsare shown in Section VI and discussed in Section VII. Finally,conclusion and future works are given in Section VIII

II. MODELS

We model the parallel application by a directed acyclicgraph (DAG) G = (V,E, C), where V is a set of nodesthat represent tasks and E is a set of edges that representdependencies between tasks (often due to communications). Cis the set of communication volume between tasks. The targetplatform is composed of a set of heterogeneous resourceseach having different capacities in terms of network speed.When there is no uncertainty we use two matrices to modelcommunication speed: T = (τi,j)1≤i≤m,1≤j≤m and L =(li,j)1≤i≤m,1≤j≤m, where m is the number of machines. τi,j

is the time to send one data element from processor i toprocessor j and li,j is the network latency from processori to processor j. To model the fact that communications areway faster between two tasks mapped on the same processorand thus negligible, we put ∀i ∈ [1,m], τi,i = li,i = 0.Hence, if task 1 is mapped to processor i and task 2 is mappedto processor j then the communication time between thesetwo tasks will be: li,j + c1,2 × τi,j , where c1,2 ∈ C is thecommunication volume between task 1 and task 2. As we

1-4244-1388-5/07/$25.00 © 2007 IEEE 2007 IEEE International Conference on Cluster Computing558

A Comparison of Robustness Metrics forScheduling DAGs on Heterogeneous Systems

Louis-Claude Canon and Emmanuel JeannotLORIA, INRIA, Nancy University, CNRS

Campus Scientique – BP 23954506 Vandoeuvre-les-Nancy Cedex, France

{louis-claude.canon,emmanuel.jeannot}@loria.fr

Abstract— A schedule is said robust if it is able to absorb somedegree of uncertainty in tasks duration while maintaining a stablesolution. This intuitive notion of robustness has led to a lot ofdifferent interpretations and metrics. However, no comparison ofthese different metrics have ever been preformed. In this paper,we perform an experimental study of these different metrics andshow how they are correlated to each other in the case of taskscheduling, with dependencies between tasks.

I. INTRODUCTION

Research in scheduling has gathered a lot of differentsolutions depending on the pursued objective. For instance,if the objective function to minimize is the makespan (thetotal execution time of the application) different heuristics havebeen proposed in the literature such as HEFT [17], CPOP [17],hybrid remapper [11], BIL [12], hybrid method [13] orGDL [16]. However, there are a lot of other possible objectivesthan minimizing the makespan. Among these objectives therobustness has recently received a lot of attention [1], [3],[5], [7], [14], [15]. A schedule is said robust if it is able toabsorb some degree of uncertainty in the task duration whilemaintaining a stable solution. Thus, it is important to note thatthe robustness alone is not a metric but it gives an idea of thestability of the solution with regards to another performancemetric such as schedule length, load balance of an application,queue waiting time of batch scheduler, etc. The reason whyrobustness is becoming an important objective is the recentfocus on large systems that can be dynamic and whereuncertainty in terms of workload or resource usage can be veryimportant. Moreover, a brief look at the literature shows thatdespite the fact that robustness is a very intuitive notion thereis no consensus on a single metric. Conversely, almost eachpaper uses its own metric depending on the studied problemand the general context of the work. Furthermore, there doesnot exist a comparison between these different metrics, henceit is not possible to decide which metric to use when designinga heuristic.

In this paper we focus on comparing different metricsof robustness in the context of scheduling task graph onheterogeneous systems: we model an application as a set oftasks having precedence constraints and a task as a set ofstatements. The performance metric we use is the makespan(the completion time of the application) and therefore, welook at the robustness of the makespan when tasks may have

variations in their duration. Moreover, we try to see to whichextend optimizing the makespan can help in optimizing therobustness. In other words, we try to answer the followingquestion: are short schedules more robust that long ones?In this work we also test some makespan-centric schedulingheuristics of the literature (BIL, HEFT, Hyb.BMCT) and seeon different scenarios how they perform in terms of robustness.

Therefore, the contribution of this paper is the following:we provide a comprehensive study of different robustnessmetrics in the case of task graph scheduling. We study howthey are correlated to each other and whether robustnessand makespan are conflicting objectives or not. Finally, wecompare the robustness of three different makespan-centricscheduling heuristics.

The remaining of the paper is organized as follows. InSection II we present the problem and the notations used inthis paper. Several works dealing with robustness are detailedin Section III. The robustness metrics we use are described inSection IV. In Section V we present the experimental setup weused for testing and comparing the different metrics. Resultsare shown in Section VI and discussed in Section VII. Finally,conclusion and future works are given in Section VIII

II. MODELS

We model the parallel application by a directed acyclicgraph (DAG) G = (V,E, C), where V is a set of nodesthat represent tasks and E is a set of edges that representdependencies between tasks (often due to communications). Cis the set of communication volume between tasks. The targetplatform is composed of a set of heterogeneous resourceseach having different capacities in terms of network speed.When there is no uncertainty we use two matrices to modelcommunication speed: T = (τi,j)1≤i≤m,1≤j≤m and L =(li,j)1≤i≤m,1≤j≤m, where m is the number of machines. τi,j

is the time to send one data element from processor i toprocessor j and li,j is the network latency from processori to processor j. To model the fact that communications areway faster between two tasks mapped on the same processorand thus negligible, we put ∀i ∈ [1,m], τi,i = li,i = 0.Hence, if task 1 is mapped to processor i and task 2 is mappedto processor j then the communication time between thesetwo tasks will be: li,j + c1,2 × τi,j , where c1,2 ∈ C is thecommunication volume between task 1 and task 2. As we

1-4244-1388-5/07/$25.00 © 2007 IEEE 2007 IEEE International Conference on Cluster Computing558

Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.

SIAM REVIEW c⃝ 2011 Society for Industrial and Applied MathematicsVol. 53, No. 3, pp. 464–501

Theory and Applications ofRobust Optimization∗

Dimitris Bertsimas†

David B. Brown‡

Constantine Caramanis§

Abstract. In this paper we survey the primary research, both theoretical and applied, in the areaof robust optimization (RO). Our focus is on the computational attractiveness of ROapproaches, as well as the modeling power and broad applicability of the methodology.In addition to surveying prominent theoretical results of RO, we also present some recentresults linking RO to adaptable models for multistage decision-making problems. Finally,we highlight applications of RO across a wide spectrum of domains, including finance,statistics, learning, and various areas of engineering.

Key words. robust optimization, robustness, adaptable optimization, applications of robust optimiza-tion

AMS subject classifications. 90C31, 93B40, 93D21

DOI. 10.1137/080734510

1. Introduction. Optimization affected by parameter uncertainty has long beena focus of the mathematical programming community. Solutions to optimizationproblems can exhibit remarkable sensitivity to perturbations in the parameters of theproblem (demonstrated in compelling fashion in [16]), thus often rendering a computedsolution highly infeasible, suboptimal, or both (in short, potentially worthless).

In science and engineering, this is hardly a new notion. In the context of op-timization, the most closely related field is that of robust control (we refer to thetextbooks [137] and [68] and the references therein). While there are many high-level similarities, and indeed much of the motivation for the development of robustoptimization (RO) came from the robust control community, RO is a distinct field,focusing on traditional optimization-theoretic concepts, particularly algorithms, ge-ometry, and tractability, in addition to modeling power and structural results whichare more generically prevalent in the setting of robustness.

In contrast to RO, stochastic optimization starts by assuming the uncertainty hasa probabilistic description. This approach has a long and active history dating at leastas far back as Dantzig’s original paper [61]. We refer the interested reader to several

∗Received by the editors September 5, 2008; accepted for publication (in revised form) October23, 2010; published electronically August 5, 2011.

http://www.siam.org/journals/sirev/53-3/73451.html†Sloan School of Management and Operations Research Center, Massachusetts Institute of Tech-

nology, E40-147, Cambridge, MA 02139 ([email protected]).‡Fuqua School of Business, Duke University, 100 Fuqua Drive, Box 90120, Durham, NC 27708

([email protected]).§Department of Electrical and Computer Engineering, The University of Texas at Austin, 1

University Station, Austin, TX 78712 ([email protected]).

464

Dow

nloa

ded

06/0

5/15

to 1

31.1

75.2

8.19

8. R

edist

ribut

ion

subj

ect t

o SI

AM

lice

nse

or c

opyr

ight

; see

http

://w

ww

.siam

.org

/jour

nals/

ojsa

.php

EWRI2016 NRM

Robustness in the WATER literature

RESEARCH ARTICLE10.1002/2014WR015338

Beyond optimality: Multistakeholder robustness tradeoffsfor regional water portfolio planning under deep uncertaintyJonathan D. Herman1, Harrison B. Zeff2, Patrick M. Reed1, and Gregory W. Characklis2

1Department of Civil and Environmental Engineering, Cornell University, Ithaca, New York, USA, 2Department ofEnvironmental Sciences and Engineering, University of North Carolina, Chapel Hill, North Carolina, USA

Abstract While optimality is a foundational mathematical concept in water resources planning and man-agement, ‘‘optimal’’ solutions may be vulnerable to failure if deeply uncertain future conditions deviatefrom those assumed during optimization. These vulnerabilities may produce severely asymmetric impactsacross a region, making it vital to evaluate the robustness of management strategies as well as their impactsfor regional stakeholders. In this study, we contribute a multistakeholder many-objective robust decisionmaking (MORDM) framework that blends many-objective search and uncertainty analysis tools to discoverkey tradeoffs between water supply alternatives and their robustness to deep uncertainties (e.g., populationpressures, climate change, and financial risks). The proposed framework is demonstrated for four intercon-nected water utilities representing major stakeholders in the ‘‘Research Triangle’’ region of North Carolina,U.S. The utilities supply well over one million customers and have the ability to collectively manage droughtvia transfer agreements and shared infrastructure. We show that water portfolios for this region that com-pose optimal tradeoffs (i.e., Pareto-approximate solutions) under expected future conditions may suffer sig-nificantly degraded performance with only modest changes in deeply uncertain hydrologic and economicfactors. We then use the Patient Rule Induction Method (PRIM) to identify which uncertain factors drive theindividual and collective vulnerabilities for the four cooperating utilities. Our framework identifies key stake-holder dependencies and robustness tradeoffs associated with cooperative regional planning, which arecritical to understanding the tensions between individual versus regional water supply goals. Cooperativedemand management was found to be the key factor controlling the robustness of regional water supplyplanning, dominating other hydroclimatic and economic uncertainties through the 2025 planning horizon.Results suggest that a modest reduction in the projected rate of demand growth (from approximately 3%per year to 2.4%) will substantially improve the utilities’ robustness to future uncertainty and reduce thepotential for regional tensions. The proposed multistakeholder MORDM framework offers critical insightsinto the risks and challenges posed by rising water demands and hydrological uncertainties, providing aplanning template for regions now forced to confront rapidly evolving water scarcity risks.

1. Introduction

Cooperative regional water portfolio planning, in which adaptive management strategies are coordinatedacross multiple water utilities, is a core component of the ‘‘soft path’’ approach for utilizing existing infra-structure more efficiently [Gleick, 2002, 2003]. Such portfolios may combine conservation measures [e.g.,Renwick and Green, 2000], water transfers [Lund and Israel, 1995; Wilchfort and Lund, 1997; Hadjigeorgalis,2008], and financial instruments [Brown and Carriquiry, 2007; Zeff and Characklis, 2013; Zeff et al., 2014] todiversify the management of scarcity risks in a flexible manner. Regional water portfolios provide an innova-tive approach to offset future demand and climate risks, yet their potential vulnerabilities to deep uncertain-ties must be recognized. Deep uncertainty acknowledges that decision makers may not be able toenumerate all sources of uncertainty in a system nor their associated probabilities [Langlois and Cosgel,1993; Lempert, 2002; Lempert et al., 2003; Polasky et al., 2011; Kasprzyk et al., 2013]. Also referred to as Knigh-tian uncertainty [Knight, 1921], a core concern is providing appropriate risk management actions despitethe uncertainties in correctly specifying all probability distributions [Friedman, 1976]. Deep uncertainty isespecially prevalent in complex economic and environmental systems, where rapidly evolving systems maycause management policies to produce severe unintended consequences on stakeholders. These traitshave long been recognized in water supply planning, making it a classic example of a ‘‘wicked’’ problem

Key Points:! We advance many-objective robust

decision making for multiplestakeholders! Stakeholders’ robustness exhibits

dependencies, vulnerabilities, andtradeoffs! A modest reduction in demand

growth rate insulates against futureuncertainty

Supporting Information:! Readme! Supplement

Correspondence to:J. Herman,[email protected]

Citation:Herman, J. D., H. B. Zeff, P. M. Reed,and G. W. Characklis (2014), Beyondoptimality: Multistakeholderrobustness tradeoffs for regional waterportfolio planning under deepuncertainty, Water Resour. Res., 50,doi:10.1002/2014WR015338.

Received 21 JAN 2014Accepted 19 AUG 2014Accepted article online 23 AUG 2014

HERMAN ET AL. VC 2014. American Geophysical Union. All Rights Reserved. 1

Water Resources Research

PUBLICATIONS

Dynamic adaptive policy pathways: A method for crafting robust decisions for adeeply uncertain world

Marjolijn Haasnoot a,b,d,*, Jan H. Kwakkel c, Warren E. Walker c, Judith ter Maat d

a Utrecht University, Department of Geosciences, P.O. Box 80115, 3508 TC Utrecht, The Netherlandsb Twente University, Department of Water Engineering & Management, P.O. Box 217,7500 AE Enschede, The Netherlandsc Delft University of Technology, Faculty of Technology, Policy and Management, P.O. Box 5015, 2600 GA Delft, The Netherlandsd Deltares, P.O. Box 177, 2600 MH Delft, The Netherlands

1. Introduction

Nowadays, decisionmakers face deep uncertainties about amyriad of external factors, such as climate change, populationgrowth, new technologies, economic developments, and theirimpacts. Moreover, not only environmental conditions, but alsosocietal perspectives and preferences may change over time,including stakeholders’ interests and their evaluation of plans(Offermans, 2010; van der Brugge et al., 2005). Traditionally,decisionmakers in many policy domains, including water manage-ment, assume that the future can be predicted. They develop astatic ‘optimal’ plan using a single ‘most likely’ future (often basedon the extrapolation of trends) or a static ‘robust’ plan that willproduce acceptable outcomes in most plausible future worlds(Dessai and Hulme, 2007; Dessai and Van der Sluijs, 2007;Hallegatte et al., 2012). However, if the future turns out to be

different from the hypothesized future(s), the plan is likely to fail.McInerney et al. (2012) liken this to ‘‘dancing on the top of aneedle’’. But, as the future unfolds policymakers learn and usuallyrespond to the new situation by adapting their plans (ad hoc) to thenew reality. Adaptation over the course of time is not onlydetermined by what is known or anticipated at present, but also bywhat is experienced and learned as the future unfolds (Yohe, 1990)and by the policy responses to events (Haasnoot et al., 2012). Thus,policymaking becomes part of the storyline, and thereby anessential component of the total uncertainty – in fact, Hallegatteet al. (2012) include the adaptation of decisions over time in anupdated definition of ‘deep uncertainty’.

To address these deep uncertainties, a new planning paradigmhas emerged. This paradigm holds that, in light of the deepuncertainties, one needs to design dynamic adaptive plans(Albrechts, 2004; de Neufville and Odoni, 2003; Haasnoot et al.,2011; Hallegatte, 2009; Hallegatte et al., 2012; Ranger et al., 2010;Schwartz and Trigeorgis, 2004; Swanson et al., 2010). Such planscontain a strategic vision of the future, commit to short-termactions, and establish a framework to guide future actions(Albrechts, 2004; Ranger et al., 2010). The seeds for this planningparadigm were planted almost a century ago. Dewey (1927) argued

Global Environmental Change 23 (2013) 485–498

A R T I C L E I N F O

Article history:Received 15 June 2012Received in revised form 3 December 2012Accepted 18 December 2012

Keywords:UncertaintyPolicymakingAdaptation pathwaysAdaptive policiesWater managementRhine delta

A B S T R A C T

A new paradigm for planning under conditions of deep uncertainty has emerged in the literature.According to this paradigm, a planner should create a strategic vision of the future, commit to short-termactions, and establish a framework to guide future actions. A plan that embodies these ideas allows for itsdynamic adaptation over time to meet changing circumstances. We propose a method fordecisionmaking under uncertain global and regional changes called ‘Dynamic Adaptive PolicyPathways’. We base our approach on two complementary approaches for designing adaptive plans:‘Adaptive Policymaking’ and ‘Adaptation Pathways’. Adaptive Policymaking is a theoretical approachdescribing a planning process with different types of actions (e.g. ‘mitigating actions’ and ‘hedgingactions’) and signposts to monitor to see if adaptation is needed. In contrast, Adaptation Pathwaysprovides an analytical approach for exploring and sequencing a set of possible actions based onalternative external developments over time. We illustrate the Dynamic Adaptive Policy Pathwaysapproach by producing an adaptive plan for long-term water management of the Rhine Delta in theNetherlands that takes into account the deep uncertainties about the future arising from social, political,technological, economic, and climate changes. The results suggest that it is worthwhile to further testand use the approach.

! 2012 Elsevier Ltd.

* Corresponding author at: Deltares, P.O. Box 177, 2600 MH Delft, TheNetherlands. Tel.: +31 88 335 81 75.

E-mail addresses: [email protected] (M. Haasnoot),[email protected] (J.H. Kwakkel), [email protected](W.E. Walker), [email protected] (J. ter Maat).

Contents lists available at SciVerse ScienceDirect

Global Environmental Change

jo ur n al h o mep ag e: www .e lsev ier . co m / loc ate /g lo envc h a

0959-3780 ! 2012 Elsevier Ltd.

http://dx.doi.org/10.1016/j.gloenvcha.2012.12.006

Open access under CC BY-NC-ND license.

Open access under CC BY-NC-ND license.

Robust Decision Making and Info-Gap Decision Theory for waterresource system planning

Evgenii S. Matrosov, Ashley M. Woods, Julien J. Harou ⇑

Department of Civil, Environmental and Geomatic Engineering, University College London, Chadwick Building, Gower Street, London WC1E 6BT, UK

a r t i c l e i n f o

Article history:Received 5 August 2011Received in revised form 5 March 2013Accepted 9 March 2013Available online 29 March 2013This manuscript was handled byKonstantine P. Georgakakos, Editor-in-Chief,with the assistance of Aris P. Georgakakos,Associate Editor

Keywords:Water resources planningRobust Decision Making (RDM)Info-Gap Decision Theory (IGDT)UncertaintyInfrastructure planning

s u m m a r y

Stationarity assumptions of linked human–water systems are frequently invalid given the difficult-to-predict changes affecting such systems. In this case water planning occurs under conditions of deep orsevere uncertainty, where the statistical distributions of future conditions and events are poorly known.In such situations predictive system simulation models are typically run under different scenarios toevaluate the performance of future plans under different conditions. Given that there are many possibleplans and many possible futures, which simulations will lead to the best designs? Robust Decision Mak-ing (RDM) and Info-Gap Decision Theory (IGDT) provide a structured approach to planning complex sys-tems under such uncertainty. Both RDM and IGDT make repeated use of trusted simulation models toevaluate different plans under different future conditions. Both methods seek to identify robust ratherthan optimal decisions, where a robust decision works satisfactorily over a broad range of possiblefutures. IGDT efficiently charts system performance with robustness and opportuneness plots summaris-ing system performance for different plans under the most dire and favourable sets of future conditions.RDM samples a wider range of dire, benign and opportune futures and offers a holistic assessment of theperformance of different options. RDM also identifies through ‘scenario discovery’ which combinations ofuncertain future stresses lead to system vulnerabilities. In our study we apply both frameworks to awater resource system planning problem: London’s water supply system expansion in the Thames basin,UK. The methods help identify which out of 20 proposed water supply infrastructure portfolios is themost robust given severely uncertain future hydrological inflows, water demands and energy prices. Mul-tiple criteria of system performance are considered: service reliability, storage susceptibility, capital andoperating cost, energy use and environmental flows. Initially the two decision frameworks lead to differ-ent recommendations. We show the methods are complementary and can be beneficially used togetherto better understand results and reveal how the particulars of each method can skew results towards par-ticular future plans.

! 2013 Elsevier B.V. All rights reserved.

1. Introduction

Water resource systems are sensitive to climate and populationchanges, making supply infrastructure planning difficult. Planningmodels grapple with the inherent uncertainty of future conditionswhen the statistical distributions of future conditions are unknownor not trusted. Under such ‘Knightian’ uncertainty (Knight, 1921)uncertainty is unquantifiable and the most likely realisation ofthe future is unknown. In such situations methods that rely on tra-ditional Bayesian decision analysis to characterise uncertaintyusing probability theory may not be appropriate (Groves and Lem-pert, 2007). In such situations of ‘deep’ or ‘severe’ uncertainty, re-cent research has argued it is more appropriate to strive forrobustness (Ben-Haim, 2001; Dessai and Hulme, 2007; Lempert

et al., 2006; Lempert and Collins, 2007) rather than optimality. A‘robust’ system performs satisfactorily, or satisfices (Simon, 1959)performance criteria, over a wide range of uncertain futures ratherthan performing optimally over the historical period or a fewscenarios.

Robust Decision Making (RDM) (Lempert and Popper, 2003) andInfo-Gap Decision Theory (IGDT) (Ben-Haim, 2001) are two deci-sion making frameworks that seek robustness. Both use trustedsimulation models to consider a wide spectrum of plausible futureseach with different input parameters to represent uncertainty.

Both approaches have been applied to water management.Groves and Lempert (2007) use RDM to identify vulnerabilities ofthe California Department of Water Resources’ California WaterPlan (CWP). Lempert and Groves (2010) apply RDM to identify cli-mate change vulnerabilities of the Inland Empire Utilities Agency’s2005 Integrated Water Resource Plan and to develop a more robustplan including adaptive strategies. Hipel and Ben-Haim (1999) use

0022-1694/$ - see front matter ! 2013 Elsevier B.V. All rights reserved.http://dx.doi.org/10.1016/j.jhydrol.2013.03.006

⇑ Corresponding author. Tel.: +44 20 7679 0536.E-mail address: [email protected] (J.J. Harou).

Journal of Hydrology 494 (2013) 43–58


Journal of Hydrology

journal homepage: www.elsevier .com/ locate / jhydrol

Many objective robust decision making for complex environmentalsystems undergoing change

Joseph R. Kasprzyk a,*, Shanthi Nataraj b, Patrick M. Reed a, Robert J. Lempert baDepartment of Civil and Environmental Engineering, Penn State University, 212 Sackett Building, University Park, PA 16802, USAbRAND Corporation, 1776 Main Street, Santa Monica, CA, USA

a r t i c l e i n f o

Article history:Received 5 July 2012Received in revised form12 December 2012Accepted 13 December 2012Available online 23 January 2013

Keywords:Multiobjective evolutionary algorithmsRobust decision makingInteractive visual analyticsUncertainty analysisWater supplyEnvironmental management

a b s t r a c t

This paper introduces many objective robust decision making (MORDM). MORDM combines conceptsand methods from many objective evolutionary optimization and robust decision making (RDM), alongwith extensive use of interactive visual analytics, to facilitate the management of complex environmentalsystems. Many objective evolutionary search is used to generate alternatives for complex planningproblems, enabling the discovery of the key tradeoffs among planning objectives. RDM then determinesthe robustness of planning alternatives to deeply uncertain future conditions and facilitates decisionmakers’ selection of promising candidate solutions. MORDM tests each solution under the ensemble offuture extreme states of the world (SOW). Interactive visual analytics are used to explore whether so-lutions of interest are robust to a wide range of plausible future conditions (i.e., assessment of theirPareto satisficing behavior in alternative SOW). Scenario discovery methods that use statistical datamining algorithms are then used to identify what assumptions and system conditions strongly influencethe cost-effectiveness, efficiency, and reliability of the robust alternatives. The framework is demon-strated using a case study that examines a single city’s water supply in the Lower Rio Grande Valley(LRGV) in Texas, USA. Results suggest that including robustness as a decision criterion can dramaticallychange the formulation of complex environmental management problems as well as the negotiatedselection of candidate alternatives to implement. MORDM also allows decision makers to characterizethe most important vulnerabilities for their systems, which should be the focus of ex post monitoringand identification of triggers for adaptive management.

! 2012 Elsevier Ltd. All rights reserved.

1. Introduction

This paper contributes the many objective robust decisionmaking (MORDM) framework, which combines many objectiveevolutionary optimization, robust decision making (RDM), andinteractive visual analytics to facilitate the management of complexenvironmental systems. The MORDM framework seeks to addressseveral key challenges for environmental systems undergoingchange. The first is how to evaluate the performance of alternativeplanning and management strategies. To make these evaluations,planners have traditionally used cost benefit analysis, in whicha project’s benefits are commensurated to their expected monetaryvalue and then compared to a project’s costs to determine whetherthe project will be funded (Griffin, 1998). Aggregating these mul-tiple performance measures into a single value can yield negativedecision biases that result because different aspects of performance

are rewarded and penalized in ways that cannot be predicteda priori (Franssen, 2005). The approach has also been shown toinadequately compensate for non-monetary benefits (Bromley andBeattie, 1973) especially when multiple policies are considered(Hoehn and Randall, 1989). Multiobjective approaches instead seekto quantify the large number of conflicting objectives that charac-terize planning. In addition to cost, it has been recognized thatcomplex planning efforts often reveal additional critical perfor-mance objectives (Hitch, 1960), such as maximizing reliable per-formance, minimizing environmental damages, and improvingsystem efficiency. The Harvard Water Program was one of theearliest efforts to advocate for the multiobjective planningapproach by emphasizing the importance of both economic ob-jectives and engineering performance objectives (Maass et al.,1962; Reuss, 2003; Banzhaf, 2009). Considering “many” objec-tives explicitly and simultaneously can also aid planners in avoidingcognitive myopia (Hogarth, 1981). Cognitive myopia arises whendecision makers inadvertently ignore aspects of the problem (suchas important decision alternatives or key planning objectives) by

* Corresponding author.E-mail addresses: [email protected], [email protected] (J.R. Kasprzyk).


Environmental Modelling & Software

journal homepage: www.elsevier .com/locate/envsoft

1364-8152/$ e see front matter ! 2012 Elsevier Ltd. All rights reserved.http://dx.doi.org/10.1016/j.envsoft.2012.12.007

Environmental Modelling & Software 42 (2013) 55e71

LETTERSPUBLISHED ONLINE: 20 JULY 2015 | DOI: 10.1038/NCLIMATE2721

Selection of climate policies under theuncertainties in the Fifth AssessmentReport of the IPCCL. Drouet1,2*, V. Bosetti1,2,3 and M. Tavoni1,2,4

Strategies for dealing with climate change must incorporateand quantify all the relevant uncertainties, and be designed tomanage the resulting risks1. Here we employ the best availableknowledge so far, summarized by the three working groupsof the Fifth Assessment Report of the IntergovernmentalPanel on Climate Change (IPCC AR5; refs 2–4), to quantifythe uncertainty of mitigation costs, climate change dynamics,and economic damage for alternative carbon budgets. Werank climate policies according to di�erent decision-makingcriteria concerning uncertainty, risk aversion and intertem-poral preferences. Our findings show that preferences overuncertainties are as important as the choice of the widelydiscussed time discount factor. Climate policies consistentwith limiting warming to 2 �C above preindustrial levels arecompatible with a subset of decision-making criteria and somemodel parametrizations, but not with the commonly adoptedexpected utility framework.

Many of the uncertainties surrounding climate change aredi�cult to quantify and depend on the judgement of experts andon the type of model used to generate future scenarios. Eachmodel produces a distribution over the possible states of nature(for example, cost of mitigation, temperature increase, or economicdamage from climate change), and these distributions might di�erfrom model to model. How should we select climate policy in theface of these uncertainties?

This paper addresses this question using a framework thataccounts for both state uncertainty (for example, the distributionover states of nature) and model uncertainty (for example, thedi�erent models (or experts) which generate distributions overstates)5. We investigate a variety of preferences and assumptionsover these two types of uncertainty. A special case is the subjectiveexpected utility6 framework, traditionally used in economicevaluations. However, an expected utility setting might not workwhen the information is incomplete and ambiguous, which isclearly the case for climate change7. Moreover, people are known toapproach risks and uncertainties di�erently8. The proposed settingallows us to explore additional decision-making criteria to deal withuncertainty, in the spirit of refs 7,9. Alternative statistical techniques,consistent with Bayesian approaches, have been developed to copewith model uncertainty10. Model weighting is an active topic inclimate research11, where historical observations provide a basis formodel evaluation, although it is not commonly used12. Althoughour framework is su�ciently flexible to accommodate di�erentprior probability measure over the set of possible models, ourbaseline model assumes a uniform prior with equal model weights.

The literature on the role of uncertainty in climate policymaking has mostly relied on either analytical or simplifiedintegrated assessment models (IAMs), such as DICE (ref. 13). Insuch contexts, di�erent decision-making criteria and preferencesover risks have been shown to have a significant impact onthe optimal abatement strategy14,15. However, these exercises lackdetail in the representation of the mitigation options and theclimate dynamics. Larger-scale models, which capture the maininterrelationships between human and natural systems, haveincorporated uncertainty only partially owing to computationallimitations. Therefore, uncertainty is mostly treated by means ofmulti-model ensembles16,17, or by single models performing MonteCarlo simulations18,19. When accounting for all the key sources ofuncertainty, the selection of optimal climate policy has been shownto be more sensitive to uncertainty about mitigation costs andimpacts than to uncertainty about warming20.

Figure 1 illustrates our approach. Using the best availableknowledge from the three working groups of IPCC AR5 (seeMethods), for each component (mitigation costs, temperature andclimate damage) we generate probability estimates for di�erentclasses of models. The decision variable is the carbon budget—that is, the cumulative CO2 emissions over the twenty-first century(2010–2100). Carbon budgets are robust policy indicators, as theyare strictly related to global warming21 and climate targets22. Weassume that uncertainty resolves immediately, but show that ourresults are robust to di�erent timings of resolution of uncertainty(in Supplementary Fig. 12).

We extract emission projections and associated mitigation costsfrom the AR5 WGIII Scenario database4, which includes theoutcomes of many long-term scenarios produced by the mostwell-established IAMs. The database spans a wide range of policystringency, and thus of associated carbon budgets, covering thewhole range of the representative concentration pathways. Therelationship between mitigation costs, harmonized across di�erentmetrics, and carbon budgets is found to be nonlinear and highlyuncertain (Supplementary Fig. 2). Furthermore, the uncertaintyof mitigation costs increases with time (Supplementary Fig. 3).From Supplementary Fig. 2 emerges a well-documented4 distinctionbetween di�erent classes of IAMs: top-down (TD) models providea more accurate description of the economy, whereas bottom-up(BU) models better represent the set of mitigation technologies. TDmodels generally show higher mitigation costs than BUmodels, butit is not obvious which class is the most reliable. We account for thismodel type uncertainty by estimating di�erent probabilistic modelsof the evolution of mitigation costs.

1Fondazione Eni Enrico Mattei (FEEM), Corso Magenta 63, 20123 Milan, Italy. 2Centro Euro-mediterraneo sui Cambiamenti Climatici (CMCC), CorsoMagenta 63, 20123 Milan, Italy. 3Bocconi University, Department of Economics, Via Sarfatti 25, 20136 Milan, Italy. 4Politecnico di Milano, Department ofManagement and Economics, Via Lambruschini 4/B, 20156 Milan, Italy. *e-mail: [email protected]

NATURE CLIMATE CHANGE | ADVANCE ONLINE PUBLICATION | www.nature.com/natureclimatechange 1

How Should Robustness Be Defined for Water SystemsPlanning under Change?

Jonathan D. Herman, S.M.ASCE1; Patrick M. Reed, Ph.D., A.M.ASCE2;Harrison B. Zeff3; and Gregory W. Characklis, Ph.D., M.ASCE4

Abstract:Water systems planners have long recognized the need for robust solutions capable of withstanding deviations from the conditionsfor which they were designed. Robustness analyses have shifted from expected utility to exploratory bottom-up approaches which identifyvulnerable scenarios prior to assigning likelihoods. Examples include Robust Decision Making (RDM), Decision Scaling, Info-Gap, andMany-Objective Robust Decision Making (MORDM). We propose a taxonomy of robustness frameworks to compare and contrast theseapproaches based on their methods of (1) alternative generation, (2) sampling of states of the world, (3) quantification of robustness measures,and (4) sensitivity analysis to identify important uncertainties. Building from the proposed taxonomy, we use a regional urban water supplycase study in the Research Triangle region of North Carolina to illustrate the decision-relevant consequences that emerge from each of thesechoices. Results indicate that the methodological choices in the taxonomy lead to the selection of substantially different planning alternatives,underscoring the importance of an informed definition of robustness. Moreover, the results show that some commonly employed methodo-logical choices and definitions of robustness can have undesired consequences when ranking decision alternatives. For the demonstratedtest case, recommendations for overcoming these issues include: (1) decision alternatives should be searched rather than prespecified,(2) dominant uncertainties should be discovered through sensitivity analysis rather than assumed, and (3) a carefully elicited multivariatesatisficing measure of robustness allows stakeholders to achieve their problem-specific performance requirements. This work emphasizesthe importance of an informed problem formulation for systems facing challenging performance tradeoffs and provides a common vocabularyto link the robustness frameworks widely used in the field of water systems planning. DOI: 10.1061/(ASCE)WR.1943-5452.0000509.© 2015 American Society of Civil Engineers.

Introduction

Decision makers in water resources systems aim to achieve multi-ple performance objectives in the projected future while remainingrobust to deviations from these projections. Robustness in thiscontext broadly refers to “the insensitivity of system design toerrors, random or otherwise, in the estimates of those parametersaffecting design choice” (Matalas and Fiering 1977), althoughspecific mathematical implementations of this concept differ dra-matically. Extensive studies have demonstrated the willingness ofdecision makers to sacrifice expected performance to improve ro-bustness to uncertainty (Hitch 1960; Maass et al. 1962; Bonder1979; Schneller and Sphicas 1983; Walker et al. 2001; Lempert2002; Clímaco 2004; Walker et al. 2013; DiFrancesco and Tullos2014), signaling a departure from traditional decision theoryrequiring a priori aggregation of costs and benefits. The challengeof simultaneously navigating these goals in complex systems hasled to the rise of a posteriori decision support, reviewed by

Tsoukias (2008), in which the identification of decision alternativesand vulnerable states is preceded by data gathering, numericalmodeling, and optimization. This process represents constructivelearning with stakeholder feedback (Roy 1971, 1990) in whichproblem formulations compete as multiple working hypotheses(Chamberlin 1890). Given a set of alternatives shown to benear-optimal in the best available projections of the expected futurestate of the world, it remains a challenge to perform a posteriorialternative selection according to a defensible robustness criterion.Several competing criteria have been proposed, and theirconsequences for decision making warrant comparison.

The term a posteriori decision support is drawn from the multi-objective optimization literature, referring to the generation of de-cision alternatives through computational search before imposingstakeholder preference on the problem (Cohon and Marks 1973,1975), also known as generate-first-choose-later (GFCL) in thesystems engineering literature (e.g., Hwang et al. 1979; Crossleyet al. 1999; Balling and Richard 2000; Messac and Mattson2002; Reynoso-Meza et al. 2014). Historically, this approach hasstood in contrast to a priori weighted preference aggregation(Reuss 2003; Banzhaf 2009), which allows simpler solution tech-niques but requires decision makers to accurately assign prefer-ence weighting prior to reviewing alternatives (Bond et al. 2008,2010). The salient implication for robustness analysis is that a sin-gle decision alternative produced by weighted aggregation, whilepromising in the projected future, may fail under deviations fromthis projection. A multiobjective, a posteriori approach provides aflexible set of alternatives that may be evaluated according to theirrobustness.

In a related context, the concept of a posteriori decision supportcan be extended to the selection of scenarios, or states of the world,for decision making under uncertainty. In a traditional scenario

1School of Civil and Environmental Engineering, 207 Hollister Hall,Cornell Univ., Ithaca, NY 14853 (corresponding author). E-mail: [email protected]

2Professor, School of Civil and Environmental Engineering,211 Hollister Hall, Cornell Univ., Ithaca, NY 14853.

3Dept. of Environmental Sciences and Engineering, Univ. of NorthCarolina, Chapel Hill, NC 27599.

4Professor, Dept. of Environmental Sciences and Engineering, Univ. ofNorth Carolina, Chapel Hill, NC 27599.

Note. This manuscript was submitted on September 19, 2014; approved onDecember 3, 2014; published online on February 10, 2015. Discussion periodopen until July 10, 2015; separate discussionsmust be submitted for individualpapers. This paper is part of the Journal of Water Resources Planning andManagement, © ASCE, ISSN 0733-9496/04015012(14)/$25.00.

© ASCE 04015012-1 J. Water Resour. Plann. Manage.

J. Water Resour. Plann. Manage.

Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

02/

13/1

5. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.

Editorial

Coping with the Wickedness of Public Policy Problems:Approaches for Decision Making under Deep Uncertainty

Jan H. KwakkelFaculty of Technology, Policy and Management, Delft Univ. of Technol-ogy, Jaffalaan 5, 2628 BX Delft, Netherlands (corresponding author).E-mail: [email protected]

Warren E. WalkerFaculty of Technology, Policy and Management, Delft Univ. ofTechnology, Jaffalaan 5, 2628 BX Delft, Netherlands. E-mail: [email protected]

Marjolijn HaasnootDeltares, P.O. Box 177, 2600 MH Delft, Netherlands; Faculty ofTechnology, Policy and Management, Delft Univ. of Technology, Jaffalaan5, 2628 BX Delft, Netherlands. E-mail: [email protected]

DOI: 10.1061/(ASCE)WR.1943-5452.0000626

In many planning problems, planners face major challenges incoping with uncertain and changing physical conditions, and rapidunpredictable socioeconomic development. How should societyprepare itself for this confluence of uncertainty? Given the presenceof irreducible uncertainties, there is no straightforward answer tothis question. Effective decisions must be made under unavoidableuncertainty (Dessai et al. 2009; Lempert et al. 2003). In recentyears, this has been labeled as decision making under deep uncer-tainty. Deep uncertainty means that the various parties to a decisiondo not know or cannot agree on the system and its boundaries; theoutcomes of interest and their relative importance; the prior prob-ability distribution for uncertain inputs to the system (Lempert et al.2003; Walker et al. 2013); or decisions are made over time in dy-namic interaction with the system and cannot be considered inde-pendently (Haasnoot et al. 2013a, b; Hallegatte et al. 2012). From adecision analytic point of view, this implies that there are a largenumber of plausible alternative models, alternative sets of weightsto assign to the different outcomes of interest, different sets of in-puts for the uncertain model parameters, and different (sequencesof) candidate solutions (Kwakkel et al. 2010).

Decision making under deep uncertainty is a particular type ofwicked problem (Rittel and Webber 1973). Wicked problems areproblems characterized by the involvement of a variety of stake-holders and decision makers with conflicting values and divergingideas for solutions (Churchman 1967). What makes wicked prob-lems especially pernicious is that even the problem formulationitself is contested (Rittel and Webber 1973). System analytic ap-proaches presuppose a separation between the problem formulationand the solution. In wicked problem situations this distinctionbreaks down. Solutions and problem formulation are intertwinedwith each other. Depending on how a problem is framed, alternativesolutions come to the fore; and, vice versa, depending on the avail-able or preferred solutions, the problem can be framed differently.Even if there is agreement on the difference between observed anddesired outcomes, rival explanations for the existence of this differ-ence are available, and, hence, different solutions can be preferred.An additional factor adding to the wickedness is that decision mak-ers can ill afford to be wrong. The consequences of any decision on

wicked problems can be profound, difficult if not impossible to re-verse, and result in lock-ins for future decision making. Planningand decision making in wicked problem situations should, there-fore, be understood as an argumentative process: in which the prob-lem formulation, a shared understanding of system functioningand how this gives rise to the problem, and the set of promisingsolutions, emerge gradually through debate among the involveddecision makers and stakeholders (Dewulf et al. 2005).

When even the problem formulation itself is uncertain and con-tested, planning and decision making requires an iterative approachthat facilitates learning across alternative framings of the problem,and learning about stakeholder preferences and trade-offs, all inpursuit of a collaborative process of discovering what is possible(Herman et al. 2015). Modeling and optimization can play a role infacilitating this learning. They can help in discovering a set of pos-sible actions that is worth closer inspection, and make the trade-offsamong these actions more transparent (Liebman 1976; Reed andKasprzyk 2009).

Under the moniker of decision making under deep uncertainty, avariety of new approaches and tools are being put forward. Emerg-ing approaches include (multiobjective) robust decision making(Kasprzyk et al. 2013; Lempert et al. 2006), info-gap decisiontheory (Ben Haim 2001), dynamic adaptive policy pathways(Haasnoot et al. 2013a, b), and decision scaling (Brown et al.2012). A common feature of these approaches is that they areexploratory model-based strategies for designing adaptive and ro-bust plans or policies. Although these frameworks are used in awide variety of applications, they have been most commonly ap-plied in the water domain, in which climate change and socialchange are key concerns that affect the long-term viability of cur-rent management plans and strategies. Liebman (1976) recognizedthat water resources planning problems are wicked problems inwhich modeling, simulation, and optimization cannot be straight-forwardly applied. In recent years, this observation has beenreiterated (Herman et al. 2015; Lund 2012; Reed and Kasprzyk2009).

If decision making under deep uncertainty is a particular typeof wicked problem, to what extent do the recent methodologicaladvances address some of the key aspects of what makes wickedproblems wicked? To answer this question, the authors look at twoexemplary approaches for supporting decision making under deepuncertainty: (multiobjective) robust decision making and dynamicadaptive policy pathways. This article first briefly outlines eachapproach, and then discusses some of the ongoing scientific workaimed at integrating the two approaches. This sets the stage for acritical discussion of these approaches and how they touch on thekey concerns of supporting decision making in wicked problemsituations.

Robust Decision Making

Robust decision making (RDM) (Lempert et al. 2006) emphasizesan iterative approach to planning in which candidate strategies aretested across a very large number of scenarios and, in light of in-sights gained from this model-based scenario analysis, candidate


J. Water Resour. Plann. Manage., 01816001

Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

01/

18/1

6. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.

Editorial





DOI: 10.1061/(ASCE)WR.1943-5452.0000626











Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

01/

18/1

6. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.

Editorial





DOI: 10.1061/(ASCE)WR.1943-5452.0000626











Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

01/

18/1

6. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.

Editorial





DOI: 10.1061/(ASCE)WR.1943-5452.0000626











Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

01/

18/1

6. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.




Introduction












Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

02/

13/1

5. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.




Introduction












Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

02/

13/1

5. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.




Introduction












Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

02/

13/1

5. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.




Introduction












Dow

nloa

ded

from

asc

elib

rary

.org

by

Li. C

o.Sa

818

1901

/mi/1

5598

5 on

02/

13/1

5. C

opyr

ight

ASC

E. F

or p

erso

nal u

se o

nly;

all

right

s res

erve

d.

EWRI2016 NRM

How robustness is implemented?

A metric 𝛙 is introduced for filtering the uncertainty in the scenarios w∈𝚵

EWRI2016 NRM

A metric 𝛙 is introduced for filtering the uncertainty in the scenarios w∈𝚵

How robustness is implemented?

• Maximin

• Optimism-pessimism rule

• Principle of insufficient reason

• Regret - deviation from best

• Regret - deviation from baseline

• Signal to noise

• Mean and standard deviation separately

• Fraction of cases that passes thresholds

the definition of the robustness metric introduces another uncertain parameter

in the problem

1. How to decide how to decide?


2. What are the impacts of mis-defining the robustness metric?


2. What are the impacts of mis-defining the robustness metric?

3. What happens if these metrics evolve in time?

EWRI2016 NRM

The Lake Como system

Como Adda River

Milano

Lake Como

160 33 66 100 km

4000 m

8

agriculturaldistricts

EWRI2016 NRM


Como Adda River

Milano

Lake Como

160 33 66 100 km

4000 m

8


Flood control in Como

EWRI2016 NRM


Como Adda River

Milano

Lake Como

160 33 66 100 km

4000 m

8



Irrigation supply

EWRI2016 NRM

Como Adda River

Milano

Lake Como

160 33 66 100 km

4000 m

8



Irrigation supply


EWRI2016 NRM

Como Adda River

Milano

Lake Como

160 33 66 100 km

4000 m

8



Irrigation supply


critical period with water deficit

EWRI2016 NRM

critical period with water deficitClimate change scenarios

Numerical results

EWRI2016 NRM

Policy performance under historical climate

flooding (storage reliability)0.7 0.75 0.8 0.85 0.9 0.95 1

irriga

tion

(volu

met

ric re

liabil

ity)

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

40%

14%

46%

co-benefitdegradation

single-objective benefit

Pareto approximate set over historyRe-evaluation over the 28-scenarios ensemble

Legend

EWRI2016 NRM

Policy performance under climate change

flooding (storage reliability)0.7 0.75 0.8 0.85 0.9 0.95 1

irriga

tion

(volu

met

ric re

liabil

ity)

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

40%

14%

46%

co-benefitdegradation

single-objective benefit

Pareto approximate set over historyRe-evaluation over the 28-scenarios ensemble

Legend

EWRI2016 NRM

The impacts of misdefining robustness: modification of tradeoffs

EWRI2016 NRM

The impacts of misdefining robustness: underestimation of system performance

EWRI2016 NRM

Multi-robustness policy design

(a): flooding (storage reliability)

(b): irrigation (volumetric reliability)

insufficientreason

maximin(pessimism)

maximax(optimism)

optimism/pessimismrule

minimax regret

direc

tion

of p

refe

renc

e

max

imin

crite

rion

direc

tion

of p

refe

renc

e0.997 0.985 1.00 0.990 0.005

0.550 0.270 0.860 0.484 0.726

0.806 0.620 0.970 0.719 0.167

0.712 0.505 0.854 0.633 0.283

a⇤ = argmax

a[

1⌅(f(a,w)), . . . ,

N⌅ (f(a,w))]

EWRI2016 NRM

Robustness evolution in time

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4

irrigation (vol. rel.)insufficient reason

irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]

J F M A M J J A S O N D

1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

EWRI2016 NRM

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend


policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

EWRI2016 NRM

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend


policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

EWRI2016 NRM

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend


policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

EWRI2016 NRM

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend


policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

policy P1

level

[m]

J F M A M J J A S O N D0.7 0.72 0.74 0.76 0.78 0.8 0.820.5

0.55

0.6

0.65

P1P2

P3

P5P4


irriga

tion

(vol.

rel.)

m

axim

in (p

essim

ism)

policy P3

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P4

level

[m]


1.2

-0.4

0.0

0.4

0.8

policy P5

level

[m]


1.2

-0.4

0.0

0.4

0.8

1.2

-0.4

0.0

0.4

0.8

policy P2

level

[m]


1.2

-0.4

0.0

0.4

0.8

release [m3/s]

0 143

water demandLegend

EWRI2016 NRM

Is robustness really robust?

EWRI2016 NRM


NO, because

• the definition of the robustness metric represents an additional uncertain parameter of the problem

EWRI2016 NRM


NO, because


• the robustness metric should be calibrated to capture the attitude of the real DM or multiple metrics should be considered

EWRI2016 NRM


NO, because



• possible dynamic changes in DM attitudes and preferences might further impact on decisions under uncertainty

EWRI2016 NRM


NO, because



• possible dynamic changes in DM attitudes and preferences might further impact on decisions under uncertainty

Comprehensive analysis including additional metrics and testing on different case study applications

thank you

http://giuliani.faculty.polimi.it www.nrm.deib.polimi.it

@MxgTeo @NRMPolimi

Matteo Giuliani [email protected]:

M, Giuliani and A. Castelletti (2016). Is robustness really robust? How different definitions of robustness impact decision-making under climate change, Climatic Change, 136: 409-424.

http://www.nrm.deib.polimi.it

mailto:[email protected]?subject=