Http:// QUANTITATIVE METHODS TO MANAGE UNCERTAINTY IN SCIENCE by Andrea Saltelli, Silvio Funtowicz, Stefano Tarantola, Joint Research

http://www.jrc.cec.eu.int/uasa 1

QUANTITATIVE METHODSTO MANAGE UNCERTAINTY IN SCIENCEby Andrea Saltelli, Silvio Funtowicz, Stefano Tarantola, Joint Research Centre of the European Communities in Ispra (I), [email protected]

MUST: Managing Uncertainty in science for suSTainability: future research challenges for Europe

FP6 launch event in Brussels, November 11, 2002


Uncertainty is not an accident of the scientific method, but its substance. Peter Høeg, a Danish novelist, writes in Borderliners (Høeg, 1995):

"That is what we meant by science. That both question and answer are tied up with uncertainty, and that they are painful. But that there is no way around them. And that you hide nothing; instead, everything is brought out into the open".

Uncertainty. Høeg.


Models mimic systems

N

Naturalsystem

F

Formalsystem

Encoding

Decoding

EntailmentEntailment

Rosen’s formalisation of the modelling process


“World” (the natural system) and “Model” (the formal system) are internally entailed - driven by a causal structure.

Nothing entails with one another “World” and “Model”; the association is hence the result of a craftsmanship.

But this does not apply to natural systems only: give 10 engineers the blueprint of the same plant and they will return you 10 model based risk assessments for the same plant.

Models mimic systems (Rosen)


It can help the craftsman that the uncertainty in the information provided by the model (the substance of use for the decoding exercise) is carefully apportioned to the uncertainty associated with the encoding process.

Models mimic systems (Rosen)


<<[…] most simulation models will be complex, with many parameters, state-variables and non linear relations. Under the best circumstances, such models have many degrees of freedom and, with judicious fiddling, can be made to produce virtually any desired behaviour, often with both plausible structure and parameter values.>>, HORNBERGER G.M., and R. C. Spear (1981) An approach to the preliminary analysis of environmental systems. Journal of Environmental management, 12, 7-18.

<<Cynics say that models can be made to conclude anything provided that suitable assumptions are fed into them.>>, The Economist, 1998.

Models maps assumptions onto inferences ... but often too narrowly


But yet models are used ...

… and a legitimate question is the following:

“If we had mapped the space of uncertain assumptions honestly and judiciously, would the space of inference still be of use1?”

1Read: do we still have peak around some useful inference (e.g. YES or NO, safe or unsafe, hypothesis accepted or rejected, policy effective or ineffective etc. ) or do we have as many YES as NO etc.?

Use of models in the scientific discourse


<<I have proposed a form of organised sensitivity analysis that I call “global sensitivity analysis” in which a neighborhood of alternative assumptions is selected and the corresponding interval of inferences is identified. Conclusions are judged to be sturdy only if the neighborhood of assumptions is wide enough to be credible and the corresponding interval of inferences is narrow enough to be useful.>>

Edward E. Leamer, “Sensitivity Analysis would help”, in Modelling Economic Series, Edited by CWJ Granger, 1990, Clarendon Press, Oxford.

Models maps assumptions onto inferences …


Space of estimated parameters

Space of plausible

models space

Simulation

...

inference


Leamer’s view of global Sensitivity Analysis (SA)

Other assumptions

(Copes with Equifinality)


Estimated parameters

Input dataModel

Uncertainty and

sensitivity analysis


(Parametric bootstrap version of UA/SA )

Inference

(Parametric bootstrap: we sample from the posterior parameter probability)

(Estimation)


Estimation of

parameters

Loop on boot-replica of the input data

Model

Bootstrapping-of-the-modelling-process version of UA/SA, after Chatfield, 1995

Inference

(Bootstrap of the modelling process)

(Estimation)

(Model Identification)


Inference

Data

Prior of Model

Bayesian Uncertainty and Sensitivity Analysis

(Draper 1995, Planas and Depoutot 2000)

Prior of Parameter

s

Prior of Model(s)

Posterior of

Parameters

Posterior of

Model(s)

(Sampling)


The space of the model induced choices (the inference) swells and shrinks by our swelling and shrinking the space of the input assumptions. How many of the assumptions are relevant at all for the choice? And those that are relevant, how do they act on the outcome; singularly or in more or less complex combinations? (ANOVA-type analysis)


… and role of uncertainty - sensitivity analysis


I desire to have a given degree of robustness in the choice, what factor/assumptions should be tested more rigorously? (=> look at how much “fixing” any given f/a can potentially reduce the variance of the output)

E.g. can I confidently “fix” a subset of the input factors/assumptions? The Beck and Ravetz “relevance” issue. How do I find these f/a? (=> total sensitivity indices).






))XX|Y(V(E

))XX|Y(E(V

))X|Y(V(E

)XX|Y(V

)Y(V

:toolsSuggested

*iix

*iix

ix

*ii

*i

*i

i*i

i*i

i

x

x

x

x sigma(0,3)1%

sigma(0.2)1%

f(0,4)38%

rest9%

sigma(0,4)1% f(0,1)

6%

f(0,2)14%

f(0,3)30%

Reduced varianceExpected reduced variance

First order effect

Total effect

16

Environmental sustainability Index, From The Economist, Green and growing, The Economist, Jan 25th 2001,

Produced on behalf of the World Economic Forum (WEF), and presented to the annual Davos summit this year.

Robustness ...

17

Mathis Wackernagel, mental father of the “Ecological Footprint” and thus an authoritative source in the Sustainable Development expert community, concludes an argumented critique of the study done presented at Davos by noting:

Robustness ...


"Overall, the report would gain from a more extensive peer review and a sensitivity analysis. The lacking sensitivity analysis undermines the confidence in the results since small changes in the index architecture or the weighting could dramatically alter the ranking of the nations.”

Robustness - Wackernagel’s critique


Monte Carlo Analysis: countries’ score for the Technology Development Index, (UN), a composite indicator. Modified as to include variability in weights (e.g. as when using budget allocation or Analytic Hierarchy Process).

Robustness of composite indicator -

a worked example


-> Monte Carlo of score of country A minus score of country B.

-> Country A generally better off…mostly due to the weight factors a and b

Robustness of composite indicator

a worked example

b

a


-> Scatter-plot of score of country A minus score of country B (colours) as function of the two most important weights

(red=negative, blue=zero, green=positive)

Robustness of composite indicator

a worked example country A – country B

wei

ght o

f 2nd

indi

cato

rweight of 1st indicator


How can I identify model structures in the simultaneous presence of several uncertainty sources ?



23

?

?

Campolongo et al., 1999, JACSaltelli et al., 1995, JAC


Is the model-induced choice weak (non robust) because there is an insufficient number of observations, or because the experts cannot agree on an accepted theory?




Useful inference versus falsification of the analysis

Example: imagine the inference is Y = the logarithm of the ratio between the two pressure-on-decision indices (Tarantola et als. 2000).

Y=Log(PI 1/PI 2)

Region where Region whereIncineration Landfillis preferred is preferred

Frequency of occurrence


Useful inference versus falsification of the analysis


What happens if I address the space of the policy options?



28

Gauging the leverage of the policy options latitude

Simulation Model

parameters

Policy Options

data

errors

resolutionlevels

model structures

uncertainty analysis

sensitivity analysismodel output

feedbacks on input data and model factors

desired target bounds

29



A broader context of knowledge production and its quality as input to the policy process. Merging formal with informal tools. Ongoing work at the JRC with Angela and Silvio.


Simulation Model

parameters

data

errors

resolutionlevels

model structures, assumptions

key indicators

feedbacks on input data and model factors

desired target bounds

Problem definition Problem definition

Policy Options(improving quality of air)

Policy Options(improving quality of air)

Policy Assumptions (10% reduction VOC level)

Policy Assumptions (10% reduction VOC level)

SocietySociety

Institutional LevelInstitutional Level

Institutional & Scientific

Level

Institutional & Scientific

Level

Scientific LevelScientific Level

Issue (health diseases: asthma,… )Issue (health diseases: asthma,… )

Normative frameworkNormative framework

Framing assumptionsFraming assumptions

Assessment: Monitoring changeAssessment: Monitoring change

Stakeholder mapping

Stakeholder mapping

Sensitivity AnalysisSensitivity Analysis

Uncertainty Analysis

Uncertainty Analysis

StakeholdersStakeholders

Scientific Pedigree

Scientific Pedigree

Extended Quality

Assurance

Extended Quality

Assurance© ULYSSES: De Marchi et. al, 1998.


Conclusions

The output from global Unc. A./Sens. A. can feed back into the extended peer review process via e.g.

- refocusing of the critical issues,

- re-assignment of weights for multiple criteria, or

- of inference falsification (or otherwise)

- identification of policy relevance/ irrelevance


Further reading on SA

Saltelli et als. Computer Physics Communications 2002, Saltelli et als. Statistical Science, 2000Saltelli et als. JASA, 2000 (available here)

Saltelli et al. Eds., Sensitivity Analysis, John Wiley & Sons publishers, Probability and Statistics series (buy it!)

Documents

Http:// QUANTITATIVE METHODS TO MANAGE UNCERTAINTY IN SCIENCE by Andrea Saltelli, Silvio Funtowicz, Stefano Tarantola, Joint Research