Click to edit Master title style Click to edit Master subtitle style A PRACTICAL LOOK AT UNCERTAINTY...

Click to edit Master title style

Click to edit Master subtitle style

A PRACTICAL LOOK AT UNCERTAINTY MODELING

Steve UnwinRisk & Decision Sciences Group

March 7, 2006

2

"The fundamental cause of trouble in the world today is that the stupid are cock-sure while the intelligent are full

of doubt.“

Bertrand Russell

3

Illuminating the Path• Visual Analytics Agenda - Recommendations

– Rec. 4.10: Develop new methods and technologies for capturing and representing information quality and uncertainty

– Rec. 4.11: Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.

– Summary Rec: Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process

4

Uncertainty Analysis as Resource to Visual Analytics

• VA Agenda

– Develop new methods and technologies for capturing and representing information quality and uncertainty

– Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.

– Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process

• UA Insight

– Probabilistic techniques• Elicitation methods• Aggregation methods• Information-theoretic approaches

– Nonprobabilistic alternatives• Dempster-Shafer• Possibility theory

– Uncertainty propagation techniques• Analytic• Numerical

– Risk communication• Risk representation• Decision-analysis methods

5

MEASURING UNCERTAINTY

CLASSICALMETHODS BAYESIAN

METHODS

NON-PROBABILISTIC

METHODS

6

Classical Statistics

• Focus on Aleatory Uncertainty– random variation inherent in the system

• Sampling produces confidence intervals• Need a sampling model

– Generally unavailable for many real-world complex situations

• Confidence intervals are not probability intervals– Propagation difficulties in even the simplest models

7

Bayesianism

• de Finetti, Ramsey, Savage (1920s-50s)• Subjectivism – Epistemic Probabilities

– Probability as a degree of belief• Classicists are coin tossers• Bayesians are believers

– What is the basis for forming probability?• “ Probabilities do not exist”

– Bruno de Finetti

8

Problems with Bayesianism• Because probabilities don’t exist, they have to be

created– but how?

• Bayes’ Theorem• Subjectivity is explicit

– judgment of evidence• Do probabilities really reflect the way we conceive

belief?– is probability theory a good theory of evidence?– what are the options?

9

One Option:Dempster-Shafer Theory

• Withholding belief distinct from disbelief• Seahawks or Steelers will win?• Set of possibilities: {sea, steel}• Probability theory:

– Weight of evidence attached to each exclusive possibility– p(sea), p(steel)

• D-S theory:– Weight of evidence attached to each subset– m(Ø), m(sea), m(steel), m(sea U steel)

• Allows: m(sea U steel) = 1, all other m=0– A compelling ignorance

10

Support and Plausibility

• Probability replaced by two belief measures:– Each calculated from weights of evidence– bel(sea) is the support for proposition ‘sea’– pl(sea) is the plausibility of ‘sea’– bel(sea) ≤ pl(sea)– Upper and lower “probabilities”

• Complete ignorance• SDU: bel(sea) = 0, pl(sea) = 1, i.e., complete

ignorance on the matter of proposition ‘sea’

11

Complementary Cumulative Belief Functions

ESD Sensor System On-Demand Failure Rate

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.0E-04 1.0E-03 1.0E-02 1.0E-01 1.0E+00

Failure Rate per Demand

Belie

f Met

ric

Complementary Cumulative Support/ Belief

Complementary Cumulative Plausibility

Complementary Cumulative Probability

12

Possibility Theory

• Genesis in fuzzy sets• Possibility is an uncertainty measure that

mirrors the fuzzy set notion of imprecision

13

The Set of Tall Men

0

0.2

0.4

0.6

0.8

1

5' 8" 5' 9" 5' 10" 5' 11" 6' 0" 6' 1" 6' 2"

Height

Mem

bers

hip

to S

et TallVery Tall

m(h)

m'(h) = m2(h)

14

Possibility Theory

• 2-tier belief: possibility and necessity• nec(X) ≤ pos(X)• Distinctive combinatorial logic

– nec(X^Y) = min[nec(X), nec(Y)]– pos(XvY) = max[pos(X), pos(Y)]

• No conceptual connection to probability– although probability/possibility can co-exist

15

Possibilistic Interpretationof Intelligence Statements (Heuer)

Probability

Possibility

Chances are slight

Little chanceBetter than even

Highly likely

16

Experience with Nonprobabilistic Methods

• Not all good:– Standardization of belief metrics?– Treatment of dependences?– Treatment of conflicting evidence?– Computational demands?– Interpretation of results?– Incorporation into decision process?

• Plan B: Confront the problems with probabilistic methods

17

Principled Basis for Probability Formulation

• Analysts uncomfortable producing probabilities– justified discomfort

• Alternative:– Produce defensible basis for probability formulation based on

nonprobabilistic judgment• Maximize expression of uncertainty subject to judged

constraints• Borrow uncertainty metrics from:

– statistical mechanics– information theory

• Entropy = -∑i pi.ln pi – discrete probability distribution, pi

18

Application of Information-Theoretic Methods

• Two USNRC programs:– QUEST- SNL

• Quantitative uncertainty evaluation of source terms

– QUASAR – BNL• Quantitative uncertainty analysis of severe accident releases

• Both studies used the same form of input to the same deterministic models– non-probabilistic input

• expert-generated input parameter uncertainty ranges

• QUEST: Bounding analysis

• QUASAR: Information Theory used to generate probability distributions from bounds

• Probabilistic analysis internal to methodology – no elicitation of probability

19

Information Theory and the Preservation of Uncertainty

Uncertainty Bands

1.00E-04 1.00E-03 1.00E-02 1.00E-01 1.00E+00

QUEST

QUASAR

QUEST

QUASARI-131

Cs-137

Release Fraction

20

Uncertainty Analysis as Resource to Visual Analytics

• VA Agenda

– Develop new methods and technologies for capturing and representing information quality and uncertainty

– Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.

– Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process

• UA Insight

– Probabilistic techniques• Elicitation methods• Aggregation methods• Information-theoretic approaches

– Nonprobabilistic alternatives• Dempster-Shafer• Possibility theory

– Uncertainty propagation techniques• Analytic• Numerical

– Risk communication• Risk representation• Decision-analysis methods

21

Merit Criteriafor Uncertainty Analysis in Intel

• Makes the analyst’s job easier• Represents strength of evidence intuitively• Can reflect dissonant evidence• Appropriately propagates uncertainty from analyst

to decision-maker• Characterizes output uncertainty in a standardized

and interpretable way• Computationally tractable• Promotes insight