1 Expert Judgment EMSE 280 Techniques of Risk Analysis and Management

1

Expert Judgment

EMSE 280 Techniques of Risk Analysis and Management

2

Expert Judgment

• Why Expert Judgement?

– Risk Analysis deals with events with low intrinsic rates of occurrence not much data available.

– Data sources not originally constructed with a Risk Analysis in mind can be in a form inadequate form for the analysis.

– Data sources can be fraught with problems e.g. poor entry, bad data definitions, dynamic data definitions

– Cost, time, or technical considerations

3

Expert Judgment

• Issues in the Use of Expert Judgement– Selection of Experts

• Wide enough to encompass all facets of scientific thought on the topic• Qualifications\criteria need to be specified

– Pitfalls in Elicitation – Biases• Mindsets – unstated assumptions that the expert uses• Structural Biases – from level of detail or choice of background scales for

quantification• Motivational Biases – expert has a stake in the study outcome • Cognitive Biases

– Overconfidence – manifested in uncertainty estimation– Anchoring – expert subconsciously bases his judgement on some

previously given estimate– Availability – when events that are easily (difficult) to recall are

likely to be overestimated (underestimated)

4

Expert Judgment

– Avoiding Pitfalls

• Be aware

• Carefully design elicitation process

• Perform a dry run elicitation with a group of experts not

participating in the study

• Strive for uniformity in elicitation sessions

• Never perform elicitation session without the presence of a qualified analyst

• Guaranteeing Anonymity of Experts

– Combination of Expert Judgements

• Technical and political issues

5

Basic Expert Judgment for Priors

• Method of Moments

– Expert provides most likely value at parameter, , say * and a range L,U

– for a distribution f() we equate

E[]=(L+4*+U)/6 Var[}= [(U-L)/6]2

– And solve for distribution parameters

• Method of Range

– Expert provides maximum possible range for say L,U

– for a distribution f() with CDF F() we equate

– F(U) = .95 F(L) = .05

– And solve for distribution parameters

6

Combining Expert Judgment: Paired Comparison

• Description

– Paired Comparison is general name for a technique used to combine several experts’ beliefs about the relative probabilities (or rates of occurrence) of certain events.

• Setup

– E # experts

– a1, …, an object to be compared

– v1, …, vn true value of the objects

– v1,r, …, vn,r internal value of object i for expert r

– Experts are asked a series (specifically a total of n taken 2 at a time) of paired comparisons ai, vs aj

– ai, >> aj by e e thinks P(ai) > P(aj)

7

• Statistical Tests– Significance of Expert e’s Preferences (Circular Triad)

Test H0 Expert e Answered Random

Ha Expert e Did Not Answered RandomlyA circular triad is a set of preferences

ai, >> aj , aj >> ak , ak >> ai

Define c # circular triads in a comparison of n objects and

Nr(i) the number of times that expert r prefers ai to another

object expert data Nr(1), …, Nr(n), r = 1, …, e. c(r) the number of circular triads in expert r’s preferences

David(1963)

n

ir niN

nnrc

1

22

12

1)(

2

1

24

)1()(


8

– Significance of Expert e’s Preferences (Circular Triad)• Kendall (1962)

– tables of the Pr{c(r) >c*} under H0 that the expert answered in a random fashion for n = 2, …, 10

– developed the following statistic for comparing n items in a random fashion,

When n>7, this statistic has (approximately) a chi-

squared distribution with df =

– perform a standard one-tailed hypothesis test. If H0 for any expert cannot be rejected at the 5% level of significance i.e. Pr{2c’(e)}>.05, the expert is dropped

24

21

n

nnn


2

1)(

34

1

4

8

)4(

)2)(1()('

2ec

n

nn

nnnec

9

• Statistical Tests– Agreement of Experts : coefficient of agreement

Test H0 Experts Agreement is Due to Chance

Ha Experts Agreement is not Due to Chance

Define

N(i,j) denote the number of times ai >> aj.

coefficient of agreement

attains a max of 1 for complete agreement


1

22

2

),(2

1 ,1

ne

jiN

u

n

i

n

ijj

10

– Agreement of Experts : coefficient of agreement• tabulated distributions of

for small values of n and e under H0

• These are used to test hypothesis concerning u. For large values of n and e, Kendall (1962) developed the statistic

which under H0 has (approx.) a chi squared distribution with .

we want to reject at the 5% level and fail if Pr{2u’}>.05


n

i

n

ijj

jiN

1 ,1 2

),(

2

23

2221

2

),(4

'1 ,1

e

eenejiN

u

n

i

n

ijj

22

12

e

een

df

11

• Statistical Tests– Agreement of Experts : coefficient of concordance Define

R (i,r) denote the rank of ai obtained expert r’s responses

coefficient of concordance

Again attains max at 1 for complete agreement


n

i

e

r

n

j

e

r

n

rjR

riRS1

2

1

1 1

),(

),(

)(121 32 nne

Sw

12

– Agreement of Experts : coefficient of concordance

• Tables of critical values developed for distribution of S under H0 for 3n7 and 3n20 by Siegel (1956)

• For n>7, Siegel (1956) provides the the statistic

Which is (approx) Chi Squared with dfn1Again we should reject a the 5% level of significance


)1(121

'

nen

Sw

13

• Assumptions

vi,r ~N(i, i2) with i = vi and i

2 = 2

Paired Comparison: Thurstone Model

Think of this as tournament play

Probability that 3 beats 2 or 3 is preferred to 2

14

• Assumptions

vi,r ~N(i, i2) with i = vi and i

2 = 2

• Implicationsvi,r vj,r ~N(i j, 22) ~N(i,j, 22) (experts assumed indep) ai is preferred to aj by expert r with probability

if pi,j is the % of experts that preferred ai to aj then


2}

2Pr{

}2

Pr{}0Pr{

,,

,,,

jiji

jirjri

Z

Zvv

22

2,

,1,

,jiji

jiji

ji porp

15

• Establishing Equations

Then we can establish a set of equations by choosing a scaling constant so that

as this is an over specified system for we solve for i such that

we get and

Mosteler (1951) provides a goodness of fit test based on an approx Chi-Squared Value


2 x Let ,

1ji,

jijip

ji ji,x

2,

ji,,..,

)(xmin1

jiji

n

n

est

n

ji

1

ji,x

)(

2

n

16

• Assumptions

Thus each paired comparison is the result of a Bernoulli rv for a single expert , a binomial rv for he set of experts

vi are determined up to a constant so we can assume

Define

then vi can be found as the solution to

Paired Comparison: Bradley-Terry Model

ji

ij vv

vaajir

} to prefered is Pr{),( i

n

iiv

1

1

e

rr iNiN

1

)()(

n

ijjji

i

hhe

iNh

,1

1

)(

17

Iterative solution Ford (1956)

Ford (1957) notes that the estimate obtained is the MLE and that the solution is unique and convergence under the conditions that it is not possible to separate the n objects into two sets where all experts deem that no object in the first set is more preferable to any object in the second set.

Bradley (1957) developed a goodness of fit test based on

(asymptotically) distributed as a chi-square distribution with df = (n1)(n2)/2

Paired Comparison: Bradley-Terry Model

1

1 1

1)()(1)1()(

)1( /)(i

j

n

ij

kj

ki

kj

ki

ki

hhhh

eiNh

n

i

n

ijj

n

i

n

i

n

ijjii vveviNjirjiNF

1 ,1 1 1 1

)ln()ln()(),(ln(),(2

18

• Motivation– If Ti~exp(i) then

– For a set of exponential random variables,we may ask experts which one will occur first

– We can use all of the Bradley-Terry machinery to estimate i – We need only have a separate estimate one particular anchor all the others

Paired Comparison: NEL Model

ji

iji TT

}Pr{

19

Combination of Expert Judgment:Bayesian Techniques

• Method of Winkler (1981) & Mosleh and Apostolakis (1986)

– Set Up

• X an unknown quantity of interest

• x1, …, xe estimates of X from experts 1, …, e

• p(x) DM’s prior density for X

• Then

– If the experts are independent

x)?|x,...,p(x

x)p(x)|x,...,p(x)x,..,x|p(x

e1

e1e1

x)?|p(x

x)|p(xx)|x,...,p(x

i

e

1iie1

20


• Method of Winkler (1981) & Mosleh and Apostolakis (1986)

– Approach

where the parameters μi and σi are selected by the DM to

reflect his\her opinion about the experts’ biases and accuracy

• Under the assumptions of the linear (multiplicative) model, the likelihood is simply the value of the normal (lognormal) density with parameters x+μi and σi .

• Then for the additive model we have

indep. and ),N(~ )Ln( Model tiveMultiplica

indep. and ),N(~ Model Additive2

i

2i

iiii

iiii

xx

xx

bygiven varianceandmean with normal is

),...,x|p(x then ,deviation standard and mean with

normal be )(prior smaker'decision Let the 10.1n Propositio

111e ee xx

xp

21


0 and with

1 ),...,x|VAR(X

and )( ),...,x|E(X

1e1e

1

2

2

1e

1

21

1e

1i1

jj

ii

ii

e

iiie

w

x

xwx

Note: i. the multiplicative model follows the same line of

reasoning but with the lognormal distribution

ii. the DM acts as the e+1st expert, (perhaps uncomfortable)

22

Combination of Expert Judgment:Bayesian TechniquesMOSLEH APOSTALAKAIS MODEL

FOR EXPERT JUDGMENT COMBINATION

Selct Model

Additive

Multiplicative x

Number of Experts

3

DM PRIOR SPECIFICATION

Mean X St Dev X Mean Ln X St Dev Ln X

4.63E-05 9.36E-06 -1.00E+01 2.00E-01

EXPERT INPUT

Estimate Expert EvaluationMult. Error Parameters Ln Error Parameters

Number Xi Mean St Dev Ln Xi Ln Mean Ln St Dev

1 2.00E-05 1.01E+00 1.01E-01 -1.08E+01 0.00E+00 1.00E-012 4.50E-06 8.00E+00 3.33E+00 -1.23E+01 2.00E+00 4.00E-013 1.00E-04 2.77E+00 5.60E-01 -9.21E+00 1.00E+00 2.00E-01

POSTERIOR INFERENCE

Mean 2.19E-05 Ln Mean -1.07E+01St Dev 1.76E-06 Ln St Dev 8.00E-02

Prob X in =

-5.00E +04

0.00E +00

5.00E +04

1.00E +05

1.50E +05

2.00E +05

2.50E +05

0.00E +00 1.00E -05 2.00E -05 3.00E -05 4.00E -05 5.00E -05 6.00E -05 7.00E -05 8.00E -05

Prior

Post.

23

Combination of Expert Judgment:The Classical Model

• Overview

– Experts are asked to assess their uncertainty distribution via specification of a 5%, 50% and 95%-ile values for unknown values and for a set of seed variables (whose actual realization is known to the analyst alone) and a set of variables of interest

– The analyst determines the Intrinsic Range or bounds for the variable distributions

– Expert weights are determined via a combination of calibration and information scores on the seed variable values

– These weights can be shown to satisfy an asymptotic strictly proper scoring rules, i.e., experts achieve their best maximum expected weight in the long run only by stating assessments corresponding to their actual beliefs

24Combination of Expert Judgment:The Classical Model

0

.5

1

ql q5 q50 q95 qu

0

.5

1

ql q5 q50 q95 qu

For a weighted combination of expert CDFs take the weighted combination at all break points (i.e. qi values for each expert) and then linearly interpolate

Expert 1

Expert 2


Var. 1

Expert 1 Expert 2 Expert 3

x| || | | | || |x x

Var. 2 x| || | | | || |x x

Var. 3 x| || | || || |x x

Var. 4 x| || | || || |x x

Var. 5 x| || | || || |x x

Expert 1 – Calibrated but not informative

Expert 2 – Informative but not calibrated

Expert 3 – Informative and calibrated

Expert Distribution Break Points

Realization


– Information• Informativeness is measured with respect to some background

measure, in this context usually the uniform distributionF(x) = [x-l]/[h-l] l < x < h

• or log-uniform distribution F(x) = [ln(x)-ln(l)]/[ln(h)-ln(l)] l < x < h

• Probability densities are associated with the assessments of each expert for each query variable by– the density agrees with the expert’s quantile assessment – the densities are nominally informative with respect to the

background measure– When the background measure is uniform, for example,

then the Expert’s distribution is uniform on it’s 0% to 5% quantile, 5% to 50% quantile, etc.


– Information

• The relative information for expert e on a variable is

• That is, r1 = F(q5(e)) F(ql(e)) , …, r4 = F(qh(e)) F(q95(e))

•

• The expert information score is the average information over all variables

i intervalfor measure background

theis r and 45,.05)(.05,.45,. is ),...,( where

)/()(

i41

4

1

pp

rpLnprIi

iii

min max| | | | |

1

0.5

Expert Distribution

Uniform Background Measure

28


• Intrinsic Range for Each Seed Variable

• Let qi(e) denote expert e’s i% quantile for seed variable X

• Let seed variable X have realization (unknown to the experts ) of r

• Determine intrinsic range as (assuming m experts)

l=min{q5(1),…, q5(m),r} and h =max{q95(1),…, q95(m),r}

• then for k the overshoot percentage (usually k = 10%)

– ql(e)=l – k(h - l)

– qh(e)=l + k(h - l)

– Expert Distribution (CDF) for seed variable X is a linear interpolation between

• (ql(e),0), (q5(e),.05), (q50(e),.5), (q.95(e),.95), (qh(e),1)

29


• Calibration

– By specifying the 5%, 50% and 95%-iles, the expert is specifying a 4-bin multinomial distribution with probabilities .05, .45, .45, and .05 for each seed variable response

– For each expert, the seed variable outcome (realization), r, is the result of a multinomial experiment, i.e.

• r [ql(e), (q5(e)), [interval 1], with probability 0.05

• r [q5(e), q50(e)), [interval 2], with probability 0.45

• r [q50(e), q95(e)), [interval 3], with probability 0.45

• r [q95(e), qh(e)], [interval 4], with probability 0.05

– Then if there are N seed variables and assuming independence

si= [# seed variable in interval i]/N is an empirical estimate of

(p1, p2, p3, p4) = (.05, .45, .45, .05)

30


• Calibration

• We may test how well the expert is calibrated by testing the hypothesis that

H0 si = pi for all i vs Ha si pi for some i

• This can be performed using Relative Information

45,.05)(.05,.45,. is ),...,( and variablesseed the

fromdensity empirical theis ),...,(s where

)/(),(

41

41

4

1

pp

s

psLnspsIi

iii

31


Note that this value is always nonnegative and only takes the value 0 when si=pi for all i.

• If N (the number of seed variables) is large enough

• Thus the calibration score for the expert is the probability of getting a relative information score worse (greater or equal to) than what was obtained

4

1

)/(),(i

iii psLnspsI

χ~ p)2NI(s, 2(3)

p)}2NI(s, Pr{χ p))(2NI(s,χ where

p))(2NI(s,χ1c(e)2(3)

2(3)

2(3)


– Weights

• Proportional to calibration score * information score

• Don’t forget to normalize

– Note

• as intrinsic range for a variable is dependent on expert quantiles, dropping experts may cause the intrinsic range

to be recalculated

• change in intrinsic range and background measure have negligible to modest affects on scores


• Optimal (DM)Weights – Choose minimum value such that if C(e) > , C(e) = 0 (some experts will get 0 weight)– is selected so that a fictitious expert with a distribution equal to that of the the weighted combination of expert distributions would be given the highest weight among experts


COMBINATION OF EXPERT JUDGMENT

Number of Experts 4

Select Type of Weights

Equal

Use Specified

Performance Based x

Intrinsic Range Adjustment 0.1

Expert Input Weights# l 5% 50% 95% h Equal User Perform

1 40 200.00 330.00 500.00 760 0.25 0.40 0.2662 40 270.00 300.00 400.00 760 0.25 0.10 0.4293 40 100.00 200.00 300.00 760 0.25 0.10 0.0004 40 200.00 300.00 700.00 760 0.25 0.40 0.306


SEED VARIABLE INPUT

Overshoot 0.1Number of Variables 5

VARIABLE 1 ql= 0

Realization 26 qh= 87.9

Experts 5% 50% 95% <5% 5%-50% 50%-95% >95% Information1 2.00E+01 5.50E+01 8.00E+01 0 1 0 0 1.56E-012 3.00E+01 5.00E+01 6.00E+01 1 0 0 0 7.37E-013 1.00E+00 1.00E+01 2.00E+01 0 0 0 1 1.22E+004 1.00E+01 5.00E+01 6.00E+01 0 1 0 0 4.80E-01

0 0 0 0 0.00E+000 0 0 0 0.00E+000 0 0 0 0.00E+000 0 0 0 0.00E+000 0 0 0 0.00E+000 0 0 0 0.00E+00


PERFORMANCE W EIGHTS

EMPITRICAL DISTRIBUTION INDIVIDUAL VARIABLE INFORMATIONEXPERT NORM WT WT CAL 0.05 0.45 0.45 0.05 INF VAR 1 VAR 2 VAR 3 VAR 4 VAR 5 VAR 6

1 0.274 0.187 0.395 0.000 0.200 0.800 0.000 0.4749 0.1564 0.3182 0.7481 0.6284 0.5236 0.00002 0.439 0.301 0.411 0.200 0.200 0.600 0.000 0.7323 0.7372 0.6301 0.5415 0.6391 1.1136 0.00003 0.000 0.000 0.000 0.000 0.000 0.200 0.800 1.2720 1.2222 0.7847 1.3326 0.7847 2.2356 0.00004 0.287 0.196 0.740 0.000 0.400 0.600 0.000 0.2656 0.4803 0.2260 0.2884 0.2260 0.1071 0.0000


Plot ComparisonPlot 1 DMPlot 2 1Plot 3 2Plot 4 3Plot 5 4

Expert CDF

0.0000.100

0.2000.300

0.4000.500

0.6000.700

0.8000.900

1.000

0 200 400 600 800 1000Varaible Value

CD

F

DM

1

2

3

4

Expert CDF

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

0 200 400 600 800 1000Varaible Value

CD

F

DM

1

2

3

4

Expert CDF

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

0 200 400 600 800 1000Varaible Value

CD

FDM

1

2

3

4

USER DEFINED WEIGHTS EQUAL WEIGHTS

PERFORMANCE BASED WEIGHTS

Documents

1 Expert Judgment EMSE 280 Techniques of Risk Analysis and Management