Eliciting numbers for BNs Samantha Low-Choy, 22 …eprints.qut.edu.au/47159/1/LowChoy-ElicitationABNMS-Part1...biosecurity built on science Eliciting numbers for BNs -Choy, 22 Nov

biosecurity built on science

Eliciting numbers for BNs

Samantha Low-Choy, 22 Nov 2011

1

Course material is based on publications Topic Hugin Expert A/S (1995-2011), Online Help, http:www.hugin.com Table Generator

James, A., Low Choy, S., Murray, J., and Mengersen, K. (2010). Elicitator: An expert elicitation tool for regression in ecology. Env Mod & Soft, 25(1):129–145.

Elicitator – How To Use, Information Technology

Johnson, S., Low-Choy, S. and Mengersen, K. (in press, 2011) “Integrating Bayesian networks and Geographic information systems”, Integrated Environ Assess and Mgmt.

Elicitator for Populating CPTs in BNs

Low Choy, S., Wilson, T. (2009) How do experts think about statistics? Hints for improving undergraduate and postgraduate training, IASE Internat Assoc for Stat Educ, Durbin South Africa, 2009, Satellite Conference Proceedings. http://www.stat.auckland.ac.nz/~iase/publications/sat09/4_3.pdf

Logical fallacies and Visualization of probabilities

Low Choy, S., O‟Leary, R. and Mengersen, K. (2009) “Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models”, Ecology, 90(1):265-277.

Framework for designing elicitation, with 5 examples.

Low Choy, S., Murray, J., James, A. and Mengersen, K. (to appear Nov 2011) “Elicitator: a user-friendly, interactive tool to support elicitation of expert knowledge in landscape ecology”, Chp3 in Expert Knowledge and Its Applications in Landscape Ecology, eds Perera, Drew, & Johnson, Springer, NY

Overview of elicitation, its purposes and uses. Introduction to Elicitator.

Low Choy, S., Murray, J., James, A. and Mengersen, K. (2010) “Indirect elicitation from ecological experts: from methods and software to habitat modelling and rock-wallabies” in The Oxford Handbook of Applied Bayesian Analysis, eds O‟Hagan, West, OUP: UK, p511-544.

Indirect elicitation – statistical methods & comparison esp. Elicitator.

Low Choy, S. (2011) “Eliciting numbers for Bayesian Networks”, slides for a Satellite Workshop for the Australian Bayesian Network Modelling Society, http://eprints.qut.edu.au/.

These slides.

http://www.stat.auckland.ac.nz/~iase/publications/sat09/4_3.pdf

http://eprints.qut.edu.au/


Eliciting numbers for Bayesian Networks

Australian Bayesian Network Modelling Society Workshop 22 Nov 2011

Samantha Low Choy (CRCNPB/QUT Maths)

Complements presentations by Marissa MacBride (ACERA)

2


Course Outline

Understanding Probabilities

Credible probabilities

Eliciting one probability

Eliciting a CPT

•Defining probabilities

•Logical fallacies

•Ambiguity

•Cognitive biases

•Calibration

•Structured design of elicitation

•Validation

•4-step elicitation method

•Outside-in (Elicitator)

•Smorgasbord

•CPT calculator

•HUGIN Table Generator

•Elicitator

3


Understanding probabilities when constructing Bayesian belief networks

“Constructing” the conceptual model

- can better reflect expert knowledge by applying precise statistical interpretation & implications of links

- can avoid redundant links or missing latent links caused by not understanding conditional probabilities

“Populating” a CPT

- can be done in a transparent, repeatable way by harnessing structured elicitation techniques

- can incorporate uncertainty by using statistical methods for encoding probabilities

- can be done more efficiently using indirect elicitation to reduce time to populate large or

complex CPTs

- can be done more accurately by adopting techniques to train and/or calibrate experts

4


Understanding Probability

Active Thinking & Learning

Quiz

Theory

•Quiz

•Theory

•Learnings for Elicitation

•Ellen & the Parrot

•Marjory & the Wallaby

•Deer, Irvine

•Zak & the Crocodile

•Philosophies

•Defining probabilities

•Alternative views

•Linguistic ambiguity

•Logical fallacies

5


AN ELICITATION EXPERIMENT How are these questions interpreted?

We want to learn how these questions are interpreted, and how this affects the answers and elicitation experience. This depends on:

• the wording, format and context of questions

• understanding & experience with probability + field

-Joint research with Therese Wilson, QUT MAC

-Previously tested on some ecologists & CRCNPB PhDs

-Refocussed to BNs by piloting on PhD students Jegar Pitchforth & Charisse Farr, Airports of the Future

6


ELICITATION Quiz, Theory & Learnings

Active learning of mathematics means: getting stuck, which means understanding what you do and do not easily recall.

We will exemplify the Bayesian learning cycle:

1. explicitly acknowledge the current state of knowledge

2. examines how this is impacted by new information

7


Question 1: Ellen and the Parrot

8


Ellen and the parrot Ellen, an expert on Australian native bird species,

states that the probability of finding an Eclectus parrot at this time of year is 0.1% (1 in a 1000).

Q1a. How would you explain what is meant by this to a first year ecology student?

Q1b. How would your explanation change if you needed to look for the parrot in a region one half the size?

9


Ellen and the parrot Ellen, an expert on Australian native bird species,

states that the probability of finding an Eclectus parrot at this time of year is 0.1% (1 in a 1000).

Q1c. If you knew that the expert had only just obtained their degree, how would this affect your interpretation of their response?

Q1d. If two people see an Eclectus parrot in the same week does this make the expert a liar? Why or why not?

10


DEFINING PROBABILITIES

Do not just focus on what you want to ask, but also how it will be interpreted by the targeted set of experts.

Linguistic ambiguity affects questionnaires of all pedigrees. Common if the designer focuses on the content rather than the testing & validation.

Pilot, pilot, pilot!

11


Defining probabilities Classically defined as a frequency

What is n, the baseline (denominator)?

What is x, the count (numerator)?

- Inclusions and exclusions

- Observation changes reality? (eg destructive)

- Missing values vs zeros

Units & Extent

- Space and time

- Other forms of dis/aggregation, eg ecological

12

xp

n


Counting

What is one thing?

Years free from pests

Individual farms that are infested

When a pest was present, we detected it

Defining the Event


Counting

What is, what is not

Years free from pests vs infested

Individual farms that are infested vs clean

When a pest was present, we detected it or not Defining the Complement


Probability: Defining the population Putting numbers in context

Probability that any year was free from pests during 1900-2010.

Probability that an individual farm is infested in the district.

Probability of detecting a pest when it is there. Defining the Population


Conditional Probability

Counting with Constraints

Area searched, infested =10 ha

Area searched, pest-free =20 ha

Area not searched =70 ha

What is the right baseline? Total area =

S = Area searched =

%area searched, Pr(S) =

%area searched & infested

Pr(S and I) =

%searched area infested, Pr(I|S) =

CountProbability

Baseline count


Conditional Probability

Counting with Constraints

Area searched, infested =10 ha

Area searched, pest-free =20 ha

Area not searched =70 ha

What is the right baseline?

Area =10+20+70=100ha

S = Area searched =

10+20=30ha

%area searched, Pr(S) =

30/100=30%

%area searched & infested

Pr(S and I) =10/100=10%

%searched area infested, Pr(I|S) = 10/30=33%

CountProbability

Baseline count


Question 2: Marjory and the Wallaby

18


Marjory is an expert on identifying rock wallabies. She is claimed to be 90% reliable at correctly identifying a rock wallaby when it really is one.

For each of the following statements tell me if you think it is true or false.

a) The probability that a rock wallaby is present is 90%.

b) If we could catch 10 of the rock wallabies identified by Marjory, then odds are that 1 of these would not be a rock wallaby.

19

Marjory and the wallaby


Marjory is an expert on identifying rock wallabies. She is claimed to be 90% reliable at correctly identifying a rock wallaby when it really is one.

For each of the following statements tell me if you think it is true or false.

a) 90 times out of a 100 Marjory will identify a rock wallaby correctly.

b) There is a 9 in 10 chance that Marjory will see the rock wallaby if it‟s there.

c) The odds are 1:9 that Marjory has not correctly identified a rock wallaby.

20

Marjory and the wallaby


INTERPRETING PROBABILITIES

Interpretation depends on Philosophy,

Which may depend on the expert‟s discipline and training.

Interpretation is guided by the order of presentation.

21


Interpreting probability Alternative Philosophies

Frequency

- Observed

Thought experiment

- Frequentist – long run of exp‟t

- Imagine, eg Schroedinger‟s cat

Parallel universe

- Useful for unique x (eg ecosystem), eg Physicist

Betting

- Odds, relies on fiduciary instinct (used by Fisher, Good, Savage)

Belief

- Degree of belief

Score (Sureness, %Right)

- Like a cross-experiment measure of performance

Be aware of multiple interpretations of probability

Discern which one(s) you and your elicitee use/prefer.

Be flexible

Consistency can be convenient (esp. for documentation, repeatability) and is feasible in disciplines where training in probability is consistent (e.g. physical sciences vs biological sciences)

22


Interpreting probability Linguistic ambiguity

90 times out of a 100 Marjory will identify a rock wallaby correctly.

- What is the Baseline? the attempted identifications?

the rock wallabies present?

- Discrimination: ID a rock wallaby vs a seal, or another wallaby?

- Frequencies are easier than fractions.

If we could catch 10 of the rock wallabies identified by Marjory, then odds are that 1 of these would not be a rock wallaby.

- More verbose but more precise

- Separates the condition into a separate clause.

- Clarifies the baseline n.

- “odds” aren‟t everyone‟s cup of tea

Linguistic ambiguity - Burgman

Aleotory (unknowable random variation) vs Epistemic (knowable structural variation) uncertainty – O‟Hagan

Write out the words exactly as you will ask them: makes the protocol explicit and therefore transparent and repeatable.

Allow some flexibility in follow-up questions.

Pilot!

23


QUESTION 3 Deer, Irvine

24


Deer, Irvine

Irvine et al (2009, Journal of Applied Ecology) assess some managers‟ knowledge of habitat use by red deer Cervus elaphus in the uplands of Scotland.

They asked managers what factors they thought impacted distribution of deer, recording interviews in transcripts. This table shows the number of times managers mentioned whether shelter or other factors were believed to impact the distribution of hinds (year-round) or stags in winter.

25

Factor

Total Shelter

Other

factors

Impact on

spatial

distribution

of deer

Hinds 19 23 42

Stags in

winter

25 33 58

Total 44 56 100

Consider the factors mentioned by managers in the transcripts.

Q3a) What‟s the probability that shelter was mentioned?

Write this down in the form of a fraction (eg 51 out of every 100 children born last year were boys).

Q3b) What‟s the probability that shelter for stags in winter was mentioned as an important factor?


Deer, Irvine

Irvine et al (2009, Journal of Applied Ecology) assess some managers‟ knowledge of habitat use by red deer Cervus elaphus in the uplands of Scotland.

They asked managers what factors they thought impacted distribution of deer, recording interviews in transcripts. This table shows the number of times managers mentioned whether shelter or other factors were believed to impact the distribution of hinds (year-round) or stags in winter.

26

Factor

Total Shelter

Other

factors

Impact on

spatial

distribution

of deer

Hinds 19 23 42

Stags in

winter

25 33 58

Total 44 56 100

Q3c) What‟s the chance that a mention related to shelter for hinds? Q3d) Consider managers‟ opinions on Stags in winter. How likely is it that managers believed shelter impacted on their distribution? Q3e) What‟s the chance that managers mention other factors (forage, comfort or disturbance) in the context of stags in winter?


DIFFERENT VIEWS OF PROBABILITIES

Visual aids to thinking about the problem

-Logic tree

-Matrix

-Venn diagram

-Mathematical

27


Visual thinking

Logic tree: % vs counts

M: Of these,

how many

match the

evidence?

G: Is the

suspect

guilty?

N-1

Innocent

1

Guilty

1 00%

Match

0%

No match

3 %

Match

97%

No match

N

suspects

M: Of these,

how many

match the

evidence?

G: Is the

suspect

guilty?

100

Innocent

1

Guilty

1

Match

0

No match

3

Match

97

No match

101

suspects

Probability of

“M given G”

Pr(M|G)=100%

Pr(not M|G)=0%

Pr(M|not G)=3%

Pr(not M|not G)=97%

Guilty is a given

Not Guilty is a given

28


Logic tree

Group by: guilt vs match

M: Of these,

how many

match the

evidence?

G: Is the

suspect

guilty?

100

Innocent

1

Guilty

1

Match

0

No match

3

Match

97

No match

101

suspects

G: Of these,

how many

are guilty?

M: How

many

match the

evidence?

97

No Match

4

Match

1

Guilty

3

Innocent

0

Guilty

97

Innocent

101

suspects

29


Visual thinking

Condition on Match

Match No Match

Guilty 1

0

1

Not Guilty

3 97 100

4 97 101

M: Of these,

how many

match the

evidence?

G: Is the

suspect

guilty?

97

No Match

4

Match

1

Guilty

3

Innocent

0

Guilty

97

Innocent

101

suspects

Column percentages:

Pr(not Guilty | Match) = 3/4 = 75%

Pr(not Guilty | no Match) = ?

30


Visual thinking

Condition on Guilt

Match No Match

Guilty 1

0

1

Not Guilty

3 97 100

4 97 101

M: Of these,

how many

match the

evidence?

G: Is the

suspect

guilty?

97

No Match

4

Match

1

Guilty

3

Innocent

0

Guilty

97

Innocent

101

suspects

Row percentage:

Pr(Match | Not Guilty) = ?

Pr(Match | Guilty) = ?

31


Visual Thinking

Venn Diagram Innocent Guilty

Marked

Only 1 crocodile

Is Guilty,

100 crocodile suspects are innocent;

3 of these match the evidence

Pr(M|not G) = 3/100 = 3%

and it also Matches

the evidence.

Pr(M|G)=100%

32


Exercise Visual Aids

Usual use Future use

Verbal description

Logic tree

Table (error matrix)

Venn diagram

Equations

Other

33


QUESTION 4 Zak and the Crocodile

34


A topical problem Zak tells a reporter that a crocodile bit his mate:

it was large, male and had a distinctive pattern along his back. Scientists tell us these characteristics co-occur on about 3 in every 100 crocs.

Someone finds a croc in the same region that fits this description.

Assume it‟s ok to kill & examine the croc if chances that it is guilty are better than even.

Q4a. What‟s your gut feeling: Is the croc innocent or guilty?

35





Q4b. Larry says there is a 3% chance that the croc would match the evidence if he were innocent, thus there is a 97% chance that he‟s guilty.

Do you agree? Explain.

36


A topical problem Zak tells a reporter that a crocodile bit his mate: it was

large, male and had a distinctive pattern along his back. Scientists tell us these characteristics co-occur on about 3 in every 100 crocs.


Q4b. Larry says there is a 3% chance that the croc would match the evidence if he were innocent, thus there is a 97% chance that he‟s guilty.

Do you agree? Explain.

Q4c. Renata says there are about 10,000 crocs in the region. Hence these matches would occur on about 300 of these. Therefore there is a 1 in 300 chance that this croc is the guilty one, which is insufficient evidence of guilt.

Do you agree? Explain why or why not.

37





Q4d. If you don‟t agree with Larry or Renata, how would you work out whether it is the offending crocodile?

38


Reversing the Conditions via

BAYES’ THEOREM

39


Probabilistic Thinking

Mathematical Events of interest

- M = crocodile matches the evidence, G = guilty

- M = DNA match, G = guilty of crime

- M = diagnostic test +ve, G = have condition

- M = sp. detected, G = sp. present

Core skills - Recognize we need Pr(G|M)

- Acknowledge we know Pr(M|G) = 1, Pr(M|not G) = 0.03

- We need Bayes Theorem to invert the logic:

Pr( | )Pr( ) Pr( | )Pr( )Pr( | )

Pr( ) Pr( | )Pr( ) Pr( | )Pr( )

M G G M G GG M

M M G G M G G

40



A word about equations An equation is

- Mathematical “shorthand”

- Pr(G|M)

- Probability of guilt (of any crocodile like this one), knowing that (given) it matches the evidence”

An equation can be

- Intimidating ... at first

- Incomplete ... without definitions

- Informative ... with definitions

41


Bayes Theorem by logic

M: Of these,

how many

match the

evidence?

G: Is the

suspect

guilty?

100

Innocent

1

Guilty

1

Match

0

No match

3

Match

97

No match

101

suspects

G: Of these,

how many

are guilty?

M: How

many

match the

evidence?

97

No Match

4

Match

1

Guilty

3

Innocent

0

Guilty

97

Innocent

101

suspects

Pr(M|not G) = 3/100 Pr(not G|M) = 3/4

Pr(not G|M) = 3/100 100/101 4/101 = 3/4

42


Bayes Theorem by table

Match No Match

Guilty 1

0

1

Not Guilty

3 97 100

4 97 101

The law of total (baseline) probability says Pr(total) = sum of Pr(parts)

assuming the parts don’t overlap, and together describe the total.

Here this baseline probability is Pr(M) = Pr(M,G) + Pr(M,not G).

So Pr(M) = Pr(M,G) + Pr(M,not G) = 1/1 x 1/101 + 3/101 = 4/101.

The definition of a conditional probability changes the baseline so

Pr(M|not G) = Pr(M,not G) / Pr(not G) = 3/101 / 100/101 = 3/100

Invert logic: Want Pr(not G|M)=3/4

If you can get Pr(M,not G)

Pr(not G|M) =

Else Using Bayes’ Theorem,

Pr(not G|M)

= Pr(M|not G) Pr(not G) / Pr(M)

= 3/100 x 100/101 4/101

= 3/4

Must know entire table to invert!

43


LOGICAL FALLACIES

Conjunction fallacy

Inversion fallacy

Neglecting base rates

Prosecutor‟s fallacy

Defendant‟s fallacy

44


Neglecting base rates

This problem occurs when you confuse a conditional probability with a joint probability.

Eg you could mistakenly say that Pr(G|M) = Pr(G and M) % marked crocs that are guilty is the same as the % crocs that match the evidence and are also guilty. but we know that Pr(G|M) = Pr(G and M) / Pr(M) which means acknowledging the base rate of matching the evidence amongst suspects! in other words you ignore the conditioning part.

Similar to the inversion fallacy,

This is a misunderstanding of how to include the base rates Pr(G) and Pr(M) Villejoubert+2002

47


The defendant’s fallacy Overestimate probability of

innocence in favour of the defendant

Dilute evidence on matching, despite rarity in population at large Thompson+2002

- Choose very large reference region, eg 40,000 crocs in NQ

- Expect 1,200 to match

- So 1 in 1,200 chance of guilt given markings

Need appropriate #suspect crocodiles

- Eg N=101 gives Pr(G|M)=25% or N=11 gives Pr(G|M)=77%

M: Of these,

how many

match the

evidence?

G: Is the

suspect

guilty?

N-1

Innocent

1

Guilty

100%

Match

0%

No match

3%

Match

97%

No match

N

suspects

49


Getting the baseline wrong

The Defendant‟s fallacy is a special case

- Diluting the number of suspects

More generally,

- Baseline population has too many irrelevant cases 40,000 crocs

naughty noughts, Austin+Meyers 96

50


How the baseline affects guilt given marking

0 200 400 600 800 1000

0.0

0.2

0.4

0.6

0.8

1.0

Number of suspects, N

Pr(

G|M

)

Pr(G|M) = 50% for N=35

Pr(G|M) = 25% for N=101

Pr(G|M) = 3.2% for N=1001

51


LOGICAL THINKING How we’d like the expert to think

52



via Equations N suspect crocodiles

Regardless of evidence

- Pr(G) = 1/N, Pr(not G) = (N-1)/N

“these characteristics match 3% of crocs [in the general population]”

- Interpret as Pr(M)=3%

“these characteristics match 3% of crocs [in the general population excluding man-eating crocodiles]”

- Or interpret as Pr(M|not G) = 3%

53



via Equations “these characteristics match 3% of crocs [in the general

population]”

- Interpret as Pr(M)=3%

Hence

So if N= 101, Pr(G|M) = 100/306 =33% but if N=1001, Pr(G|M) = 100/3006= 3.3%

11Pr( | ) Pr( ) 1001Pr( | )3Pr( ) 3( 1)100

M G G NG MM N

54



via Equations “these characteristics match 3% of crocs [in the general

population excluding man-eating crocodiles]

- Interpret as Pr(M|not G) = 3%

Pr(M) = Pr(M|G)Pr(G) + Pr(M|not G)Pr(not G) = 1x1/N + .03(N-1)/N

Hence p = Pr(G|M) = 1/(1+ .03(N-1))

So if N= 101, p = 1/(1+3) =25% but if N=1001, p = 1/(1+30)= 3.2%

55


Learnings In this case, for both interpretations, we got similar

answers, and so would conclude the crocodile is likely guilty if N 100.

As the number of suspects decreases, the calculated probability of guilt given matching changes more and more.

But in general, answers may not be similar, so conclusions may depend heavily on the interpretation

Determine hidden assumptions

Elicit the baseline separately

If necessary, elicit the probability of the event conditioning on different hypothetical baselines

56


Learnings Challenge of science is often to operate under

incomplete information; post-normal science Functowitz

- Elicitation from text often requires interpretation of ambiguous statements.

- Elicitation from groups in a workshop situation can be difficult to revisit.

Paraphrase to make explicit any underlying assumptions required to fully define the number.

Ensure authorship (not anonymity) and ownership by group/individual is clear.

Ideal to confirm interpretation or fill gaps via follow-up with authors.

57


LOGICAL THINKING How we help the expert to be logical

Bayes Theorem: Pr(G|M) = Pr(M|G) Pr(G) / Pr(M)

Bayesian cycle of learning can be thought of as a model for rational thought. Dawid

Bayesian statistical modelling and cycle of learning: Pr(model|data) = Pr(data|model) Pr(model) / Pr(data)

But do ordinary (rational) people really think that way? El Gamal; Luc Bovens & Stephan Hartmann 2003

Elicitation need not expect, but can guide, logical thinking.

58


LOGICAL FACILITATION Helping them get their logic right

Elicitation is not a test;

it is a collaboration.

We want to find out what the experts know, and we hope they want us to understand what they are trying to say.

Design elicitation so that the elicitor focuses on what the expert knows; not the same as asking for “the answer”.

59


Avoid conjunction fallacy Does the expert view additional conditions as

affecting the probability or their sureness? - Knowing more about crocodile makes it easier to judge their

guilt, but does not make it more likely that they are guilty.

- Eg Pr(marking) Pr(marking, male, big) Pr(marking| guilty) Pr(marking, male, big | guilty)

- NB marginal probabilities include all possibilities for other unmentioned conditions. Here the first group of marked crocs includes females (big or small) and small (male or female).

Clarify hidden aggregation for marginal probabilities.

Elicit joint probabilities and conditional probabilities.

Show the expert tables or graphs of probabilities

- to help comparison (across conditions),

- ensure coherence (summation across margins).

60


Avoid inversion fallacy Elicitation is NOT a test, it’s a collaboration!

Minimize the need for “mental gymnastics” by the expert, even if this requires more thought by the elicitor.

Discern which direction of conditioning is more natural for expert(s), and frame questions from that perspective.

Separate the condition from the event.

Reflect back what the expert has said, both verbally and visually.

Ask the expert to paraphrase their answer to help you confirm the direction of reasoning.

61


Avoid baseline issues Get the baseline right

Choose units that are meaningful to the expert, eg half hectare Low Choy et al 2005, 2009

Use whole numbers where possible, eg 10 out of 1000, rather than fractions Kynn 2008

Give numerator and denominator separately (separate condition from event) Girotto+2001

Consider important sub-populations separately, during elicitation and modelling Murray et al, 2010

Explain how small or large probabilities derived Kynn 2008

62


Avoid taking sides for Prosecution or Defence

Account for motivations by different stakeholders

Assess motivational bias

- Elicit expert backgrounds, positions & employers.

- From their training and history, determine their school of thought (academic peerage).

- Can request a statement re: conflict of interest (eg funding by agencies with motivational bias).

Reveal implicit assumptions; esp, explain how small or large probabilities derived Kynn 2008

Separate elicitation of scientific knowledge vs political decisions Low Choy et al 2005,2008.

63


Exercise Which fallacies are you prone to?

Previously In the Future

Linguistic uncertainty

Conjunction fallacy

Inversion fallacy

Prosecutor‟s fallacy

Defendant‟s fallacy

Baseline misrepresentation

Other

64


Elicitation technique helps to address Am

big

uit

y

Co

heren

ce

Co

nju

ncti

on

In

versio

n

Po

liti

cs

Baselin

e

Clarify hidden aggregation for marginal probabilities.

Elicit joint probabilities and conditional probabilities.

Show tables or graphs to compare probabilities.

Separate statement of condition, from event.

Show tree or flow diagrams to show logical sequence

Seek/reflect a paraphrase of statement

Choose the right units; seek numbers not fractions

Explain basis for small or large probabilities

Determine motivational perspectives

Model decomposition

65

In summary ...


COGNITIVE BIASES A summary

Pioneered by Kahnemann & Tversky in the 1970s, Recently reviewed by Kynn (2008, JRSSA) and O‟Hagan et al (2006, chapters 1-3).

Cognitive biases: how people (not nec. expert) can get it wrong

Miscalibration: over/under-confidence; over/under-estimates

Group dynamics overlay individual cognitive, motivational &

numerical biases (Plous 1993)

66


Bias

Elicitation is potentially subject to several sources of uncertainty, including biases, being conscious and subconscious discrepancies between the subject‟s responses and an accurate description of his underlying knowledge Spetzler & Staël von Holstein 1975

67


Bias

Experts can be conditioned to be aware of common biases so that they are more easily avoided. ... displacement bias when experts over- or underestimate ... variability bias or conservatism where typically experts underestimate the variability in the quantity of interest ... motivational biases due to the expert‟s lack of neutrality Spetzler & Staël von Holstein 1975; Tversky & Kahneman 1981

68


Biases & Bandaids Anchoring bias:

minimize via order of questions

Garthwaite & Dickey 1985,

Phillips & Wisbey 1993

Representativeness bias (SD vs SEM): structure to avoid tacit conditioning

Spetzler & Stael von Holstein 1975

Implicit conditioning: list reasons, esp. very

high/low values

Siu & Kelly 1998, Kynn 2008

Coherence biases: automatically check

for logical consistency

Kadane & Wolfson 1998

Feedback helps experts maintain self-

consistency

Kynn 2006, Low-Choy et al 2009b,2011

Comparing encodings from using different elicitation methods

Gavasakar 1988

Questioning multiple experts can be

representative, or outperformed by best!

Clemen & Winkler 1999

List sources of relevant expertise

Tversky & Kahneman 1973

„what is being elicited is expert, not perfect,

opinions, and thus they should not be

adjusted‟

Kadane & Wolfson 1998

69


A final thought ... On eliciting the current state of knowledge

For elicitation, is there a right or wrong answer? “No amount of experimentation can ever prove me right;

a single experiment can prove me wrong.” Albert Einstein

If a gold standard exists, then why consult experts? What do they contribute beyond measurement? (Important for enlisting experts!)

Semper disco I always learn

Some (but not enormous) pressure on experts to get it right.

Be open to their considered judgment, with the information at hand.

Accurately capture the current state of knowledge, which will change!

Understanding is a two-way street Eleanor Roosevelt

Accurately reflect relevant expert knowledge

Ensure the target of elicitation is appropriate, including an understanding of context

Aim for effective questions, questioning and collaboration

Understanding requires more work when experts are new to conceptualizing their knowledge or to quantification

70

Documents

Eliciting numbers for BNs Samantha Low-Choy, 22 …eprints.qut.edu.au/47159/1/LowChoy-ElicitationABNMS-Part1...biosecurity built on science Eliciting numbers for BNs -Choy, 22 Nov