View
1
Download
0
Category
Preview:
Citation preview
biosecurity built on science
Eliciting numbers for BNs
Samantha Low-Choy, 22 Nov 2011
1
Course material is based on publications Topic Hugin Expert A/S (1995-2011), Online Help, http:www.hugin.com Table Generator
James, A., Low Choy, S., Murray, J., and Mengersen, K. (2010). Elicitator: An expert elicitation tool for regression in ecology. Env Mod & Soft, 25(1):129–145.
Elicitator – How To Use, Information Technology
Johnson, S., Low-Choy, S. and Mengersen, K. (in press, 2011) “Integrating Bayesian networks and Geographic information systems”, Integrated Environ Assess and Mgmt.
Elicitator for Populating CPTs in BNs
Low Choy, S., Wilson, T. (2009) How do experts think about statistics? Hints for improving undergraduate and postgraduate training, IASE Internat Assoc for Stat Educ, Durbin South Africa, 2009, Satellite Conference Proceedings. http://www.stat.auckland.ac.nz/~iase/publications/sat09/4_3.pdf
Logical fallacies and Visualization of probabilities
Low Choy, S., O‟Leary, R. and Mengersen, K. (2009) “Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models”, Ecology, 90(1):265-277.
Framework for designing elicitation, with 5 examples.
Low Choy, S., Murray, J., James, A. and Mengersen, K. (to appear Nov 2011) “Elicitator: a user-friendly, interactive tool to support elicitation of expert knowledge in landscape ecology”, Chp3 in Expert Knowledge and Its Applications in Landscape Ecology, eds Perera, Drew, & Johnson, Springer, NY
Overview of elicitation, its purposes and uses. Introduction to Elicitator.
Low Choy, S., Murray, J., James, A. and Mengersen, K. (2010) “Indirect elicitation from ecological experts: from methods and software to habitat modelling and rock-wallabies” in The Oxford Handbook of Applied Bayesian Analysis, eds O‟Hagan, West, OUP: UK, p511-544.
Indirect elicitation – statistical methods & comparison esp. Elicitator.
Low Choy, S. (2011) “Eliciting numbers for Bayesian Networks”, slides for a Satellite Workshop for the Australian Bayesian Network Modelling Society, http://eprints.qut.edu.au/.
These slides.
biosecurity built on science
Eliciting numbers for Bayesian Networks
Australian Bayesian Network Modelling Society Workshop 22 Nov 2011
Samantha Low Choy (CRCNPB/QUT Maths)
Complements presentations by Marissa MacBride (ACERA)
2
biosecurity built on science
Course Outline
Understanding Probabilities
Credible probabilities
Eliciting one probability
Eliciting a CPT
•Defining probabilities
•Logical fallacies
•Ambiguity
•Cognitive biases
•Calibration
•Structured design of elicitation
•Validation
•4-step elicitation method
•Outside-in (Elicitator)
•Smorgasbord
•CPT calculator
•HUGIN Table Generator
•Elicitator
3
biosecurity built on science
Understanding probabilities when constructing Bayesian belief networks
“Constructing” the conceptual model
- can better reflect expert knowledge by applying precise statistical interpretation & implications of links
- can avoid redundant links or missing latent links caused by not understanding conditional probabilities
“Populating” a CPT
- can be done in a transparent, repeatable way by harnessing structured elicitation techniques
- can incorporate uncertainty by using statistical methods for encoding probabilities
- can be done more efficiently using indirect elicitation to reduce time to populate large or
complex CPTs
- can be done more accurately by adopting techniques to train and/or calibrate experts
4
biosecurity built on science
Understanding Probability
Active Thinking & Learning
Quiz
Theory
•Quiz
•Theory
•Learnings for Elicitation
•Ellen & the Parrot
•Marjory & the Wallaby
•Deer, Irvine
•Zak & the Crocodile
•Philosophies
•Defining probabilities
•Alternative views
•Linguistic ambiguity
•Logical fallacies
5
biosecurity built on science
AN ELICITATION EXPERIMENT How are these questions interpreted?
We want to learn how these questions are interpreted, and how this affects the answers and elicitation experience. This depends on:
• the wording, format and context of questions
• understanding & experience with probability + field
-Joint research with Therese Wilson, QUT MAC
-Previously tested on some ecologists & CRCNPB PhDs
-Refocussed to BNs by piloting on PhD students Jegar Pitchforth & Charisse Farr, Airports of the Future
6
biosecurity built on science
ELICITATION Quiz, Theory & Learnings
Active learning of mathematics means: getting stuck, which means understanding what you do and do not easily recall.
We will exemplify the Bayesian learning cycle:
1. explicitly acknowledge the current state of knowledge
2. examines how this is impacted by new information
7
biosecurity built on science
Question 1: Ellen and the Parrot
8
biosecurity built on science
Ellen and the parrot Ellen, an expert on Australian native bird species,
states that the probability of finding an Eclectus parrot at this time of year is 0.1% (1 in a 1000).
Q1a. How would you explain what is meant by this to a first year ecology student?
Q1b. How would your explanation change if you needed to look for the parrot in a region one half the size?
9
biosecurity built on science
Ellen and the parrot Ellen, an expert on Australian native bird species,
states that the probability of finding an Eclectus parrot at this time of year is 0.1% (1 in a 1000).
Q1c. If you knew that the expert had only just obtained their degree, how would this affect your interpretation of their response?
Q1d. If two people see an Eclectus parrot in the same week does this make the expert a liar? Why or why not?
10
biosecurity built on science
DEFINING PROBABILITIES
Do not just focus on what you want to ask, but also how it will be interpreted by the targeted set of experts.
Linguistic ambiguity affects questionnaires of all pedigrees. Common if the designer focuses on the content rather than the testing & validation.
Pilot, pilot, pilot!
11
biosecurity built on science
Defining probabilities Classically defined as a frequency
What is n, the baseline (denominator)?
What is x, the count (numerator)?
- Inclusions and exclusions
- Observation changes reality? (eg destructive)
- Missing values vs zeros
Units & Extent
- Space and time
- Other forms of dis/aggregation, eg ecological
12
xp
n
biosecurity built on science
Counting
What is one thing?
Years free from pests
Individual farms that are infested
When a pest was present, we detected it
Defining the Event
biosecurity built on science
Counting
What is, what is not
Years free from pests vs infested
Individual farms that are infested vs clean
When a pest was present, we detected it or not Defining the Complement
biosecurity built on science
Probability: Defining the population Putting numbers in context
Probability that any year was free from pests during 1900-2010.
Probability that an individual farm is infested in the district.
Probability of detecting a pest when it is there. Defining the Population
biosecurity built on science
Conditional Probability
Counting with Constraints
Area searched, infested =10 ha
Area searched, pest-free =20 ha
Area not searched =70 ha
What is the right baseline? Total area =
S = Area searched =
%area searched, Pr(S) =
%area searched & infested
Pr(S and I) =
%searched area infested, Pr(I|S) =
CountProbability
Baseline count
biosecurity built on science
Conditional Probability
Counting with Constraints
Area searched, infested =10 ha
Area searched, pest-free =20 ha
Area not searched =70 ha
What is the right baseline?
Area =10+20+70=100ha
S = Area searched =
10+20=30ha
%area searched, Pr(S) =
30/100=30%
%area searched & infested
Pr(S and I) =10/100=10%
%searched area infested, Pr(I|S) = 10/30=33%
CountProbability
Baseline count
biosecurity built on science
Question 2: Marjory and the Wallaby
18
biosecurity built on science
Marjory is an expert on identifying rock wallabies. She is claimed to be 90% reliable at correctly identifying a rock wallaby when it really is one.
For each of the following statements tell me if you think it is true or false.
a) The probability that a rock wallaby is present is 90%.
b) If we could catch 10 of the rock wallabies identified by Marjory, then odds are that 1 of these would not be a rock wallaby.
19
Marjory and the wallaby
biosecurity built on science
Marjory is an expert on identifying rock wallabies. She is claimed to be 90% reliable at correctly identifying a rock wallaby when it really is one.
For each of the following statements tell me if you think it is true or false.
a) 90 times out of a 100 Marjory will identify a rock wallaby correctly.
b) There is a 9 in 10 chance that Marjory will see the rock wallaby if it‟s there.
c) The odds are 1:9 that Marjory has not correctly identified a rock wallaby.
20
Marjory and the wallaby
biosecurity built on science
INTERPRETING PROBABILITIES
Interpretation depends on Philosophy,
Which may depend on the expert‟s discipline and training.
Interpretation is guided by the order of presentation.
21
biosecurity built on science
Interpreting probability Alternative Philosophies
Frequency
- Observed
Thought experiment
- Frequentist – long run of exp‟t
- Imagine, eg Schroedinger‟s cat
Parallel universe
- Useful for unique x (eg ecosystem), eg Physicist
Betting
- Odds, relies on fiduciary instinct (used by Fisher, Good, Savage)
Belief
- Degree of belief
Score (Sureness, %Right)
- Like a cross-experiment measure of performance
Be aware of multiple interpretations of probability
Discern which one(s) you and your elicitee use/prefer.
Be flexible
Consistency can be convenient (esp. for documentation, repeatability) and is feasible in disciplines where training in probability is consistent (e.g. physical sciences vs biological sciences)
22
biosecurity built on science
Interpreting probability Linguistic ambiguity
90 times out of a 100 Marjory will identify a rock wallaby correctly.
- What is the Baseline? the attempted identifications?
the rock wallabies present?
- Discrimination: ID a rock wallaby vs a seal, or another wallaby?
- Frequencies are easier than fractions.
If we could catch 10 of the rock wallabies identified by Marjory, then odds are that 1 of these would not be a rock wallaby.
- More verbose but more precise
- Separates the condition into a separate clause.
- Clarifies the baseline n.
- “odds” aren‟t everyone‟s cup of tea
Linguistic ambiguity - Burgman
Aleotory (unknowable random variation) vs Epistemic (knowable structural variation) uncertainty – O‟Hagan
Write out the words exactly as you will ask them: makes the protocol explicit and therefore transparent and repeatable.
Allow some flexibility in follow-up questions.
Pilot!
23
biosecurity built on science
QUESTION 3 Deer, Irvine
24
biosecurity built on science
Deer, Irvine
Irvine et al (2009, Journal of Applied Ecology) assess some managers‟ knowledge of habitat use by red deer Cervus elaphus in the uplands of Scotland.
They asked managers what factors they thought impacted distribution of deer, recording interviews in transcripts. This table shows the number of times managers mentioned whether shelter or other factors were believed to impact the distribution of hinds (year-round) or stags in winter.
25
Factor
Total Shelter
Other
factors
Impact on
spatial
distribution
of deer
Hinds 19 23 42
Stags in
winter
25 33 58
Total 44 56 100
Consider the factors mentioned by managers in the transcripts.
Q3a) What‟s the probability that shelter was mentioned?
Write this down in the form of a fraction (eg 51 out of every 100 children born last year were boys).
Q3b) What‟s the probability that shelter for stags in winter was mentioned as an important factor?
biosecurity built on science
Deer, Irvine
Irvine et al (2009, Journal of Applied Ecology) assess some managers‟ knowledge of habitat use by red deer Cervus elaphus in the uplands of Scotland.
They asked managers what factors they thought impacted distribution of deer, recording interviews in transcripts. This table shows the number of times managers mentioned whether shelter or other factors were believed to impact the distribution of hinds (year-round) or stags in winter.
26
Factor
Total Shelter
Other
factors
Impact on
spatial
distribution
of deer
Hinds 19 23 42
Stags in
winter
25 33 58
Total 44 56 100
Q3c) What‟s the chance that a mention related to shelter for hinds? Q3d) Consider managers‟ opinions on Stags in winter. How likely is it that managers believed shelter impacted on their distribution? Q3e) What‟s the chance that managers mention other factors (forage, comfort or disturbance) in the context of stags in winter?
biosecurity built on science
DIFFERENT VIEWS OF PROBABILITIES
Visual aids to thinking about the problem
-Logic tree
-Matrix
-Venn diagram
-Mathematical
27
biosecurity built on science
Visual thinking
Logic tree: % vs counts
M: Of these,
how many
match the
evidence?
G: Is the
suspect
guilty?
N-1
Innocent
1
Guilty
1 00%
Match
0%
No match
3 %
Match
97%
No match
N
suspects
M: Of these,
how many
match the
evidence?
G: Is the
suspect
guilty?
100
Innocent
1
Guilty
1
Match
0
No match
3
Match
97
No match
101
suspects
Probability of
“M given G”
Pr(M|G)=100%
Pr(not M|G)=0%
Pr(M|not G)=3%
Pr(not M|not G)=97%
Guilty is a given
Not Guilty is a given
28
biosecurity built on science
Logic tree
Group by: guilt vs match
M: Of these,
how many
match the
evidence?
G: Is the
suspect
guilty?
100
Innocent
1
Guilty
1
Match
0
No match
3
Match
97
No match
101
suspects
G: Of these,
how many
are guilty?
M: How
many
match the
evidence?
97
No Match
4
Match
1
Guilty
3
Innocent
0
Guilty
97
Innocent
101
suspects
29
biosecurity built on science
Visual thinking
Condition on Match
Match No Match
Guilty 1
0
1
Not Guilty
3 97 100
4 97 101
M: Of these,
how many
match the
evidence?
G: Is the
suspect
guilty?
97
No Match
4
Match
1
Guilty
3
Innocent
0
Guilty
97
Innocent
101
suspects
Column percentages:
Pr(not Guilty | Match) = 3/4 = 75%
Pr(not Guilty | no Match) = ?
30
biosecurity built on science
Visual thinking
Condition on Guilt
Match No Match
Guilty 1
0
1
Not Guilty
3 97 100
4 97 101
M: Of these,
how many
match the
evidence?
G: Is the
suspect
guilty?
97
No Match
4
Match
1
Guilty
3
Innocent
0
Guilty
97
Innocent
101
suspects
Row percentage:
Pr(Match | Not Guilty) = ?
Pr(Match | Guilty) = ?
31
biosecurity built on science
Visual Thinking
Venn Diagram Innocent Guilty
Marked
Only 1 crocodile
Is Guilty,
100 crocodile suspects are innocent;
3 of these match the evidence
Pr(M|not G) = 3/100 = 3%
and it also Matches
the evidence.
Pr(M|G)=100%
32
biosecurity built on science
Exercise Visual Aids
Usual use Future use
Verbal description
Logic tree
Table (error matrix)
Venn diagram
Equations
Other
33
biosecurity built on science
QUESTION 4 Zak and the Crocodile
34
biosecurity built on science
A topical problem Zak tells a reporter that a crocodile bit his mate:
it was large, male and had a distinctive pattern along his back. Scientists tell us these characteristics co-occur on about 3 in every 100 crocs.
Someone finds a croc in the same region that fits this description.
Assume it‟s ok to kill & examine the croc if chances that it is guilty are better than even.
Q4a. What‟s your gut feeling: Is the croc innocent or guilty?
35
biosecurity built on science
A topical problem Zak tells a reporter that a crocodile bit his mate:
it was large, male and had a distinctive pattern along his back. Scientists tell us these characteristics co-occur on about 3 in every 100 crocs.
Someone finds a croc in the same region that fits this description.
Q4b. Larry says there is a 3% chance that the croc would match the evidence if he were innocent, thus there is a 97% chance that he‟s guilty.
Do you agree? Explain.
36
biosecurity built on science
A topical problem Zak tells a reporter that a crocodile bit his mate: it was
large, male and had a distinctive pattern along his back. Scientists tell us these characteristics co-occur on about 3 in every 100 crocs.
Someone finds a croc in the same region that fits this description.
Q4b. Larry says there is a 3% chance that the croc would match the evidence if he were innocent, thus there is a 97% chance that he‟s guilty.
Do you agree? Explain.
Q4c. Renata says there are about 10,000 crocs in the region. Hence these matches would occur on about 300 of these. Therefore there is a 1 in 300 chance that this croc is the guilty one, which is insufficient evidence of guilt.
Do you agree? Explain why or why not.
37
biosecurity built on science
A topical problem Zak tells a reporter that a crocodile bit his mate:
it was large, male and had a distinctive pattern along his back. Scientists tell us these characteristics co-occur on about 3 in every 100 crocs.
Someone finds a croc in the same region that fits this description.
Q4d. If you don‟t agree with Larry or Renata, how would you work out whether it is the offending crocodile?
38
biosecurity built on science
Reversing the Conditions via
BAYES’ THEOREM
39
biosecurity built on science
Probabilistic Thinking
Mathematical Events of interest
- M = crocodile matches the evidence, G = guilty
- M = DNA match, G = guilty of crime
- M = diagnostic test +ve, G = have condition
- M = sp. detected, G = sp. present
Core skills - Recognize we need Pr(G|M)
- Acknowledge we know Pr(M|G) = 1, Pr(M|not G) = 0.03
- We need Bayes Theorem to invert the logic:
Pr( | )Pr( ) Pr( | )Pr( )Pr( | )
Pr( ) Pr( | )Pr( ) Pr( | )Pr( )
M G G M G GG M
M M G G M G G
40
biosecurity built on science
Probabilistic Thinking
A word about equations An equation is
- Mathematical “shorthand”
- Pr(G|M)
- Probability of guilt (of any crocodile like this one), knowing that (given) it matches the evidence”
An equation can be
- Intimidating ... at first
- Incomplete ... without definitions
- Informative ... with definitions
41
biosecurity built on science
Bayes Theorem by logic
M: Of these,
how many
match the
evidence?
G: Is the
suspect
guilty?
100
Innocent
1
Guilty
1
Match
0
No match
3
Match
97
No match
101
suspects
G: Of these,
how many
are guilty?
M: How
many
match the
evidence?
97
No Match
4
Match
1
Guilty
3
Innocent
0
Guilty
97
Innocent
101
suspects
Pr(M|not G) = 3/100 Pr(not G|M) = 3/4
Pr(not G|M) = 3/100 100/101 4/101 = 3/4
42
biosecurity built on science
Bayes Theorem by table
Match No Match
Guilty 1
0
1
Not Guilty
3 97 100
4 97 101
The law of total (baseline) probability says Pr(total) = sum of Pr(parts)
assuming the parts don’t overlap, and together describe the total.
Here this baseline probability is Pr(M) = Pr(M,G) + Pr(M,not G).
So Pr(M) = Pr(M,G) + Pr(M,not G) = 1/1 x 1/101 + 3/101 = 4/101.
The definition of a conditional probability changes the baseline so
Pr(M|not G) = Pr(M,not G) / Pr(not G) = 3/101 / 100/101 = 3/100
Invert logic: Want Pr(not G|M)=3/4
If you can get Pr(M,not G)
Pr(not G|M) =
Else Using Bayes’ Theorem,
Pr(not G|M)
= Pr(M|not G) Pr(not G) / Pr(M)
= 3/100 x 100/101 4/101
= 3/4
Must know entire table to invert!
43
biosecurity built on science
LOGICAL FALLACIES
Conjunction fallacy
Inversion fallacy
Neglecting base rates
Prosecutor‟s fallacy
Defendant‟s fallacy
44
biosecurity built on science
Neglecting base rates
This problem occurs when you confuse a conditional probability with a joint probability.
Eg you could mistakenly say that Pr(G|M) = Pr(G and M) % marked crocs that are guilty is the same as the % crocs that match the evidence and are also guilty. but we know that Pr(G|M) = Pr(G and M) / Pr(M) which means acknowledging the base rate of matching the evidence amongst suspects! in other words you ignore the conditioning part.
Similar to the inversion fallacy,
This is a misunderstanding of how to include the base rates Pr(G) and Pr(M) Villejoubert+2002
47
biosecurity built on science
The defendant’s fallacy Overestimate probability of
innocence in favour of the defendant
Dilute evidence on matching, despite rarity in population at large Thompson+2002
- Choose very large reference region, eg 40,000 crocs in NQ
- Expect 1,200 to match
- So 1 in 1,200 chance of guilt given markings
Need appropriate #suspect crocodiles
- Eg N=101 gives Pr(G|M)=25% or N=11 gives Pr(G|M)=77%
M: Of these,
how many
match the
evidence?
G: Is the
suspect
guilty?
N-1
Innocent
1
Guilty
100%
Match
0%
No match
3%
Match
97%
No match
N
suspects
49
biosecurity built on science
Getting the baseline wrong
The Defendant‟s fallacy is a special case
- Diluting the number of suspects
More generally,
- Baseline population has too many irrelevant cases 40,000 crocs
naughty noughts, Austin+Meyers 96
50
biosecurity built on science
How the baseline affects guilt given marking
0 200 400 600 800 1000
0.0
0.2
0.4
0.6
0.8
1.0
Number of suspects, N
Pr(
G|M
)
Pr(G|M) = 50% for N=35
Pr(G|M) = 25% for N=101
Pr(G|M) = 3.2% for N=1001
51
biosecurity built on science
LOGICAL THINKING How we’d like the expert to think
52
biosecurity built on science
Probabilistic Thinking
via Equations N suspect crocodiles
Regardless of evidence
- Pr(G) = 1/N, Pr(not G) = (N-1)/N
“these characteristics match 3% of crocs [in the general population]”
- Interpret as Pr(M)=3%
“these characteristics match 3% of crocs [in the general population excluding man-eating crocodiles]”
- Or interpret as Pr(M|not G) = 3%
53
biosecurity built on science
Probabilistic Thinking
via Equations “these characteristics match 3% of crocs [in the general
population]”
- Interpret as Pr(M)=3%
Hence
So if N= 101, Pr(G|M) = 100/306 =33% but if N=1001, Pr(G|M) = 100/3006= 3.3%
11Pr( | ) Pr( ) 1001Pr( | )3Pr( ) 3( 1)100
M G G NG MM N
54
biosecurity built on science
Probabilistic Thinking
via Equations “these characteristics match 3% of crocs [in the general
population excluding man-eating crocodiles]
- Interpret as Pr(M|not G) = 3%
Pr(M) = Pr(M|G)Pr(G) + Pr(M|not G)Pr(not G) = 1x1/N + .03(N-1)/N
Hence p = Pr(G|M) = 1/(1+ .03(N-1))
So if N= 101, p = 1/(1+3) =25% but if N=1001, p = 1/(1+30)= 3.2%
55
biosecurity built on science
Learnings In this case, for both interpretations, we got similar
answers, and so would conclude the crocodile is likely guilty if N 100.
As the number of suspects decreases, the calculated probability of guilt given matching changes more and more.
But in general, answers may not be similar, so conclusions may depend heavily on the interpretation
Determine hidden assumptions
Elicit the baseline separately
If necessary, elicit the probability of the event conditioning on different hypothetical baselines
56
biosecurity built on science
Learnings Challenge of science is often to operate under
incomplete information; post-normal science Functowitz
- Elicitation from text often requires interpretation of ambiguous statements.
- Elicitation from groups in a workshop situation can be difficult to revisit.
Paraphrase to make explicit any underlying assumptions required to fully define the number.
Ensure authorship (not anonymity) and ownership by group/individual is clear.
Ideal to confirm interpretation or fill gaps via follow-up with authors.
57
biosecurity built on science
LOGICAL THINKING How we help the expert to be logical
Bayes Theorem: Pr(G|M) = Pr(M|G) Pr(G) / Pr(M)
Bayesian cycle of learning can be thought of as a model for rational thought. Dawid
Bayesian statistical modelling and cycle of learning: Pr(model|data) = Pr(data|model) Pr(model) / Pr(data)
But do ordinary (rational) people really think that way? El Gamal; Luc Bovens & Stephan Hartmann 2003
Elicitation need not expect, but can guide, logical thinking.
58
biosecurity built on science
LOGICAL FACILITATION Helping them get their logic right
Elicitation is not a test;
it is a collaboration.
We want to find out what the experts know, and we hope they want us to understand what they are trying to say.
Design elicitation so that the elicitor focuses on what the expert knows; not the same as asking for “the answer”.
59
biosecurity built on science
Avoid conjunction fallacy Does the expert view additional conditions as
affecting the probability or their sureness? - Knowing more about crocodile makes it easier to judge their
guilt, but does not make it more likely that they are guilty.
- Eg Pr(marking) Pr(marking, male, big) Pr(marking| guilty) Pr(marking, male, big | guilty)
- NB marginal probabilities include all possibilities for other unmentioned conditions. Here the first group of marked crocs includes females (big or small) and small (male or female).
Clarify hidden aggregation for marginal probabilities.
Elicit joint probabilities and conditional probabilities.
Show the expert tables or graphs of probabilities
- to help comparison (across conditions),
- ensure coherence (summation across margins).
60
biosecurity built on science
Avoid inversion fallacy Elicitation is NOT a test, it’s a collaboration!
Minimize the need for “mental gymnastics” by the expert, even if this requires more thought by the elicitor.
Discern which direction of conditioning is more natural for expert(s), and frame questions from that perspective.
Separate the condition from the event.
Reflect back what the expert has said, both verbally and visually.
Ask the expert to paraphrase their answer to help you confirm the direction of reasoning.
61
biosecurity built on science
Avoid baseline issues Get the baseline right
Choose units that are meaningful to the expert, eg half hectare Low Choy et al 2005, 2009
Use whole numbers where possible, eg 10 out of 1000, rather than fractions Kynn 2008
Give numerator and denominator separately (separate condition from event) Girotto+2001
Consider important sub-populations separately, during elicitation and modelling Murray et al, 2010
Explain how small or large probabilities derived Kynn 2008
62
biosecurity built on science
Avoid taking sides for Prosecution or Defence
Account for motivations by different stakeholders
Assess motivational bias
- Elicit expert backgrounds, positions & employers.
- From their training and history, determine their school of thought (academic peerage).
- Can request a statement re: conflict of interest (eg funding by agencies with motivational bias).
Reveal implicit assumptions; esp, explain how small or large probabilities derived Kynn 2008
Separate elicitation of scientific knowledge vs political decisions Low Choy et al 2005,2008.
63
biosecurity built on science
Exercise Which fallacies are you prone to?
Previously In the Future
Linguistic uncertainty
Conjunction fallacy
Inversion fallacy
Prosecutor‟s fallacy
Defendant‟s fallacy
Baseline misrepresentation
Other
64
biosecurity built on science
Elicitation technique helps to address Am
big
uit
y
Co
heren
ce
Co
nju
ncti
on
In
versio
n
Po
liti
cs
Baselin
e
Clarify hidden aggregation for marginal probabilities.
Elicit joint probabilities and conditional probabilities.
Show tables or graphs to compare probabilities.
Separate statement of condition, from event.
Show tree or flow diagrams to show logical sequence
Seek/reflect a paraphrase of statement
Choose the right units; seek numbers not fractions
Explain basis for small or large probabilities
Determine motivational perspectives
Model decomposition
65
In summary ...
biosecurity built on science
COGNITIVE BIASES A summary
Pioneered by Kahnemann & Tversky in the 1970s, Recently reviewed by Kynn (2008, JRSSA) and O‟Hagan et al (2006, chapters 1-3).
Cognitive biases: how people (not nec. expert) can get it wrong
Miscalibration: over/under-confidence; over/under-estimates
Group dynamics overlay individual cognitive, motivational &
numerical biases (Plous 1993)
66
biosecurity built on science
Bias
Elicitation is potentially subject to several sources of uncertainty, including biases, being conscious and subconscious discrepancies between the subject‟s responses and an accurate description of his underlying knowledge Spetzler & Staël von Holstein 1975
67
biosecurity built on science
Bias
Experts can be conditioned to be aware of common biases so that they are more easily avoided. ... displacement bias when experts over- or underestimate ... variability bias or conservatism where typically experts underestimate the variability in the quantity of interest ... motivational biases due to the expert‟s lack of neutrality Spetzler & Staël von Holstein 1975; Tversky & Kahneman 1981
68
biosecurity built on science
Biases & Bandaids Anchoring bias:
minimize via order of questions
Garthwaite & Dickey 1985,
Phillips & Wisbey 1993
Representativeness bias (SD vs SEM): structure to avoid tacit conditioning
Spetzler & Stael von Holstein 1975
Implicit conditioning: list reasons, esp. very
high/low values
Siu & Kelly 1998, Kynn 2008
Coherence biases: automatically check
for logical consistency
Kadane & Wolfson 1998
Feedback helps experts maintain self-
consistency
Kynn 2006, Low-Choy et al 2009b,2011
Comparing encodings from using different elicitation methods
Gavasakar 1988
Questioning multiple experts can be
representative, or outperformed by best!
Clemen & Winkler 1999
List sources of relevant expertise
Tversky & Kahneman 1973
„what is being elicited is expert, not perfect,
opinions, and thus they should not be
adjusted‟
Kadane & Wolfson 1998
69
biosecurity built on science
A final thought ... On eliciting the current state of knowledge
For elicitation, is there a right or wrong answer? “No amount of experimentation can ever prove me right;
a single experiment can prove me wrong.” Albert Einstein
If a gold standard exists, then why consult experts? What do they contribute beyond measurement? (Important for enlisting experts!)
Semper disco I always learn
Some (but not enormous) pressure on experts to get it right.
Be open to their considered judgment, with the information at hand.
Accurately capture the current state of knowledge, which will change!
Understanding is a two-way street Eleanor Roosevelt
Accurately reflect relevant expert knowledge
Ensure the target of elicitation is appropriate, including an understanding of context
Aim for effective questions, questioning and collaboration
Understanding requires more work when experts are new to conceptualizing their knowledge or to quantification
70
Recommended