Uncertainty, Variability and Weight of Evidence:

How Well Do We Know Environmental Risks?

Glenn Suter

U.S. Environmental Protection Agency

EFSA Scientific Conference, Milan, Italy

The views expressed in this presentation are those of the author and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency

What do decision makers & stakeholders need? • Qualitative information

• Does this chemical cause cancer?

• Is this ecosystem impaired?

• What is the cause of observed effects?

• Quantitative information • What concentration is protective?

• How much salmon habitat would be lost?

• How long will natural remediation require?

• Confidence in both

What is Confidence?

What is Scatter? Anything expressed as a numerical distribution • Variability

• natural variation in some quantity • Cannot be reduced

• Uncertainty • Incomplete or imprecise data • Can be reduced

• Degree of belief • Subsumes variability and uncertainty for subjectivists • If you were betting on the teratogenicity of a chemical in use,

What odds would you accept?

This is statistics—controversial but familiar

What is weight of evidence?

• Metaphor based on the scales of justice

• The influence evidence should have on an inference • The weight of a piece of evidence dropped in a pan

• A summary interpretation of multiple pieces of evidence • The weight of the body of evidence in one pan relative to the other

• An inferential approach that uses this concept • Often simply an expert narrative

• We are trying to make it more formal, transparent and defensible

How do we weigh evidence?

Assemble Evidence

Evaluate and Weight Evidence

Integrate and Weigh Body of

Evidence

Assembling evidence

Assemble Evidence

Screen

Design and Conduct Studies

Search Literature

Extract and Analyze

Categorize

Weight Evidence

Weigh Body of Evidence

Assembling evidence

Assemble Evidence

Screen

Design and Conduct Studies

Search Literature

Extract and Analyze

Categorize

Weight Evidence

Weigh Body of Evidence

Systematic Review thorough and unbiased

Assembling evidence

Assemble Evidence

Screen

Design and Conduct Studies

Search Literature

Extract and Analyze

Categorize

Weight Evidence

Weigh Body of Evidence

Systematic Review Complement existing evidence to create a coherent body of evidence

Assembling evidence

Assemble Evidence

Screen

Design and Conduct Studies

Search Literature

Extract and Analyze

Categorize

Weight Evidence

Weigh Body of Evidence

Systematic Review Complement existing Evidence to create a coherent body of evidence

Eliminate irrelevant and unacceptable studies

Assembling evidence

Assemble Evidence

Screen

Design and Conduct Studies

Search Literature

Extract and Analyze

Categorize

Weight Evidence

Weigh Body of Evidence

Systematic Review Complement existing Evidence to create a coherent body of evidence

Eliminate irrelevant and unacceptable studies Put the studies in bins

that may be treated as a type of evidence

Assembling evidence

Assemble Evidence

Screen

Design and Conduct Studies

Search Literature

Extract and Analyze

Categorize

Weight Evidence

Weigh Body of Evidence

Systematic Review Complement existing evidence to create a coherent body of evidence

Eliminate irrelevant and unacceptable studies Put the studies in bins

that may be treated as a type of evidence Extract the data,

reanalyze, and combine with other information to create evidence

Weighting evidence

Weight Evidence

Combine Weights

Weight Reliability

Weight Relevance

Weight Strength

Assemble Evidence

Weigh Body of Evidence

Evaluate with respect to 3 attributes and assign scores

Weighting evidence

Weight Evidence

Combine Weights

Weight Reliability

Weight Relevance

Weight Strength

Assemble Evidence

Weigh Body of Evidence

Evaluate with respect to 3 attributes and assign scores

Combine into weight of piece or type of evidence

Weighing bodies of evidence

Weigh Body of Evidence

Interpret Bodies of Evidence

Explain Ambiguities & Discrepancies

Integrate Pieces or Categories of Evidence

Weight Evidence

Assemble Evidence

Determine the overall weight of the body of evidence based on the weights of pieces, or types of evidence and consistency

Weighing bodies of evidence

Weigh Body of Evidence

Interpret Bodies of Evidence

Explain Ambiguities & Discrepancies

Integrate Pieces or Categories of Evidence

Weight Evidence

Assemble Evidence

Which, if any, hypothesis is supported by the weight of evidence and with how much weight?

Determine the overall weight of the body of evidence based on the weights of pieces, or types of evidence and consistency

Weighing bodies of evidence

Weigh Body of Evidence

Interpret Bodies of Evidence

Explain Ambiguities & Discrepancies

Integrate Pieces or Categories of Evidence

Weight Evidence

Assemble Evidence

Which, if any, hypothesis is supported by the weight of evidence and with how much weight?

Determine the overall weight of the body of evidence based on the weights of pieces, or types of evidence and consistency

Deal with cases in which the evidence does not support any hypothesis

Expressing qualitative results

• Example Conclusion The chemical is a vertebrate teratogen

• Example Weight High relevance

Moderate strength

Moderate reliability

High consistency

Linking qualitative to quantitative results

Summary of previous multicolored boxes

Weigh Evidence for the

Quality to be Quantified

Merge Quantitative

Evidence

Choose Best Quantitative

Evidence

Define Confidence in Quantitative

Evidence

Linking qualitative to quantitative results

Weighted mean or other meta-analysis

Summary of previous multicolored boxes

Weigh Evidence for the

Quality to be Quantified

Merge Quantitative

Evidence

Choose Best Quantitative

Evidence

Define Confidence in Quantitative

Evidence

Linking qualitative to quantitative results

Weighted mean or other meta-analysis

Weigh evidence to choose best source of the quantity

Summary of previous multicolored boxes

Weigh Evidence for the

Quality to be Quantified

Merge Quantitative

Evidence

Choose Best Quantitative

Evidence

Define Confidence in Quantitative

Evidence

Linking qualitative to quantitative results

Weighted mean or other meta-analysis

Weigh evidence to choose best source of the quantity

Integrate results

Summary of previous multicolored boxes

Weigh Evidence for the

Quality to be Quantified

Merge Quantitative

Evidence

Choose Best Quantitative

Evidence

Define Confidence in Quantitative

Evidence

Expressing quantitative results

• Quality Protection from deformities

• Number 22

• Units mg/L

• Scatter ± 3

• Weight Moderate

Each has supporting information and detail, but this is the basic information

(tip of the hat to Ravetz and Funtowicz’s NUSAP system)

Summary

• Formal weighing of evidence can • Provide greater utility of results

• Increase defensibility

• Increase transparency

• With flexibility and adaption to purpose,

• WoE need not be onerous

• In combination with conventional statistics

• WoE can convey confidence in results

The views expressed in this presentation are those of the author and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency