#1
Environmental Management:Risk Assessment, Multi-Criteria Decision
Analysis, and Adaptive Management Techniques for Addressing Model Uncertainty and
Reliability
Igor Linkov & Pat DelimanUS Army Engineer Research and Development Center
Environmental [email protected], Phone: 617-233-9869
[email protected], Phone: 601-634-3623
#2
Adaptive Risk-Based Planning
Ad hoc Process· Include / Exclude?· Detailed / Vague?· Certain / Uncertain?· Consensus / Fragmented?· Rigid / Unstructed?
Scoping
Remedial Investigation
Feasibility Study
Remedy Selection & Record of Decision
Remedial Design/ Remedial Action
Five-Year Review
Record of Decision Modification
Stakeholder's Opinions and
Values
Risk Analysis
Cost
Modelling/ Monitoring
Political / Litigation Pressure
Decision
Decision Analysis Framework
Scoping
Remedial Investigation
Feasibility StudyMonitor System
Response
Data Interpretation, Model Analysis
and Improvement
Implement Management
Strategy
Model Prediction and Management Plan Improvement
Stakeholder's Opinions and
Values
· Agency-Relevant· Currently Available Software· Variety of Structuring Techniques· Iteration / Reflection Encouraged· Facilities Stakeholder Input
Risk AnalysisCost
Modelling/ Monitoring
Monitoring Results
Modelling Results
a) Ad hoc Decision Process
b) Adaptive Decision Analysis Framework
#3
• The Government Performance and Results Act (GPRA) – “provide for the establishment of strategic planning and
performance measurement in the Federal Government” (OMB, 1993).
– embodied a push for better planning, greater accountability, and straightforward performance evaluation in government.
• OMB Program Assessment Rating Tool (PART) – rates the performance of a program through a series of yes/no
questions– Scores on four primary areas: program purpose & design,
strategic planning, management, and results & accountability. – performance metrics used by the program are essential to
PART.
Agencies need to relate response to the mission goals and track the progress and performance
Risk-based Planning: Top-down Drivers
#4
• For stakeholders, the root issue is: fear of becoming a victim to (uncompensated) loss– Layperson: Risk = Hazard x Perception– Expert: Risk = Hazard x Exposure x Consequence
• Core concerns tend to be: trust, control, process, information and timing
Local communities need to understand actions by the Agenciesand like to see their values accounted for
Risk-based Planning: Bottom-up Drivers
#5
• What are the flood and storm threats to coastal Louisiana/Mississippi?
• What do we have to lose and how vulnerable are we?
• What should be our planning timeframe?
• What can be done to reduce risks for our planning timeframe?
• How is the public, agencies, and others involved?
• For taking action at varying scales, what is the cost and risk reduction?
• For taking action at varying scales, what are the adverse impacts to significant resources?
• How do you decide what actions to take?
• How much information is necessary to make decisions?
Coastal Louisiana Restoration Planning:What Questions are We Trying to Answer?
After E. Russo
Risk and/or Uncertainty elements are present almost in every question
#6
Information and Planning/Decision CyclesInformation gathering and
decision-making are two separate cycles in environmental
management
Modeling/Software/GIS…Technology-based Fix in Information Age
Integration – Need for Revolutionary
Changes
After Roman, 1996
#7
Main Points
• Risks and benefits associated with alternative management strategies are difficult to quantify.
• Model, Parameters and Scenario uncertainty and variability associated with predicting efficiency of management options as well as stakeholder value judgment are important to consider
• Challenges of risk assessment and planning for situations with a limited knowledge base and high uncertainty and variability require coupling traditional risk assessment and planning with multi-criteria decision analysis (MCDA) to support regulatory decision making
#8
• Risk-based Planning: Top-down and Bottom-up Drivers
• Risk and Uncertainty– Traditional Way of Dealing with Uncertainty – Need for Formal Decision Analysis
• MCDA -Summary
• Example:– MCDA Use to Select Performance Metrics for Oil Spill
Response Planning– RA/MCDA Application for Sediment Management
• Conclusion
• References
Presentation - Overview
#9
AD HOC Process
Quantitative? Qualitative?
Current Decision-Making Processes
Decision-Maker(s)
Include/Exclude?•Detailed/Vague?
•Certain/Uncertain?•Consensus/Fragmented?
• Iterative?• Rigid/unstructured?
Risk Analysis
Modeling / Monitoring
Stakeholders’ Opinion
Cost or BenefitsTools
Challenge: Multiple & Uncertain Criteria
#10
Challenges to Complex Decision-making
• “Humans are quite bad at making complex, unaided decisions” (Slovic et al., 1977).
• Individuals respond to complex challenges by using intuition and/or personal experience to find the easiest solution.
• At best, groups can do about as well as a well-informed individuals if the group has some natural systems thinkers within it.
• Groups can devolve into entrenched positions resistant to compromise
• “There is a temptation to think that honesty and common sense will suffice” (IWR-Drought Study p.vi)
#11
• Model Uncertainty– Differences in model structure resulting from:
model objectives computational capabilities data availability knowledge and technical expertise of the
group– Can be addressed by
considering alternative model structures weighting and combining models Eliciting expert judgment
Problem: Model UncertaintyLinear Model
y = 3x - 0.6667
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4
X
Y
Polynomial Modely = 2x2 - 5x + 6
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4
X
Y
Mechanistic models for environmental risk assessment are very uncertain and expert judgment is required
#12
• Parameter Uncertainty– Uncertainty and variability in model
parameters resulting from data availability expert judgment empirical distributions
– Can be addressed by Probabilistic Simulations (Monte-
Carlo) Analytical techniques (uncertainty
propagation) Expert estimates
Problem: Parameter Uncertainty
Many parameters and factors important for risk assessment are not well known, reported ranges are large and often
unquantifiable
Mean+SDMean-SD
Mean+SEMean-SE
Mean
Oil and Grease in Sediment
Co
nc
en
tra
tio
n (
pp
m)
49
51
53
55
57
59
VAR1
VAR155.00050.00057.000
#13
Subjective Interpretation of the Problem at Hand
What is the relative influence of modeler perception on model predictions?
Problem: “Modeler/Scenario Uncertainty”
#14
Subjective Interpretation of the Problem at Hand
What is the relative influence of modeler perception on model predictions?
Problem: “Modeler/Scenario Uncertainty”
#15
Multi-Criteria Decision Analysis and Tools
• Multi-Criteria Decision Analysis (MCDA) methods:– Evolved as a response to the observed inability of people to
effectively analyze multiple streams of dissimilar information– Many different MCDA approaches based on different theoretical
foundations (or combinations)
• MCDA methods provide a means of integrating various inputs with stakeholder/technical expert values
• MCDA methods provide a means of communicating model/monitoring outputs for regulation, planning and stakeholder understanding
• Risk-based MCDA offers an approach for organizing and integrating varied types of information to perform rankings and to better inform decisions
#16
Risk Analysis
Modeling / Monitoring
Stakeholders’ Opinion
Cost
Decision Analytical Frameworks• Agency-relevant/Stakeholder-selected
• Currently available software•Variety of structuring techniques • Iteration/reflection encouraged
•Identify areas for discussion/compromise
Decision-Maker(s)
Sharing Data,Concepts and Opinions
Evolving Decision-Making Processes
Tool Integration
Decision Integration
#17
Simplified Decision Matrix
Plan Cost EcoHealth
Human Health
A 100 10 5
B 100 5 10
C 150 10 10
D 150 10 15
#18
Criteria 1 Criteria 2 Criteria 3 Criteria 4
Alt. 1 Monitoring Results
Stakeholder Preference
Economic Cost Non-monetary benefit
Alt. 2 Monitoring Results
Stakeholder Preference
Economic Cost Non-monetary benefit
Alt. 3 Monitoring Results
Stakeholder Preference
Economic Cost Non-monetary benefit
Alt. 4 Monitoring Results
Stakeholder Preference
Economic Cost Non-monetary benefit
How to interpret these results?
How to combine these criteria?H
ow
to
co
mp
are
th
ese
alt
ern
ativ
es?
Example Decision Matrix
#19
Decision Analysis Methods and Tools
Elements of Decision Process
Ad Hoc Decision-Making Comparative Risk Assessment Multi-Criteria Decision Analysis
Define problems Stakeholder input limited or non-existent. Therefore, stakeholder concerns may not be addressed by alternatives.
Stakeholder input collected after the problem is defined by decision-makers and experts. Problem definition is possibly refined based on stakeholder input.
Stakholder input incorporated at beginning of problem formulation stage. Often provides higher stakeholder agreement on problem definition. Thus, proposed solutions have a better chance at satisfying all stakeholders.
Generate alternatives Alternatives are chosen by decision-maker usually from pre-existing choices with some expert input.
Alternatives are generated through formal involvement of experts in more site-specific manner.
Alternatives are generated through involvement of all stakeholders including experts. Involvement of all stakeholders increases likelihood of novel alternative generation.
Formulate criteria by which to judge alternatives
Criteria by which to judge alternatives are often not explicitly considered and defined.
Criteria and subcriteria are often defined.
Criteria and subcriteria hierarchies are developed based on expert and stakeholder judgment.
Gather value judgments on relative importance of criteria
Non-quantitative criteria valuation weighted by decision maker
Quantitative criteria weights are sometimes formulated by the decision maker, but in a poorly justified manner.
Quantitative criteria weights are obtained from decision makers and stakeholders.
Rank/select final alternatives
Alternative often chosen based on implicit weights in an opaque manner.
Alternative chosen by aggregation of criteria scores through weight of evidence discussions or qualitative considerations.
Alternative chosen by systematic, well-defined algorithms using criteria scores and weights.
#20
Problems
Alternatives
Criteria
Weights
Synthesis
Decision
Decision Matrix
Evaluation
RA
MCDAFeedsRA
MCDARAFeedsMCDA
AdaptiveManagement
Linking RA, AM and MCDA
#21
Specify Problems& Opportunities
Inventory & ForecastConditions
FormulateAlternative Plans
Evaluate Effects ofAlternative Plans
CompareAlternative Plans
SelectRecommended Plan
Corps Planning
Problems
Alternatives
Criteria
Evaluation
Decision Matrix
Weights
Risk and DecisionAnalysis Framework
Synthesis
Decision
Decision Analysis ToolsMAUT
Risk Analysis ModelsWave/Storm SurgeInfrastructure ModelsEcosystem ModelsEconomic Models
Figure 2. RIDF
Scenario Analysis
Risk Informed Decision Framework: Restoration Planning for Coastal LA and MS
#22
Example 1: Performance Metrics for Oil Spills Response Planning*
• Framework for selecting metrics– Multiple stakeholders
Agencies (federal, state, local) Responsible parties Local residents NGOs (business, environmental, etc.)
– Integrate deliberation and science to link goals, objectives, metrics, and measures
– Compatible with existing planning, decision-making, and assessment processes
– Completed as part of preparedness planning*based on Linkov, Seager, Figueira, Tkachuk, Levchenko, Trevonnen (2007), funding provided by
NOAA through CRRC, UNH.
#23
Examples (oil spills response)
• Endpoint – Miles of shoreline impacted or cleaned vs. areas protected (e.g., by redirecting or
containing oil). – Number of fish, birds or other wildlife killed or injured (per unit search area). – Number of “appropriate” (not exotics) animals rehabilitated and released. – Degree of change to beaches and sandbars from clean-up actions. – Types of animals and vegetation present after spill cleanup.
• Process – Did getting required permits delay response action? – Rate of bird handling at rehabilitation center. – Time to deploy booming and double-booming in sensitive areas.
• Resource – Amount of oil containment boom deployed. – Number of volunteers deployed.– Number sandbags deployed.
#24
Challenges
• Challenges to defining “good” metrics– What is most relevant may be very difficult to measure.– Metrics may be indirect measures of what people really care about.– What is easy to measure may not be relevant to what people care
about.– There can be disagreements about thresholds to differentiate
“good” versus “bad.”– Accuracy and reliability of data recording is a challenge.– Paucity of baseline data.– Timing of measurement can affect assessment of performance.– Difficult to communicate to the public – or at least that is
managers’ perception.– Weightings and aggregation.
#25
Characteristics of Good Measures
– scientifically verifiable– cost-effective– easy to communicate to a wide audience– relevant to what people care about– decision or action relevant– credible– scalable over an appropriate time period and geographic
region – sensitive to change
#26
Oil Spill Response Metrics Taxonomy by Type of Information Measured
Economic Thermodynamic Environmental Ecological Human Health Socio-Political
Clean up costs.Property & eco-
system damage.
Ecosystem damages or lost services.
Lost marginal profits.Volunteer
opportunity costs.
Volume of oil spilled, recovered, des-troyed, or contained.
Slick area and thickness.
Mass of clean up wastes generated.
Volume cleaning agent deployed.
Chemical concen-tration & toxicity.
Habitat suitability, e.g., acres shellfish bed.
Length of oiled shoreline.
Degradation rates.Residual Risk
Wildlife deaths or populations, fecundity and recovery rates.
Biodiversity.Catch sizes.Plantings, seedings.Habitat Suitability
Threatened pop-ulation
Quality-adjusted-life-years (QALYS)
Disability-adjusted-life-years (DALYS)
Life-expectancyInjuries
Newspaper column inches, minutes TV coverage, web hits.
Volunteerism.Public meeting
attendance.Critical sites pro-
tectedHistoric sites pro-
tected
#27
Assessment Criteria
#28
Metric Assessment by Criteria
#29
Criteria Weight
#30
Rank Acceptability Analysis
#31
Pairwise Metrics Domination
#32
Sensitivity Analysis
COST Most Important
ENVIRONMENTAL Relevance Most Important
#33
Example 2: NY/NJ Harbor
#34
Issues• Harbor among most
polluted in U.S.
• >106 yd3 fail regional criteria for ocean disposal
• Existing disposal site closed 1 Sep. 97
• Proposed deepening
Example: NY/NJ Harbor
#35
Example: Decision Methodology
• Proof of Concept Study
• Objectives– Integrate comparative risk assessment results with cost and stakeholder
decision criteria– Use decision criteria/performance measures from published data and
proposed costs– Test decision tools, methodology and results
• Set contaminated sediment management options
• Set decision criteria/performance measures
• Software - Criterium DecisionPlus
• Stakeholder Values / Expert Surveys – USACE/EPA dredged material managers meetings (New Orleans 2004)– SRA/USACE/Contaminated Sediments Meeting (Palm Beach 2004)
#36
Conceptual Illustration of Disposal Alternatives
Landfill Upland CDF Nearshore CDF CAD Pit No-Action Island CDF
Water Line
In-place Sediment
Dredged Material
Effluent
Manufactured Liner
Dike Wall
Cap
Standard Landfill Waste
KEY:
In-place Soil
Kane Driscoll, S.B., W.T. Wickwire, J.J. Cura, D.J. Vorhees, C.L. Butler, D.W. Moore, T.S. Bridges. 2002. A comparative screening-level ecological and human health risk assessment for dredged material management alternatives in New York/New Jersey Harbor. International Journal of Human and Ecological Risk Assessment 8: 603-626.
Manufactured SoilCement Lock
#37
$ / Cubic Yard
Contaminated Sediment Management Decision
Impacted Area / Capacity
Cost Ecological Health
Human Health
Public Acceptance
# of complete ecological exposure pathways
Largest Ecological Hazard Quotient (HQ) calculated for
any one pathway
# of complete human
exposure pathways
Largest Cancer Risk calculated for any one pathway
Estimated Fish COC Concentration / Hazard Level
Decision Criteria: NY/NJ Harbor
Source: Kane Driscoll et al. (2002).
Source: NY/NJ Dredged Material Management Plan and Expert Opinion
#38
NY/NJ Harbor in Criterium DecisionPlus
Goal Criteria Sub-Criteria
#39
NY/NJ Harbor in Criterium DecisionPlus
Goal Criteria Alternatives
Hierarchy Rating Technique: Weights
Alternatives Rating Technique: SMART
with linear value functions
Sub-Criteria
#40
Criteria Levels for Each DM Alternative
Cost Public Acceptability
Ecological Risk Human Health Risk
DM Alternatives
($/CY) Impacted Area/Capacity (acres / MCY)
Ecological Exposure Pathways
Magnitude of Ecological HQ
Human Exposure Pathways
Magnitude of Maximum
Cancer Risk
Estimated Fish
COC / Risk Level
CAD 5-29 4400 23 680 18 2.8 E -5 28
Island CDF 25-35 980 38 2100 24 9.2 E -5 92
Near-shore CDF 15-25 6500 38 900 24 3.8 E -5 38
Upland CDF 20-25 6500 38 900 24 3.8 E -5 38
Landfill 29-70 0 0 0 21 3.2 E –4 0
No Action 0-5 0 41 5200 12 2.2 E –4 220
Cement-Lock 54-75 0 14 0.00002 25 2.0 E -5 0
Manufactured Soil 54-60 750 18 8.7 22 1.0 E –3 0
Blue Text: Most Acceptable ValueRed Text: Least Acceptable Value
#41
USACE/EPA DM Managers Meeting: NY/NJ Harbor Weighting FormAttribute Swung from
Worst to bestConsequence to compare Rank
(1-9)Rate(0-
100)
Benchmark: Worst case on everything
Impacted Area/Capacity of Facility = 6500 (acres/ 106 cubic yards) Magnitude of Ecological Hazard Quotient – Maximum Exposure = 5200 Number of Complete Ecological Exposure Pathways = 41 Number of Complete Human Exposure Pathways = 25 Magnitude of Maximum Cancer Probability (Non-barge worker) = 1* 10-3 Ratio of Estimated Concentration of COCs in Fish to Risk-Based Concentrations = 220 Cost = 54-75 $/CY
9 0
Impacted Area/Capacity of Facility
Change from 6500 (acres/ 106 cubic yards) to 0 (acres/ 106 cubic yards)
Magnitude of Ecological Hazard Quotient –Maximum Exposure
Change from 5200 to 0
Number of Complete Ecological Exposure Pathways
Change from 41 to 0
Number of Complete Human Exposure Pathways
Change from 25 to 12
Magnitude of Maximum Cancer Probability (Non-barge worker)
Change from 1* 10-3 to 0.028 * 10-3
Ratio of Estimated Concentration of COCs in Fish to Risk-Based Concentrations
Change from 220 to 0
Cost Change from (54-75 $/CY) to (0-5 $/CY)
#42
USACE/EPA Survey Results: Criteria Weights (%)
EPA USACE SRA
Public Acceptability 7.4 12.5 10.77
Ecological Health 35.6 27.1 32.45
Human Health 47.0 40.7 44.10
Cost 10.0 19.7 12.67
#43
Criteria Contributions to Decision ScoreUSACE weighting
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Cost
Maximum Cancer Probability (Non-Barge Worker)
Ecological Hazard Quotient
Est. COC Conc in Fish / Risk-based Conc
Complete Human Health Exposure Pathways
Complete Ecological Exposure Pathways
Ratio of Impacted Area to Facility Capacity
EPA weighting
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Cost
Maximum Cancer Probability (Non-Barge Worker)
Ecological Hazard Quotient
Est. COC Conc in Fish / Risk-based Conc
Complete Human Health Exposure Pathways
Complete Ecological Exposure Pathways
Ratio of Impacted Area to Facility Capacity
#44
Solution vs. MCDA Method:Does it Matter?
Software & Method
Alternatives (NY Case Study)
CAD Island CDF
Near-ShCDF
UplandCDF
Landfl Cemnt Lock
ManufSoil
NoAct
ExpertChoice,AHP
5 8 6 7 2 1 3 4
DecisionLab,PROMETHEE
3 8 5 6 2 1 4 7
CritDecPlus,SMART
2 7 4 5 1 3 6 8
#45
People:
Tools:
Process:
Environmental Assessment/Modeling (Risk/Ecological/Environmental Assessment and Simulation Models)
Decision Analysis (Group Decision Making Techniques/Decision Methodologies and Software)
Policy Decision Maker(s)
Scientists and Engineers
Stakeholders (Public, Business, Interest Groups)
Monitor System Response
Data Interpretation, Model Analysis and
Improvement
Implement Management
Strategy
Model Prediction and Management Plan Improvement
Determine Performance of
Alternatives on the Criteria
Identify Criteria to Compare Alternatives
Gather Value Judgments on Relative
Criteria Importance
Define Problem and Generate Alternatives
Risk Assessment, Adaptive Management and MCDAImplementation Framework
#46
Key Take-Away Points
Risk-Based MCDA offers planners:
• Reproducible and defensible management of complex multiple criteria
• A means to define and gauge what is important
• Balancing of expert opinion and stakeholder values
• Better responses, better reporting with opportunities to more clearly get it “out on the table”
#47
• MCDA workshop sites with posted lectures– http://www.risktrace.com/sediments http://www.risktrace.com/nato http://www.risk-trace.com/ports/index.php
• Papers– Yatsalo, B., Kiker, G., Kim, J., Bridges, T., Seager, T., Gardner, K., Satterstrom, K., Linkov, I.
2006. Application of Multi-Criteria Decision Analysis Tools for management of contaminated Sediments. Integrated Environmental Assessment and Management.
– Seager, T., Satterstrom, K., Linkov, I., Tuler, S., Kay, R. 2006. Typological Review of Environmental Performance Metrics (with Illustrative Examples for Oil Spill Response). Integrated Environmental Assessment and Management.
– Linkov, I., Satterstrom, K., Kiker, G., Bridges, T., Benjamin, S., Belluck, D. (2006). From Optimization to Adaptation: Shifting Paradigms in Environmental Management and Their Application to Remedial Decisions. Integrated Environmental Assessment & Management 2:92-98.
– Linkov, I., Satterstrom, K., Seager, T.P., Kiker, G., Bridges, T., D. Belluck, A. Meyer (2006). "Multi-Criteria Decision Analysis: Comprehensive Decision Analysis Tool for Risk Management of Contaminated Sediments". Risk Analysis 26:61-78.
– Linkov, I., Satterstrom, K., Kiker, Batchelor, C., G., Bridges, T.(2006). From Comparative Risk Assessment to Multi-Criteria Decision Analysis and Adaptive Management: Recent Developments and Applications. Environment International 32: 1072-1093.
References