"Review of major types of uncertainty in fisheries modeling and how to deal with them"

Randall M. Peterman

School of Resource and Environmental Management (REM)

Simon Fraser University,Burnaby, British Columbia, Canada

National Ecosystem Modeling Workshop II, Annapolis, Maryland, 25-27 August 2009

Outline

• Five sources of uncertainty - Problems create - What scientists have done

• Adapting those approaches for ecosystem modelling

• Recommendations

Single-speciesstock assessments

General risk assessment methods

Uncertaintiesconsidered

My background

Scientific advice: including risk communication

Decision makers, stakeholders

Risk management

Multi-species ecosystemmodels Impressive!!

Risk management

Multi-species ecosystemmodels

Single-speciesstock assessments Uncertainties

considered

Risk management

Multi-species ecosystemmodels

2. Provide broad strategic advice

3. Provide specific tactical advice

Purposes of ecosystem models from NEMoW 1

1. Improve conceptual understanding

Uncertainties are pervasive ...

1. Natural variability

Sources of uncertainty

Uncertainties

2. Observation error (bias and imprecision)

Uncertainties

3. Structural complexity

Uncertainties

UncertaintiesResult:Parameteruncertainty

4. Outcomeuncertainty(deviation from target)

Result:Imperfect forecastsof system's dynamics

5. Inadequate communicationamong scientists,decision makers,and stakeholders

Uncertainties

Result:Poorlyinformed decisions

Result:Parameteruncertainty

Uncertainties

Biological risks

(ecosystems)

Economic risks

(industry)

Social risks

(coastalcommunities)

Magnitude of variable/event and probability of that magnitude occurring

Sensitivity analyses across:

1. Which components to include

3. Parameter values

2. Structural forms of relationships

4. Management objectives

5. Environmental conditions

• Focus: - Which parts most affect management decisions?- Which parts are highest priority for more data?

6. Management options

2008 Mutton snapperU.S. South Atlantic & Gulf of Mexico

SSB / SSBF30%

F / F30%

Overfished

2. Observation error

3. Unclear structure of fishery system

4. Outcome uncertainty

5. Inadequate communication

Sources of uncertaintyProblems

Resolution

UncertaintyProblems created by

not adequately accounting for that uncertainty

1. Natural variation in

space and time

• Poor estimates of model parameters and variables

• Inappropriate dynamics of model due to nonstationarity: - "Regime shifts" in productivity - "Phase shifts" in system structure

1. Simulate stochastically

2. Make parameters a function of age, size, density, ...

3. Include other components (static or dynamic) - Predators, prey, competitors - Bycatch/discards - Environmental variables...

What scientists have done to deal with ...

Uncertainty Problems created

2. Observation error (bias and imprecision in input data)

• Biased/imprecise estimates of model parameters - Incorrect functional form of error term

• Biased output indicators

• Wrong probability distributions

1. Assume % of total variance due to observation error

2. Conduct sensitivity analyses

3. Use hierarchical models that "pool" information to help "average out" annual observation error

- Jerome Fiechter et al. using hierarchical Bayesian models on NEMURO (NPZD-based)

Stocknumber Pink salmon

-0.5 0.0 0.5 1.0

40Separate single- stock analyses

Multi-stock, mixed-effects model

Alaska

B.C., Wash.

, change in salmon productivity, loge(R/S),

per oC increase in summer sea-surface temperature

Mueter et al. (2002a)

4. Separately estimate natural variation and observation error -- Errors-in-variables models

-- State-space models-- Kalman filter

Example 1: tracking nonstationary productivity parameter (Ricker value)

2. Observation error ... (continued)

0 10 20 30 40 50 60 70 80 90 100

Productivity parameter

Low High Decreasing

0 20 40 60 80 100

"True"Standard methodKalman filter

Simulation test

Productivity(Ricker parameter)

Peterman et al. (2000)

• Kalman filter with random-walk system equation was best across all types of nonstationarity

Example 2 of observation error and natural variation

Simplest possible model: spawner-recruit relationshipSu and Peterman (2009, in prep.)

- Used operating model to determine statistical properties of various parameter-estimation schemes:-- Bias-- Precision-- Coverage probabilities (accuracy of estimated

width of probability interval for a parameter)

2. Observation error ... (continued)

Operating model (simulator to test methods)

User-specified "true" underlying parameter values

("What if ...?")

Test performance of an estimator

Generate "observed data"from natural variation and observation error

Parameters estimated

("What if ...?")

Compare"true" and

estimated values

("What if ...?")

Compare"true" and

estimated values

200trials

("What if ...?")

Proportion of total variance due to measurement error

True = 2

0.25 0.75 0.25 0.75 0.25 0.75

% relativebiasin

Harvest-rate history Low Variable High

Extended Kalman filterErrors-in-variablesBayesian state-spaceStandard Ricker

• Results also change with true

Results for 95% coverage probabilities- Uncertainty in estimated is too narrow (overconfident) for all 4 estimation methods

Ricker

Probability

Actual

Estimated

- Trade-off between bias and variance (Adkison 2009, Ecol. Applic. 19:198)

Recommendation• Test parameter estimation methods before applying them (Hilborn and Walters 1992)

• Use results with humility, caution - Parameter estimates for ecosystem models may inadvertently be quite biased!

3. Unclear structure of fishery system (model uncertainty, structural uncertainty, model misspecification)

• Biased/imprecise parameters

• Wrong system dynamics

• Overconfidence in results if only use one model

1. Choose single "best" model among alternatives1a. Informally1b. Formally using model selection criterion (AICc)

Caution!! - Not appropriate for giving management advice- Asymmetric loss functions

(Walters and Martell 2004, p. 101)

SSB / SSBmsy

Case 2Case 1

SpeciesA B

Asymmetric loss: Which case is preferred?

Spawning favoured Harvest favoured

H T+Q T

0% 25% 50% 75% 100%

O:U ratio

0.25 0.5 1 2 4

0.30 1.44

0.68 2.09

1.18 1.33

Cummings (2009)

0.25 0.5 1.0 2 4 Preference ratio

Symmetric Asymmetric withharvest obj. favored

Asymmetric withspawning obj. favored

Recommendation• To develop appropriate indicators, ecosystem scientists should understand asymmetry in managers' objectives, especially given many species.

Fraser River Early Stuart sockeye salmon:Best "management-adjustment" model (H, T, Q, T+Q)

1c. Adaptive management experiment - Sainsbury et al. in Australia

More commonly, we have to consider a range of alternative models ...

1. Choose single "best" model among alternatives ... ...

3. Unclear structure of fishery system ... (cont'd.)

0 10 20 30 40 50 60

0.70.10.20.30.40.50.6 0.7

0.100.150.20

0.250.30

SSB ('000 t)

F * 1000

Stock-synthesis

Delay-diff.

(R. Mohn 2009)SSB (thousands of tonnes)

F*1000

Mvalues

Eastern Scotian Shelf cod (closed in mid-1990s)

2. Retain multiple models; conduct sensitivity analyses2a. Analyze separately

2. Retain multiple models; conduct sensitivity analyses2a. Analyze separately2b. Combine predictions from alternative models

- Unweighted model averaging - Weighted with AIC weights or posterior probab.,

then calculate expected values of indicators

• But weighting assumes managers useexpected value objectives

- Many use mini-max objectives (i.e., choose action with lowest chance of worst-case outcome)

1 2 3 4 5 6 7 8 9 10 11

0.5 1.00.750.25 1.250.50.25

Probabilitywith manage-ment action A

Limit referencepoint

Expected SSB (weighted average)

Worst-caseoutcome(unlikely, but choose action withlowest probability )

SSB/SSBtarget

Recommendation• Ecosystem scientists should work iteratively with managers to find the most useful indicators to reflect management objectives.

2. Retain multiple models; conduct sensitivity analyses ...

...2c. Evaluate alternative ecosystem assessment

models by using an operating model to determine their statistical properties

(e.g., Fulton et al. 2005 re: community indicators)

2. Retain multiple models; conduct sensitivity analyses......... 2d. Evaluate alternative ecosystem assessment

models within closed-loop simulation (MSE) to determine robust management strategies acrossrange of operating models

Caution!!!! Elaborated upon later.

Recommendation• Ecosystem scientists should compare management advice from multiple models.

• Models are "sketches" of real systems, not mirrors- Only essential features

Appropriate ecosystem model sketches?

ESAMMRMGADGETSEAPODYMEwEAtlantis...?

• "A model should be as simple as possible, but no simpler than necessary" [and no more complex either!]

- Morgan and Henrion (1990)

• "A model should be as simple as possible, but no simpler than necessary" [and no more complex either!]

- Morgan and Henrion (1990)

Appropriate model complexity depends on:- Type of questions/advice (Plagányi 2007)- Knowledge and data

Model complexity

Effectiveness, predictive power

(Fulton et al. 2003, others)

"Adaptive radiation" of ecosystem models

HighLow

• Build multiple (nested) models of a given system- Which model is best for the questions?- Yodzis (1998) could omit 44% of interactions

Recommendation: How ecosystem scientists can deal with structural uncertainty ... (continued)

• Conduct closed-loop management strategy evaluations(MSEs) across a wide range of hypothesizedoperating models of aquatic ecosystem

- "Best practice" -- Plagányi (2007)-- Tivoli meeting (FAO 2008)-- NEMoW I report (Townsend et al. 2008)

4. Outcome uncertainty, i.e., deviation from target (implementation uncertainty, implementation error, management uncertainty) Results from:

- Non-compliance

- Inappropriate regulations

- Physical and biological factors affecting q

• Overconfidence in:

- Meeting management objectives

- Avoiding undesirable outcomes

• Surprises!

1. Empirically estimate it (historical deviations from targets)

0.0 0.5 1.0 1.5 2.00.0

0.8Target

Realized

"Outcome uncertainty"Early Stuart sockeye salmon, B.C. (1986-2003)

Harvest rate

Forecast of adults (millions)Holt and Peterman (2006)

Outcome uncertainty:Both imprecise and biased

2. Add outcome uncertainty as a stochastic process

3. Conduct sensitivity analyses on nature of outcome uncertainty

Relativeaveragecatch

Ricker

Ricker AR(1)

Kalman filter

Distance-based HBM

Non-spatial HBM

Outcome uncertainty

MSE with CLIM2, a 15-popul. salmon model

(Dorner et al.2009, in press)

Ricker

Ricker AR(1)

Kalman filter

Distance-based HBM

Non-spatial HBM

None Imprecise and unbiased

Outcome uncertainty

None Imprecise and unbiased

Ricker

Ricker AR(1)

Kalman filter

Distance-based HBM

Non-spatial HBM

24% decrease

Outcome uncertainty

Imprecise andbiased

5. Inadequate communication among scientists, decision makers, and stakeholders

• Missing indicators due to unclear operational objectives

• Misinterpretation

• Overconfidence by decision makers if uncertainties not clear

• Decision makers may under- value scientific advice

1. Work iteratively with stakeholders and decision makers - Clarify management objectives and indicators

-- Maximize expected value, mini-max, or ...?

2. Conduct sensitivity analyses on mgmt. objectives

-- Cumulative probability distributions

-- Frequency format, not decimal probability format

5. Inadequate communication ... (continued)

(due to six interpretations of "probability", only one of which is "chance")

Recommendation:3. Show indicators with uncertainties - Use cognitive psychologists' findings about how people think about uncertainties and risks

“Chance" of an outcome for a given set of management regulations:

Probability format"There is a probability of 0.2 that SSB will drop below its limit reference point"

Frequency format"In two out of every 10 situations like this, SSB will drop below its limit reference point".

Gerd Gigerenzer et al.

4. Creatively display multiple indicators, and trade-offs among them

5. Inadequate communication ... (continued)

Recommendation

(Fulton, Smith, and Smith 2007)

Microfauna

Bycatch

Target

Habitat

Pelagic: demersal

TEP (marine mammals, seabirds)

Microfauna

Bycatch

Target

Habitat

Pelagic: demersal

TEP (marine mammals, seabirds)

Bycatch

Target

Habitat

Pelagic: demersal

Piscivore:planktivore

Bycatch

Target

Habitat

Pelagic: demersal

Scenario 1

Scenario 4

Radar plots,kite diagrams

Biomass size spectra

Biomass size spectra Piscivore:planktivore

PLESOL

MACNOP

Status Quo Effort

HADCODWHG

PLESOL

MACNOP

Precautionary Effort

Collie et al. (2003)

Bpa = precautionary biomass

AMOEBA plots for North Sea

0 200 400 600 800 10000.0

Average spawners (1000s)Yukon R.fall chumsalmon

Target spawners (in 1000’s)

Collie et al. (in prep.)

Harvest rateon run exceedingtarget spawners

0 200 400 600 8000.0

1.0Average spawners (1000s)Yukon R.

fall chumsalmon

Avg. subsistence catch (1000s)

Avg. commercial catch (1000s)% years commercial closed

700 180

20 100

100100

140300 60

Target spawners (in 1000’s)

Harvest rateon run exceedingtarget spawners

Booshehrian, Moeller, et al.Spawning target (1000s) of chum salmon

Avg. subsistence catch (1000s)

s)Vismon software, in prep.

Comment on tradeoffs• Remind managers and stakeholders:

Stated Actual

Uncertainty

Ecologicalindicators

Stated Actual

Uncertainty

Socio-economicindicators

Stated Actual

- Apply same standards to economists/social scientists and ecologists!!!

Stated Actual

Uncertainty

Socio-economicindicators

Stated Actual

1. Formal training:

Recommendations to deal with inadequate communication

2. "User studies" about effectiveness of communication methods

ScientistsDecision makersand stakeholders

3. Develop interactive, hierarchical information systems to show: - Management options - Consequences - Trade-offs - Uncertainties

4. Develop communications strategies like Intergovernmental Panel on Climate Change (IPCC):

Recommendations to deal with inadequate communication ...

• Advises decision makers and stakeholders

• Communication challenges:

- Complexity - Uncertainty - Risks - Credibility

How IPCC solves these communication challenges

1. Multi-level information systems: IPCC (2007) reportsa. Aim at multiple audiencesb. Hierarchicalc. Numerous footnotes (~ hypertext links)d. Diverse graphics

IPCC (2007) reports

How IPCC solves these communication challenges ...

2. Standardized format for describing uncertaintiesassociated with "essential statements":

- Chance of an outcome- Confidence in that estimated chance of that outcome

- "...very high confidence that there is a high chance of ..."

- "We have medium confidence that ..."

• Similar to recent Marine Stewardship Council guidelines

• Simulations of entire fishery systems- Closed-loop simulations (Walters 1986) - Management strategy evaluations (MSEs) (Punt and Butterworth early 1990s) - Which management procedure is most robust

to uncertainties-- A single management procedure includes:

--- Data collection method--- Stock or ecosystem assessment model--- State-dependent harvest rule

Combination of first 4 sources of uncertainty

Closed-loop simulation or MSE: ~ flight simulator

Robustprocedures for responding to unexpectedevents

Unusualweather

Uncertainties

Randomevents

Equipmentfailure

Natural Aquatic System Sampling, data collection

Operating model such as Atlantis

ESAMMRMEwEGADGET...

Ecosystemassessment model

What What we we know don't know

Stakeholders

Decision makers(harvest rules)

Stakeholders

Natural Aquatic System

Managementobjectives

Sampling, data collection

Harvesting

Fishing regulations(harvest quotas, closed areas, ...)

Stakeholders

Natural Aquatic System

Observation error

Managementobjectives

Outcome uncertainty

Sampling, data collection

Harvesting

Naturalvariability

Structuraluncertainty

Fishing regulations(harvest quotas, closed areas, ...)

Peterman (2004)

Operating model

Entire diagram = closed-loop simulation (MSE)

Inadequate communic.

MSEs include iterating across all major hypotheses about operating model

Result of MSE:Identifies relative merits of management procedures

for meeting management objectives

Caution: Substantial challenges ahead!

1. Characterizing operating model - Range of alternative hypotheses- Reliability of predictions from ecosystem models- Nonstationary environment (what if ...?)

2. Simulating ecosystem assessment process based on "observed" data using GADGET, an ESAM, ... - Automation of assessment process

3. Engaging scientists with decision makers, stakeholders

Conducting MSEs of ecosystem models

4. Simulating outcome uncertainty (deviation from target)

- Lack of data

5. Simulating state-dependent decision-making process- Lack of clear operational ecosystem objectives and indicators

- Complex objectives: optimize for one, make tradeoffs for others (Smith et al., Mapstone et al., and others

in Dec. 2008 Fisheries Research)

plus ...

(Link et al. 2002)

Can indicators of ecosystems from PCAs be used as measures of system state for input to harvest rules?

Ecosystem status (similarity to PCA category)

6. Interpreting results- Across multiple indicators and sensitivity analyses

7. Computations- CPU time

1. Need standards for evaluating reliability of models

2. If fitting ecosystem models to data, use operating models to check adequacy of estimation methods

3. Evaluate how much difference will be made by proposed "improvements" to ecosystem models (more complex not necessarily better)

4. Clarify operational management objectives and indicators that reflect ecosystem concerns

5. Analyze multiple models

Recommendations for next steps for ecosystem models

6. If use MSE approach (Tivoli, Plagányi, NEMoW I) - Start simply (ESAMs, MRMs) for assessment models (Butterworth and Plagányi 2004)

- Choose operating model (e.g., Atlantis) - Build experience- Determine feasibility of MSEs for evaluating more

complex assessment models (GADGET, EwE, ...)

7. Add to "Best practices" - Standardized protocol for determining performance of multiple assessment models for a given aquatic

ecosystem. - Training/gaming workshops to improve communication

Recommendations for next steps for ecosystem models

Reminders

• Sensitivity analyses should focus on finding which components cause changes in management advice.

• We probably underestimate the magnitude of uncertainty in estimates of parameters, state variables

• C.S. Holling: "The domain of our ignorance is larger than the domain of our knowledge."

"Review of major types of uncertainty in fisheries modeling and how to deal with them"

Documents

Objectives Learn the meaning of uncertainty and explore some theories designed to deal with it Find out what types of errors can be attributed to uncertainty

Rhode Island Commercial Fisheries Blueprint for Resilienceresilientfisheriesri.org/wp-content/uploads/2018/...cast shadows over this heritage industry and magnify uncertainty for the

How do neurons deal with uncertainty? Sophie Deneve Group for Neural Theory Ecole Normale Supérieure Paris, France

Spatial Bioeconomics under Uncertainty (with Application) Christopher Costello* September, 2007 American Fisheries Society Annual Meeting San Francisco,

Fiscal Uncertainty and How to Deal with Itauerbach/Auerbach Hutchins... · 2014. 11. 30. · Fiscal Uncertainty and How to Deal with It . Alan Auerbach . University of California,

Cold water fisheries in the FISHERIES FAO trans-Himalayan ... · to the host country Nepal. The six country reviews in the second section deal with cold water fish and fisheries in

Model uncertainty in the ecosystem approach to fisheries. · 2012-08-08 · 4 Introduction Implementing the ecosystem approach to fisheries (EAF: Garcia et al. 2003) will require,

Context-aware decision making under uncertainty for voice ... · systems are able to deal with uncertainty, the Markov Logic Network approach brings a uniﬁed theory for dealing

Challenges and opportunities for fleet- and métier-based … · fisheries are an ongoing “governance headache” combining management complexity, scientific uncertainty and political

Big data : challenges and opportunities for …Topic: Uncertainty Quantification Virtual prototyping using mathematical models. UQ does not deal with 1. unknown uncertainty in the

The Return of Uncertainty, and How Investors Can Deal With It (Safal Niveshak)

Return of Uncertainty, and How Investors Can Deal With It (Safal Niveshak)

Beyond Spending Levels: Revenue Uncertainty and …...wish to engage in strategic planning.4 There is evidence that districts grow their cash balances to deal with the uncertainty

"Review of major types of uncertainty in fisheries modeling and how to deal with them" Randall M. Peterman School of Resource and Environmental Management

Canadian Industry & Natural Resources...resources •Fisheries, Forestry, Mining •Secondary Industries •Those that deal with manufacturing or construction •Fish processing, lumber

How culture influences the way entrepreneurs deal with ...centaur.reading.ac.uk/81615/3/Upload culture influences entrepreneu… · 2. Uncertainty and entrepreneurship Uncertainty

The Return of Uncertainty, and How Investors Can Deal With It

Addressing Uncertainty in Fisheries Science and Management #5 · 2019-01-22 · reduced, and managed for in fisheries science and management. The Panel focused on science and management

A Fuzzy Approach to Deal with Uncertainty in Image Forensics

VALUE, PRICE AND INFORMATION: CLOSING THE DEALpeople.stern.nyu.edu/adamodar/podcasts/valfall16/valsession16a.pdfanalysis and deal with the consequences. ¨ Uncertainty about gap closing: