Ecological Risk Assessment: Major steps and issues - … · 2010-10-21 · Ecological Risk Assessment: Major steps and issues ... UM genetic biocontrol symposium, June 2010: Slide

Ecological Risk Assessment: Major steps and issues

Keith Hayes

CSIRO Mathematics, Informatics and Statistics

UM genetic biocontrol symposium, June 2010: Slide 2 of 57

Overview

Major steps

• Ia Scope: What is the problem?

• Ib Hazard analysis: What can go wrong?

• II Risk and uncertainty assessment

• III Monitor and review

Major issues

• qualitative risk assessment

• elicitation

• uncertainty and dependency

• statistical inference

Identify hurdles to

• scientific risk assessment

• honest risk assessment


I. Risk Assessment Framework

I: IDENTIFY, DEFINE & AGREE

Identify assessment

option

Clearly define assessment boundaries and scope

Define conceptual

model(s) of the system

Identify and prioritise hazards

Record hazards that are ignored

Agree on assessment methods and

models

Identify and quantify

uncertainty

Sampling strategies and power analysis

Evidence

Identify risk management

strategies

Define assessment & measurement

endpoints

Agree on risk acceptance

criteria

Risk acceptable?

Evaluate risk management

strategies

NO

YES

Calculate residual risk

Outcomes acceptable?

Theory, models &

existing data

Monitor outcomes and

validate assessment

Stop activity and re-evaluate

End

YES

NO

II: CALCULATE, EVALUATE & MANAGE III: MONITOR, VALIDATE & COMPARE

Uncertainty analysis

Address linguistic

uncertainty

Identify stakeholders

Risk assessment

Hazard id & prioritisation

Reject assessment

option

Calculate risk

Frequency/exposure

assessment

Consequence/effect

assessment

Engage stakeholders

Problem identified


I. Risk assessment quality criteria

Science quality criteria

• transparent and repeatable: would other investigators be able to duplicate your result?

• falsifiable: does it make predictions that are measurable and (at least theoretically) falsifi-able?

Decision quality criteria

• precision: does it provide estimates with tight confidence intervals?

• accurate: are its estimates correct?

Honest risk assessments are (Burgman, 2005)

• faithful to the assumptions about the kinds of uncertainty embedded in the assessment;

• carry these uncertainties through the analysis; and,

• represent and communicate them reliably and transparently.


I. Scoping the problem

Objectives of the scoping step

• identify and engage stakeholders

• identify assessment options

• define spatial and temporal extent of the assessment

• identify assessment endpoints

• identify acceptance criteria for assessment endpoints

Acceptance criteria

• define what separates acceptable risk from non-acceptable risk (not a science question!)

• ideally defined before the assessment but often evolve through the process of the assess-ment

Assessment endpoints

• an expression of the values that we are trying to protect by performing the risk assessment(not a science question!)

• should be linked to measurement endpoints


I. Hierarchy of assessment and measurement endpoints

Individuals Change in metabolismInhibition or induction of enzymesIncreased susceptibility to pathogensGrowth rate

Populations Genotypic and phenotypic diversityMortality/fecundityGrowth rateAbundance of harmful organisms

Species Commercial extinctionActual extinctionCreation of new harmful species (virus)

Community Decreased biodiversityDecreased food web diversityDecreased productivity

Ecosystem Decreased community diversityAltered bio- and geo-chemical cyclesLoss of rare or unique ecosystems

Landscape Physical processes (floods, fires, erosion)Resource quality (air, water, soil)

Respiration rate, assimilation efficiencyLiver enzymes Frequency of individual morbidity Age/weight ratio

Genotype frequencyPopulation morbidityAge/weight ratioSize/frequency – blooms/ pest outbreaks

Yield/production, CPUENumbers/density Occurrence

Diversity indicesSpecies diversitySpecies evenness

Diversity indicesCarbon, nitrogen, phosphorus fluxExtent and area

Frequency of floods, fires, low flowsPollutant concentrations


I. Hazard analysis

What are hazards?

• an act or phenomenon that, under certain circumstances, could lead to harm (The RoyalSociety, 1983)

• a substances or activitys propensity to produce harm (NRC, 1996)

• the answer to the question: “what can go wrong?”

How to identify hazards

• wait and see what happens: not very proactive!

• informal approaches: experience, unstructured brainstorming, checklists

• formal “top-down” and “bottom-up” approaches

Formal approaches

• in my experience always identify more hazards

• BUT are harder to implement


I. HAZOP analysis

Selectprocess

Postulatedeviation

Guide words:NO or NOTMORE THANLESS THANAS WELL ASPART OFREVERSEOTHER THAN

Is the deviationcredible?

Is the deviationhazardous?

Identifiedhazard

Identifypossiblecauses

Identifypossible

consequences

Fault treeanalysis

Event treeanalysis

Nex

tde

viat

ion


I. HHM for GM canola

BIOLOGICALHIERARCHY

BIOLOGICALCOMPONENTS

BIOLOGICALPROCESSES

PHYSICALCOMPONENTS

PHYSICALPROCESSES

Genes

Organisms

Populations,Foodwebs,

Communities

Species

Habitat

Bioregions

Bacteria,Viruses, Fungi

Plants

Insects

Otherinvertebrates

Birds

Mammals

Reptile, Fish,Amphibians

Man

Development,Reproduction,

Growth

Excretion

Movement,Behaviour

Predation,Nutrition,

Parasitism

Death

Selection,Mutation

Competition

Bioaccumulation

Atmospheric &interstitial air

Atmospheric &surface water

Groundwater& Interstitial

water

Seawater

Gravity,Magnetism &

Static elec

Windmovement

Watermovement

Soilmovement

Evaporation,Precipitation

Fire

Freezing

Lightning

GM CANOLAENVIRONMENT

CHEMICALCOMPONENTS& PROCESSES

Cycles

Creation &destruction

Inorganics

Organics

Geneexpression

MAN-MADECOMPONENTS

Machinery

Buildings

Roads, Tracks

Fences

Clothes

Fertilisers,Pesticides

MAN-MADEPROCESSES

Geneconstruct

Ploughing,Planting

Irrigation,Spraying,Weeding

Harvest,transport,

process, store

Cleaning

Recreation,Conservation

Husbandry

Temperature

Inversion Criminal

QC,Monitoring


I. Fault tree analysis for HT weed


II. Risk functions and assessment methods

Generic risk functionRisk = f(x1, x2, · · · , xn) (1)

where f(·) is the risk model, and xi:n are the risk factors. If xi are uncertain (probabilistic)variables then Equation 1 is a function of a joint probability distribution.

Qualitative example (BA, 2008)

Risk = Import× Distr.× Est.× Spread

Quantitative example (Leung et al., 2004)

P (Est.|N) = 1− exp [(αN)c]

where N is the number of released individuals, α = − ln(1 − p), p is the probability of singlepropagule establishing, and c is a shape parameter that allows the model to reflect inversedensity dependence (“Allee” effect).

Both approaches predicated on a conceptual model of the system

• one of the important (science quality) differences between qualitative and quantitative riskassessment is the transparency of the conceptual model


II. Qualitative issues: risk matrix problems

CL

Negligible Low Medium High

Negligible Negligible Negligible Low Medium

Low Negligible Low Medium Medium

Medium Low Medium Medium High

High Medium Medium High High

Sorites paradox

Arbitrary categorization

Bias

Linguistic uncertainty

Heuristics and cognitive biasNon-

associative combination

rules


II. Qualitative issues: non-associativity and bias


II. Qualitative issues: non-associativity and bias


II. Qualitative issues: Linguistic uncertainty

Linguistic uncertainty

• very prevalent but often unrecognised source of uncertainty

Many sources

• ambiguity - arises when words have more than one meaning and it is not clear which oneis meant

• context dependence - caused by a failure to specify the context in which a term is to beunderstood: “large scale escape”

• under-specificity - occurs when there is unwanted generality: “in a small percentage (gen-erally <10%) of cases, zinc level exceed WHO guidelines”

• vagueness - arises when terms allow borderline cases: “medium risk”

• indeterminacy - arises because the future use of a theoretical term may not be completelyfixed by it current use (e.g. taxonomic revisions)


II. Strategies for linguistic uncertainty

Ignore it

• surprisingly prevalent

Treat it by defining terms (Regan et al., 2002)

• define terms, specify context, remove unnecessary generality, etc.

• but use and interpretation shown to vary even when numerical definitions are provided(Patt and Desai, 2005; Budescu et al., 2009)

• crisp definitions of terms cause problems at the margins similar to Sorites Paradox - onegrain of sand 6= a pile, two grains of sand 6= a pile...so when is a pile of sand a pile?

• vagueness a motivation for fuzzy sets

Treat it by careful elicitation (O’Hagan et al., 2006; Spiers-Bridge et al., 2010)

• convert language to numbers: range, mean, degree of confidence

• design and implement to overcome other well known heuristics and bias

UM genetic biocontrol symposium, June 2010, June 2010: Slide 17 of 57

II. Major issue: Elicitation

Elicitation

• converts information into data

• can be designed to minimise heuristics and bias

• key to high quality, honest risk assessment, particularly in the absence of data

Two fundamental approaches to elicitation

• structural: specify problem, elicit parameters and/or models

• predictive: specify scenarios, with information “keys”, elicit outcome


II. Heuristics and bias

Some well documented sources

• overconfidence: assessors tend to overestimate the accuracy of their beliefs or underesti-mate the uncertainty in a process

• availability: assessors link their probability estimates to the frequency with which they canrecall an event

• anchoring: assessors tend to anchor around any initial estimate and adjust their finalestimate from this value irrespective of the accuracy of the initial estimate

• framing: assessors response to the same problem tends to vary depending on the scaleand manner in which information is presented

• motivational bias: assessors provide inaccurate or unreliable estimates because it is ben-eficial for them to do so

• conditionally incoherent response: assessors fail to provide probability estimates that ad-here to the axioms of conditional probability


II. Heuristics eg: overconfidence

Opinions of geotechnical experts on two standard problems. The correct (measured) valuefor settlement depth was 1.5 cm and for height to failure was 4.9 m. The y-axis for both was rescaledso the maximum value was 1. Correct values are shown as dashed horizontal lines. The intervalsshow ‘minimum’ and ‘maximum’ values reported by the experts (after Hynes and Vanmarcke 1975in Krinitzsky 1993).


II. Heuristics eg: framing effects

0

0.1

0.2

0.3

0.4

0.5

Laypeople AmericanAcademy of

Psychiatry andLaw

AmericanPsychology-Law

Society

Pre

dict

ed p

roba

bilit

y of

an

offe

nce Scale 1

Scale 2 (1-100%)(1->40%)

Two scales used to guide judgments about probabilities that violent criminals will re-offend (after Slovic et al. 2000)


II. Heuristics eg: motivational bias

Loss of gross world product resulting from a doubling of atmospheric CO2 by 2050, Nordhaus WD (1994), Expert Opinion on Climatic Change, American Scientist, Jan/Feb: 45-51

-5

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Loss

of g

ross

wor

ld p

rodu

ct EconomistsEcologists


II. Elicitation how to guides

Kynn (2008) recommends inter alia:

• familiarize the expert with the elicitation process (good time to calibrate experts also)

• use familiar measurements and ask questions within area of expertise

• decompose elicitation into small distinct parts and check coherence with probability axioms

• be specific with wording use a frequency representation

• do not provide sample numbers for expert to anchor on

• ask the expert to discuss estimates and give evidence for and against

• provide feedback and allow expert to re-consider

See also

• O’Hagan et al. (1998)

• Spiers-Bridge et al. 2010


II. Spiers-Bridge et al. (2010) four-step elicitation


II. Structural elicitation eg


II. Uncertainty

Some definitions

• a state of incomplete knowledge (Cullen and Frey, 1999)

• a departure from the unattainable state of complete determinism (Walker et al., 2003)

• a degree of ignorance (Beven, 2009)

Many taxonomies and classification schemes but basically

• linguistic uncertainty

• variability

• epistemic uncertainty

• decision uncertainty


II. Uncertainty nomenclature and classification

EPISTEMIC UNCERTAINTY VARIABILITY

Descriptive uncertainty 1

Measurement error 2, 3, 6, 9, 10, 11

Systematic error 2, 6

Natural variation 2, 6, 11

Subjective judgement 2, 16

Inherent randomness 2

Model uncertainty 2, 5, 6, 12, 24

Process stochasticity 3

Model error 3, 11

Random error 16

Incertitude 4,17

Completeness 5, 12, 20

Scenario uncertainty 5, 18, 19

Type A uncertainty 8

Type B uncertainty 8

Epistomological uncertainty 7 Ontological

uncertainty 7

Uncertainty 9, 11

Demographic stochasticity 10, 25

Process noise 10

Process variability 10

Random variability 10,17

Environmental stochasticity 10, 25

Process error 10, 21

Anthropogenically induced variation 11

Parameter uncertainty 12, 22, 24,

28

Measurement uncertainty 15, 26

Aleatory uncertainty 13, 14, 17

Irreducible uncertainty 15

Reducible uncertainty 15,17 Objective

uncertainty 17

Stochastic uncertainty 17, 27

Dissonance 17

Subjective uncertainty 17, 27

Ignorance 17

Nonspecificity 17

Structural uncertainty 22

Random fluctuations 22

Intrinsic uncertainty 23

Data uncertainty 24

Sampling uncertainty 26

Random sampling error 29

Vagueness 2, 4

Context dependence 2, 4

Ambiguity 4

Indeterminancy 2

Under-specificity 2, 4

Linguistic imprecision 16

Implementation error 3

DECISION UNCERTAINTY

Describing & summarising risk 28

Discounting 28

Utility functions 28

Value uncertainty 16Acceptance criteria 28

LINGUISTIC UNCERTAINTY


II. How do we measure uncertainty?

With language

• “highly certain”, “low confidence”, etc.

Numerically

• probability theory (precise and imprecise)

• evidence theory (Dempster-Shafer structures)

• possibility theory (Fuzzy sets)

• plausibility theory

Probability is best (in my opinion!)

• language confounds linguistic uncertainty with variability and epistemic uncertainty

• by far the most widely used theory of uncertainty supported by an enormous amount ofpredictive and inferential methodology

• imprecise probability constructs (pboxes) equivalent to constructs of evidence theory buteasier to convolve


II. Variability

Variability

• inherent fluctuations or differences in a quantity or process, within or between time, loca-tion or category/group

• can be characterised but not reduced with additional data

Sources of variability

• inherent randomness - many processes appear inherently random (coin toss) but genuineexamples are hard to find. For practical applications the actual (epistemic uncertainty) istreated as irreducible

• natural variability - some authors distinguish two types of temporal variability (demographicand environmental), the former refers to chance variation in the fates of individuals, thelatter refers to variation in the mean vital rates of populations caused by abiotic and bioticfactors

Population sensitive

• variability is the irreducible diversity of a population, hence its characterisation is sensitiveto the definition of the population, time frame and space domain


II. Uncertainty nomenclature and classification


II. Epistemic uncertainty

Epistemic uncertainty

• our incomplete knowledge of the world, theoretically reducible with additional study

Many sources

• model uncertainty - uncertainty in our conception or description of a system

• completeness - have all elementary outcomes or possible states of the world been enu-merated?

• scenario uncertainty - uncertainty generated when a model is applied to situations outsidethe one under study. particularly the future

• subjective judgement - occurs as a result of interpretation of information or data particu-larly where empirical evidence is lacking

• systematic error - difference between the true value and the value to which the meanconverges as sample size increases

• measurement error - apparently random variation in the measured value of a quantity

• sampling error - the epistemic uncertainty about the distribution function of a variable (i.e.its variability) that arises because only a portion of the individuals in a population haveactually been measured


II. Propagative UA methods and statistics

Second order Monte Carlo Simulation

STATISTICS

Do you have observations of the

process?NOYES

OBJECTIVE: Elicit outcomes, models and/

or parameters

OBJECTIVE: Prediction, explanation and/or

classificationPREDICTIVE

ELICITATION: Specify scenarios, elicit

outcomes

STRUCTURAL ELICITATION: Specify

process, elicit models and/or parameters

PARAMETRIC METHODS: Specify

stochastic model

NON-PARAMETRIC METHODS: Do not

specify model

Rule sets

Generalised Linear Models

and mixed models

Non-linear least

squares regression

Regularised regression (lasso and

ridge)

Time series analysis

Bayesian Hierarchical

Models

Linear regression,

ANOVA

Linear discriminant

analysis

Nearest neighbour analysis

Classification and

regression trees

Boosted regression

trees

Bayesian Belief

Networks

Fuzzy cognitive maps

Qualitative modelling

(loop analysis)

Interval analysis

Fuzzy sets

Info-gap theory

Variance propagation (Delta

method)

First order Monte Carlo Simulation

Probability and dependency

bounds analysis

UNCERTAINTY ANALYSIS

SEMI-PARAMETRIC METHODS

Semi-parametric regression


II. UA method citations


II. Strategies for variability and epistemic uncertainty

General strategies for uncertainty

• ignore - hardly defensible in risk assessment

• treat - various methods (first and second order analysis)

• compare - compare answer with different approaches

• envelope - bound answer around alternative approaches

• average - average over alternative approaches

Associated strategies for dependency

• ignore

• envelope

• model

• factorise


II. Strategies for variability

Ignore it

• undefensible in risk assessment?

Model or eliminate it

• model the process that creates the variability (more complex model)

• eliminate it via a simpler model (simpler endpoint)

Treat it (first order)

• choose a distribution to represent variability in a parameter

• justify choice on a) theory, b) maximum entropy; or c) data

• propagate via first order MCS

• but hundreds of observations needed to distinguish tails if σµ> 1 (Haas, 1990)


II. Strategies for variability cont..

Compare alternatives

• sensitivity analysis for parametric uncertainty

Treat it (second order analysis)

• assign distributions to moments of the distribution functions for variable parameters

• bootstrap to generate sampling distribution of the moments of the distribution from sampledata (Frey and Burmaster, 1999)

• propagate through second order MCS

Envelope it

• capture uncertainty in variability via probability boxes (Ferson et al., 2004)

• propagate via dependency bounds analysis


II. Probability boxes


II. Dependence

Dependence

• acknowledging variability imposes an important practical challenge: dependence

• dependence implies that the probability of observing values of one of the factors is relatedin some fashion to the probability of observing values in one or more of the other factors

Many potential sources, including

• common cause failure modes

• spatial and temporal autocorrelation

• biological

Can be complex and not-necessarily linear

• its effect may not be accurately characterised by varying a correlation coefficient between+ 1 and -1.

• low or zero correlation between two random variables does not generally mean that theyare independent (Ferson and Hajagos, 2006)


II. First order MCS and dependence

UM genetic biocontrol symposium, June 2010:: Slide 39 of 57

II. Strategies for dependency

Ignore it

FX,Y (x, y) = FX(x) · FY (y)

σ2z=x+y = σ2

x + σ2y

Envelope it

FX·Y (Z) = minz=x·y

{min [FX(x) + FY (y)− 1, 0]

}FX·Y (Z) = min

z=x·y

{min [FX(x) + FY (y), 1]

}Model it (linear and non-linear)

σ2z=x+y = σ2

x + σ2y + 2ρσxσy

C [F (x), F (y)] = − 1

αln

[1 +

(exp(−αx)− 1)(exp(−αy)− 1)

exp(−α)− 1

]


II. Probability Bounds Analysis eg


II. Model dependency via Copulas


II. Factorise dependency

Bayes nets

P (x1, x2, x3) = P (x3|x1, x2)P (x2|x1)p(x1)

p(X) =K∏k=1

p(xk|pak)

MCMC Gibbs sampling

p(y|λ) =n∏i=1

Poisson(yi|λi)

p(λi|α, β) = Gamma(λi|α, β)

p(λ, α, β|y1:n) ∝

Conjugate pair︷︸︸︷n∏i=1

Poisson(yi|λi)n∏i=1

Gamma(λi|α, β) p(β)p(α)

More advanced methods

• Metropolis Hastings, MH within Gibbs, adaptive MCMC, PMCMC, AdPMCMC..


II. Bayes network eg (Hood and Barry, 2010)

ZoneLow riskHigh risk

70.030.0

DiseaseInfectedUninfected

1.0099.0

Test_resultPositiveNegative

0.9599.0

Age_classYoungMatureOld

40.047.013.0


II. Models in risk assessment

Necessary abstractions of real world complexity

• all risk assessments are predicated on a model

Model desiderata

• precise, generalisable and realistic (Levins, 1993)

• relevant, flexible and realistic (Pastorok et al., 2002)

• impossible to maximise all three properties!

Model complexity

• models range from highly abstract to highly complex

• realism associated with complexity but..

• complexity is not associated with accuracy (Reckhow, 1994; Arhonditsis and Brett, 2004;Fulton et al., 2004)


II. Model caricatures

QUALITATIVEMODELS

GENERALITY

PRECISION

REALISM

MECHANISTICMODELS

GENERALITY

PRECISION

REALISMGENERALITY REALISM

STATISTICALMODELS

PRECISION

Real world process represented by a statistical model:

regression model, time series model, etc...

Input variables

Input data

Response variables

Real world process treated as unknown

Input variables

Input data

Machine learning

techniques

Response variables

Real world process represented by set of

difference or differential equations: populations,

biogeochemical.. Input variables

Input data

Response variables

Real world process represented by a

statistical model or a mechanistic model.

Input variables

Input data

Response variables

Observation model

A

D

B

E

Real world process represented graphically: influence diagram, loop

analysis, etc..Input

variables

Response variables

C

Hyper-parameters

Input data

CSS TCP RA Workshop, Coogee, May 2010: Slide 46 of 57

II. Strategies for model uncertainty

Ignore it

• Defensible in rare circumstances (e.g. model mandated by legislation or internationalguidelines)

Compare alternatives

• statistical models I: manual parametric model choice (AIC, DIC, BIC, Bayes factors) andautomatic parametric choice (ridge and lasso regression)

• statistical models II: automatic non-parametric choice (machine learning methods)

• qualitative models: easily constructed and well suited to comparing alternatives

• mechanistic models: can compare alternatives but its harder

• BUT without data comparison strategy is unconstrained


II. Strategies for model uncertainty cont.

Average over alternatives

• statistical models: Bayesian model averaging

• process models: assign a prior mass to each model and weight predictions accordingly(e.g. Bernoulli random variable for two competing models)

• BUT difficulty with priors and can end up averaging over incompatible theories

Envelope alternatives

• mechanistic models: classic example IPCC global warming projections

• BUT problem still unconstrained?


II. Qualitative Modelling

iiiii eir

00

00

2,3

3,21,2

2,11,1

A

Community matrix

Signed digraph

Lotka–Volterra equations for a simple trophic chain

133,122,111,11

1 rNNaNdtN

dN

233,211,222,22

2 rNNaNdtN

dN

311,322,333,33

3 rNNaNdtN

dN

12, 23,21, 32,

11,

(Levins 1968, 1974)



Predator-prey

Mutualism

Commensalism

Competition

Self-effects

Amensalism



Kurle et al. 2008 PNAS


III. Monitor, review and inference


III. Over-dispersed Poisson process


III. GLM for journey survival

General Linear Model (with over-dispersion) for journey survival

• assume a Poisson sampling distribution for data (counts)

• log link function

• single covariate (time)

• Gaussian distributed error terms in the linear model

yi ∼ Pois(θi)

ln(θi) = xβ = β0 + β1xi + εi

εi ∼ N(0, σ2)

Full model for n data points factors the joint distribution function.

p(β, θ, σ|x,y, β0,V0, s1, s2) =n∏i=1

Pois(yi|θi)n∏i=1

(ln θi|xi, β, σ)N2 (β|b0,V0) IG(σ|s1, s2)


III. Posterior distribution for journey survival


III. AdPMCMC for SSM Peters et al. submitted

State Space Modelling for single species population dynamics

log(Nt+1) = log(Nt) + g(Nt) + εt yt = f(Nt) + ωt

Latent state models

g(Nt) = β0 β0 = r Exponential (M0)

g(Nt) = β0 + β1Nt β1 = − r

KRicker (M1)

g(Nt) = β0 + β2Nβ3t β2 = − r

Kβ3, β3 = θ Theta-Ricker (M2)

g(Nt) = M1− log(β4 +Nt) β4 = “Allee N” Mate-limited Ricker (M3)

g(Nt) = β5 + β6Nt + β7N2t Flexible Allee (M4)

K =−β6 −

√−beta2

6 − 4β5β7

2β7

, r = −β7K2


III. Bayes factors for synthetic data M4

CSIRO Mathematics, Informatics and Statistics

Keith Hayes

Phone: +61 3 6232 5260

Email: [email protected]

Web: www.csiro.au

Contact UsPhone: 1300 363 400 or +61 3 9545 2176

Email: [email protected] Web: www.csiro.au

mailto:[email protected]

mailto:[email protected]

http://www.csiro.au

Documents

Ecological Risk Assessment: Major steps and issues - … · 2010-10-21 · Ecological Risk Assessment: Major steps and issues ... UM genetic biocontrol symposium, June 2010: Slide