1 of 31 The EPA 7-Step DQO Process Step 6 - Specify Error Tolerances 60 minutes (15 minute Morning...

1 of 31

The EPA 7-Step DQO Process

Step 6 - Specify Error Tolerances

60 minutes(15 minute Morning Break)

Presenter: Sebastian Tindall

DQO Training CourseDay 3

Module 15

2 of 31

Step Objective: To specify the decision

makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data– Since analytical data can only

provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error

Step 6: Specify Error Tolerances

Step 4: Specify Boundaries

Step 2: Identify Decisions

Step 3: Identify Inputs

Step 1: State the Problem

Step 5: Define Decision Rules

Step 6: Specify Error Tolerances

Step 7: Optimize Sample Design

3 of 31

Objectives

To be able to define, for a specific project:1. The variability for each COPC,2. the decision errors, 3. the consequences of the errors, 4. the null hypothesis,5. the error rates (Alpha and Beta) 6. the Lower Bound of the Gray Region & Delta

4 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Step 6- Specify Error Tolerances

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

5 of 31

Decision Error Tolerances The goal of the planning team is to develop

a data collection design that reduces the chance of making a decision error to a tolerable level

Step 6 provides a mechanism for allowing the decision maker to define tolerable limits on the probability of making a decision error

6 of 31

Two Reasons Why Decision Makers Make Decision Errors

Sampling error occurs because the sampling design is unable to capture and control the complete extent of heterogeneity that exists in the true state of the environment

Measurement error occurs because analytical methods and instruments are not absolutely perfect

7 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

In order to calculate the number of samples needed(in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable.

• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information

– Existing data– Process knowledge– Professional judgment

• Estimate of the population standard deviation– Reference source – Method of calculating

Step 6- Specify Error Tolerances

8 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

In order to calculate the number of samples needed(in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable.

• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information

– Existing data– Process knowledge– Professional judgment

• Estimate of the population standard deviation– Reference source – Method of calculating

Estimate the standard deviation by using the Deming approach of dividing the range by 2 or 3, depending on the frequency distribution.

Step 6- Specify Error Tolerances

9 of 31

Estimated Standard DeviationsEstimated Standard Deviations for NaI Example

Environmental VariableRange of

EnvironmentalVariable

Estimate of Population Standard DeviationDR#

Attribute Unit ofMeasure

LowerLimit

UpperLimit

StandardDeviation Source How Estimated?

1

True mean ofCs-137estimated by95% UCL ofsample mean

pCi/g 0.031 1.89 0.4052000HPGedata

From 116-H-1 CVP data

1

True mean ofCs-137estimated by95% UCL ofsample mean

cpm(convertedto pCi/g)

-5.17 99.92

7.48or

7.48/2=3.74

1999 to2000 NaIdata

From 116-H-1 CVP data

The choice of an estimate of a standard deviation has a large impact on the number of samples required. Avoid underestimating the standard deviation. Always be conservative when estimating the standard deviation.

10 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray regionDefine both types of decision error:Determine which one occurs above and which one occurs below the action level.

Two Types of Decision Error:• Cleaning up a clean site• Walking away from a dirty site

Step 6- Specify Error Tolerances

11 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

For each Alternative Action:• Create a list of possible decision error(s) that may occur if an action

is incorrectly taken• Discuss the consequences of making each decision error• Rate the severity of the consequences of a decision error

(i.e., low, moderate, severe) at a point:– Far below the action level– Below but near the action level– Above but near the action level– Far above the action level

• Indicate which decision error has the most severe consequencenear the action level

Step 6- Specify Error Tolerances

12 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray regionProvide rationale for rating the severity of consequences as low or severe

Step 6- Specify Error Tolerances

13 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray regionDefine the null hypothesis (baseline condition) and the alternative hypothesis:The decision error that has the most adverse potential consequences should bedefined as the null hypothesis.

The null hypothesis should state the OPPOSITE of what the project hopes to demonstrate.

• Site is assumed to be contaminated until shown to be clean• Site is assumed to be clean until shown to be contaminated

Step 6- Specify Error Tolerances

14 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

The gray region is a range of possible parametervalues within which the consequences of a decision errorare relatively minor.

Step 6- Specify Error Tolerances

15 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

The gray region is bounded on one side by the action level, and on the other side by the parametervalue where the consequences of decision error beginsto be significant. This point is labeled LBGR, whichstands for lower bound of the gray region.

Step 6- Specify Error Tolerances

16 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the possible range of the parameter of interest

Choose the null hypothesis.

Identify the decision errors.

Specify the boundaries of the gray region

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

It is necessary to specify the gray region because variability in the population and unavoidable imprecision in the measurement system combine to produce variability in the data such that a decision maybe “too close to call” when the true parameter value is very near the action level.

Step 6- Specify Error Tolerances

17 of 31

Width of the Gray Region () :UBGR - LBGR or AL - LBGR

= Analytical + Sampling Error– Estimated based on past data and general knowledge

= 1/2 of the AL – For each COPC, calculate and set LBGR

= 10 - 30% of the AL – For each COPC, calculate and set LBGR

= PDF method– Use PDF for worst COPC to set LBGR

18 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

Present the rationale of how the LBGR was calculated or determined.

Step 6- Specify Error Tolerances

19 of 31

Lower Bound of the Gray Region

Because the null hypothesis is that the site is contaminated, the upper bound of the gray region is set equal to the action level

The LBGR should be set at a value where the consequences of the decision error begin to be significant

20 of 31

How to Set the LBGR

LBGR set by the Analytical + Sampling Error LBGR set to 1/2 Action Level LBGR set to ~ 50 to 90% of AL (Decision-

Maker “whim”) LBGR set by the Probability Density Function (PDF)

method

UBGR - GR = LBGR

AL - = LBGR

21 of 31

The LBGR is often based on unavoidable variability in the concentration data– The GR may be estimated based on the precision

that the analytical methods allow plus an estimate as to the sampling variance

– LBGR = AL – GR (Analytical + Sampling Error) 100 ppm – (10 ppm + 31 ppm) = 59 ppm

MARSSIM suggests the LBGR be set as:– LBGR = AL – GR (1/2 AL)

100 ppm – 50 ppm = 50 ppm

How to Set the LBGR (cont.)

22 of 31

The LBGR is often set at some other value– This is based on the decision makers’ choice and

is not scientifically based– LBGR = AL – GR (20% of AL);

100 ppm – 20 ppm = 80 ppm

How to Set the LBGR (cont.)

23 of 31

Use the Probability Density Function (PDF) method– The LBGR may be estimated based the Probability Density

Function– Place the Action Level on the mean of the PDF– Ask: “Does a substantial amount of contaminant

concentration values exceed the Action Level?”– Ask: “Is there a reasonably high probability that data from a

small sample size will result in declaring the site clean?”– If “yes” to the first question and “no” to the second question,

begin moving the PDF backwards along thex-axis towards zero concentration

– Pause and ask both questions again– When the answer is “no” to the 1st and “yes” to the 2nd, you

have set the LBGR(e.g., where the mean of the PDF lies on the x-axis is now the LBGR)

• Use probability theory to show this

How to Set the LBGR (cont.)

24 of 31

Methods for Evaluating the Attainment of Cleanup Standards - Volume 1: Soils and Solid Media

EPA, February 1989

PB89-234959

How to set the LBGR

2

121

211

2

5.0

ZAL

ZZn

1 is a hypothetical “true mean concentration where the site should be declared clean with a high probability”. (1 = LBGR)

25 of 31

Show Probability Density Function Distribution

Demonstration

Show VDT File:Moving Gray Region

26 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

Assign probability values that reflect the decision maker’s tolerable limits for making an incorrect decision.

• At the action level (Alpha error)• At the other bound of the gray region (Beta error)• At a point far below the action level• At a point far above the action level

Step 6- Specify Error Tolerances

Note: EPA QA/G9 recommends that you set both Alpha and Beta error rates to 1% to start.

27 of 31

Site is dirtySite is clean

100

True State of Site

Alternative ActionWalk away from site Clean up site

75

Probability of deciding

that the site is dirty

0.0

0.5

1.0

Lower Bound of Gray Region

Four Decision-MakerError Tolerance

Locations

Null Hypothesis:

Site is dirty.

True mean COPC Concentration

Action Level

The Gray Region

28 of 31

Step 6 Summary

Determine the variability for each COPC Define the two types of error

– Incorrectly walking away from a dirty site, or– Incorrectly cleaning a clean site

Evaluate severity of the incorrect decisions both below, above, and near the action level

Select the null hypothesis

29 of 31

Step 6 Summary Establish a LBGR based on one of the four

methods shown previously Provide the basis for selecting the LBGR Remember the closer the LBGR is to the

action level, the more samples are needed Assign probability limits on either side of the

gray region (Delta)– Specify the error rates (Alpha and Beta) decision

makers are willing to accept and provide rational for the rates

30 of 31

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

Step 6- Specify Error Tolerances

31 of 31

End of Module 15

Thank you

Questions?

We will now take a 15 minute break.Please be back in 15 minutes.