1 of 48 The EPA 7-Step DQO Process Step 6 - Specify Error Tolerances 3:00 PM - 3:30 PM (30 minutes)…

1 of 48

The EPA 7-Step DQO Process

Step 6 - Specify Error Tolerances

3:00 PM - 3:30 PM (30 minutes)

Presenter: Sebastian Tindall

Day 2 DQO Training CourseModule 8

2 of 48

Terminal Course ObjectiveTo be able to define the decision errors, consequences of the errors, the null hypothesis, and the lower bound of the gray region for a specific project

3 of 48

Step Objective: To specify the decision

makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data– Since analytical data can only

provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error

Step 6: Specify Error Tolerances

Step 4: Specify Boundaries

Step 2: Identify Decisions

Step 3: Identify Inputs

Step 1: State the Problem

Step 5: Define Decision Rules


Step 7: Optimize Sample Design

4 of 48

Step Objective: To specify the decision

makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data– Since analytical data can only

provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error


Step 4: Specify Boundaries

Step 2: Identify Decisions

Step 3: Identify Inputs

Step 1: State the Problem

Step 5: Define Decision Rules


Step 7: Optimize Sample Design

Estimation Error

Inherent in the process of estimation is error (deviationfrom the true value).

That’s why it’s called estimation.

Error Mistake

Error = Deviation

5 of 48

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Step 6- Specify Error Tolerances

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

6 of 48

Decision Error Tolerances The goal of the planning team is to develop

a data collection design that reduces the chance of making a decision error to a tolerable level

Step 6 provides a mechanism for allowing the decision maker to define tolerable limits on the probability of making a decision error

7 of 48

Two Reasons Why Decision Makers Make Decision Errors

Sampling error occurs because the sampling design is unable to capture the complete extent of heterogeneity that exists in the true state of the environment

Measurement error occurs because analytical methods and instruments are not absolutely perfect

8 of 48

Where do errors occur?

Planning

Sampling

Analysis

Data Vs

Decision

9 of 48

Types of Decision Errors

Before we can talk about acceptable limits for making decision errors, we must first understand what correct decisions and decision errors look like

There are two types of correct decisions and two types of decision errors that can be made

10 of 48

Null Hypothesis:

Site is dirty. Site is dirty.Site is clean.

100

True State of Site

Alternative Action

Walk away from site. Clean up site.

75

Probability of deciding

that the site is dirty

0.0

0.5

1.0

Action LevelLower Bound of Gray Region

Typical Curve

Decision Performance

GoalDiagram

Sample MeanUCL

True Mean Sample

MeanUCL

Sample MeanUCL

True Mean

True Mean

11 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





In order to calculate the number of samples needed(in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable.

• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information

– Existing data– Process knowledge– Professional judgment

• Estimate of the population standard deviation– Reference source – Method of calculating


12 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





In order to calculate the number of samples needed(in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable.

• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information

– Existing data– Process knowledge– Professional judgment

• Estimate of the population standard deviation– Reference source – Method of calculating

Estimate the standard deviation by using the Deming approach of dividing the range by 2 or 3, depending on the frequency distribution.


13 of 48

Estimated Standard DeviationsEstimated Standard Deviations for NaI Example

Environmental VariableRange of

EnvironmentalVariable

Estimate of Population Standard DeviationDR#

Attribute Unit ofMeasure

LowerLimit

UpperLimit

StandardDeviation Source How Estimated?

1

True mean ofCs-137estimated by95% UCL ofsample mean

pCi/g 0.031 1.89 0.4052000HPGedata

From 116-H-1 CVP data

1

True mean ofCs-137estimated by95% UCL ofsample mean

cpm(convertedto pCi/g)

-5.17 99.92

7.48or

7.48/2=3.74

1999 to2000 NaIdata

From 116-H-1 CVP data

14 of 48

Estimated Standard DeviationsPerimeter Side-Slope Soil

CS

DR # EnvironmentalVariable

Range ofEnvironmental

Variable

Estimate of PopulationStandard Deviationa


LowerLimit

UpperLimit

StandardDeviation Source How

Estimated

Co-60 pCi/g 0.018U 0.49 0.1711

Cs-137 pCi/g 0.083 11.6 4.31

Eu-152 pCi/g 0.13 12.8 3.62

Sr-90 pCi/g 0.11U 5.8 1.56

RI/FSData

Computedfrom

RI/FS data

a The choice of an estimate of a standard deviation has a large impact on the number of samples required. Avoid underestimating thestandard deviation. Always be conservative when estimating the standard deviation.

U = Sample was undetected, value shown is the analytical detection limit.

15 of 48

Estimated Standard DeviationsTrench Footprint Soil

CS

DR # EnvironmentalVariable

Range ofEnvironmental

Variable

Estimate of PopulationStandard Deviationa


LowerLimit

UpperLimit

StandardDeviation Source How

Estimated

Cs-137 pCi/g 37.2 75 19.72

Pu-239/240 pCi/g 1.4 3 0.795

RI/FSData

Computedfrom RI/FS

dataa The choice of an estimate of a standard deviation has a large impact on the number of samples required. Avoid overly optimistic

estimates of the standard deviation. Always be conservative when estimating the standard deviation.U = Sample was undetected, value shown is the analytical detection limit.

16 of 48

Analytical +

Sub-sampling +

Natural heterogeneity of the site=

Total Uncertainty

Uncertainty is Additive!Remember the uncertainty is additive for

all steps in sampling and analysis

17 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5




Specify the boundaries of the gray regionDefine both types of decision error:Determine which one occurs above and which one occurs below the action level.

Two Types of Decision Error:• Cleaning up a clean site• Walking away from a dirty site


18 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





For each Alternative Action:• Create a list of possible decision error(s) that may occur if an action

is incorrectly taken• Discuss the consequences of making each decision error• Rate the severity of the consequences of a decision error (i.e.,

low, moderate, severe) at a point:– Far below the action level– Below but near the action level– Above but near the action level– Far above the action level

• Indicate which decision error has the most severe consequencenear the action level


19 of 48

Decision Error Consequences CS

Severity of Consequences of Decision Error When TrueParameter Is …

(Rated as Low/Moderate/Severe)

DR # AA # PossibleDecision Error

Consequencesof Decision

ErrorFar Belowthe Action

Level

Below ButNear the

Action Level

Above ButNear the

Action Level

Far Above theAction Level

Decision ErrorThat Has More

SevereConsequences

Near the ActionLevel

1: Conductremedialaction

Remediating anuncontaminatedsite

Expense andscheduleimpacts ofremediating anuncontaminatedsite

Moderate Moderate None None11

2: Take nofurtheraction

Failing toremediate acontaminatedsite

Leaving a sitein place theposes a threat tohuman healthand safety

None None Moderate Severe

Not remediating acontaminated site

1: Conductremedialaction

Remediating anuncontaminatedsite

Expense andscheduleimpacts ofremediating anuncontaminatedsite

Moderate Moderate None None22

2: Take nofurtheraction

Failing toremediate acontaminatedsite

Leaving a sitein place theposes a threat tohuman healthand safety

None None Moderate Severe

Not remediating acontaminated site

1 Applies to the perimeter side-slope soil around the –20 level of the trench footprint.2 Applies to the bottom of the trench footprint after excavation.

20 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5




Specify the boundaries of the gray regionProvide rationale for rating the severity of consequences as low or severe


21 of 48

Rationale for Error Consequence Ratings

CS

Table 3b. Justification for Rating Severity of Consequences as “Severe”

DS/DR

AA Justification

#1a,#2b

#2 The decision errors for not remediating both perimeter side slope andtrench footprint soils when the concentrations are far above the action levelare also rated as “severe.” In these cases, soil with concentrations that maypose serious health risks would remain at the site.

a) Applies to decision statements and rules related to perimeter side-slope soil afterexcavation

b) Applies to decision statements and rules related to trench footprint soil afterexcavation

22 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5




Specify the boundaries of the gray regionDefine the null hypothesis (baseline condition) and the alternative hypothesis:The decision error that has the most adverse potential consequences should bedefined as the null hypothesis.

The null hypothesis should state the OPPOSITE of what the project hopes to demonstrate.

• Site is assumed to be contaminated until shown to be clean• Site is assumed to be clean until shown to be contaminated


23 of 48

Null Hypothesis

DR Null Hypothesis Statement (examples) IndicateSelection

The site is contaminated. #1-2 a

The site is uncontaminated.a Applies to both perimeter side-slope soils and trench footprint

Contaminated:H0 : > Action Level

Uncontaminated:HA : < Action Level

CS

24 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





The gray region is a range of possible parametervalues within which the consequences of a decision errorare relatively minor.


25 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





The gray region is bounded on one side by the action level, and on the other side by the parametervalue where the consequences of decision error beginsto be significant. This point is labeled LBGR, whichstands for lower bound of the gray region.


26 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5

Determine the possible range of the parameter of interest

Choose the null hypothesis.

Identify the decision errors.





It is necessary to specify the gray region because variability in the population and unavoidable imprecision in the measurement system combine to produce variability in the data such that a decision maybe “too close to call” when the true parameter value is very near the action level.


27 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





Present the rationale of how the LBGR was calculated or determined.


28 of 48

Lower Bound of the Gray Region

Because the null hypothesis is that the site is contaminated, the upper bound of the gray region is set equal to the action level

The LBGR should be set at a value where the consequences of the decision error begin to be significant

29 of 48

How to Set the LBGR

LBGR = AL - (Analytical + Sampling Error) LBGR = AL - 1/2 Action Level LBGR = Decision-Maker “whim” - AL - 0.2 AL LBGR = Frequency Distribution method

30 of 48

The LBGR is often based on unavoidable variability in the concentration data– The LBGR may be estimated based on the

precision that the analytical methods allow plus an estimate as to the sampling variance

– LBGR = AL - (Analytical + Sampling Error) MARSSIM suggests the LBGR be set as:

– LBGR = AL - 1/2 AL

How to Set the LBGR (cont.)

31 of 48

The LBGR is often set at some other value– This is based on the decision makers’ choice and is not

scientifically based– LBGR = AL - (10 - 20% of AL)


32 of 48

Use the Frequency Distribution method– The LBGR may be estimated based the Probability

Distribution Function (PDF)– Place the Action Level on the mean of the PDF– Ask: “Does a substantial amount of contaminant

concentration values exceed the Action Level?”– If yes, begin moving the PDF backwards along the

x-axis towards zero concentration– Pause and ask again– When the answer is no, you have set the LBGR

(e.g., where the mean of the PDF lies on the x-axis is now the LBGR)

• Use probability theory to show/prove this


33 of 48

Show Probability Density Function Distribution

Demonstration

34 of 48

Methods for Evaluating the Attainment of Cleanup Standards - Volume 1: Soils and Solid Media

EPA, February 1989

PB89-234959

How to set the LBGR

212

1

211

25.0

Z

ALZZ

n

1 is a hypothetical “mean concentration where the site should be declared clean with a high probability”

35 of 48

“Normal” FD

“Skewed” FD

Computer Simulations: “Badly skewed” or Any FD• Evaluate and errors to select n

Using the LBGR to Estimate n

212

1

211

25.0

Z

ALZZ

n

212

1

211

25.016.1

ZAL

ZZn

36 of 48

Example 1 Balancing the width

of the gray region controls the cost of sampling and analysis

The closer the LBGR lies to the action level, typically the greater the number of samples, and thus the cost increases

Mean = LBGR AL

concentration

Width of the Gray Region

Mean = LBGR AL

concentration


37 of 48


GR = Analytical + Sampling Error– Estimated based on past data and general knowledge

GR = 1/2 of the AL – For each COPC, calculate and set LBGR

GR = 20% of the AL – For each COPC, calculate and set LBGR

GR = PDF method– Use PDF for worst COPC to set LBGR

38 of 48

Setting the GR Based on the (Analytical + Sampling Error)

CS

Environmental Variable

DR # Attribute Unit ofMeasure

ActionLevel LBGRa Gray

Region

Co-60 pCi/g 1.4 1.23 0.17

Cs-137 pCi/g 6.2 1.89 4.3

Eu-152 pCi/g 3.3 -0.32 3.62

1

Sr-90 pCi/g 4.5 2.94 1.56

Cs-137 pCi/g 500,000 500,000 19.82

Pu-239/240 pCi/g 718,600 718,000 0.795a For DR #1,2 the LBGR = AL – total error. Standard deviation of RI/FS sampling data was used

as the total error. Avoid overly optimistic estimates of the standard deviation. Always beconservative when estimating the standard deviation.

39 of 48

Setting the GR Based on Regulator Input (20% of the AL)

CS




Region

Co-60 pCi/g 1.4 1.11 0.28

Cs-137 pCi/g 6.2 5.0 1.24

Eu-152 pCi/g 3.3 2.64 0.66

1

Sr-90 pCi/g 4.5 3.6 0.9

Cs-137 pCi/g 500,000 400,000 100,0002

Pu-239/240 pCi/g 718,600 574,880 144,000a For DR #1,2 LBGR = AL – 80% of the AL. Avoid overly optimistic estimates of the standard

deviation. Always be conservative when estimating the standard deviation.

40 of 48

Setting the GR Based on ½ of the Action Level

CS




Region

Co-60 pCi/g 1.4 0.7 0.7

Cs-137 pCi/g 6.2 3.1 3.1

Eu-152 pCi/g 3.3 1.65 1.65

1

Sr-90 pCi/g 4.5 2.25 2.25

Cs-137 pCi/g 500,000 250,000 250,0002

Pu-239/240 pCi/g 718,600 359,300 359,300a For DR #1,2 LBGR = AL - 50% of the AL. Avoid overly optimistic estimates of the

standard deviation. Always be conservative when estimating the standard deviation.

41 of 48

Setting the GR Based on the PDF Method

CS



ActionLevel LBGR Gray Region

Co-60 pCi/g 1.4 TBD: FAM/DWP TBD: FAM/DWP

Cs-137 pCi/g 6.2 TBD: FAM/DWP TBD: FAM/DWP

Eu-152 pCi/g 3.3 TBD: FAM/DWP TBD: FAM/DWP

1

Sr-90 pCi/g 4.5 TBD: FAM/DWP TBD: FAM/DWP

Cs-137 pCi/g 500,000 TBD: FAM/DWP TBD: FAM/DWP2

Pu-239/240 pCi/g 718,600 TBD: FAM/DWP TBD: FAM/DWP

42 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





Assign probability values that reflect the decision maker’s tolerable limits for making an incorrect decision.

• At the action level• At the other bound of the gray region• At a point far below the action level• At a point far above the action level


43 of 48

Site is dirty.Site is clean.

100

True State of Site

Alternative Action

Walk away from site. Clean up site.

75

Probability of deciding

that the site is dirty

0.0

0.5

1.0

Action LevelLower Bound of Gray Region

Typical Curve

Null Hypothesis:

Site is dirty.

44 of 48

Decision Error Consequences CS

Probability of Deciding the Site is Dirty When the Mean is…DR #Far Belowthe Action

Level

At the LBGR At the ActionLevel

Far Above theAction Level

11, 22 0%Error = 0.0

20%Type II (beta)Error = 0.2

95%Type I (alpha)Error = 0.05

100%Error = 0.0

1 Applies to the perimeter side-slope soil around the –20 ft level of the trench footprint.2 Applies to the bottom of the trench footprint after excavation.

45 of 48

Step 6 Summary Define the two types of error

– Incorrectly cleaning a dirty site or– Incorrectly cleaning a clean site

Evaluate severity of the incorrect decisions both below, above, and near the action level

Select the null hypothesis Specify the error rates decision makers are willing

to accept and provide rational for the rates

46 of 48

Establish a LBGR based on one of the four methods shown previously

Step 6 Summary

Provide the basis for selecting the LBGR Remember the closer the LBGR is to the action level,

the more samples are needed More samples can mean real-time measurements or

traditional laboratory measurements Assign probability limits on either side of the gray

region

47 of 48

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5






48 of 48

End of Module 8

Thank you

Documents

1 of 48 The EPA 7-Step DQO Process Step 6 - Specify Error Tolerances 3:00 PM - 3:30 PM (30 minutes)…