Evaluating Stormwater Technology Performance …...Eric Winkler, Ph.D. University of Massachusetts 160 Governors Drive Amherst, MA 01003 (413) 545-2853 (Voice) (413) 545-1027 (Fax)

1

University of Massachusetts, Amherst, © 20041

Evaluating Stormwater Technology PerformanceModule II

Guidance for the Technology Acceptance Reciprocity Partnership (TARP) Stormwater Protocol: Stormwater Best Management Practice Demonstrations

October 2004

Prepared byEric Winkler Ph.D. and Nicholas Bouthilette

Center for Energy Efficiency and Renewable EnergyUniversity of Massachusetts – Amherst

Stormwater pollution, especially in developed urban areas is a leading cause of water quality degradation in U.S. rivers, lakes, streams, and other surface waters. Water quality problems associated with nonpoint sources of pollution, particularly stormwater, are being addressed by federal mandates that affect all states. The National Pollutant Discharge Elimination System (NPDES) Phase II, Storm Water Regulations requires stormwater plans from thousands of municipalities nationwide, and there is a renewed focus nationally on the total maximum daily load provisions (TMDL) in the Clean Water Act. These programs are predicted to have a significant influence on the rate at which new technologies enter the marketplace, as they bring more attention and increased resources to stormwater control issues.

As new products are added to the marketplace list of stormwater best management practices, there is a growing demand for good data and information on these technologies. To address this issue, the Technology Acceptance and Reciprocity Partnership (TARP) states of CA, MA, MD, NJ, PA, VA, and WA formed the Stormwater Work Group to layout a cost-effective path for evaluating innovative stormwater technologies by producing scientifically credible data, as well as useful information, which demonstrates how to apply the technology in the field to manage stormwater runoff. To support responsible use of stormwater technologies, the TARP Stormwater Work Group developed two Internet-based training modules. The two modules cover aspects of stormwater technology evaluation and are supported by the “TARP Protocol for Stormwater Best Management Practice Demonstrations.”Module I: Planning for a Stormwater BMP DemonstrationFactors Affecting Stormwater SamplingData Quality Objectives and the Test QA Plan Module II: Collecting and Analyzing Stormwater BMP Data Statistical AnalysesSampling DesignData Adequacy: Case Study

The purpose of the Protocol is to provide a uniform method for demonstrating stormwater technologies and developing test quality assurance (QA) plans for certification or verification of performance claims. In this training course you will learn:

to use the Protocol to identify gaps and inconsistencies in a test plan for evaluation of a stormwater technology and understand differences in state requirements;to recognize data deficiencies;to develop, implement, and review a test plan;to understand, evaluate, and implement statistical methods.

The goal of this training course is to encourage collaboration, data sharing and reciprocal review amongst states and practitioners, while providing a uniform method for demonstrating stormwater technologies and developing Test Quality Assurance (QA) Plans. As a result of these efforts there will be an increase in expertise for assessment of stormwater technology performance and a reduction decision-making timeframes for technology implementation.For access to the training modules:(1) http://www.clu-in.org/conf/tarp/stormwater/ (2) For more information on TARP and to download a copy of the TARP Protocol for Stormwater Best Management Practice Demonstrations, please visit: http://www.dep.state.pa.us/dep/deputate/pollprev/techservices/tarp/. EPA Office of Superfund Remediation and Technology Innovation http://clu-in.org/studio/ECOS – Environmental Council of States http://www.sso.org/ecos/CEIT - Center for Environmental Industry & Technology, U S EPA New England http://www.epa.gov/ne/assistance/ceit/CEERE – Center for Energy Efficiency and Renewable Energy, University of Massachusetts http://www.ceere.org/

2


Meet the Instructors

Nancy BakerMassachusetts Dept. of Environmental Protection1 Winter StreetBoston, MA 02108(617) 654-6524 (Voice)(617)292-5850 (Fax)[email protected]

Eric Winkler, Ph.D.University of Massachusetts160 Governors DriveAmherst, MA 01003(413) 545-2853 (Voice)(413) 545-1027 (Fax)[email protected]

Instructor Biographies:Nancy Baker is an environmental analyst in the Massachusetts Department of Environmental Protection, Northeast Regional Office, where she provides oversight and commentary on stormwater issues and coordinates the region’s reviews of development projects that require permits from the Department. Nancy has chaired the Technology Acceptance Reciprocity Partnership Stormwater Work Group since it’s inception in 1999 and has assisted with the Massachusetts Strategic Envirotechnology Partnership (STEP) evaluation of stormwater technologies, including the preparation of fact sheets, which complement the STEP verification reports.Previously, Nancy supervised the Department’s staff in Outreach Services, which produced publications and training materials, including the Massachusetts Stormwater Management Handbooks. She formerly was an analyst with the Massachusetts DEP Drinking Water Program and the Massachusetts Environmental Policy Act (MEPA) Unit in the Executive Office of Environmental Affairs. Nancy has a B.S. degree in biology, an M.S. degree in marine biology, and enjoys landscape gardening and gourmet cooking.

Eric Winkler, Ph.D. is Director of Technical Services in the Center for Energy Efficiency and Renewable Energy (CEERE) at the University of Massachusetts Amherst. He is primarily responsible for outreach and technical support services on energy and environmental systems to public and private sector clients. He is also Director of the Massachusetts Energy Efficiency Partnership leading state and region wide initiatives with industry, public utilities and state agencies on energy and resource conservation. Dr. Winkler is part of the implementation team for two other DOE funded programs at CEERE, the Industrial Assessment Center and the Northeast CHP Application Center.

Dr. Winkler also directs a university based research and development program for innovative energy and environmental technologies. He was technical lead for the Massachusetts Executive Office of Environmental Affairs/UMass Strategic Envirotechnology Partnership focusing on technology validation of environmental and energy technologies. Dr. Winkler has served on state and federal environmental advisory review panels, including the MA DEP Storm Water Advisory Panel and the EPA ETV Washington DOT Project Review Panel. Dr. Winkler has been Principal Investigator on a number of US EPA S319 projects, most recently leading a project to develop a clearing house of information on proprietary stormwater technologies and a prioritization process for investment in technology demonstrations and testing. He has published numerous reports and journal articles on innovative wastewater and stormwater technologies. Dr. Winkler received his Doctorate in Environmental Chemistry from the University of Massachusetts Amherst in 1995 and holds Masters Degrees in Public Health and Soil Science.

3


Meet the Sponsors

TARP Stormwater Work Group

CaliforniaMarylandMassachusettsNew JerseyPennsylvaniaVirginia

State of Washington, Illinois, New York, and ETV are collaborating with TARP

TARP Member State

TARP – Technology Acceptance Reciprocity Partnershiphttp://www.dep.state.pa.us/dep/deputate/pollprev/techservices/tarp/A state tool to promote scientifically sound, cost-effective environmental decision making. Information on the performance of new technologies is critical to state environmental protection efforts. Regulatory standards and permits often rely on technology performance data. Unfortunately, few, if any, standardized methods have been established to guide the collection and evaluation of technology performance across the states. And, while states look to new environmental technologies as cost-effective opportunities to achieve better environmental performance, reliable performance information is hard to find. As a result, new technologies often face unnecessary and financially burdensome regulatory and permit hurdles that slow down or prevent their use. That is why TARP - The Technology Acceptance and Reciprocity Partnership - was formed by the states of California, Illinois, Maryland, Massachusetts, New Jersey, New York, Pennsylvania, and Virginia

4


Goals of TARP Stormwater Work Group

Use protocol to test new BMPsApprove effective new stormwater BMPsGet credible data on BMP effectivenessShare information and dataIncrease expertise on new BMPsUse protocol for appropriate state initiatives

TARP Member State

No associated notes. For more information on TARP and to download a copy of the TARP Protocol for Stormwater Best Management Practice Demonstrations, please visit: http://www.dep.state.pa.us/dep/deputate/pollprev/techservices/tarp/.

5


Evaluating Stormwater Technology Performance

Learning ObjectivesUse the TARP Stormwater Demonstration Protocol to review a test plan and a technology evaluationRecognize data gaps and deficienciesDevelop, implement, and review a test planUnderstand, evaluate, and use statistical methods

Logistical RemindersPhone audience

Keep phone on mute* 6 to mute your phone and again to un-muteDo NOT put call on hold

Simulcast audience

Use at top of each slide to

submit questions

Course time = 2 hours

3 question & answer periods

Links to additional resources

Your feedback

For more information on TARP and to download a copy of the TARP Protocol for Stormwater Best Management Practice Demonstrations, please visit: http://www.dep.state.pa.us/dep/deputate/pollprev/techservices/tarp/.

To access training materials on-line: http://www.clu-in.org/conf/tarp/stormwater/

6


Project NoticePrepared by The Center for Energy Efficiency and Renewable Energy, University of Massachusetts Amherst for submission under Agreement with the Environmental Council of States. The preparation of this training material was financed in part by funds provided by the Environmental Council of States (ECOS).

This product may be duplicated for personal and government use and is protected under copyright laws for the purpose of author attribution.

“Publication of this document shall not be construed as endorsement of the views expressed therein by the Environmental Council of States/ITRC or any federal agency."


7


DISCLAIMER

The contents and views expressed are those of the authors and do not necessarily reflect the views and policies of the Commonwealth of Massachusetts its agencies or the University of Massachusetts. The contents of this training are offered as guidance. The University of Massachusetts and all technical sources referenced herein do not (a) make any warranty or representation, expressed or implied, with respect to accuracy, completeness, or usefulness of the information contained in this training, or that the use of any information, apparatus, method, or process disclosed in this report may not infringe on privately owned rights; (b) assume any liabilities with respect to the use of, or for damages resulting from the use of, any information, apparatus, method or process disclosed in this report. Mention or images of trade names or commercial products does not constitute endorsement or recommendation of use.


8


Module I: Planning for A Stormwater BMP Demonstration

1. Factors Affecting Stormwater Sampling2. Data Quality Objectives and the Test QA Plan

3. Sampling Design4. Statistical Analyses5. Data Adequacy: Case Study

Module II: Collecting and AnalyzingStormwater BMP Data


9


3. Sampling Design

Stormwater data collection guidance

Locating samples and stations

Selecting water quality parameters

QA/QCSample handling and record keepingField measures


10


Sampling Plan Elements TARP Data Collection Criteria

(Section 3.3, TARP Protocol)

Any relevant historic dataMonthly mean rainfall and snowfall data (12 months over the period of record)Rainfall intensity over 15 minute increments

These elements are specified in the TARP as required components of the sampling plan.

11


Protocol Minimum Criteria Identifying Qualifying Storm Event

(Section 3.3.1.2 and Section 3.3.1.3, TARP Protocol)

Minimum rainfall event depth is 0.1 inchMinimum inter-event duration of 6 hours (duration beginning a cessation of flow to unit)Base flow should not be sampled

Identification of qualifying event needs to verify flow to the unit and rainfall concurrently

Protocol specifies a minimum of a 6 hour inter-event period. Other national protocols (WASHDOT) may specify longer periods, up to 72 hours.Inter-event period is a value is based on a reasonable time in which pollutant buildup may occur.Sampling first flush, may be more important in capturing certain pollutants. Sampling long duration storms or storms that are closely link may not produce useful information. Also, consider concentration effects on treatment potential and sampling long durations storms.

12


Analysis for Determining Number of Samples

Paired sampling approach-Significant difference in mean concentration

Coefficient of variationPower of 80%

95% confidence(Urban Stormwater BMP Performance Monitoring. USEPA, 2002.)

Coefficient of Variation

Diff

eren

ce in

sam

ple

set

mea

ns (

%)

Example: 80% difference in means, 100% CV = 20 sample pairs

Higher variability between sample means (CV) will require more sample pairs to get the same level of confidence for the difference between means.Smaller differences between paired samples will require more sample pairs to get the same level of confidence for significant differences.

13


Qualifying Event Sample TARP Protocol Criteria

10 water quality samples per event10 influent and 10 effluentIf composite - 2 composites, 5 sub-samples

Data for flow rate and flow volumeAt least 50% of the total annual rainfall

CA – monitor 80-90% of rainfall

Sampling criteria.

14


Qualifying Event Sample (continued)

Preferably 20 storms, 15 minimumSampling over the course of a full year of sampling to account for seasonal variationCompositing flow-weighted samples cover at least 70% of storm flow (and as much of the first 20% as possible)Examples of variation within TARP community

PA - Temporary BMPs sized using 2 year eventNJ – Water Quality design based on volume from a 1.25 inch event

These requirements may vary from region to region to rainfall characteristics.

15


Sampling Locations

Location in close proximity to BMP technology inlet and outletConsider field personnel safety during equipment accessSecure equipment to avoid vandalismProvide a scaled plan view of site that indicates

All buildingsLand usesStorm drain inletsOther control devices

•Sampling can occur in catch basin, which offers a well mixed environment.•Also can sample within BMP structure, when sampling equipment will not interfere with operation of unit and will not be biased by location to close to treatment section.•This is often a good choice due to the convenience of building in testing structures or placement of test equipment.

16


Automated Sampling Equipment

Water surface elevation should be less than 15 feet below the elevation of the pump in the sampler

Access to the equipment should take into consideration confined space safety

Flow measurement equipment should be located to avoid measuring turbulent flow

Area velocity meter placement is typically position in a location 10 times the diameter of the pipe from any point in which turbulence occurs (a bend, inlet, tee). Grab sampling or manual sampling may be needed for certain pollutants including:pH, Temperature, Cyanide, Total phenols, Residual chlorine, Oil and grease, E. coli, Total coliform, Fecal coliform, Fecal streptococci, Enterococci, Total petroleum hydrocarbons

17


Adequacy of Sampling and Flow Monitoring Procedures

Primary and secondary flow measurement devices are requiredProgrammable automatic flow samplers with continuous flow measurements are recommendedTime-weighted composite samples are not acceptable unless flow is monitored and the event mean concentration can be calculated


18


Monitoring Location Recommendations

Monitoring location design should consider whether the upgradient catchment system is served by a separate storm drain systemPay attention to potential combined sewer system or illicit connections which may contaminate stormwater systemThe storm drain system should be well understood to allow reliable delineation and description of catchment areaFlow-measuring monitoring stations in open channels should have suitable hydraulic control and where possible the ability to install primary flow measurement devices


19


Monitoring Location Recommendations (continued)

Avoid steep slopes, pipe diameter changes, junctions, and irregular channel shaping (due to breaks, roots, debris)Avoid locations likely to be affected by backwater and tidal conditionsPipe, culvert, or tunnel stationing should be located to avoid surcharging (pressure flow) over the normal range of precipitationUse of reference watershed and remote rainfall data are discouraged


20


Selecting Applicable Water Quality Parameters

Designated uses of the receiving water – consider stormwater discharge constituentsOverall program objectives and resources – adjust parameter list according to resources (test method capability, personnel, funds, and time)Use of “Keystone” pollutant may vary from state to state

It should be noted that some State to State variability exists in this area.Reciprocity between states may be compromised if the study design does not include consideration of other state’s water quality criteria.VA uses P as Keystone pollutant parameter.

21


Resources for Standardized Test Methods (Section 3.1, TARP Protocol)

EPA Test Methods – pollutant analysis www.epa.gov/epahome/Standards.htmlASME Standards and Practices – pressure flow measurementsASCE Standards – hydraulic flow estimationASTM Standards – precision open-channel flow measurements for water constituent analysisNational Field Manual for Collection of Water Quality Data, Wilde et al., USGS water.usgs.gov/owq/FieldManual/“Guidance Manual: Stormwater Monitoring Protocols” –Caltrans. http://www.dot.ca.gov/hq/env/stormwater/special/newsetup/index.htm#monitoring


22


Examples of Analytical Laboratory Methods

Total Phosphorus – SM4500-PENitrate and Nitrite – EPA353.1Ammonia – EPA350.1Total Kjeldahl Nitrogen – EPA351.2TSS – SM2540DSSC – ASTM D3977-97Enterococci – SM9230CFecal Coliform – SM9222DChronic Microtox Toxicity Test – Azur Environmental

Reference: ASTM, EPA, American Public Health Association, (Other non-standard methods or tests approved through acceptable regulatory process, EPA or state authority)

The process of choosing a method is dependent upon the method sensitivity and the data quality objectives. In addition field conditions including predicted pollutant concentrations may impact the choice of sample procedures, including interferences from other dissolved or suspended constituents.Furthermore, as is generally a part of “good” study design, consider the capability of available resources to provide testing methods (e.g. some labs cannot meet sensitivity requirements specified by EPA even if they physically can perform the test procedure).

23


Constituent Listing with Detection Limits

Urban Stormwater BMP Performance Monitoring. US EPA, 2002.


24


Representative Sampling

Sample CollectionUniformity varies between staff and techniqueAutomated sampling allows for reproducible samplesStandard Operating Procedures (SOPs) and Quality Assurance help track variability


25


Required Elements in Lab QA/QC (Section 3.3.5, TARP Protocol)

The QAPP Test QA Plan and Sampling and Analysis Plan should include

Laboratory and sampling equipment decontaminationSample preservation and holding timeSample volumesQC samples (spikes, blanks, splits, field and blank lab duplicates)QA on sampling equipment (calibration)Packaging and shippingIdentification and labelingChain of custodyLab certification

QA on equipment needs to include: (Calibration of Automatic Samplers and Flow Measurement Devices)

Certified by a state or federal agency regulating laboratory certification or accreditation programs.

NELAC or ELAP Program used to perform standardized test methods and procedures

26


Quality Control Tests and Acceptance Limits for Physical/Chemical Parameters Analyzed in the Laboratory

Each sample batch of ≤ 20

samples (≥5%)≤ 25Duplicates


samples (≥5%)75-125% rec.QCS

Total Suspended Solids




samples (≥5%)75-125% rec.QCS

Suspended Sediment

Concentration

80-120% rec.LFM

85-115% rec.LFBEach sample batch of ≤ 20

samples (≥5%)≤28Duplicates


samples (≥5%)

90-110% rec.QCS

Total Kjeldahl N

80-120% rec.LFM

85-115% rec.LFBEach sample batch of ≤ 20



samples (≥5%)

90-110% rec.QCS

Ammonia-N

80-120% rec.LFM

85-115% rec.LFB

Each sample batch of ≤20 samples

(≥ 5%)

≤45DuplicatesEach sample batch of ≤ 20

samples (≥5%)

90-110% rec.QCSNitrate and Nitrite

N

80-120% rec.LFMd

85-115% rec.LFBcEach sample batch of ≤20

samples (≥5%)≤20Duplicates


samples (≥5%)

90-110% rec.QCSb

Total Phosphorus

FrequencyAcceptance

Limits(RPD)

QC TestFrequencyAcceptance

Limits (RPD)a

QC Test

PrecisionAccuracy

Parameter

aRPD = relative percent difference among duplicates cLFB= laboratory fortified blank samplebQCS = quality control sample from source outside of laboratory dLFM = laboratory fortified matrix sample


27


Laboratory Records

Sample dataNumber of samples, holding times, location, deviation from SOP’s, time of day, and dateCorrective procedures for samples inconsistent with the protocol

Management of recordsConsider electronic filingDocument data validation, calculations and analysis, and data presentationReview data reports for completeness, including requested analyses and all required QA


28


Field Sample QC

Field matrix spikeA sample prepared at the sampling point by adding a known mass of the target analyte to a specified amount of sampleUsed to determine the effect of sample preservation, shipment, storage, and preparation on analyte recovery efficiency

Field split sampleThe split of a sample into two representative portions to be sent off to different laboratoriesEstimates inter-laboratory precision


29


Quality Control

Field blanksA clean sample carried to the sampling site, exposed to sampling conditions, and returned to the laboratoryProvides useful information about pollutants and error that may be introduced during the sampling process

Field duplicatesIdentical samples taken from the same sampling location and time (not split)Identical sampling and analytical procedures to assess variance of sampling and analysis

Some issues regarding applicability of field duplicates with respect to realistically being able to perform such a task.Can you really get a valid duplicate?

30


Quality Assurance / Quality ControlDocumentation and Records

DocumentationInclude all QC dataDefine critical records and information, as well as the data reporting format and document control procedures

ReportingField operation recordsLaboratory recordsData handling records


31


Field Operation Records

Sample collection recordsShow that the proper sampling protocol was used in the field by indicating persons names, sample numbers, collection points, maps, equipment/methods, climatic conditions, and unusual observations

Chain of custody recordsDocument the progression of samples as they travel from samplinglocation to the lab to disposal area

QC sample recordsDocument QC samples such as field, trip, blanks, and duplicate samplesProvide information on frequency, conditions, level of standards, and instrument calibration history


32


Field Operation Records (continued)

General field proceduresRecord field procedures and areas of difficulty in gathering samples

Corrective action reportsWhere deviations of standard operating procedures occurs, report methods used and/or details of procedure, and a plan to resolve noncompliance issues


33


Questions and Answers (1 of 3)


34







35


4. Statistical Analyses

Data reporting and presentationStatistical method reviewAppropriate forms of analysesResults interpretation


36


Presenting Statistical Data and Applicability Efficiency Calculation Implications

Determine the category of BMPBMPs with well-defined inlets and outlets whose primary treatment depends upon extended detention storage of stormwaterBMPs with well-defined inlets and outlets that do not depend upon significant storage of waterBMPs that do not have a well defined inlet and/or outletWidely distributed BMPs that use reference watersheds to evaluate effectiveness


37


Analysis to Address Data Quality Objectives

What degree of pollution control does the BMP provide under typical operating conditions?How does efficiency vary from pollutant to pollutant?How does efficiency vary with input concentrations?How does efficiency vary with storm characteristics?How do design variables affect performance?How does efficiency vary with different operational and/or maintenance approaches?

Idealized questions one may ask.

38


Analysis to Address Data Quality Objectives (Continued)

Does efficiency improve, decay, or remain stable over time?How does efficiency, performance, and effectiveness compare to other BMPs?Does the BMP reduce toxicity?Does the BMP cause an improvement or protect downstream biotic communities?Does the BMP have potential downstream negative impacts?


39


Recommended Statistical Methods for Performance Evaluation

Efficiency ratioSummation of loadsRegression of loadsMean concentrationEfficiency of individual storm loads Lines of Comparative Performance Method


40


Efficiency Ratio (ER)TARP Protocol Recommended Method

Where Event Mean Concentration (EMC):

V=volume of flow during period in=total number of events C=average concentration associated

with period jm=number of events measured

ER = 1 - = Average Outlet EMCAverage Inlet EMC

Average Inlet EMC - Average Outlet EMCAverage Inlet EMC

EMC =∑ CjVjj=1

n

∑ Vjj=1

n


41


Efficiency Ratio Interpretation

EMCs weight all storms equallyMost useful when loads are directly proportional to the relative magnitude of the storm – accuracy varies with BMP typeMinimizes impacts of smaller/cleaner stormsAllows for use of data where portions of data are missing –would not significantly effect the average EMC

Can apply log normalization to avoid equal weighting of events

Mean of the Log EMCs =∑ Log(EMCj)j=1

m

m

Can also apply log normalization to avoid equal weighting of events.

42


Summation of Loads

The sum of loads can be calculated using concentration and flow volume, as follows:

Removal efficiencies calculated by the summation of loads tend to be dominated by larger storm events

The summation of loads method uses a mass balance approach

SOL = 1 -Sum or outlet loadsSum of inlet loads

Sum of Loads = ∑(∑ CiVi) = ∑ EMCj*Vjj=1

m

j=1

m

i=1

n

Removal efficiencies calculated by the summation of loads tend to be dominated by larger storm events. The summation of loads method uses a mass balance approach.

43


Regression of Loads

The Regression of Loads is defined as

Percent reduction in loads is approximated as

β - slope term in regression analysis

Data can be dominated by large storm eventsIt is recommended that the line not be forced through originMay require polynomial fit to achieve higher R2

Loads out = ß * Loads in = ß -Loads outLoads in

Percent Removal = 1- ß = 1 -Loads outLoads in

Data can be dominated by large storm events.It is recommended that the line not be forced through origin.May require polynomial fit to achieve higher R2

44


Mean Concentration

The Mean Concentration is defined as:

May be useful for threshold level pollutants, e.g., bacterial or toxicsWeights samples equally and may result in bias due to variances in

sampling protocolsNot amenable to mass balance approachFlow measure must represent total event characteristics

MC = 1 -Average of outlet concentrationAverage of inlet concentration

May be useful for threshold level pollutants, e.g., bacterial or toxics.Weights samples equally and may result in bias due to variances in sampling protocols.Not amenable to mass balance approach.Flow measure must represent total event characteristics

45


Efficiency of Individual Storm Loads

The efficiency of the BMP for a single storm is given by:

Average efficiency can be calculated as follows:

Weights all storms equallyMust have paired data for inflow and outflowDoes not account for interrelationship between storm events

Storm Efficiency = 1 -Loadout

Loadin

Average Efficiency =m

∑ Storm Efficiencyjj=1

m

Weights all storms equally.Must have paired data for inflow and outflow.Does not account for interrelationship between storm events.

46


Interpreting Results and Inappropriate Analyses

Efficiency Ratio – Most useful when loads are directly proportional to the relative magnitude of the storm – accuracy varies with BMP typeSummation of Loads – A small number of large storms can significantly influence resultsRegression of Loads – Assumes removal efficiency is uniform over a range of operating conditions and concentrationsMean Concentration – Not appropriate where flow-weighted sampling is performed – weighs all events equally


47


Using Statistics Inappropriately

Each method is likely to produce a different efficiencyMethod should be chosen by its relevance and applicability to each scenario, not by the efficiency value it producesBe aware of statistics being misused to support claimsReporting of ranges may be more appropriate under certain test conditions, e.g., determination that data is qualitative versus quantitative


48




49







50


5. Case Study I: Test Plan and Data Adequacy

Review of dataTest QA Plan and Data Quality Assurance Project PlanField and lab data adequacyData reporting and depiction of performance claims

Potential problemsField and lab violationsData reduction issues


51


Background

Study of structural BMPPerformance claim made – 80% TSS removalStudy commenced prior to TARP Protocol as well as other published, public domain stormwater BMP monitoring protocolsResources for conducting study borne by single private entityStudy design included limited information on site, sampling equipment, sampler programming, calibration, sample collection and analysisTSS/SSC primary water quality parameter of interestFlow measurement and rain gauge equipment to be installedAnalytical testing to be performed by outside laboratorySample collection by technology developer


52


Sample Plan Adequacy – Issues Identified

Provide complete engineering drawings of system includingEntire drainage area connected to systemPipe sizing and inlet locationsDescription of pervious and impervious surfacesDesign calculations used to size unitClimatic data used to design structuresAny additional structures or site details that may have bearing on system

Provide use characteristics of site includingVehicle and industrial usageMaintenance of site relative to sweeping, gutter maintenance, spill containment, and snow stockpiling/disposal

Indicate condition and maintenance of system prior to commencing test, e.g., was system cleaned out during or before initiation of this phase of the study?


53


Sample Plan Adequacy –Test Equipment and Sampling Issues

Provide a description and/or reference to manufacturer’s documentation showing how flow meter and sampling equipment is calibrated and their location in the system (cross-section and plan view)Detailed description of sampling program by design and storm eventIndicate when and how much sample to be takenProvide explanation and methodology for discreet sampling. Indicate who, when, and where in the system these samples were takenIdentify equipment used and calibrations used for samplingProvide discussion/explanation for sampling without detention delay in the system between inlet and outlet


54


Sample Plan Adequacy –Lab and Data Analysis Issues

Identify laboratory and controls for sampling handling, QA/QC on sample dataIdentify methodology used for PSD, including sieve sizes and sample sizesAnalytical method for TSS stated in protocol - EPA 160.2Need explanation and equations used to calculate EMC, including calculation worksheets


55


Field Test Issues

Flow meters not calibrated for the first 10 eventsTruncated sampling protocolNo documented instrument calibrationOnce flow measurement error was identified, an adjusted flow factor is applied to the first 10 events, based on the outcome of the second 10 eventsMeasured and calculated storm volumes vary by up to a factor of 10 resulting in adjustment to flow volume


56


Data Analysis Issue

Values in italics are untransformed flow values in ft3

63 %70.3188.4377,7530.302063 %2157.4326,6140.2519

73 %25.593.3336,8580.2718

56 %102.4233.68029,3781.0217

43 %93.2164.14071,6071.9116

62 %12.633.1409,5430.4315

79 %14.769.2122,9080.1314

57 %19.846.27013,2100.5313

58 %47.3111.6401,2840.4812

33 %59.188.8123147,5865.4511

38 %3861124,559 (608)0.1710

10 %33.637.262,183 (291)0.109

95 %521088.84018,045 (2406)0.758

28 %31433014,441 (1925)0.557

89 %57.8533.2186,831 (911)0.466

60 %145.9367.6119,630 (1284)0.535

94 %49.4857.61313,203 (1760)0.324

95 %63.61364.9129,117 (1216)0.323

85 %149.21010.71712,051 (1607)0.522

24 %50.365.92821,600 (2880)0.31

Event Removal Efficiency1

Effluent EMC(mg/L)

Influent EMC(mg/L)

Number of Sub SamplesFlow Volume M (ft3)1

Rainfall Depth(in)Storm

What is notable about this data set is that flows and rainfall depth do not correlate. See next slide.The entity analyzing this data modified the flows based on rainfall depths as they are correlated with events 11 thru 20.While this modification was noted, it may not be justified and may impact EMC values used to calculate performance efficiency.

57


Predicted / Measured Runoff Values

0

5,000

10,000

15,000

20,000

25,000

30,000

0 0.5 1 1.5 2

Adjusted Runoff

Measured Runoff

95% Conf +

95% Conf -

Predicted Runoff

Linear (95% Conf +)

Linear (95% Conf -)

The values highlighted in red indicate that the relationship between rainfall and total runoff is inconsistent

Applying the rational method (Q=CiA) leads reviewers to believe that measured runoff is inaccurate when compared to predicted values

Runo

ff (

ft3 )

Precipitation (inches)

Illustrates an analysis of data which demonstrates that flow measurement equipment was not properly calibrated.

58


Variation in Performance Values

59%Removal Efficiency by Efficiency of Individual Storm Events:

40%Removal Efficiency by Regression of Loads:

44%Removal Efficiency by Summation of Loads:

57%Removal Efficiency by Efficiency Ratio:

Removal Efficiencies for Events 11-20:

60%Removal Efficiency by Efficiency of Individual Storm Events:

77%Removal Efficiency by Regression of Loads:

73%Removal Efficiency by Summation of Loads:

83%Removal Efficiency by Efficiency Ratio:

Removal Efficiencies for all Events:

Using various statistical methods, a comparison of TSS removal efficiencies between all events and those where the flow measures are more representative of rainfall, demonstrates significant divergence of performance values.Analysis also demonstrates that efficiency measures by storm is not affected by errors in flow measurement.

59


Data Analysis and Presentation Issues

Impact of adjusted flow on average net TSS/SSC removal resulted in positive bias in performance efficiencyMissing raw data including documented deviations from sampling plan, lab analysis and data managementMissing laboratory and data management QA/QCNo independent validation of calibrations, sampling and analysisMissing statistical analysis including relative percent differences (precision) and percent recovery (accuracy) as well as number of sample (n), standard deviations, means (specify arithmetic or geometric) and standard errorsMissing documentation of chain of custody protocol


60


Ensuring Adequate Data

Provide QC activities including blanks, duplicates, matrix spikes, lab control samples, surrogates, or second column confirmationState the frequency of analysis and the spike compound sources and levelsState required control limits for each QC activity and specify corrective actions and effectiveness if limits exceeded


61


Summary of Data Quality Assessment

5 Steps1. Review the DQOs and sampling design2. Conduct a preliminary data review3. Select the statistical test4. Verify the assumptions of the statistical test5. Draw conclusions from the data


62


Summary: Sampling Design

Use of TARP Protocol data collection criteriaConsideration of issues relating to sampling locationsSampling water quality parametersLab analysis of water quality parameters


63


Summary: Statistical Analysis

Protocol recommends the efficiency ratioSummation of loads, regression of loads, and mean concentration may be applicable Each method gives somewhat different resultsCheck statistics to be sure they support claims


64


Summary: Case Study

Learn to recognize limitations in test plans, equipment data and documentation deficiencies, and problems with the statistical analysisMaking decisions on the usability of the dataSharing the data among TARP states and others


65


Module II: Retrospective

Sampling designPlanning, contingency planning, and flexibility

Statistical analysisMethods show different results --- use caution interpreting results

Data adequacy: case studyLearning to deal with sampling and data deficiencies


66


What Have You Accomplished?

Guidance for using the TARP protocolExposure to key issues in a technology demonstration field testKnowledge of TARP stormwater work group and others evaluating technologies


67




68


Evaluating Stormwater Technology Performance

TARP: http://www.dep.state.pa.us/dep/deputate/pollprev/techservices/tarp/

Links to Additional Resources

For more information on TARP go to: http://www.dep.state.pa.us/dep/deputate/pollprev/techservices/tarp/For links to additional resources for this presentation go to: http://www.clu-in.org/conf/tarp/stormwater/resource.cfm

Documents

Evaluating Stormwater Technology Performance …...Eric Winkler, Ph.D. University of Massachusetts 160 Governors Drive Amherst, MA 01003 (413) 545-2853 (Voice) (413) 545-1027 (Fax)