INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental

INDUSTRIAL ECONOMICS, INCORPORATED

IEc


Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental & Quasi-Experimental Evaluation Design

Tracy Dyke RedmondSenior Associate

June 23, 2011


Typology of Evaluation Designs*

Experimental Design

Quasi-Experimental Design

Non-Experimental Design

2

IncreasingStatistical Strength*

*Notnecessarilyoverall evaluation strength

• Pre-test + post-test comparison with statistical matching

• Regression discontinuity• Pre-test + post-test

comparison with judgmental matching

• Pipeline control group• Pre-test + post-test

comparison with delayed baseline

• Pre-test + post-test treatment group with post-test only comparison group

• Post-test only for treatment and comparison group

* Not all possible evaluation designs shownAdapted from Bamberger, Rugh, and Margy: RealWorld Evaluation,Sage Publications 2006.


“Statistically Valid” Pilot Project for Auto Body Shops








Treatment and comparison areas selected judgmentally, with subjects randomly selected from within these areas

Pro:•Flexible•Reasonably good estimate of project impact when there are good matching criteriaCon:•Assumes comparison group similar to treatment group and equally willing to participate•Does not assess project implementationAdapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.










Project implemented in phases: subjects in phase 2 serve as control group for subjects in phase 1

Pro:•Does not require external control group; design relatively inexpensive and easy to useCon:•Assumes phase 1 and 2 groups are similar (which may not be true)•Requires that phase 2 group does not have access to phase 1 treatment (spillover problem)

Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.


• Goal: test impact of EPA compliance assistance (e.g., workshops, webinars, materials) on auto body shop compliance with air and hazardous waste regulations

• Context: existing hazardous waste regulations, new air regulations coming into effect 2011 (Surface Coating Rule)

• Requirements: • Conduct representative measurement of all regulated

entities, not just voluntary participants

• Do not prevent shops from receiving compliance assistance (or not for very long)

• Also test phone survey validity (that methodology not covered in this presentation)



Control and Comparison Groups

• Population = auto body shops located in areas with elevated air toxics risks and subject to the Surface Coating Rule

• Massachusetts selected as study area because EPA Region 1 planned compliance assistance campaign

• Treatment and control group in Massachusetts: randomly assigned, considered equivalent

• Includes auto body shops in eastern MA with elevated risk

• Excludes communities with pre-existing aggressive assistance/enforcement campaigns

• Comparison group selected in Tidewater and Piedmont regions of Virginia on the basis of:

• No expected compliance assistance from EPA or state

• State regulations related to RCRA and air emissions

• Number of shops located in areas of elevated-risk


“Statistically Valid” Pilot Project Design

MA - A

MA - B

VA

October 2009 –January 2010

March – Early July 2010

ComplianceAssistance (CA):• Mailings• Workshops/ Webinars

On-site surveysFollowed by on-site CA

On-site surveys

Summer 2010-January 2011

CA:• Mailings• Webinars









MA - A

MA - B

VA





On-site surveys









Compare randomly assigned treatment vs. control group in MA



MA - A

MA - B

VA





On-site surveys









Compare “difference-in-differences”: Pre-test to Post-test in MA, compared to Pre-test to Post-test in VA


Methodological Notes and Next Steps

• Among valid shops, response rates between 80 – 85% for both states and both years

• However, list problems (e.g., shops going out of business) led to need for many “backup shops;” increased cost and effort of project

• Evidence of shops not on the list, operating “under the radar” in VA

• Some evidence of spillover problems (e.g., some VA shops may have accessed EPA Region 1 webinars)

• Currently working to analyze 2011 data and develop comparisons; report expected in late 2011


Thanks to:

• EPA HQ for developing the vision and supporting the project

• EPA Region 1 for identify the list of shops, implementing the assistance, and helping conduct site visits

• EPA Region 3, Virginia, and Massachusetts for participating

• ERG and Ski Fabyanic for conducting hundreds of site visits

• Chris Leggett and Michael Crow for helping develop the methodology and analyzing the data

• The auto body shops, for letting us in the door!


IEcINDUSTRIAL ECONOMICS, INCORPORATED

617.354.0074

Documents

INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental