Http:// Getting Started: Identifying, Designing, and Using Environmental Compliance & Enforcement Indicators Meredith R. Reeves INECE Secretariat

http://www.inece.org

Getting Started: Getting Started: Identifying, Designing, and Using Identifying, Designing, and Using

Environmental Compliance & Enforcement Environmental Compliance & Enforcement IndicatorsIndicators

Meredith R. ReevesMeredith R. ReevesINECE SecretariatINECE Secretariat

ECENA Workshop on Benchmarking and Setting Environmental Compliance and Enforcement Indicators

Szentendre, Hungary

29 March 2007


OverviewOverview

1.1. Introduction Introduction

2.2. FilmFilm

3.3. Identifying IndicatorsIdentifying Indicators

4.4. Exercise Exercise

5.5. Designing and Using IndicatorsDesigning and Using Indicators

6.6. Country ExampleCountry Example

7.7. QuestionsQuestions


What is INECE?What is INECE?• Global network of environmental compliance and Global network of environmental compliance and

enforcement practitionersenforcement practitioners– +4,000 from over 120 countries+4,000 from over 120 countries– Government officials, IGOs, and NGO partnersGovernment officials, IGOs, and NGO partners

• Key partners:Key partners: Netherlands, United States, other national Netherlands, United States, other national governments, World Bank, OECD, European governments, World Bank, OECD, European Commission, UNEP, regional enforcement networksCommission, UNEP, regional enforcement networks

• Secretariat based in Washington, D.C.Secretariat based in Washington, D.C.

• Executive Planning Committee guides Executive Planning Committee guides implementation of mission and goalsimplementation of mission and goals


INECE GoalsINECE Goals

• Raise Awareness to Environmental Compliance and Raise Awareness to Environmental Compliance and Enforcement at each step in the Regulatory CycleEnforcement at each step in the Regulatory Cycle– International ConferencesInternational Conferences– Issue Analysis and PublicationsIssue Analysis and Publications

• Build Capacity for Environmental Compliance and Build Capacity for Environmental Compliance and EnforcementEnforcement– Annotated Principles of Environmental Enforcement Annotated Principles of Environmental Enforcement – Training Course on ECE & Legal Aspects of Water GovernanceTraining Course on ECE & Legal Aspects of Water Governance

• Strengthen Networks for Enforcement Cooperation Strengthen Networks for Enforcement Cooperation – Hazardous Wastes, Water Governance, Environmental Crime, Hazardous Wastes, Water Governance, Environmental Crime,

Judiciary, Illegal Logging, Ozone Depleting SubstancesJudiciary, Illegal Logging, Ozone Depleting Substances


Environmental Compliance and Environmental Compliance and Enforcement IndicatorsEnforcement Indicators

• Flagship INECE projectFlagship INECE project• Initiated in 2002Initiated in 2002• Main Activities & Outputs: Main Activities & Outputs:

– Expert Working GroupExpert Working Group– Performance Measurement Guidance for Compliance Performance Measurement Guidance for Compliance

and Enforcement Practitionersand Enforcement Practitioners– Capacity Building ProgramCapacity Building Program– Executive-level BrochureExecutive-level Brochure– Assist in the formation of pilot projects in various Assist in the formation of pilot projects in various

countriescountries


What are Environmental Compliance & What are Environmental Compliance & Enforcement Indicators? Enforcement Indicators?

• Indicators Indicators – From Latin verb From Latin verb indicare, indicare, to make known or point outto make known or point out– pieces of information that provide evidence on pieces of information that provide evidence on

matters of broader concernmatters of broader concern

• Environmental compliance and enforcement Environmental compliance and enforcement indicatorsindicators– Inform users about the performance of and the results Inform users about the performance of and the results

achieved by ECE programsachieved by ECE programs


What Types of Indicators Are Needed What Types of Indicators Are Needed for ECE Programs?for ECE Programs?

• Input:Input: personnel, funds, other resources personnel, funds, other resources that contribute to an activity that contribute to an activity

• Output:Output: quantitative or qualitative quantitative or qualitative measures of activities, work products, or measures of activities, work products, or actions actions – Example:Example: number of enforcement actions taken in number of enforcement actions taken in

20052005


What Types of Indicators Are Needed What Types of Indicators Are Needed for ECE Programs?for ECE Programs?

• Intermediate Outcome:Intermediate Outcome: changes in changes in knowledge, behavior, or conditions that knowledge, behavior, or conditions that result from program activities and are result from program activities and are needed to achieve the end outcomeneeded to achieve the end outcome– Example:Example: pounds of pollution reducedpounds of pollution reduced

• End Outcome:End Outcome: the ultimate results a the ultimate results a program is designed to achieveprogram is designed to achieve– Example:Example: improvement in air qualityimprovement in air quality


Benefits of Indicators to BenchmarkingBenefits of Indicators to BenchmarkingQuantitative and qualitative performance Quantitative and qualitative performance

measurement indicators are key components of measurement indicators are key components of the benchmarking methodologythe benchmarking methodology

1.1. AssessAssess current program operation parameterscurrent program operation parameters

2.2. CompareCompare program activities with external partnerprogram activities with external partner

3.3. MeasureMeasure progress towards recommendationsprogress towards recommendations

4.4. IdentifyIdentify and help correct performance issuesand help correct performance issues

5.5. InformInform resource allocation decisions resource allocation decisions

6.6. CommunicateCommunicate progress to decision-makers and the progress to decision-makers and the publicpublic


What Purposes Do Indicators Serve?What Purposes Do Indicators Serve?

Purpose:Purpose: Evaluate Evaluate

Question:Question: How far have we progressed How far have we progressed towards the recommended changes? towards the recommended changes?

Adapted from Behn (2003)



Purpose:Purpose: Budget Budget

Question:Question: How should we spend public How should we spend public funds?funds?



Purpose:Purpose: Motivate Motivate

Question:Question: How can we use information How can we use information from an external program to motivate from an external program to motivate improvements in performance?improvements in performance?



Purpose:Purpose: Promote Promote

Question:Question: How can we demonstrate our How can we demonstrate our agency is doing a good job?agency is doing a good job?



Purpose:Purpose: Recognize Recognize

Question:Question: What accomplishments are What accomplishments are worthy of recognition?worthy of recognition?



Purpose:Purpose: Control Control

Question:Question: How can we ensure that the How can we ensure that the activities of our program staff are leading activities of our program staff are leading to improved performance?to improved performance?



Purpose:Purpose: Learn Learn

Question:Question: What is working or not What is working or not working? Why or why not?working? Why or why not?



Purpose:Purpose: Improve Improve

Question:Question: What should we do differently What should we do differently to improve performance?to improve performance?


INECE Film on INECE Film on Environmental Compliance Environmental Compliance and Enforcement Indicatorsand Enforcement Indicators

what gets measured, what gets measured, gets managedgets managed


Best PracticesBest Practices

Determine scope

Apply logic model

Develop guiding principles

Select criteria for evaluating indicators

Develop common definitions for key terms

Inventory existing data sources

Look beyond existing data

Select appropriate combination of indicators

Stage 1Identifying

Potential Indicators

Consult with stakeholders and staff

Best Practices

Use internal teams to determine how to design

and test

Conduct pilot projects

Develop in phases

Consult with experts

Monitor design and testing

Create and distribute development plan

Ensure timely and accurate reporting

Stage 2Designing Indicators

Stage 3Using

Indicators

Monitor performance with regular reports

Assess and adapt indicators

Analyze performance of organizational units

Review effectiveness of specific programs

Report to external audiences

Analyze behind the numbers

Three-Stage Model


Stage 1: Stage 1:

Identifying IndicatorsIdentifying Indicators


Identifying ECE IndicatorsIdentifying ECE Indicators

• Define scope of indicators neededDefine scope of indicators needed– Are you conducting process benchmarking Are you conducting process benchmarking

or performance benchmarking? or performance benchmarking? – Measuring national, regional or local Measuring national, regional or local

program?program?– Measuring whole program or specific Measuring whole program or specific

initiative or problem?initiative or problem?



• Conduct meetings with external Conduct meetings with external stakeholders and internal staff stakeholders and internal staff to solicit:to solicit:

– Potential performance measuresPotential performance measures– Criteria for evaluating potential Criteria for evaluating potential

measures measures – Principles to guide the overall Principles to guide the overall

effort to develop and use effort to develop and use indicatorsindicators


• Establish definitions for types of Establish definitions for types of indicatorsindicators– Define inputs, outputs, outcomesDefine inputs, outputs, outcomes

• Create a “map” of activities and Create a “map” of activities and results, perhaps using logic modelresults, perhaps using logic model– Identify full range of results of program Identify full range of results of program

activities, with particular attention to activities, with particular attention to intermediate outcomesintermediate outcomes



Sample Logic ModelSample Logic Model

InputsInputs

resourcesresources

OutputsOutputs

activitiesactivities

Intermediate Intermediate OutcomesOutcomes

behavior changebehavior change

Final OutcomesFinal Outcomes

environmental environmental impactimpact

PersonnelPersonnel

Funds for Funds for salaries, salaries, contracts, contracts, etc.etc.

Inspections Inspections conductedconducted

Enforcement Enforcement actions actions takentaken

Fines Fines assessed assessed and and collectedcollected

Greater Greater understanding of understanding of how to complyhow to comply

Improved Improved environmental environmental management management practicespractices

Increased Increased compliancecompliance

Reduced pollution Reduced pollution emissionsemissions

Improved ambient Improved ambient water qualitywater quality

Reduced contaminant Reduced contaminant burden in wildlife burden in wildlife speciesspecies



• Inventory existing data sourcesInventory existing data sources– Determine whether existing data will support Determine whether existing data will support

potential indicatorspotential indicators– Limitations of data sources should not limit Limitations of data sources should not limit

consideration of potential indicatorsconsideration of potential indicators

• Select appropriate combinations of indicators Select appropriate combinations of indicators for further developmentfor further development

– Use Selection CriteriaUse Selection Criteria– Make important audiences aware of which Make important audiences aware of which

indicators have been selected for development indicators have been selected for development


Sample Criteria for EvaluatingSample Criteria for EvaluatingPerformance Indicators Performance Indicators

• RelevantRelevant: to goals, objectives, and priorities : to goals, objectives, and priorities • TransparentTransparent: promotes understanding of program: promotes understanding of program• CredibleCredible: based on data that is complete and : based on data that is complete and

accurateaccurate• FunctionalFunctional: encourages constructive behavior: encourages constructive behavior• FeasibleFeasible: value to program outweighs cost: value to program outweighs cost• ComprehensiveComprehensive: covers important operational : covers important operational

aspectsaspects


Indicator Type IdentificationIndicator Type Identification

ExerciseExercise


Stage 2: Stage 2:

Designing IndicatorsDesigning Indicators


Designing ECE IndicatorsDesigning ECE Indicators

• Select indicators for design and Select indicators for design and implementationimplementation

• Establish internal work teams to:Establish internal work teams to:– Define measures in more detailDefine measures in more detail– Review relevant data in existing data systemsReview relevant data in existing data systems– Develop new information collection and Develop new information collection and

reporting processesreporting processes– Establish schedule for testing and developing Establish schedule for testing and developing

indicatorsindicators



• Conduct pilot projects to test indicators Conduct pilot projects to test indicators and correct problemsand correct problems

• Use consultants as needed to resolve Use consultants as needed to resolve technical and methodological issuestechnical and methodological issues

• Execute implementation in phasesExecute implementation in phases



• Monitor the design and testingMonitor the design and testing

• Create and distribute a development planCreate and distribute a development plan

• Ensure timely and accurate reportingEnsure timely and accurate reporting


Stage 3: Stage 3:

Using IndicatorsUsing Indicators


Using IndicatorsUsing Indicators

• Monitor performance through regular Monitor performance through regular reportsreports

• Review performance of organizational Review performance of organizational units units

• Evaluate effectiveness of specific Evaluate effectiveness of specific programsprograms


Suggested Practices for Using ECE Suggested Practices for Using ECE IndicatorsIndicators

• Analyze behind the numbersAnalyze behind the numbers

• Report to external audiencesReport to external audiences

• Assess and adapt indicatorsAssess and adapt indicators

• Assess and adapt enforcement programAssess and adapt enforcement program


Possible Challenges in Using IndicatorsPossible Challenges in Using Indicators

• Duration of implementationDuration of implementation

• Lack of interpretive skillsLack of interpretive skills

• Misuse by external audiencesMisuse by external audiences

• Inherent limitations of indicatorsInherent limitations of indicators


Country ExampleCountry Example

US Environmental Protection US Environmental Protection Agency Agency

Office of Enforcement and Office of Enforcement and Compliance Assurance (OECA)Compliance Assurance (OECA)


Program FoundationsProgram Foundations

• Government Performance and Results Act Government Performance and Results Act (GPRA) of 1993(GPRA) of 1993 – RequiresRequires Annual Performance Report on results and Annual Performance Report on results and

outcomes achieved through the activities of major outcomes achieved through the activities of major federal programs federal programs

• Executive level focus on: Executive level focus on: – outcome measures to monitor program performance outcome measures to monitor program performance – performance-based management in ECE Programs performance-based management in ECE Programs


Program GoalsProgram Goals

Goal 1: Report on results achieved to Goal 1: Report on results achieved to external audiencesexternal audiences

• the public, the public, • the U.S. Congress, and the U.S. Congress, and • the U.S. Office of Management and the U.S. Office of Management and

Budget (OMB)Budget (OMB)


Examples of EPA Outcome IndicatorsExamples of EPA Outcome Indicators

• Pounds of pollutants reduced by Pounds of pollutants reduced by enforcement actionsenforcement actions

• Pounds of soil treated or removedPounds of soil treated or removed

• Gallons of groundwater treatedGallons of groundwater treated

• Number of companies/facilities self-Number of companies/facilities self-disclosing and correcting violations under disclosing and correcting violations under EPA Audit PoliciesEPA Audit Policies


Criminal Enforcement ProgramCriminal Enforcement ProgramFY 2001 – FY 2005FY 2001 – FY 2005

FY 2005 Data Source: Criminal Docket, October 27, 2005

Environmental Crime Investigations

482 484471

425

372

0

100

200

300

400

500

Fiscal Year (FY)

Tot

al

20012002200320042005 20012002200320042005

Defendants Charged

372

325

247

293320

0

100

200

300

400

Fiscal Year (FY)

Tot

al

Disclaimer: Due to enhanced data quality reviews, minor corrections may have been made to previously reporteddata. As such, this FY 2005 End-of-Year Report contains updated enforcement and compliance data for prior years.


Sentencing Results FY 2001 – FY 2005Sentencing Results FY 2001 – FY 2005

FY 2005 Data Source: Criminal Docket, October 27, 2005

77

Incarceration

212 215

146

77

186

0

50

100

150

200

250

Fiscal Year (FY)

Tot

al Y

ears

20012002200320042005

Fines and Restitution

$71

$95

$62$47

$100

$0

$25

$50

$75

$100

$125

Fiscal Year (FY)

Tota

l V

alue

in M

illions of D

ollar

s20012002200320042005



551590

721 731

612

502

673

870

1,164

1,443

0

200

400

600

800

1,000

1,200

1,400

1,600

2001 2002 2003 2004 2005

Fiscal Year (FY)

Tota

l Entities R

each

ed in T

housa

nds

Numbers of Entities Reached with Compliance Assistance (except Center Users)Number of Users of EPA-Sponsored Web-Based Compliance Assistance Centers

FY 2005 Data Sources: Integrated Compliance Information System (ICIS), October 27, 2005 and on-line usage report.


Entities Reached with Entities Reached with Compliance AssistanceCompliance Assistance


FY 2005 HighlightsFY 2005 HighlightsIn FY 2005, actions from the enforcement and compliance

program produced significant results including…

• Enforcement actions projected to reduce pollution by Enforcement actions projected to reduce pollution by 1.1 billion pounds1.1 billion pounds

• Defendants will spend a record $10 billion to reduce Defendants will spend a record $10 billion to reduce pollution and achieve compliance pollution and achieve compliance

• Defendants will pay $100 million in criminal fines and Defendants will pay $100 million in criminal fines and restitution and were sentenced to 186 years in jail restitution and were sentenced to 186 years in jail

Results like this reflect a strong and active enforcement program,effective targeting, and a focus on results-based performance.


Program GoalsProgram Goals

Goal 2: Goal 2: Monitoring, Analyzing, and Improving Monitoring, Analyzing, and Improving PerformancePerformance

• Monthly Management ReportsMonthly Management Reports• year-to-date status of activities reported to senior year-to-date status of activities reported to senior

managersmanagers

• Regional Data Profiles Regional Data Profiles • twice yearly status on performance in the current fiscal twice yearly status on performance in the current fiscal

year, three-year trends on key outputs and outcomes, year, three-year trends on key outputs and outcomes, • comparisons between regionscomparisons between regions

• Watch List for Significant NoncomplianceWatch List for Significant Noncompliance • List of facilities where significant noncompliance has List of facilities where significant noncompliance has

been identified but has not been addressed after a been identified but has not been addressed after a prolonged period prolonged period


Questions for ConsiderationQuestions for Consideration

• What do you anticipate being your biggest What do you anticipate being your biggest institutional hurdle? institutional hurdle?

• Who should be participating on your team Who should be participating on your team to design the indicators? What groups to design the indicators? What groups should be represented? should be represented?


Contact InformationContact Information

INECE SecretariatINECE Secretariat2300 Wisconsin Ave, NW Suite 300B2300 Wisconsin Ave, NW Suite 300B

Washington, DC 20007 Washington, DC 20007

Phone: 202-338-1300Phone: 202-338-1300 Email: [email protected]: [email protected]

Web: Web: http://www.inece.orghttp://www.inece.org

Documents

Http:// Getting Started: Identifying, Designing, and Using Environmental Compliance & Enforcement Indicators Meredith R. Reeves INECE Secretariat