Evaluating OSHA inspections for intended and unintended outcomes

Mike ToffelAssociate Professor

Harvard Business School

Liberty Mutual Research Institute for Safety

August 20121

May 18, 2012

2

OSHA, a much-criticized agency

Too lenient! (?)• Inspections too seldom• Penalties too small• Lengthy process to adopt new regulations

compromises worker safety

Too costly! (?)• Stifling job creation / job killer• Increasing labor costs• Eroding America's competitiveness 3

4

Challenges evaluating impact of OSHA inspections:Causality

Most OSHA inspections are not random: After accidents and deaths When employees complain

If accidents/deaths are rare events, outcomes will feature mean reversion: Problems likely decline after inspections…

but even without inspections

Our approach Examine random inspections and compare

to a control group.5

Several studies have relied on company logs But inspections can lead companies to

improve logs’ comprehensiveness, increasing reported injuries

This cloaks changes in actual injury rates

Our approach Rely on workers’ compensation claims Annual number of claims Annual cost of all claims

Challenges evaluating impact of OSHA inspections:

Measuring outcomes

6

3 data sources

WCIRBNumber of claims

Cost of claimsOccupation

classesPayroll

OSHA IMISInspections

Dun & Bradstreet

IndustrySingle-plantEmployment

Sales

Data restrictions California Cal/OSHA Some high hazard industries (randomization

targets) Single-plant firms7

Data issuesWCIRB data• MOU between California Department of

Insurance (CDI) and Commission on Health and Safety and Worker’s Compensation listing us as subcontractors

• Required to keep data anonymous

OSHA data• Complex data structure• Limited documentation• Many conversations with Cal/OSHA to

understand implementation of randomization process8

Developing a matched sampleTreatments Single-plant establishments randomly selected for

a programmed inspection High-hazard industries Cal/OSHA targeted for

random inspection each year

Matched controls Find population of single-plant establishments at

risk of random inspection, but not selected Exclude if < 10 employees or recently

inspected For each treatment, select one control:

Same industry and region Closest sizeResult: 409 matched pairs

9

Matching led to a balanced sample of very similar treatments and controls

In the two years before the match year, the 409 matched pairs had very similar: Sales Employment Payroll Credit scores (PAYDEX, Comprehensive

Credit Appraisal) Annual number of WC claim Annual total cost of WC claims

10

11

Figure S1: Indistinguishable levels

12

Figure S2: Indistinguishable trends

Industry distribution of matched sample (Table S2)

13

Evaluation model

Did treatments experience a greater decline in annual injuries (or injury-related costs) after inspections than the controls, examined over the same time period? Fixed effects regression Control for establishment characteristics Difference-in-differences approach Compares two groups over time

14

Randomized inspections reduce annual injuries by 9.4%

Persistent

15

Randomized inspections reduce annual injuries by 9.4%

Persistent

16

Randomized inspections reduce annual injuries by 9.4%

Persistent effect

17

Randomized inspections reduce annual injury costs by 26%

Persistent

18

Randomized inspections reduce annual injury costs by 26%

Persistent

19

Randomized inspections reduce annual injury costs by 26%

Persistent effect

20

Unanticipated consequences of inspections?

Inspections (and consequences) cause interruptions, but are they substantial? Sales impact? Credit worthiness? Employment, payroll? Firm survival?

21

Unanticipated consequences of inspections?

22

Unanticipated consequences of inspections?No difference in employment, payroll, sales (Table 2) Tight confidence intervals enable us to rule out

that inspections caused big declines of employment or payroll

No difference in credit ratings (Table S8) Late bills, etc. more sensitive to financial burden

than firm death Two D&B metrics of financial distress: PAYDEX &

CCA

No difference in firm survival (Table S7) Approx. 5% of treatments and of controls died

Difference not statistically significantResult robust to survival regressions

23

Summary of resultsEvidence of intended results Annual injuries reduced by 9.4% Annual injury costs reduced by 26%

No evidence of unintended consequences Sales impact Credit worthiness Employment, payroll Firm survival

24

ConclusionsCal/OSHA is not entirely toothless• Injuries and injury costs decline a lot• Social value of very approximately $350,000 per inspection• Includes medical costs and lost wages, but not

productivity loss, pain and suffering, etc.• With many assumptions (similar effects across all states

and inspection type, etc.) translates to $22 B per year nationwide

OSHA is not detectably a job killer• But we cannot rule out some costs

25

Levine, David I., Michael W. Toffel, and Matthew S. Johnson. "Randomized Government Safety Inspections Reduce Worker Injuries with No Detectable Job Loss." Science 336, no. 6083 (May 18, 2012)

Future research in this domain1. When do random inspections bolster regulatory compliance?

High vs. low hazardous industries Strong vs. weak compliance history VPP participants versus others Prior OSHA consultation versus not

2. Do random inspections have spillover effects? Organizational: bolster compliance at corporate siblings? Geographic: bolster compliance at neighboring facilities? Regulated domains: bolster compliance with EPA regulations?

3. Your ideas? Loss prevention vs. OSHA inspections?

26

Other research…

Environmental reporting & carbon disclosure• Responding to public and private politics: Corporate

disclosure of climate change strategies, Strategic Management Journal (2009)• Engaging supply chains in climate change. Working Paper

(2012)• When do firms greenwash? Corporate visibility, civil society

scrutiny, and environmental disclosure, Working Paper (2012)

Environmental ratings• How well do social ratings actually measure corporate social

responsibility? Journal of Economics & Management Strategy (2009)• How firms respond to being rated, Strategic Management

Journal (2010)27

Environmental policy evaluationVoluntary violations disclosure (EPA Audit Policy)• Coerced confessions: Self-policing in the shadow of the

regulator, Journal of Law, Economics and Organization (2008)• Coming clean and cleaning up: Does voluntary self-reporting

indicate effective self-policing, Journal of Law and Economics (2011)• Making self-regulation more than merely symbolic: The

critical role of the legal environment, Administrative Science Quarterly (2010)

Mandatory performance disclosure• How firms respond to mandatory information disclosure,

Strategic Management Journal (forthcoming)

Tailpipe emissions fraud• Competition and illicit quality, Working Paper (2012)• The role of organizational scope and governance in

strengthening private monitoring, Working Paper (2012)28

Quality management: ISO 9001 evaluation• Quality management and job quality: How the ISO 9001

standard for quality management systems affects employees and employers, Management Science (2010).

Occupational health and safety • Randomized government safety inspections reduce worker

injuries with no detectable job loss, Science (2012).  

29

Thank you!

Prof. Michael ToffelHarvard Business School

Morgan Hall 497Boston, MA 02163mtoffel@hbs.edu

www.people.hbs.edu/mtoffel/

30