1

A Short Comparative Interrupted Time-Series

Analysis of the Impacts of Jobs-Plus

Howard S. Bloom

MDRC

Presented at the HHS Conference on Building Strong Evidence in Challenging Contexts:

Alternatives to Traditional Randomized Control Trials, Washington, DC, September 23,

2016.

Introduction

What is short comparative interrupted time-series (CITS)

analysis?

It compares deviations from trends for a treatment and comparison

group

It is an extension of difference-in-differences analysis

When might we use such an analysis?

For a retrospective study of a policy change (e.g. raising or lowering speed limits or drinking ages)

For a small-N study of a place-based initiative (e.g. a community employment, crime or health intervention)

To study the impacts of environmental, economic or social

disruptions (e.g. storms, earthquakes, plant closings or wars)

For a longitudinal comparison-group study of a social program (e.g. federal employment and training programs)

2

Some Benefits of Short CITS Analysis

What you see is what you get!

You can use it prospectively or retrospectively.

You can use it with administrative data.

Aggregate level

Individual level

You can use it when a conventional RCT is not feasible.

3

4

A Hypothetical Killer Application of Short CITS:

Measuring the Impact of an Oprah Book Endorsement

0

200

400

600

800

1,000

1,200

An

nu

al S

ale

s (

1,0

00

s)

Anna Karenina (Endorsed) War and Peace (Not Endorsed)

Before Oprah Endorses After Oprah Endorses

5

Conditions for a Successful

Short CITS Analysis

An outcome that is measured consistently over time

A baseline trend that is sufficiently long, frequent and stable

Impacts that are sufficiently pronounced and immediate

A follow-up period that is long enough to account for program

implementation but short enough to avoid other major changes

A comparison group with the same data (matching can help but is not always necessary)

Covariates can be used to account for sample composition that

changes markedly over time

6

Estimating Intervention Effects

The basic estimator is a treatment- and comparison-group difference in deviations from their baseline trends.

Baseline trends can be simple means or linear and (infrequently) non-linear functions of time.

Serial correlation can sometimes be accounted for.

Multi-level data can be accommodated.

Matching can be used to choose a comparison group.

Covariate adjustments can be made, if needed.

Origins of the Jobs-Plus Community Revitalization

Initiative for Public Housing Families

Jobs-Plus was an MDRC demonstration project designed to build mixed-income communities from within Response to growing concentration of joblessness, underemployment,

welfare receipt, and poverty in public housing and surrounding neighborhoods

Public and private Jobs-Plus sponsors US Dept. of Housing and Urban Development (HUD)

The Rockefeller Foundation

Other public and private funders

7

8

The Jobs-Plus Program Model

Three intervention components focused on selected public housing developments:

Employment and training services

Convenient on-site job centers

New rent rules to make work pay

Rents rise less than usual as earnings grow

Community support for work

Neighbor-to-neighbor outreach; sharing work leads,

babysitting for working mothers, etc.

Saturation-level outreach

Aimed at all working-age residents

The Jobs-Plus Sites

The local public housing authorities (PHAs) involved

50 PHAs invited

42 PHAs applied

7 PHAs selected

6 PHAs began participation

3 PHAs had stronger implementation (LA, Dayton and St. Paul)

1 PHA had stronger implementation but could not continue (Seattle)

2 PHAs had very weak implementation (Baltimore and Chattanooga)

Selection of treatment and comparison developments

2 or 3 candidate developments per site

Random assignment to Jobs-Plus of one candidate development per site

Remaining candidate developments formed the comparison group for each site

9

The Jobs-Plus Short CITS Analysis

Time series

Baseline: 4 6 years before 1998 launch

Follow-up: 6 8 years after 1998 launch

Outcome measures

Quarterly earnings and employment rates

Quarterly welfare receipt and receipt rates

Analytic perspectives

People (1998 cohort members)

Place (local public housing developments)

Substantive focus

Implementation

Impacts

Findings reported

Overall, by site implementation level, and by site

For sample subgroups10

Pooled Average Quarterly Earnings for the 1998 Cohort

At the Three Stronger Implementation Sites

Rollout period

0

500

1,000

1,500

2,000

2,500Q

1 1

99

2

Q1

19

93

Q1

19

94

Q1

19

95

Q1

19

96

Q1

19

97

Q1

19

98

Q1

19

99

Q1

20

00

Q1

20

01

Q1

20

02

Q1

20

03

Time Period

Mea

n Q

ua

rter

ly E

arn

ing

s R

ecei

pt

(in

20

03

do

lla

rs)

Jobs-Plus Group

Comparison GroupDifference due

to Jobs-Plus =

+$ 1,141/year

or + 14%

Pooled Difference in Average Quarterly Earnings for the

1998 Cohort at the Three Stronger Implementation Sites

-600

-400

-200

0

200

400

600

Q 1

19

92

Q 1

19

93

Q 1

19

94

Q 1

19

95

Q 1

19

96

Q 1

19

97

Q 1

19

98

Q 1

19

99

Q 1

20

00

Q 1

20

01

Q 1

20

02

Q 1

20

03

Time Period

Dif

feren

ce in

Mean

Qu

arte

rly

Earn

ing

s R

eceip

t (in

20

03

do

llars)

Jobsplus Program vs Comparison Developments

Jobs-Plus Quarterly Impact Estimation Model

Model of Quarterly Mean T/C Earnings Differences

= + + and

= 1+

where

= the difference in mean earnings for the treatment and

comparison groups in quarter t,

= one if quarter t is follow-up quarter m and zero otherwise,

and 1= error terms with a first-order autoregressive structure,

= a random error term that is independently and identically

distributed.

13

Additional Impact Estimation Steps

Estimate annual impacts: by summing quarterly impact estimates

Estimate standard errors of annual impact estimates: based on estimated standard errors and covariances of the quarterly impact

estimates.

Adjust p-values for the multiplicity of annual impact

estimates: using a layered Bonferroni approach

14

Jobs-Plus Earnings Impacts for the 1998 Cohort from the

Three Stronger Implementation Sites

15

Estimated

Percentage

Observed Estimated Estimated Change

Outcome with Effect of Outcome Without in Outcome Due to

Follow-Up Period Jobs-Plus Jobs-Plus Jobs-Plus Jobs-Plus

1998 6,089 173 5,916 2.9

1999 7,619 209 7,410 2.8

2000 8,793 714 ** 8,079 8.8

2001 9,256 1,135 *** 8,121 14.0

2002 9,419 1,171 *** 8,248 14.2

2003 9,443 1,543 *** 7,900 19.5

2000-2003 9,228 1,141 *** 8,087 14.1

Selected References

Original Sources

Campbell, D.T. and J.C. Stanley (1963) Experimental and Quasi-experimental Designs

for Research. Chicago: Rand McNally, 37 43.

Cook, T.D. and D.T. Campbell (1979) Quasi-Experimentation: Design and Analysis

Issues for Field Settings, Chicago: Rand McNally, 207 232.

Recent Sources:

St. Clair, T., K. Hallberg and T.D. Cook (2016) The Validity and Precision of the

Comparative Interrupted Time-Series Design, Journal of Educational and Behavioral

Statistics, 41(3): 269 299.

Wong, M., T.D. Cook and P.M. Steiner (2015) Adding Design Elements to Improve

Time-Series Designs: No Child Left Behind as an Example of Causal Pattern

Matching, Journal of Research on Educational Effectiveness, 8(2): 245 279.

16

Selected References

(continued)

Recent Sources (continued)

St. Clair, T., T.D. Cook and K. Hallberg (2014) Examining the Internal Validity and

Statistical Precision of the Comparative Interrupted Time Series Design by Comparison

with a Randomized Experiment, American Journal of Evaluation: 1098214014527337.

Dee, T.S. and B. Jacob (2011) The Impact of No Child Left Behind on Student

Achievement. Journal of Policy Analysis and Management. 30 (3): 418 486.

Personal Sources

Bloom, H.S., J. Riccio and N. Verma (2005) The Effectiveness of Jobs-Plus. New York:

MDRC.

Bloom, H.S. (1984) Estimating the Effect of Job-training Programs Using Longitudinal

Data: Ashenfelters Findings Reconsidered, Journal of Human Resources (Fall):

544 556.

Bloom, H.S. and H.F. Ladd (1982) Property Tax Revaluation and Tax Levy Growth,

Journal of Urban Economics, Vol. 11.

17

Selected References

(continued)

Personal Sources (continued)

Jacob, R., M.A. Somers, P. Zhu and H. Bloom (2016) The Validity of the

Comparative Interrupted Time Series Design for Evaluating the Effect of

School-Level Interventions, Evaluation Review. DOI: 10.1177/0193841X16663414.

Jobs-Plus Long-Term Follow-up Source

Riccio, J.A. (2010) Sustained Earnings Gains for Residents in a Public Housing

Jobs Program: Seven-Year Findings From the Jobs-Plus Demonstration. New

York: MDRC (January).

18