Applied Psychometric Strategies Lab Applied Quantitative

Joe Waddington, PhD

Regression Discontinuity

October 3, 2017

Applied Psychometric Strategies LabApplied Quantitative and Psychometric Series

Overview

• Introduction to Regression Discontinuity• Example of Regression Discontinuity using Angrist & Lavy (1999)

• Mechanics• Modeling Considerations• Internal/External Validity• Analysis using Stata

• Questions

Introduction to Regression Discontinuity

• A regression discontinuity design (RDD) is a powerful quasi-experimental approach to determine the effect of an intervention.

• Used when assignment to treatment condition is based on an exogenous “cut-point” along a variable.

• The variable upon which the cut-point occurs is the “assignment” or “forcing” variable determining the treatment and comparison groups.

• If an individual’s value of the assignment variable falls to one side of the cut-point, the individual is assigned to the treatment group.

• If an individual’s value of the assignment variable falls to the other side of the cut-point, the individual is assigned to the comparison group.

• The assignment variable needs to be a continuous variable, with a range of values on either side of the cut-point.

Introduction to Regression Discontinuity

• The RDD design shares similarities with the difference-in-differences approach.

• The assignment to treatment or comparison is determined exogenously.• We need to difference out the pre-existing trends within the comparison

group to arrive at the treatment effect.

• Examples of RDD designs• Effect of Class Size on Student Achievement (Angrist & Lavy, 1999)• Effect of Head Start Funding on Child Mortality (Ludwig & Miller, 2007)• Effect of Summer Remedial Coursework on Student Achievement (Jacob &

Lefgren, 2004)

Example of RD using Angrist & Lavy (1999)

• In this article, Angrist and Lavy use a RDD to investigate the relationship between class size and student reading achievement.

• Students are not randomly assigned to large and small classes. Instead, they use an exogenous class size assignment mechanism:

• This mechanism is based on Maimonides’ Rule. This rule states that class sizes should have one teacher (one class) for a cohort of up to 40 students. Once there are 41 students in a cohort, there should be two teachers (two classes). This “cut-point” would continue to occur at every 40-student increments.

• We can visualize 40 to 41 students in a cohort as the “cut-point” between large classes and small classes.

• The causal question of interest: “What is the effect of class size on student reading achievement?”

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)

0 50 100 150 200 250size of september enrollment cohort

We are going to use data from the Angrist & Lavy (1999) article examining the effect of class size on student reading scores.

Here, we have a scatterplot of the entire dataset (aggregated at the class level).

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)

36 37 38 39 40 41 42 43 44 45 46size of september enrollment cohort

Maimonides’ Rule states that class sizes should have one teacher up to 40 students per class. Once there are 41 students in a cohort, there should be two classes. We can visualize 41 students in a cohort as the “cut-point.”

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


At first glance, there does not appear to be much of a relationship between cohort size and test score. We have to keep in mind that we are trying to get at the effect of intended class size.

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


We can start by examining the relationship by looking directly around the cut-point for the assignment of smaller classes at 41 students in a cohort.

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


A cohort with 40 students should be assigned to one large class of 40 students. A cohort with 41 students should be assigned to two small classes of 20 and 21 students.

4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


To simplify things, we could just look at the cohort-size average reading achievement. Here we can calculate the first difference, or our naïve estimate of the effect of intended small class size.

First Difference = 5.75 points

4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


Just as is the case in a diff-in-diff approach, we need to know more about the existing trends associated with cohort size and reading scores.


4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


To determine existing trends, we might look to the nearest cohort sizes that are not impacted by the treatment. These would be cohorts of 38 and 39 students.


4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


There is still a one-student difference between these cohorts. And, regarding the overall cohort background characteristics, these should be quite similar to the 40 and 41 student cohorts.


4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


However, the 38 and 39 student cohorts are intended to be assigned to a single class. So, we can calculate the second difference as an estimate of the trend in achievement.

First Difference = 5.75 pointsSecond Difference

= 1.02 points

4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


First Difference = 5.75 pointsSecond Difference = 1.02 points

Removing the second difference from the first difference, we arrive at the diff-in-diffestimate. This is a less biased estimate than the first difference estimate.

Diff-in-Diff = 4.73 points

4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


If we only look at the nearest similar pair of cohorts, we could be throwing away information. For example, we might want to understand the trend between cohort size and reading scores within 5 students.

4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


Rather than just taking a difference between two cohorts, we could plot the average trend across these cohorts. Accounting for this full trend gives us a new estimate of the treatment effect.

4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


If we extend the trend forward to the cut-point (i.e. the relationship between cohort size and test score if there was no change in class size, we could estimate the adjusted difference based on this trend.

First Difference = 5.75 pointsAdjusted Difference = 5.12 points

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


So that we are clear, we could estimate this same adjusted difference using all of the data (each individual class) versus the cohort averages. The result would be the same, and we’ll have additional stability in our estimates.

Adjusted Difference = 5.12 points

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)

-5 -4 -3 -2 -1 0 1 2 3 4 5treatment centered size of cohort

To help us think of this in the regression context, we could re-center the cohort size variable relative to 41. All points before 41 are prior to the cut-point and zero (41) is the actual cut-point.


4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


And to be even more clear, there are still individual classes at cohorts of size 41. We have just been displaying the cohort average test score for the sake of simplicity.


4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


Now let’s extend our analysis to account for cohorts on either side of the cut-point. We previously made an assumption about the trend before the cut-point, but there is one after it, too, to be concerned about.

4060

8010

0ve

rbal

sco

re (e

nrol

lmen

t coh

ort a

vera

ge)


If we plot the trends on either side of the cut-point, we arrive at a new adjusted difference that is accounting for “pre-cut-point trends” as well as “post-cut-point trends.”

Adjusted Difference

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


Adjusted Difference

Remember, all of the classes are lying underneath the cohort averages…

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


…and we can also look at this from a centered cohort size perspective, too.

Adjusted Difference

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


Rather than making an assumption about the trend on either side of the cut-point, we could look at the picture as a whole. In other words, determine the entire trend between reading scores and cohort size.

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


If we use this same trend on either side of the cut-point, but let the cut point serve as the “jump” due to the treatment, we have ourselves the regression discontinuity treatment effect.

Treatment Effect = 3.85

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)


This picture came from estimating the following OLS regression:�𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒊𝒊 = �𝜷𝜷𝟎𝟎

+�𝜷𝜷𝟏𝟏𝒔𝒔𝒔𝒔𝒓𝒓𝒔𝒔𝒔𝒔𝒊𝒊+�𝜷𝜷𝟐𝟐𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒓𝒓𝒄𝒄𝒔𝒔𝒊𝒊𝒄𝒄𝒓𝒓𝒊𝒊

�𝜷𝜷𝟏𝟏

�𝜷𝜷𝟎𝟎

�𝜷𝜷𝟐𝟐

�𝜷𝜷𝟐𝟐

�𝜷𝜷𝟎𝟎+�𝜷𝜷𝟏𝟏

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)




�𝜷𝜷𝟎𝟎 is the mean reading score at the cut-point for the comparison large class group.

�𝜷𝜷𝟏𝟏

�𝜷𝜷𝟎𝟎

�𝜷𝜷𝟐𝟐

�𝜷𝜷𝟐𝟐

�𝜷𝜷𝟎𝟎+�𝜷𝜷𝟏𝟏

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)



+�𝜷𝜷𝟏𝟏𝒔𝒔𝒔𝒔𝒓𝒓𝒔𝒔𝒔𝒔𝒊𝒊+�𝜷𝜷𝟐𝟐𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒓𝒓𝒄𝒄𝒔𝒔𝒊𝒊𝒄𝒄𝒓𝒓𝒊𝒊�𝜷𝜷𝟏𝟏is the mean difference in reading score at the cut-point between small and large classes. It’s the average treatment effect!

�𝜷𝜷𝟏𝟏

�𝜷𝜷𝟎𝟎

�𝜷𝜷𝟐𝟐

�𝜷𝜷𝟐𝟐

�𝜷𝜷𝟎𝟎+�𝜷𝜷𝟏𝟏

4060

8010

0ve

rbal

sco

re (c

lass

ave

rage

)




�𝜷𝜷𝟐𝟐is the slope between the forcing variable (cohort size) and test scores. It is a test score trend control.

�𝜷𝜷𝟏𝟏

�𝜷𝜷𝟎𝟎

�𝜷𝜷𝟐𝟐

�𝜷𝜷𝟐𝟐

�𝜷𝜷𝟎𝟎+�𝜷𝜷𝟏𝟏

Modeling Considerations

�𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒊𝒊 = �𝜷𝜷𝟎𝟎 +�𝜷𝜷𝟏𝟏𝒔𝒔𝒔𝒔𝒓𝒓𝒔𝒔𝒔𝒔𝒊𝒊 +�𝜷𝜷𝟐𝟐𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒓𝒓𝒄𝒄𝒔𝒔𝒊𝒊𝒄𝒄𝒓𝒓𝒊𝒊• We made an assumption (that fits pretty well here) that there is a

continuous linear relationship between cohort size and reading scores.

• This is reflected by a single slope estimate for �𝜷𝜷𝟐𝟐, the “secular trend.”

• We could modify this approach to better fit the data.• We could fit an overall trend that takes on various functional forms, just as in

any linear regression analysis (e.g., non-linear or non-parametric).• We could fit two different trends taking any functional form—one for pre-

treatment and one for post-treatment.

• Any of these considerations go back to regression analysis principles.

Modeling Considerations

�𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒊𝒊 = �𝜷𝜷𝟎𝟎 +�𝜷𝜷𝟏𝟏𝒔𝒔𝒔𝒔𝒓𝒓𝒔𝒔𝒔𝒔𝒊𝒊 +�𝜷𝜷𝟐𝟐𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒓𝒓𝒄𝒄𝒔𝒔𝒊𝒊𝒄𝒄𝒓𝒓𝒊𝒊• We have also proposed the simplest regression model for uncovering

the treatment effect using RDD.• As with any regression analysis, we may consider accounting for covariates.�𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒓𝒊𝒊 = �𝜷𝜷𝟎𝟎 +�𝜷𝜷𝟏𝟏𝒔𝒔𝒔𝒔𝒓𝒓𝒔𝒔𝒔𝒔𝒊𝒊 +�𝜷𝜷𝟐𝟐𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒓𝒓𝒄𝒄𝒔𝒔𝒊𝒊𝒄𝒄𝒓𝒓𝒊𝒊 +𝜹𝜹𝐗𝐗𝐢𝐢

• Choosing the bandwidth on either side of the cut-point:• Can test various bandwidths of observations.• Changing bandwidth has estimation implications as well as validity

implications.

Internal Validity of RDD

• Key assumption: Observations close to one another on either side of the cut-point are equal in expectation on pre-treatment covariates.

• In the Angrist and Lavy (1999) example, there should be little difference, on average, between students in cohorts of sizes 36-40 vs. 41-45.

• Expect prior achievement, funding, school resources, etc. to be equal in expectation.• Difference between two groups is having one large class vs. two smaller classes.

• Implications of bandwidth choice:• Smaller bandwidth (i.e. closer to cut-point), treat./comp. more similar.• Larger bandwidth (i.e. further from cut-point), treat./comp. less similar.

• Key threat: Manipulation of treatment status around cut-point.• Conduct McCrary Test (2008) to examine density of observations at cut-point.

Internal Validity of RDD

• Key threat: Manipulation of treatment status around cut-point.• Conduct McCrary Test (2008) to examine density of observations at cut-point.• If there is a discontinuity, suggests some observations able to “manipulate”

assignment to treatment condition.

• What do you do if either a) manipulation occurs or b) individuals are not perfectly assigned to treatment or comparison conditions at the cut-point (i.e. there is not a “sharp” discontinuity)?

• Turn to “Fuzzy” RDD and an Instrumental Variables (IV) approach.• The forcing variable is now used as an instrumental variable to predict the likelihood of

receiving the treatment (first stage).• Use the predicted likelihoods of receiving treatment as instrument for estimating the

effect of the treatment on the outcome (second stage).

External Validity of RDD

• Key assumption: Observations close to one another on either side of the cut-point are equal in expectation on pre-treatment covariates.

• From an external validity perspective, we are estimating an average treatment effect generalizable only to those observations within the bandwidth we have chosen.

• Implications of bandwidth choice:• Smaller bandwidth (i.e. closer to cut-point), less generalizable across all

values of forcing variable, and by extension, the population.• Larger bandwidth (i.e. further from cut-point), more generalizable across all

values of forcing variable, and by extension, the population.

Thank You!

• Any Questions?• [email protected]

mailto:[email protected]

Documents

Applied Psychometric Strategies Lab Applied Quantitative