ANNUAL LECTURE

Heterogeneous Agents, Social Interactions and Causal Inference

This talk is based on “Heterogeneous Agents, Social Interactions, and Causal Inference” by Guanglei Hong and Stephen W. Raudenbush, to appear in Morgan, S. (Ed.) Handbook of Causal Analysis for Social Research (Springer 2012) and draws

on two examples originally reported in:

Savitz-Verbitsky, N. and Raudenbush, S.W. (in press). Evaluating community policing program in Chicago: A case study of causal inference in spatial settings. To appear in Epidemiologic Methods; and

Raudenbush, S.W., Reardon, S. and Nomi, T. (in press). Statistical analysis for multi-site trials using instrumental variables. To

appear in Journal of Research and Educational Effectiveness.

The research reported here was supported by a grant from the Spencer Foundation entitled “Improving Research on Instruction: Models Designs, and Analytic Methods;” and a grant from the W.T. Grant Foundation entitled “Building Capacity for

Evaluating Group-Level Interventions.”

21ST MARCH 2012

Stephen W. RaudenbushLewis-Sebring Distinguished Service

Professor in the Department of Sociology at the University of Chicago andChairman of the Committee on Education

Abstract

This talk will focus on two pervasive features of social interventions designed to increase human health, skills, or productivity. First, the interventions are usually delivered by human agents – physicians, teachers, case workers, therapists, police officers, or workplace managers - who tend to be ‘heterogeneous’ in the sense that they differ in their beliefs, training, and experience. These agents enact the intervention and shape its effects. Second, the participants in these interventions – patients, pupils, employees or offenders - are typically clustered in organizational settings, and social interactions among these participants influence the success of the intervention. In this presentation, Stephen will argue that causal models conventionally used in medical research are not well suited to study these interventions. Instead, he proposes a model in which the heterogeneous agents and social interactions among participants shape participants’ response to an intervention. Stephen will illustrate this model with studies of community policing and high-school curricular reform.

OutlineCounter-Factual Account of Causation

The “drug-trial paradigm” for causal inference

An alternative paradigm for social interventionsHeterogeneous agentsSocial interactions among participants

ExamplesCommunity policingHigh School Curricular Reform

Conclusions

Counter-factual Account of Causality

In statistics (Neyman, Rubin, Rosenbaum)

In economics (Haavelmo, Roy, Heckman)

Potential Outcomes in a Drug Trial

Y(1): Outcome if the patient receives Z = 1

(the “new drug”)

Y(0): Outcome if the patient receives Z = 0

(the “standard treatment”)

Y(1) – Y(0): Patient-specific causal effect

E (Y(1) – Y(0)) = : Average causal effect

Stable Unit Treatment Value Assumption (Rubin, 1986)

• Each patient has two potential outcomes• Implies

– Only one “version” of each treatment– No “interference between units”

• Implies the doctor and the other patients have no effect on the potential outcomes

Formally…

)();,...,,( 11211 zYdzzzY n

Failure of SUTVA in Education

• Teachers enact instruction in classrooms– Multiple “versions of the treatment”

• Treatment assignment of one’s peers affects one’s own potential outcomes– EG Grade Retention

– Hong and Raudenbush, Educational Evaluation and Policy Analysis, 2005

– Hong and Raudenbush, Journal of the American Statistical Association, 2006

Group-Randomized Trials

Potential outcome

Thus, each child has only two potential outcomes – if we have “intact classrooms”– if we have “no interference between classrooms”

controltoassignedisjiftY

treatmenttoassignedisjiftY

tzzzY

jj

jj

jnjjjj

);0,...,0,0(

);1,...,1,1(

);,...,,(

1

1

211

Limitations of cluster randomized trial

Mechanisms operate within clusters

* Example: 4Rs

teachers vary in response

classroom interactions spill over

We may have interference between clusters

* Example: community policing

Alternative ParadigmTreatment setting (Hong, 2004):

A unique local environment for each treatment composed of * a set of agents who may implement an intervention and* a set of participants who may receive it

Each participant possesses a single potential outcome within each possible treatment setting

Causal effects are comparisons between these potential outcomes

);,...,,( 21 jnjjij tzzzYj

Example 1: Community Policing (joint work with Natalya Verbitsky-Shavitz)

• Let Zj=1 if Neighborhood j gets community policing

• Let Zj=0 if not

• Under SUTVA

)0()1( jjj YY

Relaxing SUTVA

Potential outcome for any unit depends on the treatment assignment of ALL units in the population,

),(),( **

jjjjjjj ZZYZZY

),( jjj ZZY

Individual Causal Effect:

Population Average Causal Effect:

)],(),([][ **

jjjjjjj ZZYZZYEE

“All or none”

)0,0()1,1(

jjj YY

1

1

1

1

10

0

0

0

0

“Shall we do it in my neighborhood?”

),0(),1( jjjjj ZYZY

1

1

0

1

00

1

0

1

0

Do it only in high-crime areas: effect on those areas

)0,0(),1(

jjjj YZY

1, HC

1, HC

0, LC

0, LC

1, HC

0, HC

0, HC

0, LC

0, LC0, HC

Do it only in high-crime areas: effect on low-crime areas

)0,0(),0( ''''

jjjj YZY

1, HC

1, HC

0, LC

0, LC

1, HC

0, HC

0, HC

0, LC

0, HC

0, LC

Spatial Causal Assumptions (1)

1)(:

))(,(),(

zwithneighborsoffractionZFEx

ZFZYZZY

j

jjjjjj

Functional Form:

1, #3

1, #1

0, #4

0, #5

1, #2

2/1

0

0

4/1

4/1

3

F

Longitudinal Design: 25 districts, 279 “beats”

91 92 93 94 95 96 97 98 99

No community policing

25 25 25 20 20 0 0 0 0

Community policing

0 0 0 5 5 25 25 25 25

Results

Having community policing was especially good if your surrounding neighbors had it

Not having community policing was especially bad if your neighbors had it

*** So targetting only high crime areas may fail***

Example 2: Double-dose Algebra

Requires 9th-graders to take Double-dose Algebra if they scored below 50 percentile on 8th-grade math test

1200 students in 60 Chicago high schools

Double-dose Algebra enrollment rate by math percentile scores (city wide)

Enro

llmen

t Rat

es

ITBS percentile scores

Conventional Mediation Model (T, M,Y model)

Cut off (T)Double-Dose Algebra (M)

Algebra Learning (Y)

• Assume no direct effect of T on Y (exclusion restriction)• Δ= Effect of double dose on the “compliers”• Δ Γ= Effect of assignment to double dose (“ITT” effect)

Nomi, T., & Allensworth, E. (2009)

Γ Δ

Effects of Double-dose Algebra:District-wide average

Effect of cutoff on taking DD (average compliance rate): Increase prob by .72

District-wide average ITT effect on Y: Average effect≈0.15

District-wide average Complier-Average Treatment EffectAverage ≈0.21

double-dose algebra effects varied across schools

But the policy changed classroom composition!!

Classroom average skill levels by math percentile scores

Pre-policy (2001-02 and 2002-03 cohorts)

Post-policy (2003-04 and 2004-05 cohorts)

Implementation varied across schools in---

• Complying with the policy • Inducing classroom segregation

Exclusion Restriction RevisedT-M-C-Y model

Cut off (T)

Double-Dose Algebra (M)

Algebra score (Y)

Classroom Peer ability (C)

Research Questions

1) What is the average effect of assignment to DD? (“ITT effect”)

2) What is the average effect of taking double-dose algebra? (effect “on the compliers”).

3) How much do these effects vary across schools?4) What is the effect of taking double-dose Algebra,

holding constant classroom peer ability?5) What is the effect of classroom peer ability, holding

constant taking double-dose Algebra?

Results

  Degree of sortingLow Average High

ITT 0.21 0.15 0.11

Complier effects 0.29 0.23 0.14

School N 19 19 22

The effect of double-dose algebra on algebra scores by the degree of sorting

We now estimate the effect of taking DDA and classroom peer composition

Statistical Models

Stage 1: the effect of Cut-off on Double Dose

and Peer Ability

Stage 2: the effect of M and C on Y

23210

23210

ijjijjijjjij

ijjijjijjjij

XXCutBelowPEERE

XXCutBelowDDE

)()(

)()(

243210 ijijjijijjij XXPeerEDDEYE )()()(

31

Stage 1 Results:the average effect T on M and C

Double-dose algebra enrollment Peer composition

Coeff 0.72*** -0.24***

SE 0.03 0.03

The effect of the cutoff score (T) on double-dose algebra enrollment (M) and peer composition (C)

Note: *** p<.001, **p<.01, * p<.05

32

Context specific effects:The effects of cutoff score on double-dose algebra

enrollment and peer ability

Th

e ef

fect

of

cut

off

sco

re o

n p

eer

abil

ity

The effect of cut off score on double-dose algebra enrollment

Stage 2 results: The effect of M and C on Y

The average effect of taking double-dose algebra (M) and peer ability (C) on Algebra test scores

Double-dose algebra enrollment

Classroom Peer composition

Coeff 0.30*** 0.40***

SE 0.06 0.12

34

5. Conclusions

The reform enhanced math instruction for low-skill students, and that helped a lot

The reform also intensified tracking and that hurt

On balance the effect was positive, but much more so in schools that implemented double dose with minimal tracking

Final Thoughts

Conventional causal paradigm:* a single potential outcome per participant under each treatment

Alternative paradigm* a single potential outcome per participant in each treatment setting

- aims to avoid bias-open up new questions

Policy implications are potentially large