Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
PUBL0050: Advanced Quantitative Methods
Lecture 6: Synthetic Control Method
Jack Blumenau
1 / 50
Course outline
1. Potential Outcomes and Causal Inference2. Randomized Experiments3. Selection on Observables (I)4. Selection on Observables (II)5. Panel Data and Difference-in-Differences6. Synthetic Control7. Instrumental Variables (I)8. Instrumental Variables (II)9. Regression Discontinuity Designs10. Overview and Review
2 / 50
Lecture outline
Motivation
Synthetic Control
Inference
Additional application
Conclusion
3 / 50
Motivation
Motivation
Comparative case studies have a long history in applied political science:
โข Qualitative: โthickโ description of the context/features of two or moreinstances of specific phenomena. Aim to describe contrasts orsimilarities across the cases and reason inductively about causality
โข Quantitative: more explicitly causal, using (normally) aggregate datafrom one treated unit and a small set of control units. Often based onโnatural experimentsโ where a shock affects one unit, but not others.
Unifying logic:
Systems as similar as possible with respect to as many features aspossible constitute the optimal samples for comparative enquiry.
Przeworski and Tenure, 1970, p. 32
4 / 50
Quantitative Comparative Case Studies
Goal:
โข Estimate effects of events or policy interventions that take place at anaggregate level
โข Types of unit: cities, states, countries, etc
โข Types of intervention: passage of laws, economic shocks, etc
Approach:
โข Compare the evolution of an aggregate outcome for the unit affectedby the intervention to the evolution of the same outcome for somecontrol group
โข e.g. Card (1990), Card and Krueger (1994), Abadie and Gardeazabal(2003)
5 / 50
Quantitative Comparative Case Studies
Advantages:
โข Policy interventions often take place at an aggregate levelโข Aggregate/macro data are often available
Problems:
โข Selection of control group is often ambiguousโข Standard errors do not reflect uncertainty about the ability of thecontrol group to reproduce the counterfactual of interest
Solution:
โข If you donโt have a good control group: synthesize one
6 / 50
Running example
Reunification of West and East GermanyWhat were the economic effects of reunification on the West Germaneconomy? Many economic historians argue that reunification had largenegative economic costs, but identification is difficult because there is noobvious country with which we can compare the growth trajectory of WestGermany. Abadie et al (2015) estimate the effects of reunification bycomparing the actual time series for West Germany with a syntheticcontrol group which provides the counterfactual.
โข Outcome: GDP per capita (inflation adjusted)โข Treatment: Reunification (1 for W. Germany after 1990, 0 otherwise)โข Time: Years (1960 to 2003)
7 / 50
What should be the control group?
What is the most appropriate control group for evaluating the effects ofreunification on West Germany in 1990?
โข Geographical/cultural: Austria?โข Economic: USA?โข Average: OECD countries?
The choice of the control group matters!
8 / 50
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West Germanyrest of OECD sample
reunification
9 / 50
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West GermanyMean of OECD sample
reunification
10 / 50
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West GermanyAustria
reunification
11 / 50
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West GermanyUSA
reunification
12 / 50
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West GermanyUSA and Austria
reunification
13 / 50
What should be the control group?
Synthetic control moves away from using a single control unit or a simpleaverage of control units.
Instead we use a weighted average of the set of control or โdonorโ units.
Rather than assuming that either the USA or Austria are similar to W.Germany, we calculate a weighted average (the synthetic control) which ismore similar to West Germany than any individual country.
IntuitionWhen we only have a few aggregate units, a โsyntheticโ combination ofcontrol units may do a better job of reproducing the characteristics of thetreated unit than any one unit alone.
14 / 50
Synthetic Control
Notation
DefinitionFor units ๐ โ 1, ..., ๐ฝ + 1:โข Unit 1 is the unit of interest (which receives the treatment)
โข Units 2 to ๐ฝ + 1 are the โdonor poolโ or potential comparison unitsTime periods โ 1, ..., ๐ :โข Pre-treatment period: ๐ก = 1, ..., ๐0
โข Post-treatment period: ๐ก = ๐0 + 1, ..., ๐Potential outcomes:
โข ๐ ๐๐๐ก = outcome for unit ๐ at time ๐ก in the absence of the intervention
โข ๐ ๐ผ๐๐ก = outcome for unit ๐ at time ๐ก when exposed to the intervention
15 / 50
Notation
Estimand
๐ฟ1๐ก = ๐ ๐ผ1๐ก โ ๐ ๐
1๐ก = ๐1๐ก โ ๐ ๐1๐ก for all ๐ก > ๐0
i.e. the treatment effect on the treated unit in the post-treatment periods.
Problem
We cannot observe ๐ ๐1๐ก
Why? โ Fundamental problem of causal inference.
The critial question, as always, is how should we impute ๐ ๐1๐ก ?
15 / 50
Imputing ๐ ๐1๐ก
1. Matching
โข For each time period ๐ก, find the ๐ โclosestโ units to unit 1 and averagethe observed outcomes:
๐ ๐1,๐ก=1 = 1
๐๐
โ๐=1
๐๐๐(1),๐ก=1
2. Diff-in-diff
โข Add the average change in outcome for the control group to thetreated unitโs outcome in the pre-treatment period
๐ ๐1,๐ก=1 = ๐1,๐ก=0 + ( ๐0,๐ก=1 โ ๐0,๐ก=0)
3. Synthetic control
โข Take a weighted average of the outcomes of the donor unitsโข Weights defined by closeness to the trend of the outcome for thetreated unit in the pre-treatment period
๐ ๐1,๐ก=1 =
๐ฝ+1โ๐=2
๐คโ๐๐๐,๐ก=1
16 / 50
Defining the synthetic control
Definition (Synthetic control)A synthetic control is a vector of weights, ๐ , associated with each of theavailable ๐ฝ donor units.
Compare to our three examples above. ๐ is a vector withโฆ
โข โฆequal weight for each unit (OECD average)โข โฆ0 weight for all units, except Austria where ๐ค๐ = 1 (Austria)โข โฆ0 weight for all units, except USA where ๐ค๐ = 1 (USA)โข โฆ0 weight for all units, except USA where ๐ค๐ = .5 and Austria where
๐ค๐ = .5 (USA and Austria)
There are many potential synthetic controls! The goal is to select ๐ suchthat the characteristics of the treated unit are best resembled by thecharacteristics of the synthetic control.
17 / 50
Estimating๐
Basic approach: compare treatment and control units in terms of theirpre-intervention characteristics ๐1 and ๐0. Put more weight on countriesin the donor pool that are similar to the treatment unit in terms of:
โข Covariates that are predictive of post-intervention outcomesโข Pre-intervention outcome values
The idea is to give more weight to units in the donor pool that closelyapproximate the treated unit in the pre-intervention period.
I.e. We look for weights, ๏ฟฝ๏ฟฝ๐, such that
๐1,๐ก โ๐ฝ+1โ๐=2
๏ฟฝ๏ฟฝ๐๐๐,๐ก for all ๐ก โ 1, ..., ๐0
18 / 50
Estimating๐
For each donor unit, define a weight ๐ = {๐ค2, ๐ค3, ..., ๐ค๐ฝ+1}. We findthe values of ๐ by minimizing the following expression:
๐โ๐=1
๐ฃ๐(๐1๐ โ ๐0๐๐)2
where
1. ๐ค๐ โฅ 0 for ๐ = 2, ..., ๐ฝ + 1 (only positive or zero weights allowed)2. โ๐ฝ+1
2 ๐ค๐ = 1 (weights must sum to 1)3. ๐ฃ๐ is a weight that reflects the importance of the ๐th variable thatwe use to measure the distance between treated and control units
19 / 50
Estimating๐
There are two components that we need:
1. ๐ค๐ โ donor unit weights, indicating which units are most similar to thetreated unit
2. ๐ฃ๐ โ covariate weights, indicating which covariates are mostpredictive of the outcome
Estimate by minimizing the mean squared prediction error for thepre-intervention periods:
๐0
โ๐ก=1
(๐1๐ก โ๐ฝ+1โ๐=2
๐ค๐(๐ )๐๐๐ก)2
IntuitionMost weight (๐ค๐) is put on control units which are similar to the treatedunits on covariates (๐1, ๐0) that are predictive of the outcome (๐ฃ๐) inthe pre-intervention period (๐ก โค ๐0)
20 / 50
Estimating๐
SC is, at heart, a sort of difference-in-differences matching estimator.
โข Diff-in-diff: establish a control group which follows a parallel trend inthe absence of treatment (note we are still assuming that theparallel-ness would continue post-treatment!)
โข Matching: ๐ค๐ calculated using observed pre-treatment observedcovariates But Abadie et al argue that SC also accounts fornon-parallel trends:
Only units that are alike in both observed and unobserved determi-nants of the outcome variableโฆshould produce similar trajectoriesof the outcome variable over extended periods of time.
Abadie et al, 2015, p. 498
21 / 50
Estimating๐ (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal โ
Syn
thet
ic)
reunification
county weightUSA 0.06UK 0.06Austria 0.06Belgium 0.06Denmark 0.06France 0.06Italy 0.06Netherlands 0.06Norway 0.06Switzerland 0.06Japan 0.06Greece 0.06Portugal 0.06Spain 0.06Australia 0.06New Zealand 0.06
The Synth package in R will automate this optimization problem for us.
22 / 50
Estimating๐ (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal โ
Syn
thet
ic)
reunification
county weightAustria 0.14Japan 0.08Netherlands 0.08USA 0.07Switzerland 0.07UK 0.06Denmark 0.06France 0.06Spain 0.06Australia 0.06Italy 0.05Norway 0.05Belgium 0.04Greece 0.04Portugal 0.04New Zealand 0.03
The Synth package in R will automate this optimization problem for us.
22 / 50
Estimating๐ (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal โ
Syn
thet
ic)
reunification
county weightAustria 0.18Netherlands 0.13USA 0.11UK 0.07Greece 0.07New Zealand 0.07Japan 0.07Switzerland 0.05Belgium 0.04Denmark 0.04France 0.04Norway 0.04Spain 0.04Australia 0.04Portugal 0.01Italy 0.00
The Synth package in R will automate this optimization problem for us.
22 / 50
Estimating๐ (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal โ
Syn
thet
ic)
reunification
county weightAustria 0.28USA 0.10Switzerland 0.09Netherlands 0.08Denmark 0.07Greece 0.07New Zealand 0.07Japan 0.07Belgium 0.03France 0.03Norway 0.03Portugal 0.03Spain 0.03Australia 0.03UK 0.00Italy 0.00
The Synth package in R will automate this optimization problem for us.
22 / 50
Estimating๐ (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal โ
Syn
thet
ic)
reunification
county weightAustria 0.42USA 0.22Japan 0.16Switzerland 0.11Netherlands 0.09UK 0.00Belgium 0.00Denmark 0.00France 0.00Italy 0.00Norway 0.00Greece 0.00Portugal 0.00Spain 0.00Australia 0.00New Zealand 0.00
The Synth package in R will automate this optimization problem for us.
22 / 50
Estimating๐ (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal โ
Syn
thet
ic)
reunification
county weightAustria 0.42USA 0.22Japan 0.16Switzerland 0.11Netherlands 0.09UK 0.00Belgium 0.00Denmark 0.00France 0.00Italy 0.00Norway 0.00Greece 0.00Portugal 0.00Spain 0.00Australia 0.00New Zealand 0.00
The Synth package in R will automate this optimization problem for us. 22 / 50
Interpreting๐ (country weights)
Country weights in synthetic Germany:
County Weight County WeightAustria 0.42 France 0.00USA 0.22 Italy 0.00Japan 0.16 Norway 0.00Switzerland 0.11 Greece 0.00Netherlands 0.09 Portugal 0.00UK 0.00 Spain 0.00Belgium 0.00 Australia 0.00Denmark 0.00 New Zealand 0.00
23 / 50
Interpreting๐ (country weights)
Country weights in synthetic Germany (SC and OLS):
Country Synth Reg Country Synth RegAustria 0.42 0.26 France 0.00 0.04USA 0.22 0.13 Italy 0.00 -0.05Japan 0.16 0.19 Norway 0.00 0.04Switzerland 0.11 0.05 Greece 0.00 -0.09Netherlands 0.09 0.14 Portugal 0.00 -0.08UK 0.00 0.06 Spain 0.00 -0.01Belgium 0.00 -0.00 Australia 0.00 0.12Denmark 0.00 0.08 New Zealand 0.00 0.12
Regression weights can be greater than 1 or less than zero โ extrapolationoutside of the support of control units.
Extrapolation is not possible in the SC case because the weights are boundbetween 0 and 1.
24 / 50
Interpreting๐ (assessing balance)
GDP predictor means:
Treated Synthetic Rest of OECD SampleGDP per-capita 15808.9 15802.2 8021.1Trade openness 56.8 56.9 31.9Inflation rate 2.6 3.5 7.4Industry share 34.5 34.4 34.2
Schooling 55.5 55.2 44.1Investment rate 27.0 27.0 25.9
25 / 50
Interpreting ๐ฃ
Which variables are most important for determining the synthetic control?
Variable ๐ฃGDP per-capita 0.44Investment rate 0.24Trade openness 0.13Schooling 0.11Inflation rate 0.07Industry share 0.00
The weights ๐ฃ1, โฆ , ๐ฃ๐ should reflect the predictive value of the covariates.
26 / 50
Break
27 / 50
Causal effects
Weighting donor units leads to a synthetic unit with a similar outcometrend in the pre-intervention period as the treated unit.
Given ๏ฟฝ๏ฟฝ, an unbiased estimator of ๐ฟ1๐ก is:
๐ฟ1๐ก = ๐1,๐ก โ๐ฝ+1โ๐=2
๐ค๐๐๐,๐ก for ๐ก โ {๐0 + 1, โฆ , ๐ }
where
โข ๐1,๐ก is the outcome for the treated unit in post-treatment period ๐กโข โ๐ฝ+1
๐=2 ๐ค๐๐๐,๐ก is the outcome for the synthetic control unit inpost-treatment period ๐ก
โข ๐ฟ1,๐ก is the ATT for time period ๐ก
28 / 50
Causal effect I
1960 1970 1980 1990 2000
050
0010
000
1500
020
000
2500
030
000
Year
Per
โca
pita
GD
P (
PP
P, 2
002
US
D)
West Germanysynthetic West Germany
Reunification
29 / 50
Causal effect II
1960 1970 1980 1990 2000
โ40
00โ
2000
020
0040
00
Year
Gap
in p
erโ
capi
ta G
DP
(P
PP,
200
2 U
SD
)
Reunification
30 / 50
Inference
Asymptotic inference in synthetic control
Standard errors from regression/t-tests are typically used to characteriseuncertainty about aggregate data:
โข i.e. use a sample of restaurants in NJ and PA to estimate employmenttrends in each state
โข standard errors reflect unavailability of aggregate data onemployment
So, if we use aggregate data, is there zero uncertainty? No!
โข We do not have perfect information about potential outcomes, evenwhen we use aggregate data
โข We have uncertainty about the potential outcome under control forthe treated unit
But, because the number of units is small in most SC applications, largesample inferential techniques are not appropriate.
31 / 50
Permutation inference
Instead, we turn to an alternative inference technique: permutationinference.
1. Calculate the the test-statistic under the actual treatment assignment2. Calculate the distribution of the test-statistic under alternative
treatment assignments assuming treatment effects of zero3. Assess whether the โtrueโ test-statistic is unlikely under the null
distribution of treatment effects
Here, this implies constructing a synthetic control for every country in oursample, summarising the treatment effect, and comparing it to thetreatment effect in West Germany.
32 / 50
Permutation inference (example)
What is the test-statistic here? Two components:
โข Treatment effect (analogous to ๐ฝ in a regression)โข Pre-treatment difference between unit and SC (analogous to ๐๐ธ( ๐ฝ))
For each unit calculate:
RMSE๐,๐1 = โโโ
1๐1
๐โ
๐ก=๐0
(๐1,๐ก โ๐ฝ+1โ๐=2
๐ค๐๐๐,๐ก)2โโโ
1/2
RMSE๐,๐0 = โโโ
1๐0
๐0
โ๐ก=1
(๐1,๐ก โ๐ฝ+1โ๐=2
๐ค๐๐๐,๐ก)2โโโ
1/2
Where RMSE๐,๐0characterises the lack of fit between a unit and itโs SC in
the pre-treatment period, and RMSE๐,๐1is the equivalent for the
post-treatment period.33 / 50
Permutation inference (example)
Given these, the test-statistic is:
RMSE๐,๐1
RMSE๐,๐0
= Pre-intervention โfitโPost-intervention โfitโ
Intuition:
โข More confident that the effect is different from zero when theestimated treatment effect is larger (RMSE๐,๐1
)
โข Less confident that the effect is different from zero when thepre-treatment fit with the SC is larger (RMSE๐,๐0
)
P-value: how likely would it be to observe a ratio as large as the one weactually observe if the treatment effects were zero and we picked onecountry at random?
34 / 50
Permutation inference (example)
Portugal
Denmark
France
Netherlands
Japan
UK
Austria
Switzerland
Belgium
Australia
Spain
USA
New Zealand
Italy
Greece
Norway
West Germany
โ
โ
โ
โ
โ
โ
โ
โ
โ
โ
โ
โ
โ
โ
โ
โ
โ
5 10 15
PostโPeriod RMSE / PreโPeriod RMSE
35 / 50
Placebos in space
1960 1970 1980 1990 2000
โ40
00โ
2000
020
0040
00
Year
Gap
in p
erโ
capi
ta G
DP
(P
PP,
200
2 U
SD
)
West GermanyDonor countries
36 / 50
Placebos in time
1960 1965 1970 1975 1980 1985 1990
050
0010
000
1500
020
000
2500
030
000
year
perโ
capi
ta G
DP
(P
PP,
200
2 U
SD
)West Germanysynthetic West Germany
placebo reunification
37 / 50
Additional application
Californiaโs Proposition 99
Anti-smoking legislation and cigarette consumptionIn 1988, California passed comprehensive tobacco control legislation. Thiswas a package of measures that included a tax increase, more earmarkedspending to anti-smoking health initiatives, and anti-smoking mediacampaigns. We will investigate the effect of this legislation on cigaretteconsumption in California using synthetic control methods.
โข Outcome variable (Y): Per capita cigarette sales (packs)โข Treatment (D): 1 for CA after 1988, 0 for all other periods/statesโข Time (T): 1970 to 2000
(All states which passed similar legislation are excluded from the donor pool.)
38 / 50
Californiaโs Proposition 99
1970 1975 1980 1985 1990 1995 2000
020
4060
8010
012
014
0
year
perโ
capi
ta c
igar
ette
sal
es (
in p
acks
)Californiarest of the U.S.
Passage of Proposition 99
39 / 50
Californiaโs Proposition 99
1970 1975 1980 1985 1990 1995 2000
020
4060
8010
012
014
0
year
perโ
capi
ta c
igar
ette
sal
es (
in p
acks
)Californiasynthetic California
Passage of Proposition 99
40 / 50
State Weights in Synthetic California
State WeightUtah 0.334Nevada 0.234Montana 0.199Colorado 0.164Connecticut 0.069
41 / 50
Predictor means: Real vs. Synthetic California
California Average ofVariables Real Synthetic 38 control statesLn(GDP per capita) 10.08 9.86 9.86Percent aged 15-24 17.40 17.40 17.29Retail price 89.42 89.41 87.27Beer consumption per capita 24.28 24.20 23.75Cigarette sales per capita 1988 90.10 91.62 114.20Cigarette sales per capita 1980 120.20 120.43 136.58Cigarette sales per capita 1975 127.10 126.99 132.81Note: All variables except lagged cigarette sales are averaged for the 1980-1988 period(beer consumption is averaged 1984-1988).
42 / 50
Californiaโs Proposition 99
1970 1975 1980 1985 1990 1995 2000
โ30
โ20
โ10
010
2030
year
gap
in p
erโ
capi
ta c
igar
ette
sal
es (
in p
acks
)
Passage of Proposition 99
Cigarette sales gap in CA (versus synthetic CA).
43 / 50
Californiaโs Proposition 99
1970 1975 1980 1985 1990 1995 2000
โ30
โ20
โ10
010
2030
year
gap
in p
erโ
capi
ta c
igar
ette
sal
es (
in p
acks
) Californiacontrol states
Passage of Proposition 99
Cigarette sales gap in all 38 states.
44 / 50
Californiaโs Proposition 99
1970 1975 1980 1985 1990 1995 2000
โ30
โ20
โ10
010
2030
year
gap
in p
erโ
capi
ta c
igar
ette
sal
es (
in p
acks
) Californiacontrol states
Passage of Proposition 99
Cigarette sales gap in states with pre-intervention MSPE < 20 โ MSPECA.45 / 50
Californiaโs Proposition 99
1970 1975 1980 1985 1990 1995 2000
โ30
โ20
โ10
010
2030
year
gap
in p
erโ
capi
ta c
igar
ette
sal
es (
in p
acks
) Californiacontrol states
Passage of Proposition 99
Cigarette sales gap in states with pre-intervention MSPE < 5 โ MSPECA.46 / 50
Californiaโs Proposition 99
1970 1975 1980 1985 1990 1995 2000
โ30
โ20
โ10
010
2030
year
gap
in p
erโ
capi
ta c
igar
ette
sal
es (
in p
acks
) Californiacontrol states
Passage of Proposition 99
Cigarette sales gap in states with pre-intervention MSPE < 2 โ MSPECA.47 / 50
Conclusion
Data requirements
Synthetic control has relatively low data requirements:
โข Can use aggregate data (often administrative)
โข e.g. economic indicators such as GDP, current-account balance, etc;political indicators such as turnout, vote share, etc
โข Causal factors can be big and important
โข e.g. legislation changes, macro-shocks, etc
โข Units of analysis can be large
โข Countries, states, regions, etc
โข Does not even require full panel data for the pre-treatment period
โข Can use averages of covariates rather than full panel data (useful whencovariates do not vary yearly)
48 / 50
Conclusion
The synthetic control approachโฆis arguably the most importantinnovation in the policy evaluation literature in the last 15 years.
Athey and Imbens, 2017
Advantages:โข Builds on D&D and Matching by essentially forcing the data to exhibitparallel trends in the pre-treatment period
โข Amenable to small-ish N comparisons (often easier to get data)โข Clear, transparent, and easily communicable comparisons(e.g. Germany is part Austria, part USA, etc)
49 / 50
Conclusion
The synthetic control approachโฆis arguably the most importantinnovation in the policy evaluation literature in the last 15 years.
Athey and Imbens, 2017
Disadvantages:โข Inference is not straightforward! Asymptotic inference does not workwith the SC method
โข Coding is not straightforward! As you will see in the seminar :)โข Pre-intervention period must be relatively large for us to trust paralleltrends holds in the post-intervention period
49 / 50
Next week
Instrumental Variables (I)
50 / 50