Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1
SLIDES FOR*
Comparing the Conley-Taber and the Standard Approaches to Inference in Difference-in-Difference Models Based on Small Policy Variation: The Case
of TennCare
John C. Ham
NYU Abu Dhabi and Wagner School of Public Service, NYU
IFAU, IRP and IZA
Ken Ueda The Office of the Comptroller
Of the Currency
Revised September 2018
* We thank Corina Mommaerts for her comments on these slides without implicating her.
2
Introduction and Motivation
Many empirical studies focus on a small
number of policy changes in a few locations.
The extreme version of this is one policy
change in one year in one state. In this case,
Conley and Taber (2011) show under
reasonable assumptions about how one does
the asymptotics, one cannot estimate the
treatment effect consistently, but can
estimate consistently a confidence interval
for the treatment effect.
3
Garthwaite, Gross and Notowidigdo (2014,
hereafter r GGN), using data from the March
CPS, analyzed the effect of a major
contraction in Medicaid coverage in
Tennessee’s TennCare program in 2000’s.
They aggregated the micro data up to the
state-year level. GGN got very large
estimated effects, and on the basis of these,
predicted the Affordable Care Act (ACA),
also known as Obamacare, would have
substantial labor market effects.
4
However several papers have investigated
this issue and found no labor market impact
of the ACA. But perhaps their analysis did
not offer enough power to reject the null
hypothesis of the ACA having no effect even
if the GGN estimates are ‘correct’.
Their regression results seemed very high
compared to other studies, see e.g.,
The survey, Buchmueller, Ham and Shore-
Sheppard (2016).
5
To investigate this, we estimated their models
for two much larger data sets i) all months
of the CPS (ALLCPS) and iii) American
community survey (ACS).
We find quite big differences from those
obtained from the March CPS (MCPS) data.
These do not go away even when we keep
only the estimates that pass diagnostic tests.
6
In terms of the latter, Diff-in-Diff (DD)
estimation using the March CPS produces a
big significantly positive impact on
employment, while the ACS produces
significantly negative impact.
But for Triple Difference (DDD) estimation
the DDD March CPS estimate doubles, while
the ACS estimate becomes significantly
positive.
We also show below that the GGN point
estimates should have enough power to reject
the null hypothesis of no labor market effect
of the ACA, although this is not true of all of
the estimates in their CI.
7
We hypothesize that the above problems of
the regression occur because the model is not
identified in the sense of Conley and Taber
(ReStat 2011, CT hereafter) – everything is
coming from one policy change in one state
in one year. This is a widely cited paper in
empirical papers and theoretical papers, but
seems to be not widely implemented to this
point.
We then estimate TC estimate consistent
Confidence Intervals (CIs) for the TennCare
effect.
8
Again we apply specification tests based on
CT’s consistent CIs for placebo coefficients
and find that we can reject the null of no
placebo effect for several specifications.
Our results for the CT CIs, using the
specifications that survive the tests, produce
a number of fairly wide confidence intervals
that are consistent with the null hypothesis of
no TennCare effect, and hence more in
keeping with the previous literature.
9
But we do not believe that one can claim that
this result is just a case of CT generally
producing noisy CIs, since their CIs were
precise enough in the placebo tests to reject
several specifications.
Too many slides here for 30 minutes but
hopefully the extra slides will help those
interested in the absence of a paper.
10
Paper Organization
1. Review of TennCare.
2. Discuss the GGN regression approach for
the MCPS – specification and results.
3. Review of i) previous research on the
impact of being offered public health
insurance on labor market outcomes ii)
research on measuring the labor market
effects of the ACA.
4. Use the regression approach on the
ALLCPS and ACS.
11
5. Conduct a power analysis for the ACA
impact using the MCPS, ALLCPS, and
ACS regression estimates. Evaluate power
at the parameter estimates and obtain a
consistent CI for the power function
evaluated at the regression estimates using
the approach in Woutersen and Ham
(2018).
6. Apply specification (placebo) tests to the
results from the MCPS, ALLCPS and ACS.
The surviving estimates are very different
across data sets.
7. Review the CT approach to estimating
consistent estimates of the CI for the
treatment effect.
12
8. Use the CT approach on the MCPS,
ALLCPS and ACS. We also conduct power
analysis using their results. Here we can
only provide CIs for the power function,
since CT does not produce consistent
parameter estimates.
9. Apply specification (placebo) tests to the
CT CIs from the MCPS, ALLCPS and
ACS. Surviving estimates are quite similar
across data sets. All CIs include zero and
are fairly wide.
10. Conclude.
13
TennCare Basics
• Started in 1994 to cover
“uninsurable”/“uninsured” individuals.
• Prevent loss of federal funds/Emergency
Room utilization.
• Enrolled all Medicaid recipients into
Managed Care plans, used savings to cover
“uninsurables” – i.e. people who were
rejected by private insurance plans; no
income restrictions. Examples: Displaced
workers, children whose parents did not have
access to workplace insurance.
14
• TennCare faced a $342 million shortfall for
2001.
• BCBST (Blue Cross Blue Shield Tennessee),
which covered almost half of all TennCare
patients, threatens to pull out of TennCare
due to rising costs.
• 2002 – changed definition of eligibility
• Enrollee must have gone through medical
review of “insurability”
• “Reverification” - enrollees must
schedule appointments to determine if
they remained eligible for benefits.
15
Enrollee income distribution (1995)
• 40% had incomes above 100% of the
poverty line.
• 6.3% had incomes between 200% to
400% of poverty line
• 1.3% had incomes above 400% of
poverty line
We don’t have later breakdown since in our
data sets we don’t see income and public
insurance in the same year.
This would seem to contradict the argument
that the big effects in GGN occur because
TennCare covered a lot of people over 200%
of the poverty line.
16
In November 2004, Governor Philip
Bredesen announced that childless
individuals would not be covered as of July 1,
2005, and disenrollment started then.
17
Regression Estimates
The fact that the contraction started in July
2005 is important since we are going to look
at data from 2000-2007, where 2000-2005
are the control years and 2006-2007 in
Tennessee are the treatment years.
In the MCPS, 2005 is actually April 2004-
March 2005. For the ALLCPS and ACS, the
2005 year covers January-December 2005.
18
But there is the question of whether we
should include 2005 for all the data sets.
For the MCPS, the 2005 comparison year
included January-March 2005 period before
the TennCare contraction. But it was
announced in November 2004 that is would
be implemented in July 1, 2005. Hence the
announcement might have lead to an
anticipation effect.
For the ALLCPS and ACS, the 2005 control
year included January-June 2005 before the
TennCare contraction and June-December
2005 after the contraction.
19
We drop 2005, but surprisingly this has no
effect on the results from any data sets.
One could also worry about 2006 treatment
year for the MCPS since it covers April 2005-
March 2006, but in the absence of
anticipation effects, was not in place April –
June 2005.
20
GGN’s Approach
To evaluate the impact of TennCare GGN
use Difference in Difference Estimation
��� = �� + �� + �� + ��[�����]
�[����� ≥ 2006]+ ��� (1)
• where
���: employment rate within state �, year �,
�: Comparison of change in Tennessee
employment rate after Tenn Care disenrollment
vs change in employment rate in other southern
states or Tennessee before 2006.
21
• Comparison States: Alabama, Arkansas,
Delaware, DC, Florida, Georgia, Kentucky,
Louisiana, Maryland, Mississippi, North
Carolina, Oklahoma, Texas, Virginia, South
Carolina, West Virginia.
• Sample MCPS 2000-2007 for GGN.
• The treatment effect is identified by the
comparison of 2005 and 2006 employment in
their analysis, or 2004 and 2006 employment
after we drop 2005.
• Crucial Assumption – trends constant across
states, and we will test this. To relax this
assumption use a triple difference
specification.
22
• Since the TennCare contraction affected only
adults without children under 17 years old, so
GGN can eliminate state specific trends by
taking the difference between the two groups
in each state.
For childless adults they hypothesize that
1 1 1
1 1
{ [ ] * [ 2006]}
, (2)
st s t
st st
L I s Tenn I t
v
For adults with children
2 2 2 2 2 . (3)st s t st stL v
23
Assume that the state specific trends 1st and
2st are equal (i.e. 1 2st st ). Then subtracting
(3) from (2) yields
1 2 1 2 1 2 1 2
1 12
( ) ( ) ( )
{ [ ]* [ 2006]} ( - )
{ [ ]* [ 2006]} .
st st s s t t st st
st t
s t st
L L
I s Tenn I t v v
I s Tenn I t v
We will test the null hypothesis that 1 2st st
using placebo tests.
24
• Sample: 21-64 years old (inclusive), no
Military, at most college degree.
• We can replicate their results exactly.
25
Table 1 Regression Estimates of the TennCare Effect on Employment (Hours) Using the
March CPS from GGN
Difference-in-Difference
Triple Differences
(1) (2)
Working > 0 hours Point Estimate 0.025** 0.046** Standard Error (0.011) (0.020)
0<Working < 20 hours Point Estimate -0.001 0.002 Standard Error (0.004) (0.009)
Working ≥ 20 hours
Point Estimate 0.026*** 0.044**
Standard Error (0.010) (0.020)
Working ≥ 20 hours, < 35 hours Point Estimate 0.001 0.018 Standard Error (0.007) (0.013)
Working ≥ 35 hours Point Estimate 0.025** 0.026 Standard Error (0.011) (0.021)
Notes: *, **, and *** denote significance at the 10%, 5% and 1% level respectively.
26
There is a question which estimates/hours
group we should focus on.
We would argue that if the we are thinking
people are getting jobs to get Employer
Sponsored Insurance (ESI), the hours > 35 is
the most plausible.
GGN argue that hours > 20 is the most
relevant because part-time workers also can
get ESI.
27
But Carroll and Miller use the MEPS data to
study the extend of ESI for full-time and part-
time workers, where part-time is < 30 hours
per week.
They show that in 2005, 85.6% of full-time
workers were eligible for ESI and 70.6% had
it.
However they show that in 2005, 39.7% of
part-time workers were eligible for ESI and
15.1% had it.
Full-Time defined as hours >=30, Part-time defined as hours <=30.
From Carroll and Miller (2018).
28
Hence we would argue that hours > 35 is the
most appropriate category. Below we also
consider hours > 0. In the next round we will
add hours > 20 to the presentation. Our
results do not change when we include them.
29
How do the MCPS estimates compare to
previous results:
There is a large body of literature on the
employment effects of Medicaid eligibility,
and most studies find small or non-existent
effects.
For example, Yelowitz (1995), from the
March CPS, found large employment effects
of Medicaid eligibility among single
mothers.
30
But Ham and Shore-Sheppard (2005), using
data from the March CPS and from SIPP, that
his results were an artifact of constraining
welfare benefits and Medicaid availability to
have the same coefficient. Once this
constraint was relaxed, welfare benefits, but
not Medicaid eligibility, continued to affect
employment.
31
Further, Meyer and Rosenbaum (2001), using
data from the Current Population Survey
(CPS) Outgoing Rotation Group Files and
from the March CPS, found an important role
for welfare benefits, but not Medicaid
provisions, in a static model of labor force
participation.
Recently Finkelstein et al. (2014) found that
offering individuals Medicaid coverage in the
Oregon Health Experiment had essentially no
effect on employment; since their result is
based on a randomized trial, this evidence is
perhaps the strongest to date.
32
What happens if we estimate the treatment
effect using the ALLCPS or ACS?
Below we drop 2005 but this has very little
effect on any of the estimates.
33
Table 2 Regression Estimates of the TennCare Effect on
Employment (Hours) by Database, 2000-2007, Omit 2005 GGN MCPS ALLCPS ACS (1) (2) (3) (4)
Panel A: Difference in Differences
Working > 0 hours Point
Estimate 0.025** 0.021** 0.015*** -0.013***
(0.011) (0.011) (0.004) (0.004) 95 Percent
CI [0.003, 0.047] [0.000, 0.042] [0.007, 0.023] [-0.021, -0.005]
Working ≥ 35 hours
Point Estimate
0.025** 0.022* 0.016*** -0.015***
(0.011) (0.012) (0.004) (0.004) 95 Percent
CI [0.003, 0.047] [-0.001, 0.045] [0.008, 0.024] [-0.024, -0.006]
Panel B: Triple Difference
Working > 0 hours
Point Estimate
0.046** 0.043** 0.005 0.002
(0.020) (0.022) (0.008) (0.006) 95 Percent
CI [0.007, 0.085] [0.000, 0.086] [-0.010, 0.020] [-0.010, 0.014]
Working ≥ 35 hours
Point Estimate
0.026 0.017 -0.006 0.013*
(0.021) (0.023) (0.009) (0.007) 95 Percent
CI [-0.015, 0.067] [-0.028, 0.062] [-0.023, 0.012] [0.000, 0.026]
Microdata
216,751 1,789,894 2,491,229
Notes: All Standard Errors calculated as in GGN.
34
If we include 2005 we get
Table 3 Regression Estimates of the TennCare Effect on Employment (Hours)
by Database, 2000-2007 GGN MCPS ALLCPS ACS (1) (2) (3) (4)
Panel A: Difference in Differences
Working > 0 hours Point Estimate 0.025** 0.025** 0.016*** -0.011***
(0.011) (0.011) (0.004) (0.004) 95 Percent CI [0.003, 0.047] [0.003, 0.047] [0.008, 0.024] [-0.019, -0.003]
Working ≥ 35 hours
Point Estimate 0.025** 0.025** 0.016*** -0.013*** (0.011) (0.011) (0.004) (0.004)
95 Percent CI [0.003, 0.047] [0.003, 0.047] [0.008, 0.024] [-0.021, -0.005]
Panel B: Triple Difference
Working > 0 hours Point Estimate 0.046** 0.046** 0.007 0.004
(0.020) (0.020) (0.008) (0.006) 95 Percent CI [0.007, 0.085] [0.007, 0.085] [-0.009, 0.023] [-0.008, 0.016]
Working ≥ 35 hours
Point Estimate 0.026 0.026 -0.002 0.015 ** (0.021) (0.021) (0.008) (0.006)
95 Percent CI [-0.015, 0.067] [-0.015, 0.067] [-0.018, 0.014] [0.003, 0.027]
Microdata 249,559 2,057,701 3,036,337
35
Summary of DD Estimates
• For both h > 0 and h > 35 specifications
get a significant positive treatment effect from the
MCPS and CPS, but a significant negative
treatment effect from the ACS.
Summary of DDD Estimates
• For h > 0 specification
MCPS estimates double when compared to the DD
estimates and significantly positive, but ALLCPS
and ACS are insignificant and have quite small CIs.
• For h > 35 specification
MCPS estimates are insignificant with a big CI.
ALLCPS estimates are insignificant with a small
CI. ACS are insignificant and have quite small CIs.
Hence the results are unstable across data sets.
36
Power calculations
If the TennCare effects are valid, should the
papers investigating the introduction of the
ACA have found significant effect?
Given a treatment effect, we simulate how
often we expect to be able to reject the null
hypothesis that there was no ACA effect
given above coefficients regression. For now
we consider the DD estimates.
The idea here is that others may have found
no ACA effect even if the above estimates are
valid because of a lack of power.
37
Let be our estimated TennCare Treatment
effect. Our power calculation is ˆ( ),POW
i.e. the fraction of time we simulate getting a
significant effect for the ACA introduction
and the TennCare parameters.
To get a consistent confidence interval for
ˆ( ),POW we need to take into account that
it is a random variable because of are two
sources of randomness here – from the
simulations and from the fact that is a
random variable.
38
If we just use the simulation at the point
estimates we not are considering randomness
from the fact that is estimated. Note that
we cannot use the delta method to calculate a
consistent confidence interval for ˆ( )POW
because ˆ( )POW is a non-differentiable
function.
39
To incorporate randomness from the fact that
is estimated, we might be tempted to a
large number of draws from the asymptotic
distribution of , call them ˆs , and then
calculate ˆ( )sPOW for each s. Then we
could take the top 2.5% of the ˆ( )sPOW
values and the bottom 2.5% of the ˆ( )sPOW
values, and drop them both to get a 95% CI
for ˆ( ).POW
Unfortunately, there is no consistency proof
for this approach, and Woutersen and Ham
(2018) construct counter-examples where it
does not produce consistent CIs.
40
• However, Woutersen and Ham (2018) show
that we can get a consistent confidence
interval for ˆ( )POW by taking a large
number of draws from the 95% confidence
interval for . Call them ˆs , and calculate
ˆ( )sPOW for each s. Then we take minimum
value of ˆ( )sPOW over these draws ˆs as the
lower limit of the CI for ˆ( )sPOW , and take
maximum value of ˆ( )sPOW over these
draws ˆs for the upper limit of the CI for
ˆ( )sPOW .
41
Table 4 Power Calculations Using DD Regression Coefficients
Point Estimate,
March CPS
March CPS
Point Estimate, All CPS
All CPS
Working>0
Expanded in 2014 with no prior expansions vs Did not expand 2014 with no prior expansions
0.97 [0.044, 1] 0.99 [0.576, 1]
Expanded in 2014 with no prior expansions vs Did not expand 2014
1.00 [0.042, 1] 1.00 [0.648, 1]
Working>35
Expanded in 2014 with no prior expansions vs Did not expand 2014 with no prior expansions
0.98 [0.042, 1] 0.99 [0.578, 1]
Expanded in 2014 with no prior expansions vs Did not expand 2014
1.00 [0.040, 1] 1.00 [0.650, 1]
(We did not do the calculations for the ACS since the
treatment effects are often negative for this data set and
because of time constraints.)
42
• We see that, at the point estimates, we would
certainly expect to see the null hypothesis of
no ACA effect rejected almost all the time.
• But the CI for the power function based on
the CIs for the TennCare treatment effect are
very wide and are consistent with low power
for rejecting the null that the ACA effect is
zero given the CIs for the TennCare
coefficients.
43
Placebo Tests
1. For the DD equations, we estimate
regressions for 2000-2004 we use year
dummies and
i) put in a dummy for 2003-2004 for
Tennessee and
ii) Put in a dummy for 2002-2004 for
Tennessee.
• This gives us 2 tests of the null hypothesis
that Tennessee has the same trend as the
comparison states for each data set.
44
2. For the DDD equations, we estimate
regressions for 2000-2004 where we use
year dummies and
i) Put in a dummy for 2003-2004 for
Tennessee and
ii) Put in a dummy for 2002-2004 for
Tennessee.
This again gives us 2 tests of the null
hypothesis that the difference in trend
between those with children and those
without in Tennessee has the same trend as
the same groups for comparison states for
each data set.
45
Table 5 Regression Estimates of Placebo Treatment Effect
2002-2004 Using 2000-2004 Data MCPS ALLCPS ACS (1) (2) (3)
Panel A: Difference in Differences
Working > 0 hours
Point Estimate -0.022** -0.009** 0.003 (0.011) (0.004) (0.004)
95 Percent CI [-0.045, 0.000] [-0.017, 0.000] [-0.005, 0.012]
Working ≥ 35 hours Point Estimate -0.015 0.001 0.000
(0.013) (0.005) (0.005)
95 Percent CI [-0.041, 0.011] [-0.009, 0.011] [-0.010, 0.009]
Panel B: Triple Difference
Working > 0 hours
Point Estimate 0.004 -0.011 -0.006 (0.024) (0.008) (0.008)
95 Percent CI [-0.044, 0.051] [-0.028, 0.005] [-0.021, 0.009]
Working ≥ 35 hours Point Estimate -0.022 -0.023** -0.011
(0.026) (0.009) (0.009) 95 Percent CI [-0.074, 0.029] [-0.041, -0.005] [-0.029, 0.006]
Microdata
149,253 1,247,650 1,355,926
• For DD h>0 we reject the model for the MCPS and the
ALLCPS.
• For DDD h>35 we reject the model for the ALLCPS.
46
Table 6 Regression Estimates of Placebo Treatment Effect
2003-2004 Using 2000-2004 Data MCPS ALLCPS ACS (1) (2) (3)
Panel A: Difference in Differences
Working > 0 hours
Point Estimate -0.031** -0.018*** 0.005 (0.012) (0.004) (0.004)
95 Percent CI [-0.054, -0.008] [-0.026, -0.010] [-0.003, 0.013]
Working ≥ 35 hours Point Estimate -0.016 -0.012*** 0.004
(0.013) (0.004) (0.004) 95 Percent CI [-0.041, 0.010] [-0.020, -0.004] [-0.004, 0.013]
Panel B: Triple Difference
Working > 0 hours
Point Estimate -0.007 -0.018** -0.006 (0.023) (0.008) (0.008)
95 Percent CI [-0.052, 0.038] [-0.033, -0.003] [-0.022, 0.010]
Working ≥ 35 hours Point Estimate -0.032 -0.018** -0.009
(0.025) (0.009) (0.009) 95 Percent CI [-0.081, 0.016] [-0.035, -0.001] [-0.026, 0.008]
Microdata
149,253 1,247,650 1,355,926
47
• For DD h > 0 we reject the model for the
MCPS and the ALLCPS. For DD h > 35 we
reject the model for the ALLCPS.
• For DDD h>0 and DDD h>35 we reject the
model for the ALLCPS.
48
Table 2 Revisited
TennCare Impacts from Table 2 in Bold that Survive Both Placebo Tests in Tables 4 and 5
MCPS ALLCPS ACS (1) (2) (3)
Panel A: Difference in Differences
Working > 0 hours
Point Estimate 0.021** 0.015*** -0.013*** (0.011) (0.004) (0.004)
95 Percent CI [0.000, 0.042] [0.007, 0.023] [-0.021, -0.005]
Working ≥ 35 hours Point Estimate 0.022* 0.016*** -0.015***
(0.012) (0.004) (0.004)
95 Percent CI [-0.001, 0.045]
[0.008, 0.024] [-0.024, -0.006]
Panel B: Triple Difference
Working > 0 hours
Point Estimate 0.043** 0.005 0.002 (0.022) (0.008) (0.006)
95 Percent CI [0.000, 0.086] [-0.010, 0.020] [-0.010, 0.014]
Working ≥ 35 hours Point Estimate 0.017 -0.006 0.013*
(0.023) (0.009) (0.007)
95 Percent CI [-0.028, 0.062]
[-0.023, 0.012] [0.000, 0.026]
Still a wide range of estimates that do not fail either placebo test.
49
• Why are the MCPS’ results different from the
ALLCPS and ACS’ results? One possibility
is that, e.g. 2003 in the MCPs covers April
2002-March 2003, while in the ALLCPS and
ACS 2003 covers January 2003-December
2003. We thought this may be more
important for those in more volatile
industries in the data sets. So we delete those
working in Construction and Manufacturing
in all the data sets, but this makes no
difference. Also it couldn’t explain
differences between the ALLCPS and ACS.
50
Table 7 Regression Estimates 2000-2007, Without 2005,
Omit Construction and Manufacturing
MCPS ALLCPS ACS (1) (2) (3)
Panel A: Difference in Difference
Working > 0 hours Point Estimate 0.026** 0.015*** -0.010**
(0.013) (0.005) (0.005) 95 Percent CI [0.001, 0.050] [0.005, 0.025] [-0.019, -0.001]
Working ≥ 35 hours
Point Estimate 0.023* 0.017*** -0.011** (0.014) (0.005) (0.005)
95 Percent CI [-0.003, 0.050] [0.007, 0.026] [-0.020, -0.002]
Panel B: Triple Difference
Working > 0 hours Point Estimate 0.047* 0.011 -0.001
(0.024) (0.009) (0.007) 95 Percent CI [-0.001, 0.094] [-0.007, 0.028] [-0.015, 0.013]
Working ≥ 35 hours
Point Estimate 0.008 -0.003 0.009 (0.026) (0.009) (0.008)
95 Percent CI [-0.043, 0.059] [-0.022, 0.015] [-0.006, 0.023]
Microdata 178,274 1,477,819 2,022,102
51
Can we make the MCPS, ALLCPS and ACS
estimates look more similar by only looking at
those living in the Metropolitan Areas because
these individuals are overrepresented in the
ACS?
52
Table 8: Regression Estimates 2000-2007, Without 2005, Only Metropolitan Areas
MCPS ALLCPS ACS (1) (2) (3)
Panel A: Difference in Difference
Working > 0 hours
Point Estimate 0.018 0.002 -0.012** (0.012) (0.004) (0.006)
95 Percent CI [-0.006, 0.042] [-0.006, 0.010] [-0.024, -0.001]
Working ≥ 35 hours Point Estimate 0.021 0.007 -0.014**
(0.013) (0.005) (0.007) 95 Percent CI [-0.005, 0.047] [-0.003, 0.017] [-0.027, -0.001]
Panel B: Triple Difference
Working > 0 hours
Point Estimate 0.039* 0.005 0.000 (0.024) (0.009) (0.009)
95 Percent CI [-0.008, 0.086] [-0.013, 0.022] [-0.017, 0.017]
Working ≥ 35 hours Point Estimate 0.010 -0.014 0.013
(0.027) (0.010) (0.009) 95 Percent CI [-0.042, 0.063] [-0.034, 0.006] [-0.006, 0.031]
Microdata
166,519 1,365,850 2,224,792
53
Move to Conley-Taber consistently estimated CIs
• Conley and Taber (2011) show you will not get
consistent estimates of the treatment effect if you assume that you are only adding comparison states.
• However they also show that you can consistently estimate the confidence interval for the treatment effects. Here we will get 87.5% CIs for their approach but can get more conventional ones (e.g 95%) by following their suggestions and adding more comparisons.
54
Table 9 Conley-Taber Consistent 87.5% Confidence Intervals for the TennCare Effect: on
Employment (Hours) by Database, 2000-2007, Omit 2005
MCPS ALLCPS ACS
(1) (2) (3)
Panel A: Difference-in-Difference
Working > 0 hours
Regression [0.004, 0.038]* [0.009, 0.021]* [-0.019, -0.007]*
Taber Conley [0.001, 0.046]* [-0.005, 0.040] [-0.025, 0.004]
Working > 35 hours
GGN [0.004, 0.040]* [0.009, 0.023]* [-0.022, -0.008]*
Taber Conley [0.002, 0.042]* [-0.001, 0.035] [-0.031, -0.002]
Panel B: Triple Difference
Working > 0 hours
GGN [0.009, 0.077]* [-0.007, 0.017] [-0.007, 0.011]
Taber Conley [0.001, 0.074]* [-0.017, 0.037] [-0.010, 0.017]
Working > 35 hours
GGN [-0.013, 0.057] [-0.020, 0.008] [0.003, 0.023]*
Taber Conley [-0.024, 0.047] [-0.036, 0.044] [-0.001, 0.023]
Microdata 216,751 1,789,894 2,491,229
55
Table 10 Power Calculations Based on Conley-Taber Consistent Confidence Intervals
MCPS
Confidence Intervals
ALLCPS Confidence
Intervals
Working>0
87.5% CIs When the ACA is expanded in 2014 for states with no prior expansions vs states that did not expand 2014 with no prior expansions
[0.042, 1] [0.102, 1]
87.5% CIs When the ACA is Expanded in 2014 for states with no prior
expansions vs Did not expand 2014 [0.070, 1] [0.150, 1]
Working>35
87.5% CIs When the ACA is expanded in 2014 for states with no prior expansions vs states that did not expand 2014 with no prior expansions
[0.048, 1] [0.048, 1]
87.5% CIs When the ACA is Expanded in 2014 for states with no prior
expansions vs Did not expand 2014 [0.098, 1] [0.088, 1]
Again consistent with a wide set of power values.
56
Placebo Tests for CT approach
Table 12 Conley-Taber Consistent 87.5% Confidence Intervals: Placebo Treatment Effect 2002-
2004, Using 2000-2004 Data
MCPS ALLCPS ACS (1) (2) (3)
Difference in Difference
Working > 0 hours
GGN [-0.040, -0.005]* [-0.016, -0.002]* [-0.004, 0.010] Conley-Taber [-0.041, -0.007]* [-0.021, 0.003] [-0.008, 0.014]
Working ≥ 35 hours
GGN [-0.036, 0.006] [-0.006, 0.009] [-0.007, 0.007] Conley-Taber [-0.033, -0.003]* [-0.007, 0.008] [-0.016, 0.014]
Triple Difference
Working > 0 hours
GGN [-0.033, 0.040] [-0.024, 0.001] [-0.018, 0.006] Conley-Taber [-0.034, -0.011]* [-0.024, 0.012] [-0.002, 0.009]
Working ≥ 35 hours
GGN [-0.063, 0.018] [-0.037, -0.009]* [-0.025, 0.003] Conley-Taber [-0.023, -0.002]* [-0.019, 0.017] [-0.004, 0.005]
Microdata
149,253 1,247,650 1,355,926
57
Table 13 87.5 % Confidence Intervals for Regression Approach and CT Approach: Placebo
Treatment Effect 2003-2004, Using 2000-2004 Data
MCPS ALLCPS ACS (1) (2) (3)
Difference in Difference
Working > 0 hours
GGN [-0.049, -0.013]* [-0.025, -0.012]* [-0.002, 0.011] Conley-Taber [-0.042, -0.012]* [-0.028, -0.008]* [-0.004, 0.015]
Working ≥ 35 hours
GGN [-0.036, 0.004] [-0.019, -0.006]* [-0.003, 0.011] Conley-Taber [-0.035, 0.010] [-0.020, -0.006]* [-0.008, 0.015]
Triple Difference
Working > 0 hours
GGN [-0.042, 0.029] [-0.030, -0.007]* [-0.018, 0.007] Conley-Taber [-0.040, -0.019]* [-0.035, -0.004]* [-0.003, 0.011]
Working ≥ 35 hours
GGN [-0.070, 0.005] [-0.031, -0.005]* [-0.022, 0.005] Conley-Taber [-0.025, -0.002]* [-0.041, 0.025] [-0.003, 0.010]
Microdata
149,253 1,247,650 1,355,926
58
Table 9 Revisited
Conley-Taber Consistent 87.5% Confidence Intervals for TennCare Impacts from Table 9 in Bold that Survive both Placebo Treatment Tests in Tables 12 and 13
MCPS ALLCPS ACS (1) (2) (3)
Panel A: Difference-in-Difference
Working > 0 hours
87.5 Percent Confidence Interval [0.001, 0.046]* [-0.005, 0.040] [-0.025, 0.004]
Working > 35 hours 87.5 Percent Confidence Interval [0.002, 0.042]* [-0.001, 0.035] [-0.031, -0.002]
Panel B: Triple Difference
Working > 0 hours
87.5 Percent Confidence Interval [0.001, 0.074]* [-0.017, 0.037] [-0.010, 0.017]
Working > 35 hours 87.5 Percent Confidence Interval [-0.024, 0.047] [-0.036, 0.044] [-0.001, 0.023]
59
Our results for the CT CIs, using the
specifications that survive the tests, produce
a number of fairly wide confidence intervals
across the data sets that do not contradict each
other and that are consistent with the null
hypothesis of no TennCare effect. Hence
more in keeping with the previous literature.
• But recall that for the Regression results,
those that survive the placebo tests have
quite different implications and
contradictory results.
60
Table 2 Revisited: TennCare Impacts from Table 2 in Bold that Survive Both Placebo Tests in Tables 4 and 5
MCPS ALLCPS ACS (1) (2) (3)
Panel A: Difference in Differences
Working > 0 hours Point Estimate 0.021** 0.015*** -0.013***
(0.011) (0.004) (0.004)
95 Percent CI [0.000, 0.042] [0.007, 0.023] [-0.021, -0.005]
Working ≥ 35 hours
Point Estimate 0.022* 0.016*** -0.015*** (0.012) (0.004) (0.004)
95 Percent CI [-0.001, 0.045] [0.008, 0.024] [-0.024, -0.006]
Panel B: Triple Difference
Working > 0 hours Point Estimate 0.043** 0.005 0.002
(0.022) (0.008) (0.006) 95 Percent CI [0.000, 0.086] [-0.010, 0.020] [-0.010, 0.014]
Working ≥ 35 hours
Point Estimate 0.017 -0.006 0.013* (0.023) (0.009) (0.007)
95 Percent CI [-0.028, 0.062] [-0.023, 0.012] [0.000, 0.026]