Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1
The Campbell Collaboration www.campbellcollaboration.org
Using Robust Standard Errors for Dependent Effect Sizes
Elizabeth Tipton Teachers College, Columbia University
Emily Tanner-‐Smith, Mark Lipsey
Peabody Research InsCtute
The Campbell Collaboration www.campbellcollaboration.org
Outline
1. Type of dependencies – Correlated Effects – Hierarchical Model
2. Dealing with dependencies – Robust Variance Estimation
3. Efficient weight estimation: What about ρ? 4. Different types of regression coefficients
– Independent effects meta-regression (I-MR) vs. Dependent effects meta-regression (D-MR)
5. Software demonstration
2
2
The Campbell Collaboration www.campbellcollaboration.org
1. Types of dependencies
The Campbell Collaboration www.campbellcollaboration.org
1. Types of dependencies
Where does the dependency come from? Recall that Ti = θi + εi, where
– Ti is the effect size estimate, – θi is the effect size parameter, and – εi is the estimation error.
Dependence can arise because the
– εi are correlated, – θi are correlated, – or both.
4
3
The Campbell Collaboration www.campbellcollaboration.org
1.1 Correlated Effects Model (εi correlated)
5
kj correlated estimates of the study specific ES
τ2 = between study variation in ES Meta-regression
D-MR Study 1, θ1
Estimate 1.1 of θ1
Estimate 1.2 of θ1
Estimate 1.3 of θ1 Study 2, θ2
Estimate 2.1 of θ2
Study 3, θ3 Estimate 3.1 of θ3
Estimate 3.2 of θ3
The Campbell Collaboration www.campbellcollaboration.org
1.2 Hierarchical Model (θi correlated)
6
ω2= between-study, within-cluster variation in ES
τ2 = between cluster variation in average
ES Meta-
regression
D-MR Cluster 1, θ1
Study 1.1 estimate of θ1.1
Study 1.2 estimate of θ1.2
Study 1.3 estimate of θ1.3
Cluster 2, θ2 Study 2.1 estimate of θ2.1
Cluster 3, θ3 Study 3.1 estimate of θ3.1
Study 3.2 estimate of θ3.2
4
The Campbell Collaboration www.campbellcollaboration.org
2. What to do about this dependence?
The Campbell Collaboration www.campbellcollaboration.org
2. Possible approaches
• Ignore it and analyze the effect sizes as if they are independent
• Create a single synthetic effect size for each sample – Use the study-average – Randomly select an effect size – Choose the “best” one
• Model the dependence with full multivariate analysis – This requires information on the covariance structure
• Use Robust Variance Estimation!
5
The Campbell Collaboration www.campbellcollaboration.org
2.1 Variance Estimation Assume T = Xβ + ε , where
T = (T1,…,Tm)’ and Tj is a kjx1 vector of effect sizes for study j X = (X1,…,Xm)’ and Xj is a kjxp design matrix for study j β=(β1,…,βp)’ is a 1xp vector of unknown regression coefficients ε = (ε1,…,εm)’ and εj is a kjx1 vector of residuals for study j E(εj) = 0, V(εj) = Σj
The Campbell Collaboration www.campbellcollaboration.org
2.1 Variance Estimation
We can estimate β by: And and the covariance matrix for this estimate is The problem is that while the variances in Σj are known, the covariances are UNKNOWN.
!" = # j$%j# j
j=1
m
!"#$%&'
&'
# j$%j(jj=1
m
!
!V (") = # j$%j# j
j=1
m
!"#$%&'
&'
# j '%j( j%j# jj=1
m
!"#$%&'
# j$%j# jj=1
m
!"#$%&'
&'
6
The Campbell Collaboration www.campbellcollaboration.org
2.1 Robust Variance Estimator
The RVE of b is where ej = Tj – Xjb is the kjx1 estimated residual vector for study j A robust test of H0: βa = 0 uses the statistic where vaa
R is the ath diagonal of the VR matrix. Note: the t-distribution with m-p d.f. should be used for critical values.
!V R = " j#$j" j
j=1
m
!"#$%&'
%&
" j '$j' j' j '$j" jj=1
m
!"#$%&'
" j#$j" jj=1
m
!"#$%&'
%&
!
"#$ =
%#&
&!'( )(##$
The Campbell Collaboration www.campbellcollaboration.org
2.2 The General Theorem
Under regularity conditions and as m ∞, VmR is a consistent estimator of
the true covariance matrix and Where Vm
R is the RVE and bm is the estimate of β computed from m studies. Regularity conditions: • The regressors and weights are uniformly bounded; • The number of estimates per study is bounded; • The first few moments of εj are uniformly bounded • Absolute moment conditions on εj’εj
!m m"m
R( )!1/2 #m ! "( ) L# $# N $, %p( )
7
The Campbell Collaboration www.campbellcollaboration.org
2.3 Notes on the Theorem • This theorem is asymptotic in the number of studies m, not
the number of effect sizes; • There are no distributional assumptions needed for the effect
sizes; • The theorem applies for any set of weights; • Correlations do not need to be known or specified, though
may impact the standard errors; • The results apply to any type of dependency.
The Campbell Collaboration www.campbellcollaboration.org
3. How to choose weights
8
The Campbell Collaboration www.campbellcollaboration.org
3. Weighting issues: ρ unknown
Recall that in general, 1. The robust standard error estimator does not require information on the
true correlation in the data. Additionally, the estimator works for any weights.
2. The most efficient weights are inverse-variance weights, – i.e for any covariance matrix Σ, W = Σ-1
In the hierarchical model, these weights can be estimated fairly easily. In the correlated effects case, while the variances are known, the
covariances between the estimates are NOT known.
15
The Campbell Collaboration www.campbellcollaboration.org
3.1 Correlated effects model
One way to estimate the efficient weights is to assume a simplified correlation structure. Assume that within each study j, 1. The correlation between all the pairs of effect sizes is a constant ρ; 2. This correlation is the same in all studies; and 3. The kj sampling variances within the study are approximately equal, with average Vj .
Then the (approximately) efficient weights can be shown to be:
Wij = 1/{(Vj + τ2)[1+(kj-1)ρ]}
16
9
The Campbell Collaboration www.campbellcollaboration.org
3.2 Where ρ occurs in weights
In these weights, the unknown correlation ρ occurs twice: 1. In the estimator of τ2:
where Jj is a kj x kj matrix of ones and V = (X′WX)-1
2. In the multiplier: [1+(kj-1)ρ].
17
!!!
"! 2 =
"# "$+ tr "%&
'&
# &$# &&=1
$
#$
%&
'
() + * #$ "
%&
'&
# &$% &# & "# &$# &+, -.
&=1
$
#$
%&
'
()
'&% & " #$ " %&2# &$% &# &
&=1
$
#$%&
'()&=1
$
#
The Campbell Collaboration www.campbellcollaboration.org
3.3 Three approaches
Three strategies for dealing with ρ in calculating efficient weights can be used: 1. Sensitivity approach
2. Conservative approach
3. External information approach
18
10
The Campbell Collaboration www.campbellcollaboration.org
3.4 Sensitivity approach
1. Sensitivity approach: – Run the model with various values of ρ in (0,1).
This approach allows you to see how robust the results are to weights based on different values of ρ.
19
The Campbell Collaboration www.campbellcollaboration.org
3.5 Conservative approach
2. Conservative approach: – Assume ρ = 1.
The weights become: Wij = 1/{kj(Vj + τ2)} . Each study gets total weight ΣWij = 1/(Vj + τ2) . These are conservative in that a study does not receive additional weight because it has multiple measures.
20
11
The Campbell Collaboration www.campbellcollaboration.org
3.6 External information approach
3. External information approach: – e.g. test reliability measures, information from a study that
reports the correlations, information from the design of the test. – In the binary outcomes case (e.g. log odds ratio), an upper
bound on ρ can be calculated from estimates of the treatment and control proportions, (πC, πT).
21
The Campbell Collaboration www.campbellcollaboration.org
3.7 Combined approaches
In practice, a combination of these three approaches will often be used. Example: HTJ recommendation
– Estimate τ2 using a sensitivity approach; but – For the multiplier, use a conservative strategy.
Or, the same strategy can be used for both estimating both τ2 and the multiplier.
22
12
The Campbell Collaboration www.campbellcollaboration.org
4. Covariate Issues in Meta-regression
The Campbell Collaboration www.campbellcollaboration.org
4. Within- vs. between- study models In detecting the effect of an intervention, there are two types of models and
analyses:
1. Within-study model: For example: • Within each study, people are randomly assigned to:
– Exercise 6 times a week (Treatment A); or – Exercise 3 times a week (Treatment B); or – Not exercise (Control).
• The ES for comparing Treatments B and A then is simply ES(Trt B vs Trt A) = ES(Trt A vs C) – ES(Trt B vs C)
• In a MR, each study contributes one such comparison. The MR estimates an average causal ES for exercising 6 (vs 3) times per week.
24
13
The Campbell Collaboration www.campbellcollaboration.org
4.1 Within- vs. between- study models
2. Between-study effects: For example: • In some studies, individuals are randomly assigned to:
– Exercise 6 times a week (Treatment A); or – Not exercise (Control).
• In other studies, individuals are randomly assigned to: – Exercise 3 times a week (Treatment B); or – Not exercise (Control).
• The ES for exercising 6 (vs 3) times a week is calculated using MR. Note that this average effect is NOT causal.
The idea of within- and between-study effects plays an important role in D-MR.
25
The Campbell Collaboration www.campbellcollaboration.org
4.2 Regression Coefficients: I-MR
Let Tj be the effect size and Xj be the length to follow-up: Tj = β0 + Xjβ1 + …
Note: here each study contributes one value of Xj. The coefficients β0 and β1 can be interpreted as:
– β0 = the average effect size when Xj = 0 • e.g. the average effect size in studies in which the intervention just occurred.
– β1 = the effect of a 1-unit increase in Xj on Tj
• e.g. the ES change from moving from a study in which the intervention just occurred to a study in which the ES was measured at a follow-up 1 month later.
26
14
The Campbell Collaboration www.campbellcollaboration.org
4.3 Regression Coefficients: D-MR For a fixed study (j=1), now assume there are multiple outcomes. This
study has its own regression equation: Ti1 = β01 + Xi1β11 +…
Note: here each outcome contributes one value of Xi1. The coefficients β01 and β11 can be interpreted as:
– β01 = the average effect size when Xi1 = 0 • e.g. the average effect size for units in the study (j=1) when the intervention
just occurred. – β11 = the effect of a 1-unit increase in Xi1 on Ti1
• e.g. the effect size change for units in the study at the time of intervention and at follow-up 1 month later.
27
The Campbell Collaboration www.campbellcollaboration.org
4.4 Between and within: D-MR In D-MR these two types of regression occur in one analysis:
– Within Study: Tij = β0j + Xijβ2 + …
– Between Study: β0j = β0 + Xjβ1 + …
Here there are two different relationships between X and T:
– The between-study effect of Xj on Tj • Note: this effect is found in I-MR
– The within-study effect of Xij on Tij • Note: this effect is NOT found in I-MR
These two regressions are combined into one analysis and model:
Tij = β0 + Xijβ2 + Xjβ1 + …
28
15
The Campbell Collaboration www.campbellcollaboration.org
4.5 The solution in D-MR: Centering!
In D-MR, how X is centered: • Affects the interpretation of the coefficients; and • Allows within- and between- study effects to be properly
separated. The best way to center is call Group Mean Centering:
Xcij = Xij – Xj
Where Xj is the mean value of Xij in group j (and where group is either
study or cluster). 29
The Campbell Collaboration www.campbellcollaboration.org
4.6 What can go wrong?
What if you don’t center Xij? Tij = β0 + Xijβ2 +…
• The effect you think you are modeling (β2) is the within-
effect, BUT • What you have actually modeled is a weighted combination
of the within- and between- effects. • This makes interpreting β2 very difficult. à This is why group centering is preferred.
30
16
The Campbell Collaboration www.campbellcollaboration.org
4.7 Interpretation issues
Including β2Xcij in addition to β1X.j in the model allows a new type of effect
to be modeled. However, in some D-MR, there may only be a few clusters with values of X
that vary within each cluster. This leads to 2 issues:
1. The estimate of β2 (associated with Xcij) will be imprecise (i.e. have
a large standard error). 2. The types of clusters in which Xij varies may be different (i.e. not
representative) of clusters in which Xij does not vary.
31
The Campbell Collaboration www.campbellcollaboration.org
4.8 Conclusions
1. When using a covariate, ask if the effect of interest is between- (Xj) or within- (Xij
C) studies. 2. Make sure to group-center your within-study variables. 3. Check your data to see if Xij
C varies in many studies and if you think these studies are representative of all studies.
32
17
The Campbell Collaboration www.campbellcollaboration.org
5. Software demonstration
The Campbell Collaboration www.campbellcollaboration.org
5.1 Stata Demonstration – Correlated Effects • Meta-analysis on the effectiveness of substance abuse
treatment for adolescents • Effect sizes represent post-treatment differences between
treatment and comparison groups on some measure of substance use (positive effect sizes represent beneficial treatment effects) – Number of effect sizes k = 172 – Number of studies m = 39 – Average number of effect sizes per study = 4.4
NOTE: Data have been manipulated for pedagogical purposes, DO NOT cite or reference actual statistical results
18
The Campbell Collaboration www.campbellcollaboration.org
5.1 Stata Demonstration – Correlated Effects • Install robumeta.ado file from the Statistical Software
Components (SSC) archive
The Campbell Collaboration www.campbellcollaboration.org
5.1 Stata Demonstration – Correlated Effects • 4 moderators of interest: 2 vary within and between studies,
2 vary between studies only
19
The Campbell Collaboration www.campbellcollaboration.org
5.1 Stata Demonstration – Correlated Effects • To model both the within- (Xc
ij) and between- effects (Xj) of outcome type alc and outcome time frame dvwks, create group-mean and group-mean-centered variables
The Campbell Collaboration www.campbellcollaboration.org
5.1 Stata Demonstration – Correlated Effects
• Intercept only model to estimate random-effects mean effect size, with robust variance estimate assuming ρ = .80
• Weighted mean effect size = .23; 95% CI (.11, .36)
20
The Campbell Collaboration www.campbellcollaboration.org
5.1 Stata Demonstration – Correlated Effects
The Campbell Collaboration www.campbellcollaboration.org
5.1 Stata Demonstration – Correlated Effects • Conduct sensitivity tests at varying levels of ρ to see how it
affects substantive conclusions • In this example, ρ has minimal impact on the statistical and
substantive findings
21
The Campbell Collaboration www.campbellcollaboration.org
5.2 Stata Demonstration – Hierarchical Effects • Meta-analysis on the effectiveness of substance abuse
treatment programs for adolescents • Effect sizes represent post-treatment differences between
treatment and comparison groups on some measure of substance use (positive effect sizes represent beneficial treatment effects) – Number of effect sizes k = 68 – Number of research labs m = 15 – Average number of effect sizes per research lab = 4.5
NOTE: Data have been manipulated for pedagogical purposes, DO NOT cite or reference actual statistical results
The Campbell Collaboration www.campbellcollaboration.org
5.2 Stata Demonstration – Hierarchical Effects • 5 moderators of interest: all vary within and between
research labs
22
The Campbell Collaboration www.campbellcollaboration.org
5.2 Stata Demonstration – Hierarchical Effects
• τ2 – between-lab variance component • ω2 –between-study-within-lab variance component • Weighted mean effect size = .25; 95% CI (.12, .38)
The Campbell Collaboration www.campbellcollaboration.org
5.2 Stata Demonstration – Hierarchical Effects
23
The Campbell Collaboration www.campbellcollaboration.org
5.3 Software Demonstration • See handouts for similar walkthroughs in SPSS and R
The Campbell Collaboration www.campbellcollaboration.org
6. Conclusions
24
The Campbell Collaboration www.campbellcollaboration.org
Overall Conclusions 1. Make sure you choose the proper model for the type of dependencies in
your data. 2. When using the correlated effects model with efficient weights:
a. If you have information on ρ, use it! b. If the Tij are functions of proportions, use this information to get an upper
bound on ρ. c. If you have no information on ρ:
• Use a sensitivity approach for estimating τ2
• Assume ρ=1 in your weights, i.e. Wij = 1/kj[V.j + τ2]
3. For each covariate Xij in your model, remember that you can include 1. The group-centered within-study variable (Xc
ij = Xij – X.j), and/or 2. The average (X.j).
47
The Campbell Collaboration www.campbellcollaboration.org
Thank you!
For more information:
http://www.northwestern.edu/ipr/qcenter/RVE-meta-analysis.html
http://peabody.vanderbilt.edu/
peabody_research_institute/methods_resources.xml
48
25
The Campbell Collaboration www.campbellcollaboration.org
P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway
E-mail: [email protected]
http://www.campbellcollaboration.org