View
1
Download
0
Category
Preview:
Citation preview
Moving Beyond the Average
Treatment Effect: A Look at Power Analyses for Moderator Effects in Cluster
Randomized Trials Jessaca Spybrook Western Michigan University December 3, 2015 (Joint work with Ben Kelcey, University of Cincinnati, Nianbo Dong, University of Missouri) *This work was funded by a grant (DGE-1437692) from the National Science Foundation.
Overview
n Background n Purpose n 2-level CRT n 3-level CRT n Implications
Slide 2
Background
n What works?
Slide 3
Background
n What works?
n Under what conditions?
Slide 4
Background
n What works?
n Under what conditions?
n For whom?
Slide 5
Purpose
n What works?
n Under what conditions?
n For whom?
Slide 6
Purpose
n Designs ¨ 2-level cluster randomized trial
(2-level CRT)
¨ 3-level cluster randomized trial (3-level CRT)
Slide 7
2-level CRT
n Suppose a team of researchers are planning an impact study to determine the effectiveness of a new 5th grade science curriculum. They are planning a 2-level CRT with students nested within schools and schools are randomly assigned to the new curriculum or the current curriculum.
Slide 8
2-level CRT – what works?
n What works?
What is the effect of the new science curriculum relative to the current curriculum on science achievement for fifth graders?
Slide 9
2-level CRT – what works?
n Model
Slide 10
( )( )τσγγ
,0~
,0~
0
200100
NrNeerTy
j
ijijjjij +++=
Main effect of Treatment
2-level CRT – what works?
n Model
Set
Slide 11
( )( )τσγγ
,0~
,0~
0
200100
NrNeerTy
j
ijijjjij +++=
Main effect of Treatment
2σττ
ρ+
=2
01
στ
γδ
+=
2-level CRT – what works?
n Minimum detectable effect size (MDES) (Bloom, 1995)
MJ-2 is multiplier that approaches 2.8 as J increases, two-tail test with 0.05 level of significance J is total number of sites n is number of individuals per site is the intraclass correlation
Slide 12
( )( )J
nMMDES J/14
2ρρ −+
= −
ρ
2-level CRT – what works?
n To increase precision ¨ Common practice to include a cluster-level
covariate ¨ School-level pretest
n Model:
Slide 13
ijjjjij erWTy ++++= 0020100 γγγ
Main effect of Treatment
2-level CRT
Question Effect of Interest
MDES
What works? Main effect of
treatment
Main effect of treatment w/covariate
Slide 14
( )( )J
nM J/14
2ρρ −+
−
( )( )J
nRM W
J
/1)1(4 2|
3
ρρ −+−−
2-level CRT – Minimum Detectable Effect Size
What Works?
Main Effect
J=30 n=10 0.39 n=50 0.29 n=100 0.27 J=60 n=10 0.27 n=50 0.20 n=100 0.19
Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80,
Slide 15 .75.0,23.0 2
| == WRρ
2-level CRT – under what conditions?
n Under what conditions?
Is there a difference in the mean effects by school type (urban vs. suburban)?
Slide 16
2-level CRT – under what conditions?
n Model:
Slide 17
ijjjjjjjij erWSTSTy ++++++= 00403020100 γγγγγ
Cluster level moderator effect
2-level CRT
Question Effect of Interest
MDES MDESD
What works? Main effect of
treatment
Main effect of treatment w/covariate
Under what conditions?
Cluster-level moderator
Slide 18
( )( )J
nM J/14
2ρρ −+
−
( )( )J
nRM W
J
/1)1(4 2|
3
ρρ −+−−
( )( )J
nRM SW
J
/1)1(16 2|
5
ρρ −+−−
2-level CRT – Minimum Detectable Effect Size
What Works?
Main Effect
Under what conditions?
CL Mod
J=30 n=10 0.39 0.75
n=50 0.29 0.53
n=100 0.27 0.49
J=60 n=10 0.27 0.52
n=50 0.20 0.37
n=100 0.19 0.34
Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80,
Slide 19 .80.0,75.0,23.0 2
|2| === SWW RRρ
2-level CRT – for whom?
n For whom?
Is there a difference in the mean effect for boys and girls?
Slide 20
2-level CRT – for whom?
n Model:
Slide 21
ijjijjijjij erXTXTy +++++= 011100100 γγγγ
Individual level moderator effect
2-level CRT
Question Effect of Interest
MDES MDESD
What works? Main effect of
treatment
Main effect of treatment w/covariate
Under what conditions?
Cluster-level moderator
For whom? Individual-level moderator
Slide 22
( )( )J
nM J/14
2ρρ −+
−
( )( )J
nRM W
J
/1)1(4 2|
3
ρρ −+−−
( )( )J
nRM SW
J
/1)1(16 2|
5
ρρ −+−−
( )( )J
nRM X
JJn
/1)1(16 2|
)2(*
ρ−−−−
2-level CRT – Minimum Detectable Effect Size
What Works?
Main Effect
Under what conditions?
CL Mod
For whom?
Ind Mod
J=30 n=10 0.39 0.75 0.51
n=50 0.29 0.53 0.23
n=100 0.27 0.49 0.16
J=60 n=10 0.27 0.52 0.36
n=50 0.20 0.37 0.16
n=100 0.19 0.34 0.11
Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80,
Slide 23 .20.0,80.0,75.0,23.0 2
|2|
2| ==== XSWW RandRRρ
3-level CRT
n Suppose a team of researchers are planning an impact study to determine the effectiveness of a school-wide science curriculum. They are planning a 3-level CRT with students nested within teachers nested within schools and schools are randomly assigned to the new curriculum or the current curriculum.
Slide 24
3-level CRT – what works?
n What works?
What is the effect of the new science curriculum relative to the current curriculum on science achievement for all students?
Slide 25
3-level CRT – under what conditions?
n Under what conditions?
Is there a difference in the mean effects by school type (urban vs. suburban)? Is there a difference in the mean effects by teacher experience level (0-3 years vs. more than 3 years)?
Slide 26
3-level CRT – for whom?
n For whom?
Is there a difference in the mean effect for boys and girls?
Slide 27
3-level CRT
Question Effect of Interest
MDES
What works?
Main effect of treatment
Slide 28
( ) ( )[ ]{ }K
JnRM LLLLW
K
//114 23232|
3
ρρρρ −−++−−
3-level CRT
Question Effect of Interest
MDES MDESD
What works?
Main effect of treatment
Under what conditions?
Cluster-level moderator
Slide 29
( ) ( )[ ]{ }K
JnRM LLLLW
K
//114 23232|
3
ρρρρ −−++−−
( ) ( )[ ]{ }K
JnRM LLLLSW
K
//1116 23232|
5
ρρρρ −−++−−
3-level CRT
Question Effect of Interest
MDES MDESD
What works?
Main effect of treatment
Under what conditions?
Cluster-level moderator
Under what conditions?
Teacher-level moderator
Note: u=J*K-(K-2) Slide 30
( ) ( )[ ]{ }K
JnRM LLLLW
K
//114 23232|
3
ρρρρ −−++−−
( ) ( )[ ]{ }K
JnRM LLLLSW
K
//1116 23232|
5
ρρρρ −−++−−
( ) ( )[ ]{ }K
JnRM LLLM
u
//1116 2322| ρρρ −−+−
3-level CRT
Question Effect of Interest
MDES MDESD
What works?
Main effect of treatment
Under what conditions?
Cluster-level moderator
Under what conditions?
Teacher-level moderator
For whom? Individual-level moderator
Note: u=J*K-(K-2), v=n*J*K-(J*K)-(K-2) Slide 31
( ) ( )[ ]{ }K
JnRM LLLLW
K
//114 23232|
3
ρρρρ −−++−−
( ) ( )[ ]{ }K
JnRM LLLM
u
//1116 2322| ρρρ −−+−
( )( )[ ]{ }K
JnRM LLX
v
//1116 232| ρρ −−−
( ) ( )[ ]{ }K
JnRM LLLLSW
K
//1116 23232|
5
ρρρρ −−++−−
3-level CRT – Minimum Detectable Effect Size
What Works?
Main Effect
J=5 n=10 0.24 n=30 0.22 J=30 n=10 0.19 n=30 0.18
Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,
Slide 32 .20.0,30.0,80.0,75.0,08.0,15.0 2|
2|
2|
2|23 ====== XPSWLL RandRRRρρ
3-level CRT – Minimum Detectable Effect Size
What Works?
Main Effect
Under what conditions?
CL Mod
J=5 n=10 0.24 0.45
n=30 0.22 0.41
J=30 n=10 0.19 0.34
n=30 0.18 0.33
Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,
Slide 33 .20.0,30.0,80.0,75.0,08.0,15.0 2|
2|
2|
2|23 ====== XPSWLL RandRRRρρ
3-level CRT – Minimum Detectable Effect Size
What Works?
Main Effect
Under what conditions?
CL Mod
Under what conditions?
TL Mod
J=5 n=10 0.24 0.45 0.29
n=30 0.22 0.41 0.23
J=30 n=10 0.19 0.34 0.12
n=30 0.18 0.33 0.09
Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,
Slide 34 .20.0,30.0,80.0,75.0,08.0,15.0 2|
2|
2|
2|23 ====== XPSWLL RandRRRρρ
3-level CRT – Minimum Detectable Effect Size
What Works?
Main Effect
Under what conditions?
CL Mod
Under what conditions?
TL Mod
For whom?
Ind Mod
J=5 n=10 0.24 0.45 0.29 0.18
n=30 0.22 0.41 0.23 0.11
J=30 n=10 0.19 0.34 0.12 0.08
n=30 0.18 0.33 0.09 0.04
Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,
Slide 35 .20.0,30.0,80.0,75.0,08.0,15.0 2|
2|
2|
2|23 ====== XPSWLL RandRRRρρ
Implications
n Under what conditions? ¨ Cluster level moderator
n Challenging given current study sizes n Effectiveness studies rather than efficacy studies n If priority, need to consider in design stage
60 60 60
30 S 30 U
15C 15C 15 T 15 T
30 T 30 C
40S 20U
20C 10C 10T 20T
30 T 30 C
40S 20U
30C 0C 20T 10T
30 T 30 C
Implications
n Under what conditions? ¨ Teacher level moderator
n Number of teachers per school matters n If priority, need to consider in design phase
Slide 37
Implications
n For whom? ¨ Individual level moderator
n In many cases, reasonable given current study sizes n If priority, need to consider in design phase
Slide 38
Implications
n Extensions ¨ Multisite studies ¨ R code available – jessaca.spybrook@wmich.edu ¨ Some parts implemented in PowerUP! -
http://web.missouri.edu/~dongn/ ¨ Add to Optimal Design Plus
Slide 39
Thank You!
Questions? Please email me: jessaca.spybrook@wmich.edu
Slide 40
Recommended