Embed Size (px)
Citation preview
Pre-Post-Control Effect Size 1
Estimating Effect Size from the Pretest-Posttest-Control Design
Scott B. Morris
Illinois Institute of Technology
April 2003
Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology,
Orlando, FL.
Pre-Post-Control Effect Size 2
Estimating Effect Size from the Pretest-Posttest-Control Design
Despite advances in the statistical models available, researchers are still faced with a number of
operational challenges when conducting a meta-analysis. One of these challenges is dealing with data from
alternate research designs. Not all studies will use the same research design, and researchers need to understand
how best to estimate effect sizes from alternate designs.
This paper will discuss the Pretest-Posttest-Control (PPC) design. In the PPC design, research
participants are assigned to treatment or control conditions, and each participant is measured both before and
after the treatment has been administered. The PPC design is a useful quasi-experimental design for examining
change over time, and is often recommended for evaluating organizational interventions and training
effectiveness. Several effect size estimates have been recommended for the PPC design. This paper compares
these alternatives in terms of their precision and usability in meta-analysis.
When choosing among alternate effect size estimates, several factors should be considered. First, the
effect size estimate should be unbiased. Second, among unbiased estimates, the most precise effect size should
be selected. In general, estimates with smaller sampling variance will provide more precise estimates of the
mean effect size, particularly when the number of studies in the meta-analysis is small. Even in large meta-
analyses, moderator analysis often requires the examination of subgroups with a relatively small number of
studies. Therefore, the selection of a more precise effect size estimate can improve the accuracy of the results.
A third consideration is that the distribution of the effect size must be known. Characteristics of the
sampling distribution, such as the degree of bias or the sampling variance are needed in order to conduct a meta-
analysis. Fore example, estimates of sampling variance are used in several meta-analysis procedures. When
computing the precision-weighted mean effect size, the weights are computed from the inverse of the sampling
variance (Hedges & Olkin, 1985). Estimates of sampling variance are also needed to build confidence intervals
around the mean effect size estimate, to test to homogeneity of effect size, and to estimate random variance
component in random effects models.
A forth factor that can be used to choose among alternate effect sizes estimates is robustness to
violations of model assumptions. Standard meta-analysis procedures make many assumptions about the nature
of the data (e.g., normality, homogeneity of variance) that may be inappropriate in many situations. Some effect
size estimates may be more resistant than others to the effects of violating these assumptions.
The current paper will consider violations of the homogeneity of variance assumption. All of the effect
sizes to be compared assume that pre- and posttest scores have equal variance. However, when the effect of
treatment is not the same for each individual, the treatment will tend to increase the variance of scores.
Therefore, posttest variances are often larger than pretest variances, resulting in smaller effect size estimates for
alternatives that use the posttest standard deviations (Carlson & Schmidt, 1999).
The following section will define an effect size for the PPC design and present three alternate estimates
of this effect size. The distribution of each effect size will be discussed, and the results of a Monte Carlo
2
Pre-Post-Control Effect Size 3 simulation will be used to compare the relative efficiency of the alternatives. Next, the effect of violating the
homogeneity of variance assumption will be examined.
Effect Size for the PPC Design
The data are assumed to be randomly sampled from two populations, corresponding to treatment and
control conditions. Pretest and posttest scores in each population have a bivariate normal distribution with
common variance σ2 and common correlation ρ, but distinct means, indicated by µE,pre for the treatment
population pretest, µE,post for the treatment population posttest, µC,pre for the control group pretest, and µC,post for
the control group posttest.
The standardized mean change in each population is defined as the mean difference between posttest
and pretest scores, divided by the common standard deviation. The standardized mean change for the treatment
group (δE) is
σ
µµδ preEpostE
E,, −
= . ( 1)
The standardized mean change for the control group (δC) is
σ
µµδ preCpostC
C,, −
= . ( 2)
The effect size for the PPC design is defined as the difference between the standardized mean change for the
treatment and control groups,
( ) ( )
σ
µµµµδδ preCpostCpreEpostE
CE,,,, −−−
=−=∆ . ( 3)
Alternate Effect Size Estimates
An individual study consists of nE participants receiving treatment, and nC participants in the control
group. The pretest and posttest means for the treatment group are indicated by Mpre,E and Mpost,E, respectively.
The pretest and posttest means for the control group are indicated by Mpre,C and Mpost,C, respectively. A separate
estimate of the standard deviation can be obtained for the treatment groups at pretest (SDpre,E) and posttest
(SDpost,E), and for the control group at pretest (SDpre,C) and posttest (SDpost,C). These standard deviations can be
combined in several different ways to derive different estimates of the effect size ∆.
Effect Size Estimate Using Separate Pretest SDs
Becker (1988) described an effect size measure for the PPC design, referred to here as gppc1,
Cpre
CpreCpost
Epre
EpreEpostppc SD
MMSD
MMg
,
,,
,
,,1
−−
−= . ( 4)
3
Pre-Post-Control Effect Size 4 This effect size estimate is biased when sample size is small. An approximately unbiased estimate can be
obtained using
−−
−=
Cpre
CpreCpostC
Epre
EpreEpostEppc SD
MMc
SDMM
cd,
,,
,
,,1 , ( 5)
where the bias adjustments cE and cC can be approximated by
( ) 11431
−−−=
jj n
c . ( 6)
Effect Size Estimate Using Pooled Pretest SD
A limitation of dppc1 is that separate estimates of the sample standard deviation are used (SDpre,E and
SDpre,C), despite the assumption that the population variances are homogeneous. Under this assumption, a better
estimate of the population standard deviation could be obtained by pooling the data from the treatment and
control groups. This suggests an alternative effect size estimate, which will provide a more precise estimate of
the population treatment effect,
( ) ( )
−−−=
Ppre
CpreCpostEpreEpostPppc SD
MMMMcd
,
,,,,2 ( 7)
where the pooled standard deviation is defined as
( ) ( )
211 2
,2
,, −+
−+−=
CE
CpreCEpreEPpre nn
SDnSDnSD ( 8 )
and
( ) 12431
−−+−=
CEP nn
c . ( 9)
Except for the bias correction, dppc2 is the same as the effect size estimate (ESPPWC) recommended by Carlson &
Schmidt (1999).
Effect Size Based on the Pooled Pre- and Posttest SD
Both of the preceding estimates consider only the pretest standard deviations. Under the assumed
model, pretest and posttest variances are homogeneous. Therefore, a more precise estimate (dPPC3) can be
obtained by pooling estimates across both pretest and posttest measurements for both treatment and control
conditions (Dunlap, Cortina, Vaslow & Burke, 1996; Taylor & White, 1992). Specifically,
4
Pre-Post-Control Effect Size 5
( ) ( )
−−−=
− Ppostpre
CpreCpostEpreEpostPPppc SD
MMMMcd
,
,,,,3 ( 10)
where the pooled standard deviation is defined as
( ) ( ) ( ) ( )
( )221111 2
,2
,2
,2
,, −+
−+−+−+−=−
CE
CpostCEpostECpreCEpreEPpostpre nn
SDnSDnSDnSDnSD . ( 11 )
For dppc3, the exact value of the bias correction is not known. If the four standard deviations were from
independent groups, the bias correction would be
( ) 1422431
−−+−=
CEPP nn
c . ( 12)
However, because pre- and posttest scores are not independent, the amount of information gained by adding the
posttest standard deviations will be less than if the groups were independent (Kish, 1965). Therefore, the degree
of bias is likely to be greater than indicated by Equation 12 (i.e., cPP will be too large), particularly when there is
a substantial correlation between pre- and posttest scores. Consequently, it was expected that the use of
Equation 12 would result in a slight overestimate the population effect size for large values of ρ.
Comparison of Alternate Effect Size Estimates
The two alternatives that involve pooling of standard deviations across groups should allow more
precise estimates of effect size from studies using the PPC design. However, factors other than precision should
also influence the choice of an effect size estimate. In order to use the alternatives in a meta-analysis, it is
necessary to first specify their sampling distribution. Estimates of the sampling variance are needed for meta-
analytic procedures, such as computing the weighted mean or estimating variance of effect sizes in a random
effects model. In addition, it is important to consider the behavior of each alternative under violations of the
model assumptions. The following sections will discuss the theoretical sampling distribution of each effect size
estimate, followed by a Monte Carlo simulation comparing the mean and variance of each estimate under a
variety of conditions. A second simulation compares the performance of each statistic when the assumption of
homogeneity of variance is violated.
Distribution of Alternate Effect Size Estimates
Distribution of dppc1. Becker (1988, c.f., Morris & DeShon, 2002) derived the asymptotic distribution of
dppc1. When sample size is large, dppc1 is approximately normally distributed, with a mean equal to the
population effect size ∆, and variance,
5
Pre-Post-Control Effect Size 6
( )( )
( )( ) .12
13112
121
3112)(
22
2
22
21
2
CCC
C
C
CC
EEE
E
E
EEppc
nnn
nc
nnn
ncd
δρ
δρ
δρ
δρσ
−
−+
−−
−+
−
−+
−−
−=
( 13)
Distribution of dppc2. The asymptotic distribution of dppc2 can be derived using the approach developed
by Becker (1988; c.f., Morris, 2000). Following Hedges (1981), the distribution of dppc2 is derived by relating
the effect size to the non-central t distribution.
The effect size estimate without the correction for small sample bias will be referred to as gppc2, which is
defined,
( ) ( )
Ppre
CpreCpostEpreEpostppc SD
MMMMg
,
,,,,2
−−−= . ( 14)
The numerator is normally distributed with a mean of σ∆ and a standard error of
nnn+n)-2(1CE
CEρσ ( 15)
Therefore,
( ) ( )
σ
ρσ
ρPpre
CE
CE
CpreCpostEpreEpost
CE
CE
ppc
SDnnn+n)-2(1
M-M-M-M
=
nnn+n)-2(1
g
,
,,,,
2 ( 16)
is distributed as a non-central t, with df = nE + nC - 2 and noncentrality parameter,
.
nnn+n)-2(1
=
CE
CE
∆
ρ
φ ( 17)
As a result, gppc2 is approximately normally distributed with expected value
6
Pre-Post-Control Effect Size 7
.c
=
nnn+n)-2(1c
nnn+n)-2(1=)E(g
P
CE
CEP
CE
CEppc
∆
∆
ρ
ρ2 ( 18)
cP is a function indicating the degree of bias in gppc2,
,1)/2]-[(df
[df/2]df2=cP
ΓΓ
( 19)
where Γ is the gamma function (Johnson and Kotz, 1970). An unbiased estimate (dppc2) can be obtained by
multiplying the effect size by cP, as in Equation 7. A close approximation to cP is provided in Equation 9.
The variance of g is times the variance of a noncentral )nn)/(n+n)(-2(1 CECEρ t. The variance of a
noncentral t is defined by Johnson & Kotz (1970) as
( )[ ]
.c(df)
-+12-df
df=(t) 2
222 φ
φσ
( 20)
Substituting df=nE + nC - 2 and the noncentrality parameter specified in Equation 17, the variance of gppc2 is
( ) .c
-
nnn+n)-2(1
+14-n+n2-n+n
nnn+n-12=)(g
p
2
CE
CE
2
CE
CE
CE
CEppc
2
∆
∆
22
ρρσ ( 21)
The variance of dppc2 is cp2 times the variance of gppc2, or
( )( ) .-
nnn+n)-2(1
+14-n+n2-n+n
nnn+n-1c2=)(d 2
CE
CE
2
CE
CE
CE
CEpppc
2 ∆
∆
ρρσ 2
2 ( 22)
Distribution of dppc3. The third alternative (dppc3) is problematic because the sampling distribution is
unknown. For dppc3, the standard deviation is pooled across non-independent scores (i.e., pretests and posttest
scores). As a result, the gain in precision will be less than if standard deviations were pooled across independent
groups. The exact sampling variance of dppc3 is currently unknown, but is expected to be smaller than the other
alternatives, and to approach dppc2 as ρ approaches 0.
Bias and Precision of Alternate Effect Size Estimates
7
Pre-Post-Control Effect Size 8 A Monte Carlo Simulation was used to investigate the mean and sampling variance of each effect size.
A Fortran program was used to conduct the simulation. The performance of each estimate was investigated
across varying levels of ∆, ρ and n. The magnitude of the population effect size, ∆, was manipulated by varying
the standardized mean change in the treatment and control groups. The standardized mean change in the
treatment group, δE , was set at 0.0, 0.5, or 1.0. The standardized mean change in the control group, δC , was set
at 0.0 or 0.2. The correlation between pre- and posttest scores, ρ, was always the same for the treatment and
control groups, and was set at 0.0, 0.45 or 0.9. The sample size was always equal across groups, and the within-
group n was 10, 25 or 50.
For each combination of the parameters, the program generated 10,000 samples with n observations
from both the treatment and control groups. For each observation, pre- and posttest scores were generated from
a bivariate normal distribution with a pre-post correlation of ρ. The data were generated using the multivariate
random number generator (DRNVMN) of the International Mathematical and Statistical Libraries (IMSL;
1884). The effect of treatment was manipulated by adding δE or δC to the posttest scores, depending on the
group.
For each sample, the three effect size estimates (dppc1, dppc2, and dppc3) were computed using the formulas
given above. The mean and variance of each effect sizes was computed across the 10,000 random samples. The
mean effect sizes were evaluated based on how closely they matched the population effect size specified in the
simulation (i.e., bias). Sampling variances were compared across the three alternatives in terms of relative
efficiency (i.e., the ratio of the variance of one estimator to the variance of another estimator), as well as how
closely the observed variance matched the theoretical variance, when known. The results of the simulation are
summarized in Tables 1 and 2.
As expected, dppc1 and dppc2 were nearly unbiased. Across all conditions, the mean effect size was
always within .008 of the population parameter. dppc3 generally provided a good estimate, except when n was
small and ρ was large, in which case there was a slight positive bias. For example, with δE = 1.0, δC = 0.0, ρ =
.9, and n=10, the mean effect size was 1.013. As expected, when ρ was large, the bias correction, which was
based on the degrees of freedom from four independent groups, underestimated the degree of bias, and an
overestimate of the population effect size. However, the degree of bias was not large.
The results also verify the accuracy of the sampling variance formulas for dppc1 and dppc2. Under most
conditions, the theoretical variance differed from the observed variance by less then 3%, and in no case was the
difference greater than 7% of the observed variance.
The analysis of relative efficiency confirmed the expectation that dppc2, where the pretest standard
deviations were pooled across treatment and control groups, would provide a more precise estimate of the
sampling variance than dppc1. In general, dppc2 had a smaller sampling variance than dppc1, as indicated by relative
efficiency less than 1.0. When both δE = 1.0 and δC = 0.0, there was little difference between the two estimates.
8
Pre-Post-Control Effect Size 9 However, as either δE or δC increased, the superiority of dppc2 over dppc1 became more apparent. The difference
was greatest when n was small and ρ was large. The variance of dppc2 was as much as 50% lower than the
variance of dppc1 (i.e., for δE = 1.0, δC = 0.2, ρ = .9, and n=10).
Contrary to expectations, pooling both pre- and posttest standard deviations in dppc3 did not substantially
reduce the sampling variance relative to dppc2. Across conditions, the variance of dppc3 was generally lower than
the variance of dppc2, but the difference was quite small. The relative efficiency of dppc3 to dppc2 was generally
greater than .96, and never fell below .94.
Heterogeneity of Variance
A common concern with the PPC design is that the posttest variance in the treatment group may be
larger than the variance of the untreated population, violating the homogeneity of variance assumption. If there
are individual differences in the effectiveness of treatment (a subject by treatment interaction), some individuals
in the treatment condition will improve more than others, and the distribution of scores at posttest will be more
spread out than at pretest (Cook & Campbell, 1979). It is likely that the pretest scores in the treatment group, as
well as both pre- and posttest scores in the control group will reflect the untreated population, and therefore,
homogeneity of variance is reasonable for these conditions. However, the variance of posttest scores in the
treatment group might be inflated relative to the other conditions. This pattern has been shown in research on
training effectiveness (Carlson & Schmidt, 1999); therefore, it is important to examine the performance of the
alternate effect size estimates under this pattern of heterogeneity.
Heterogeneity of variance raises the question of which standard deviation to use in the definition of the
population effect size. This study followed the recommendations of Becker (1988) and Carlson and Schmidt
(1999) by defining the population effect size using the standard deviation of the untrained population. Because
the larger posttest standard deviation in the treatment group results from a Subject x Treatment interaction, it
may depend on the magnitude of the treatment effect, which may not be the same across studies. The standard
deviation of the untreated population is more likely to be comparable across studies.
It was expected that heterogeneity of variance would be most problematic for dppc3. While homogeneity
of variance is assumed by all three estimates, only dppc3 uses the standard deviation of posttest scores when
computing the effect size estimate. The inflated posttest standard deviation in the treatment group will tend to
increase the pooled standard deviation, resulting in an effect size estimate that is too small. dppc1 and dppc2,
which use only the pretest standard deviations, should provide an unbiased estimate of the population effect size,
even when posttest scores are heterogeneous.
A Monte Carlo simulation was conducted to examine the performance of the alternate effect size
estimates when the variance of posttest scores in the treatment group was inflated. The posttest standard
deviation in the treatment groups was set at 1.5 times the standard deviation of the untreated population. Scores
in each group were initially generated from a bivariate normal distribution with all means equal to zero and
9
Pre-Post-Control Effect Size 10 variances equal to one. Then, posttest scores were computed by first multiplying each score by the posttest
standard deviation (1.5), and then adding the population standardized mean change, δE or δC, depending on the
group. Otherwise the simulation was the same as the first simulation, except that the standardized mean change
in the control group was always δC = 0.0. The results of the simulation are presented in Table 3.
As expected, dppc3 underestimated the population effect size, except when δE = 0. In general, the mean
effect size tended to underestimate the population parameter by about 12%, regardless of the sample size or the
magnitude of ρ. dppc1 and dppc2 were nearly unbiased, providing mean estimates comparable to those when the
homogeneity assumption was satisfied.
Although dppc1 and dppc2 were unbiased estimates, introducing heterogeneity of variance had a negative
impact on their precision. The sampling variance of both estimates was substantially larger than in the previous
simulation. In addition, the inflated sampling variance was systematically larger than the theoretical variance
estimated using Equations 13 and 22. The theoretical variance underestimated the actual variance by between
21% and 48%, with more extreme errors occurring when ρ was large.
Conclusion
When conducting a meta-analysis, it is important to obtain the best possible effect size estimate from
each study. The choice of an effect size depends on a number of considerations, such as bias, precision, having
a known sampling distribution, and robustness to violations of assumptions.
For the PPC design, three effect size estimates have been recommended in the literature. Becker (1988)
suggested a measure of effect size using separate estimates of the pretest standard deviation in the treatment and
control groups (dppc1). Carlson and Schmidt (1999) used an effect size based on the pooled pretest standard
deviation (dppc2). Others (Dunlap et al., 1996; Taylor & White, 1992) have recommend pooling standard
deviations across both pretest and posttest scores (dppc3).
The results support dppc2 as the best choice. dppc2 provides an unbiased estimate of the population effect
size, and has a known sampling variance that is smaller than the sampling variance of dppc1.
Pooling standard deviations across both pre- and posttest scores, as in dppc3 is problematic. Because pre-
and posttest scores are not independent, the sampling variance of dppc3 is more complex than the other two
estimates, and is currently unknown. In addition, dppc3 demonstrated very little improvement over dppc2 in terms
of sampling variance. Because meta-analytic procedures require estimates of sampling variance, the use of dppc3
is not recommended.
Under a common violation of the homogeneity of variance assumption (i.e., when posttest variance in
the treatment group was inflated), dppc3 produced a biased estimate of the population effect size. Inflation of the
posttest variance resulted in values of dppc3 that tended to underestimate the population effect size. Both dppc1
and dppc2 were unbiased, which is not surprising, because these two estimates use only the pretest standard
deviations.
10
Pre-Post-Control Effect Size 11
When the homogeneity of variance assumption was violated, the sampling variance of both dppc1 and
dppc2 were inflated. Thus, the theoretical estimates of variance due to sampling error (which assume
homogeneity of variance) tended to underestimate the observed variance. The theoretical variance is used in a
number of meta-analytic procedures, and therefore, inaccuracy in this value could lead to erroneous conclusions.
Underestimates of sampling variance will result in confidence intervals that are two small, and will artificially
inflate tests for homogeneity of variance. Because the estimated variance is too small, a meta-analyst might
incorrectly conclude that effect sizes are heterogeneous, and potentially search for moderators of effect size
when none exist. Additional work is therefore needed to better estimate the sampling variance of effect sizes
from the PPC design when variances are heterogeneous.
References
Becker, B. J. (1988). Synthesizing standardized mean-change measures. British Journal of
Mathematical and Statistical Psychology, 41, 257-278.
Carlson, K. D., & Schmidt, F. L. (1999). Impact of experimental design on effect size: Findings from
the research literature on training. Journal of Applied Psychology, 84, 851-862.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field
settings. Boston, MA: Houghton Mifflin.
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with
matched groups or repeated measures designs. Psychological Methods, 1, 170-177.
Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators.
Journal of Educational Statistics, 6, 107-128.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
International Mathematical and Statistical Library (1984). User's Manual: IMSL Library, Problem-
solving software system for mathematical and statistical FORTRAN programming (vol. 3, ed. 9.2) Houston, TX:
Author.
Johnson, N. L., & Kotz, S. (1970). Continuous univariate distributions. NY: John Wiley & Sons.
Kish, L. (1965). Survey Sampling. NY: Wiley.
Morris, S. B. (2000). Distribution of the standardized mean change effect size for meta-analysis on
repeated measures. British Journal of Mathematical and Statistical Psychology, 53, 17-29.
Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated
measures and independent-groups designs. Psychological Methods, 7, 105-125.
Taylor, M. J., & White, K. R. (1992). An evaluation of alternative methods for computing standardized
mean difference effect size. Journal of Experimental Education, 61, 63-72.
11
Pre-
Post
-Con
trol E
ffec
t Siz
e
12
Ta
ble
1
Mea
n an
d V
aria
nce
of A
ltern
ate
Effe
ct S
ize
Estim
ates
Whe
n δ C
= 0
.0.
d p
pc1
d p
pc2
d p
pc3
Rel
ativ
e Ef
ficie
ncy
δ E
ρ n
M
Obs
erve
d V
aria
nce
Theo
retic
al
Var
ianc
e
M
Obs
erve
d V
aria
nce
Theo
retic
al
Var
ianc
e
M
Obs
erve
d V
aria
nce
dpp
c2/ d
ppc1
dpp
c3/ d
ppc2
0.
0
0.00
100.
003
0.42
5 0.
430
0.00
20.
409
0.41
3 0.
000
0.40
4
0.96
0.
99
0.0
0.00
250.
004
0.16
30.
164
0.00
40.
161
0.16
20.
004
0.15
90.
990.
990.
00.
0050
0.00
30.
079
0.08
10.
004
0.07
80.
080
0.00
30.
078
0.99
1.00
0.0
0.45
10-0
.002
0.23
00.
236
-0
.004
0.22
10.
227
-0.0
030.
218
0.96
0.99
0.0
0.45
25-0
.001
0.09
10.
090
-0
.001
0.
090
0.08
9-0
.001
0.09
00.
991.
000.
00.
4550
0.00
10.
044
0.04
40.
001
0.04
40.
044
0.00
10.
044
1.00
1.00
0.0
0.90
10-0
.001
0.04
40.
043
-0
.001
0.
042
0.04
1-0
.001
0.04
30.
961.
030.
00.
9025
0.00
10.
016
0.01
60.
001
0.01
60.
016
0.00
10.
016
0.99
1.01
0.0
0.90
50-0
.001
0.00
80.
008
-0
.001
0.
008
0.00
8-0
.001
0.00
81.
001.
010.
50.
0010
0.50
70.
451
0.44
80.
508
0.41
70.
421
0.50
50.
405
0.92
0.97
0.5
0.00
250.
499
0.16
80.
169
0.49
90.
163
0.16
40.
500
0.16
10.
970.
990.
50.
0050
0.49
50.
082
0.08
40.
496
0.08
10.
082
0.49
60.
080
0.98
0.99
0.5
0.45
100.
497
0.25
30.
255
0.49
70.
234
0.23
50.
499
0.23
00.
930.
980.
50.
4525
0.50
70.
094
0.09
60.
506
0.09
00.
092
0.50
80.
089
0.96
0.99
0.5
0.45
500.
499
0.04
80.
047
0.49
90.
046
0.04
60.
500
0.04
50.
970.
990.
50.
9010
0.49
90.
060
0.06
10.
500
0.04
90.
049
0.50
80.
049
0.81
1.01
0.5
0.90
250.
495
0.02
20.
022
0.49
50.
019
0.01
90.
498
0.01
90.
851.
000.
50.
9050
0.50
00.
011
0.01
10.
499
0.00
90.
009
0.50
10.
009
0.87
0.99
1.0
0.00
100.
998
0.49
90.
503
0.99
50.
440
0.44
40.
996
0.41
80.
880.
951.
00.
0025
0.99
90.
187
0.18
70.
999
0.17
10.
173
0.99
90.
164
0.92
0.96
1.0
0.00
500.
989
0.09
10.
092
0.98
90.
085
0.08
60.
990
0.08
20.
930.
971.
00.
4510
1.00
00.
308
0.31
01.
003
0.25
40.
259
1.00
80.
239
0.83
0.94
1.0
0.45
251.
003
0.11
30.
113
1.00
20.
101
0.10
01.
004
0.09
60.
890.
951.
00.
4550
0.99
90.
055
0.05
51.
000
0.04
90.
049
1.00
00.
047
0.90
0.96
1.0
0.90
100.
994
0.11
60.
117
0.99
60.
073
0.07
31.
013
0.07
10.
630.
971.
00.
9025
1.00
00.
038
0.03
91.
000
0.02
70.
027
1.00
60.
026
0.69
0.96
1.0
0.90
501.
000
0.01
90.
019
1.00
00.
013
0.01
31.
003
0.01
30.
720.
96
Pre-
Post
-Con
trol E
ffec
t Siz
e
13
Ta
ble
2
Mea
n an
d V
aria
nce
of A
ltern
ate
Effe
ct S
ize
Estim
ates
Whe
n δ C
= 0
.2.
d p
pc1
d p
pc2
d p
pc3
Rel
ativ
e Ef
ficie
ncy
δ E
ρ n
O
bser
ved
Var
ianc
e Th
eore
tical
V
aria
nce
M
O
bser
ved
Var
ianc
e Th
eore
tical
V
aria
nce
M
O
bser
ved
Var
ianc
e d
ppc2
/ dpp
c1 d
ppc3
/ dpp
c2
0.0
0.
0010
-0.1
950.
439
0.43
2 -0
.197
0.41
9 0.
414
-0.1
970.
412
0.
96
0.98
0.
0
0.00
25-0
.203
0.16
50.
165
-0.2
030.
162
0.16
2-0
.203
0.16
20.
981.
000.
00.
0050
-0.2
000.
081
0.08
1-0
.200
0.08
00.
081
-0.1
990.
080
0.99
0.99
0.0
0.45
10-0
.201
0.24
30.
239
-0.2
010.
231
0.22
8-0
.202
0.22
90.
950.
990.
00.
4525
-0.2
010.
092
0.09
1-0
.201
0.09
00.
089
-0.2
010.
090
0.98
1.00
0.0
0.45
50-0
.200
0.04
50.
045
-0.1
990.
045
0.04
4-0
.199
0.04
50.
991.
000.
00.
9010
-0.1
980.
045
0.04
6-0
.198
0.04
20.
043
-0.2
010.
043
0.93
1.03
0.0
0.90
25-0
.202
0.01
70.
017
-0.2
010.
017
0.01
7-0
.203
0.01
70.
961.
010.
00.
9050
-0.2
000.
009
0.00
9-0
.199
0.00
80.
008
-0.2
000.
008
0.98
1.00
0.5
0.00
100.
296
0.43
80.
451
0.29
80.
410
0.41
60.
300
0.40
30.
940.
980.
50.
0025
0.30
00.
173
0.17
00.
300
0.16
40.
163
0.30
10.
164
0.95
1.00
0.5
0.00
500.
296
0.08
30.
084
0.29
60.
080
0.08
10.
296
0.08
00.
971.
000.
50.
4510
0.30
50.
253
0.25
80.
302
0.22
70.
230
0.30
30.
224
0.90
0.99
0.5
0.45
250.
299
0.09
80.
097
0.29
90.
091
0.09
00.
299
0.09
10.
940.
990.
50.
4550
0.30
20.
048
0.04
80.
302
0.04
60.
045
0.30
20.
045
0.94
0.99
0.5
0.90
100.
296
0.06
30.
064
0.29
80.
045
0.04
40.
303
0.04
60.
721.
020.
50.
9025
0.30
00.
023
0.02
30.
301
0.01
70.
017
0.30
30.
017
0.75
1.01
0.5
0.90
500.
301
0.01
10.
011
0.30
10.
008
0.00
90.
302
0.00
80.
771.
001.
00.
0010
0.80
60.
512
0.50
60.
805
0.43
80.
433
0.80
40.
419
0.86
0.96
1.0
0.00
250.
805
0.18
80.
188
0.80
40.
168
0.16
90.
805
0.16
40.
890.
981.
00.
0050
0.80
20.
092
0.09
20.
803
0.08
40.
084
0.80
30.
082
0.91
0.98
1.0
0.45
100.
806
0.30
80.
313
0.80
80.
241
0.24
70.
810
0.23
30.
780.
961.
00.
4525
0.80
40.
118
0.11
40.
804
0.09
80.
096
0.80
50.
095
0.83
0.97
1.0
0.45
500.
799
0.05
60.
056
0.79
70.
048
0.04
80.
798
0.04
60.
850.
971.
00.
9010
0.80
30.
123
0.12
00.
799
0.06
30.
062
0.81
30.
062
0.51
0.99
1.0
0.90
250.
804
0.04
00.
040
0.80
10.
023
0.02
30.
806
0.02
30.
580.
991.
00.
9050
0.80
00.
019
0.01
90.
800
0.01
10.
011
0.80
30.
011
0.60
0.98
M
13
Pre-
Post
-Con
trol E
ffec
t Siz
e
14
14
Tabl
e 3
Mea
n an
d V
aria
nce
of A
ltern
ate
Effe
ct S
ize
Estim
ates
Whe
n δ C
= 0
.0 a
nd H
eter
ogen
eous
Var
ianc
e.
d p
pc1
d p
pc2
d p
pc3
δ E
ρ n
M
Obs
erve
d V
aria
nce
Theo
retic
al
Var
ianc
e
M
Obs
erve
d V
aria
nce
Theo
retic
al
Var
ianc
e
M
Obs
erve
d V
aria
nce
0.0
0.
0010
-0.0
130.
572
0.43
0 -0
.012
0.55
2 0.
413
-0.0
100.
417
0.0
0 8
4
5
2
6
3
7 1
7 0
1 4
6
1 7
7
1 0
0
9
2
1 0
4
1 9
4
0 5
1
3 3
2
9 1
2
2 2
1 6
2
0 6
2
4
0 5
8
0 5
8
0 1
8
9 8
7
7 1
2
0
1 0
9
2 5
4
8 2
9 0
4
9 9
2
6 2
8
5 5
7
5 5
6
1
1 9
6
0 4
2
8 4
1
4 7
1
2 6
8
7 1
7 1
1
4 9
6
6
9 6
2
8 3
9
8 5
8
7 1
7
6 9
6
2
4 4
3
6 2
4
0 5
0 4
7
1 1
3
4 3
9
7 2
7
1 6
3
3
7 3
0
8 7
9
9 8
2
4 3
3
1 0
8
7 7
2 5
7
6 9
1
9
6 8
7
5 1
3
5 6
7
3 9
8
2 7
7
2 0
7 9
0
1 3
6
6
0.00
250.
000.
200.
160.
001
0.20
0.16
0.00
10.
150.
00.
0050
0.00
0.10
0.08
0.00
30.
100.
080.
002
0.08
0.0
0.45
10
-0
.006
0.
320.
23-0
.006
0
0.31
0.22
-0.0
05
0 0.
230.
00.
4525
-0.0
010.
120.
090.
000.
120.
080.
000.
090.
00.
4550
0.00
0.06
0.04
0.00
0.05
0.04
0.00
0.04
0.0
0.90
100.
000.
080.
040.
000.
070.
040.
000.
060.
0 0.
90
25
0.00
0.03
0.01
0.00
0.03
0.01
0.00
0.02
0.0
0.90
500.
000.
010.
000.
000.
010.
000.
000.
010.
50.
0010
0.50
0.57
0.44
0.50
0.54
0.42
0.44
0.41
0.5
0.00
250.
500.
220.
160.
500.
210.
160.
430.
160.
5 0.
00
50
0.49
0.11
0.08
0.49
0.10
0.08
0.43
0.08
0.5
0.45
100.
500.
330.
250.
500.
310.
230.
440.
240.
50.
4525
0.50
0.12
0.09
0.50
0.12
0.09
0.43
0.09
0.5
0.45
500.
500.
060.
040.
500.
060.
040.
430.
040.
5 0.
90
10
0.50
0.09
0.06
0.50
0.08
0.04
0.44
0.06
0.5
0.90
250.
490.
030.
020.
490.
030.
010.
430.
020.
50.
9050
0.49
0.01
0.01
0.49
0.01
0.00
0.43
0.01
1.0
0.00
101.
000.
640.
501.
000.
580.
440.
880.
421.
0 0.
00
25
1.00
0.23
0.18
1.00
0.22
0.17
0.87
0.16
1.0
0.00
500.
990.
110.
090.
990.
110.
080.
870.
081.
00.
4510
0.99
0.39
0.31
0.99
0.34
0.25
0.87
0.25
1.0
0.45
251.
000.
140.
111.
000.
130.
100.
870.
091.
0 0.
45
50
0.99
0.07
0.05
0.99
0.06
0.04
0.87
0.04
1.0
0.90
100.
990.
150.
110.
990.
110.
070.
880.
081.
00.
9025
0.99
0.05
0.03
0.99
0.04
0.02
0.87
0.03
1.0
0.90
50
1.
000.
020.
011.
000.
020.
010.
870.
01
