Embed Size (px)
Citation preview
MONTE CARLO SIMULATION WITH PARAMETRIC AND NONPARAMETRIC
ANALYSIS OF COVARIANCE
FOR NONEQUIVALENT CONTROL GROUPS
by
Mary Bender
Dissertation submitted to the Faculty of the
Virginia Polytechnic Institute and State University
in partial fulfillment of the requirements for the degree of
DOCTOR OF PHILOSOPHY'
in Educational Research and Evaluation
APPROVED:
c:J?. Fortune, Co-Chairman w. Seaver, co-Chairman
R. McKeen
'L •
9 unaerwoOd
April, 1987
Blacksburg, Virginia
MONTE CARLO SIMULATION. WITH PARAMETRIC AND NONPARAMETRIC
ANALYSIS OF COVARIANCE
FOR NONEQUIVALENT CONTROL GROUPS
by
Mary Bender
Conunittee Co-Chairmen: Jinunie c. Fortune William L. Seaver
Educational Research and Evaluation
(ABSTRACT)
There are many parametric statistical models that have
been designed to measure change in nonequivalent control
group studies, but because of assumption violations and
potential artifacts, there is no one form of analysis that
always appears to be appropriate. While the parametric
analysis of covariance and parametric ANCOVAS with a
covariate correction are some of the more frequently
completed analyses used in nonequivalent control group
research, comparative studies with nonparametric
counterparts should be completed and results compared with
those more conunonly used forms of analysis.
The current investigation studied and compared the
application of four ANCOVA models: the parametric, the
covariate-corrected parametric, the rank transform, and the
covariate-corrected rank transform. Population parameters
were established; sample parameter intervals determined by
Monte Carlo simulation were examined; and a best ANCOVA
model was systematically and theoretically determined in
light of population assumption violations, reliability of
the covariate correction, the width of the sample
probability level intervals, true parent population
parameters, and results of robust regression.
Results of data exploration on the parent population
revealed that, based on assumptions, the covariate-
corrected ANCOVAS are preferred over both the parametric
and rank analyses. A reliability coefficient of r=.83 also
indicated that a covariate-corrected ANCOVA is effective in
error reduction. Robust regression indicated that the
outliers in the data set impacted the regression equation
for both parametric models, and deemed selection of either
model questionable.
The tightest probability level interval for the
samples serves to delineate the model with the greatest
convergence of probability levels, and, theoretically, the
most stable model. Results of the study indicated that,
because the covariate-corrected rank ANCOVA had by far the
tightest interval, it is the preferred model. In addition,
the probability level interval of the covariate-corrected
rank model is the only model interval that contained the
true population parameter.
Results of the investigation clearly indicate that the
covariate-corrected rank ANCOVA is the model of choice for
this nonequivalent control group study. While its use has
yet to be reported in the literature, the covariate-
corrected rank analysis of covariance provides a viable
alternative for researchers who must rely upon intact
groups for the answers to their research questions.
ACKNOWLEDGMENTS
Sincere appreciation and gratitude are extended to my
co-chairmen, Dr. Jimmie Fortune and Dr. Bill Seaver, for
their inspiration in the planning and development of the
study; to my other committee members, Dr. Larry McCluskey,
Dr. Ron McKeen, and Dr. Ken Underwood, for their enthusias-
tic and positive attitudes; to my husband, Lou Bender, for
the sacrifices he made throughout my graduate studies; to
my son, Ryan Bender, for his assistance in random number
generation and his allowing me to use my computer; to my
daughter, Elizabeth Bender, for her loving patience; to
Susan Becker for her computing expertise and insistence
that we're not looking for perfect; to my dear friend, Dr.
Virginia Vertiz, and my boss (and friend), Dr. Bruce
Chaloux, for their ongoing support and encouragement
throughout the analysis and writing of this dissertation.
v
TABLE OF CONTENTS
TITLE PAGE • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. i ABSTRACT ••. . ...................................... . . ii ACKNOWLEDGMENTS. ...................................... . . v
TABLE OF CONTENTS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
LIST OF TABLES .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
LIST OF FIGURES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Chapter
I. INTRODUCTION AND BACKGROUND OF THE STUDY ..•......... 1
II.
III.
Statement of the Problem .. • • 9 Purpose of the Study .. .10 Research Questions... . ...•• . .• 10 General Questions..... . .....•.•.....•. Definitions of Terms. ............. . .....•
..11
.. 12
..19 Limitations of the Study....... . .•.• Significance of the Study......... . .... . ..... 2 a REVIEW OF THE RELATED LITERATURE. .23
The Analysis of Covariance....... .23 Assumptions of the Parametric ANCOVA .....••........ 26 The Measurement of Change............. ..39 Nonequivalent Control Groups.......... . .....•... 47 The ANCOVA in Nonequivalent Control Group Research.SO The Nonparametric ANCOVA. ......•. .54 SUllUllary • •.•••••..••••••••
METHODS AND PROCEDURES ..
overview ..•.•....... Study Population .....••.. Data Identification .•. Research Design .....•
........ 67
.68
.68
.69
.71 . ....... 71
Design on Groups............ . ..........• .71 ..73 • • 79
Design on Comparisons.. . ..••• Analytical Procedures .•.......••••. Variable Production ...•......• Data Exploration •.••.•.•.
• ••• 7 9 .80
The ANCOVAS on the Population .• Robust Regression on the Population.
. ........ 83
Monte Carlo Samples and the ANCOVA ......•...... Assumption Diagnosis and ANCOVA Comparisons.
vi
.84 ..85
• •• 86
IV.
v.
Implementation Guidelines .......................... 89
RESULTS •. . ....................................... . .93
overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93 Population Results ......................•.....•.... 94
Data Exploration on the Population.. ..94
Data
Linearity •..•............. . ............ 94 Normality. . . . . . . . . . . . . . . . . . . . . . . ..... 10 5 Homogeneity of Variances •............. 108 Homogeneity of Within-Group Regressions.108 Monotonicity. . . . . . . . . • . . . . . . . . . . • . . . .• 111 Equality of Marginal Distributions .•.• 111 Analysis on the Population •....••....••. 116 Covariate Correction •.•..••...•........• 116 ANCOVAS on Pre Measures •....•..•...... 117 Pearson and Spearman Correlations •.... 122 The ANCOVAS on the Population..... ..124
The Parametric ANCOVA........ .124 The Covariate-Corrected Parametric ANCOVA ••••••••••••••••••••••••••••• 129 The Rank ANCOVA •.•.....•..•........ 13 2 The Covariate-Corrected Rank ANCOVA
Robust Regression on the Population .. .136 .140 .149 .149 .151 .157 .165 .173
Sample Results . ......................... . Monte Carlo Samples and the ANCOVA •.. Probability Level Interval Description ..
General Questions.. ..••• . ..... Research Questions.. . ..••...•. SUJ1UY1ary. • • • • • • • • • • • • ••••
DISCUSSION.
Issues Related to the Study .• Further Research .• SUJ1UY1ary • ••••••••••
.174
. .... 17 5 .177
••• 181
LITERATURE CITED. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182
APPENDICES •.•...• .188
A. 100 Random Numbers for Monte Carlo Sample Generation . ................................. . 188
B. Sample Probability Levels from Four ANCOVA Mode ls ••••.•.••..•......•.•.•......•..•..•... 19 0
VITA . ...............................••........•....... • 194
vii
List of Tables
Table 1. Purposes for the Evaluation of Change and Corresponding Model Classification Types, as Reported by Fortune & Hutson (1984) ......................••.•........ 43
Table 2. Assumptions Underlying the Parametric and Nonparametric Analysis of Covariance ...........••......•. 59
Table 3. Simulation Studies Comparing Power and Robustness of the Parametric ANCOVA and the Rank Transform ANCOVA in Relation to Parametric Assumption Violations .•...•....•.• 66
Table 4. Sample Size of the Study Population ............ 70
Table 5. Design on Groups ..••....••..•........•..•..•.•. 72
Table 6. Design on Comparisons Matrix for Population Parameters . ..••........••..•••...••••.•.....••••...•••••. 7 4
Table 7. Design on Subsample Comparisons Matrix for Subsample Probability Intervals .•.•....••....•..•.......• 76
Table 8. Design on Comparisons Matrix for Subsample Probability Intervals in Relation to Population Parameters ••.........•••....•.•••............•.•••.....•.•.......•. 78
Table 9. Pearson and Spearman Correlations between the Covariate and the Dependent Measure by Program Type .••.. 104
Table 10. Goodness-of-Fit Test on Normality of Y Scores.107
Table 11. Bartlett-Box Homogeneity of Variance Tests on Y Scores .. ............................................... . 109
Table 12. Homogeneity of Within-Group Regressions ....••• 110
Table 13. Kruskal-Wallis Table to Assess Equality of Marginal Distributions and Differences in the Parametric X scores .. ............................................... . 112
Table 14. Kruskal-Wallis Table to Assess Equality of Marginal Distributions and Differences in the Covariate-corrected Parametric X Scores .•.....•.........•..••..... 113
Table 15. Kruskal-Wallis Table to Assess Equality of Marginal Distributions and Differences in the Rank X Scores ........................................................ 114
Table 16. Kruskal-Wallis Table to Assess Equality of Marginal Distributions and Differences in the Covariate-Corrected Rank X Scores ••...•..•..•••.•.....•....•..•.•. 115
viii
Table 17. Analysis of Variance Table to Assess Differences in the Parametric X Scores .............................. 118
Table 18. Analysis of Variance Table to Assess Differences in the Covariate-Corrected Parametric X Scores .........• 119
Table 19. Analysis of Variance Table to Assess Differences in the Rank X Scores ......•.••.....•.•....•...•......... 120
Table 20. Analysis of Variance Table to Assess Differences in the Covariate-Corrected Rank X Scores .•....•••....•.• 121
Table 21. Pearson and Spearman Correlations ....•....•... 123
Table 22. Parametric Analysis of Covariance Table ....••• 126
Table 23. Observed and Adjusted Means for the Parametric Analysis of Covariance •••..•.•........••.•••.•.........• 127
Table 24. Matrix Delineating Specific Group Differences by Post Hoc Analysis for the Parametric Model ....•••.....•• 128
Table 25. Covariance
covariate-Corrected Parametric Analysis of Table ........................................ 130
~~~~~~~---
Table 26. Observed and Adjusted Means for the Covariate-Corrected Parametric Analysis of Covariance •.••.•.•.••.. 131
Table 27. Rank Analysis of Covariance Table •....•.....•. 133
Table 28. Observed and Adjusted Means for the Rank Analysis of Covariance ........................................... 134
Table 29. Matrix Delineating Specific Group Differences by Post Hoc Analysis for the Rank Model •..••.•....••....... 135
Table 30. Covariate-Corrected Rank Analysis of Covariance Table ................................................... 137
Table 31. Observed and Adjusted Means for the Covariate-Corrected Rank Analysis of Covariance •••.....••..•..••.. 138
Table 32. Matrix Delineating Specific Group Differences by Post Hoc Analysis for the Covariate-Corrected Rank Model ........................................................ 139
Table 33. ANCOVA Table for the Parametric Model with
Table 34. Adjusted Dependent Measure Means for the Parametric ANCOVA with Outliers Deleted ..•••..•••.....•• 143
ix
Table 35. Matrix Delineating Specific Group Differences by Post Hoc Analysis for the Parametric ANCOVA with Outliers Deleted . ............................................... . 144
Table 36. ANCOVA Table for the Covariate-Corrected Parametric Model with Outliers Deleted •..••.....••••.... 146
Table 37. Adjusted Dependent Measure Means for the Covariate-Corrected Parametric ANCOVA with Outliers Deleted ................................................. 147
Table 38. Matrix Delineating Specific Group Differences by Post Hoc Analysis for the Covariate-Corrected Parametric ANCOVA with Outliers Deleted .•...•••••..••....••••••.... 148
Table 39. Summary of Population and Sample Probability Levels and Intervals for the Four ANCOVA Models .•••..•.. 164
Table 40. Analysis of the Assumptions Underlying the Parametric and Rank Analysis of Covariance •..••.•.••.••. 169
x
List of Figures
Figure 1. Scatterplots of the dependent and the covariate for the parametric ANCOVA model for the Follow Through and Bilingual groups ......................................... 96
Figure 2. Scatterplots of the dependent and the covariate for the parametric ANCOVA model for the groups with both programs and neither program .•.••...........•.......•.... 97
Figure 3. Scatterplots of the dependent and the covariate for the covariate-corrected parametric ANCOVA model for the Follow Through and Bilingual groups .•••...............•.. 98
Figure 4. Scatterplots of the dependent and the covariate for the covariate-corrected parametric ANCOVA model for the groups with both programs and neither program •...•.•..... 99
Figure 5. Scatterplots of the dependent and the covariate for the rank ANCOVA model for the Follow Through and Bilingual groups .. ..................................... . 100
Figure 6. Scatterplots of the dependent and the covariate for the rank ANCOVA model for the groups with both programs and neither program ..................................... 101
Figure 7. Scatterplots of the dependent and the covariate for the covariate-corrected rank ANCOVA model for the Follow Through and Bilingual groups •...•.....•.....•..•. 102
Figure 8. Scatterplots of the dependent and the covariate for the covariate-corrected rank ANCOVA model for the groups with both programs and neither program .•......••. 103
Figure 9. Boxplots of Y scores of program types (Follow Through, Bilingual, both programs, and neither program) according to population and ranked data •••...•••..•.••.. 106
Figure 10. Probability level intervals and true parent population parameters for the four ANCOVA models ........ 150
Figure 11. Frequency distribution of the sample probability levels for the parametric ANCOVA •.•.•...•.....••..•..... 152
Figure 12. Frequency distribution of the sample probability levels for the covariate-corrected parametric ANCOVA •.•• 153
Figure 13. Frequency distribution of the sample probability levels for the rank ANCOVA •..........•...•.•••••..•.•... 155
xi
Figure 14. Frequency distribution of the sample probability levels for the covariate-corrected rank ANCOVA •.•••...•. 156
xii
Chapter One
INTRODUCTION AND BACKGROUND OF THE STUDY
One of the basic purposes of education is to promote
desirable change or growth in the educational attainment of
students (Richards, 1975). The terms "education" and
"students" should not be associated solely with the
numerous school systems operating across our country and
world; they encompass an unlimited number of institutions
and organizations that use systematic and/or developmental
programs and processes with the intent of bringing about
change in one or more individuals. In addition to the
numerous types of educational institutions and associated
students, there are as many, if not more, content areas
that are presented and goals that are sought through the
educational processes that serve as catalysts for growth
and change. Fortune and Hutson (1984) explain several
types of educational change:
Trainers seek to change the skills and abilities
of their trainees through presentation and guided
practice. Teachers seek to change students'
knowledge and understanding of a given body of
content through instruction, classroom activities
and assignments, and interaction. Social reform
programs are directed toward change of certain
aspects of a selected target group or specific
1
2
institutional practices. Compensatory education
and remedial programs seek to induce change which
has previously failed to occur. Almost all
intervention programs attempt to
change in reading skills,
proficiency, in job skills,
induce change--
in language
in functional
literacy, in attitudes, etc. (p. 197)
An important part of any instructional process is
evaluation, and rightly so. Evaluation is crucial to
ensure that educational programs are effective and
beneficial both for the students who receive the
instruction and the educational institution that provides
the programs and disseminates information to be learned by
either direct interaction with the students or through
materials geared to individualization. While evaluation
should be ongoing throughout the entire instructional
process, that is not always the case. Program evaluators,
under the pressure of accountability, limited funding, or
federal mandates, lean on numerical representations that
supposedly reflect entire projected and/or actual student
growth or change.
"Intuitively, the measurement of change is simple. The
discrepancy between criterion measures ad.ministered before
and after program participation should represent the change
due to the program" (Fortune & Hutson, 1984, p. 197), but
this is true only in ideal conditions where perfectly
3
reliable criterion measures and pure experimental design
are used. Most typically, "perfect" and "pure" exist in
textbook examples where students are taught the "ins and
outs" of research design, methodology, and probability
statistics.
Pure experimental design has random assignment as its
basis; students are randomly assigned to different
treatment groups/learning situations or to a control group.
With randomization, there are less bias and measurement
error to confound the treatment effect and distort the
statistic chosen to explain mean differences between or
among groups. In addition, random assignment is one of the
basic assumptions underlying the validity of parametric
statistical tests (tests where the underlying parent
population of the random variables being measured is
assumed to be distributed normally.)
Although most researchers would like to incorporate
randomization into their designs, they are often prevented
from doing so by a variety of practical, ethical, and
political reasons. Thus, many studies turn out to be
quasi-experiments, with the possibility that treatment
groups are different from each other in important respects
(Bryk & Weisberg, 1977). For most quasi-experiments, both
sampling error and biased group selection contend as
reasonable hypotheses in explaining treatment effects,
instead of the actual treatment itself (Kenny, 1975).
4
One quasi-experimental design that is a common
alternative to a true experimental design is called the
nonequivalent control group design (Cook & Campbell, 1979).
"Pretest and posttest scores are obtained on a group of
subjects who were exposed to an intervention and on a
control group who were not exposed" (Bryk & Weisberg, 1977,
p. 950). According to Dicostanzo and Eichelberger (1980),
these two measures are then frequently compared by an
analysis of covariance (ANCOVA) to "compensate
statistically for the lack of experimental control" (p.
419). While the ANCOVA can be a very powerful statistical
device, it must be used carefully. Dicostanzo and
Eichelberger (1980) state:
Use and interpretation of the ANCOVA technique is
extremely complex, requiring that numerous
assumptions and conditions be met if meaningful
interpretations are to be applied to educational
settings. These assumptions are never precisely
met in an evaluation setting, so the extent of
the deviations and their impact on meaningful
interpretations must be assessed in the
evaluation. (p. 419)
Numerous other researchers have warned against the
inordinant use of the ANCOVA with the nonequivalent control
group design and its potential accompanying artifacts or
assumption violations (e.g., Aiken, 1981; Bryk & Weisberg,
5
1977; Campbell & Erlebacher, 1970; Cox & Mccullagh, 1982;
Elashoff, 1969; Evans & Anastasio, 1968; Fortune & Hutson,
1984; Glass, Peckham, & Sanders, 1972; Griffey, 1982;
Hendrix, Carter, & Scott, 1982; Karabinus, 1983; Kenny,
1975, 1979; Levy, 1980; Linn & Slinde, 1977; Lord, 1963,
1969; Locascio & Cordray, 1983; Magidson, 1977; Olejnik &
Algina, 1984; Olejnik & Porter, 1978; Overall & Woodward,
1977; Porter, 1967; Werts & Linn, 1969; Yow-Wu, 1984).
Fortune and Hutson (1984) have delineated over fifty
variations of five generic types of parametric statistical
models that have been devised to make estimates of change
when true experimental conditions do not exist. The models
differ in the ways they adjust or estimate data; "how they
address the different artifacts, statistical assumptions,
and measurement requirements" (p. 199); and how they relate
to four major purposes for the evaluation of change. The
authors do not say any one model or analysis is better than
another, or without widespread criticism. In addition to
presenting problems addressed by using a specific model,
they include potential problems created by its selection.
There are very few comparative data as to how these models
function and adjust, and while a comprehensive comparison
is far beyond the scope of this study, comparisons
including two of the five generic types of models and
different forms of the analysis of covariance have been
made.
6
While several of the models include or could include
some form of the analysis of covariance, this study
concentrates on the two model classification types that
include the basic parametric ~COVA and parametric ~COVAS
with a covariate correction. Analyses using both types of
parametric ~COVAS and their nonparametric counterparts
(Porter & Mcsweeney, 1974) have been completed on four
intact groups with the intent of determining population
parameters. Subsequently, a Monte Carlo simulation was
used to generate samples upon which ~COVAS were completed,
and sample parameters were compared to those of the
population.
Nonparametric tests require few assumptions about the
underlying populations from which the data are obtained
(Hollander & Wolfe, 1973). Glass et al. (1972) indicate
that the "assumptions of most mathematical models are
always false to a greater or lesser extent" (p. 237), so
because there are many assumptions that must be met for the
parametric analysis of covariance to be valid, a
nonparametric equivalent is a logical alternative. Daniel
(1978) believes that because nonparametric procedures
depend on a minimum of assumptions, there is small chance
of their being improperly used. In addition, for many
analyses, nonparametric statistical methods often involve
less computational work and are easier and quicker to apply
than other statistical methods (Conover, 1980). While
these advantages
analysis quite
7
of nonparametric
attractive, the
tests make that form of
use of either the
nonparametric ANCOVA or a nonparametric ANCOVA with a
covariate correction to study nonequivalent control groups
is not apparent in the literature.
Such a void in the literature is also apparent with
model selection for the analysis of covariance and the
nonequivalent control group by Monte Carlo simulation, a
computer-intensive technique, whereby one or more analyses
(like the four ANCOVA models in the current investigation)
are completed on numerous samples which are generated at
random with replacement from the same data set. An
interval based on a statistic of interest is then found to
estimate the accuracy of the statistic. A Monte Carlo
simulation can be applied to any parameter, and in the
current study, that parameter of interest is the
probability level associated with the F statistic for the
ANCOVA grouping factor. With nonequivalent control group
studies and model selection for the "delicate" analysis of
covariance (Elashoff, 1969, p. 383), a Monte Carlo
simulation should prove to be a valuable form of research
methodology.
"Robust regression procedures [are] designed to
dampen the effect of observations that would be highly
influential if least squares [regression] were used"
(Montgomery & Peck, 1982, p. 364). Highly influential
8
observations, called outliers, create heavy tails in a
distribution, and are different in magnitude from the bulk
of the data (Birch & Myers, 1982). They may affect
parameter estimates and cause least squares regression to
be inappropriate. "Robust regression techniques are a set
of iterative procedures which seek to find these outlying
points and minimize their influence on the parameter
estimates" (Number Cruncher Statistical System, 1986, p.
16.1). With each iteration, a weight is assigned; small
weights are given to outlier data points, and larger
weights are given to observations close to the regression
line. As a result, outliers in the data set can be
identified, deleted, ANCOVAS rerun, and comparisons made to
the original ANCOVA findings. Large differences in
probability levels in original and subsequent ANCOVAS
indicate that nonnormality has invalidated the findings of
the original analyses and impacted model selection.
While each of the statistical models that has been
used in this study has its own proponents and is efficient
and powerful under controlled laboratory settings and in
field studies when its own assumptions are met, no one form
of analysis works in all instances. Locascio and Cordray
(1983) write:
When it is not possible to accurately specify the
statistical and measurement model, performing
multiple analyses on the same data is possible--
9
when the results converge and different assump-
tions underlie the analytic strategies, more
faith can be placed in the conclusions. (p. 124)
As a result, performing multiple analyses of
covariance on population samples by Monte Carlo simulation
and making subsequent comparisons to population parameters
should serve as a theoretical basis for model selection and
an effective form of research methodology. Because
research studies using a Monte Carlo simulation for model
selection with the analysis of covariance and nonequivalent
control groups has not been reported in the literature,
results of the current investigation should provide a
valuable contribution to the field of behavioral science
research.
Statement of the Problem
There are many parametric statistical models that have
been designed to measure change in nonequivalent control
group studies, but because of assumption violations and
potential artifacts, there is no one form of analysis that
always appears to be appropriate. While the parametric
analysis of covariance and parametric ANCOVAS with a
covariate correction are some of the more frequently
completed analyses used to measure change in nonequivalent
control groups, comparative studies with _nonparametric
10
counterparts should be completed and results compared with
those more commonly used forms of analysis. As the means
of comparison, sample parameter intervals determined by
Monte Carlo simulation were examined in light of "true"
parent population parameters, population assumption
violations, reliability of the covariate correction, and
the size/stability of the sample probability intervals for
each ANCOVA model. As an additional means of model
specification, robust regressions were
determine if outlier-induced nonnormality
ANCOVA regression estimates.
Purpose of the Study
completed to
impacted the
The purpose of the current methodological study was to
examine alternative models of the analysis of covariance in
relation to the nonequivalent control group design, with
the intent of delineating the "best" method(s) of analysis.
Research Questions
While the study ultimately sought to answer five
specific research questions leading to a sixth focal
question, a number of general questions were addressed. The
specific research questions were:
1. How do the results of data exploration on the
11
population assist in delineating a preferred ANCOVA model?
2. Is the reliability correction on the covariate
strong enough for error reduction?
3. Which Monte Carlo sample model shows the tightest
probability interval?
4. Are population parameters contained in the
sample probability intervals?
5. How do the results of robust regression aid in
ANCOVA model specification?
6. In light of assumption violations, the reliability
of the covariate correction, probability interval size,
true parent population parameters, and the results of
robust regression, what is the "best" overall model for
this nonequivalent control group investigation?
General Questions
General questions based on population and sample
comparisons of the four ANCOVA models include:
1. With the parent population, how does the basic
parametric analysis of covariance compare to the parametric
analysis of covariance with a covariate correction?
2. With the parent population, how does the rank
transform analysis of covariance compare to the rank
transform analysis of covariance with a covariate
correction?
12
3. With the parent population, how do results of the
nonparametric rank transformation procedures compare with
those of the parametric ANCOVAS?
4. With samples generated by Monte Carlo simulation,
how do both the parametric and the nonparametric ANCOVAS
compare with each other and to the parent population?
Definitions of Terms
Analysis of Covariance: a "method of statistical
correction for initial differences between groups" (Overall
& Woodward, 1976, p. 588); test "used most often by
researchers to compare group means on a dependent variable,
after these group means have been adjusted for differences
between the groups on some relevant covariate (concomitant)
variable" (Huck, Cormier, and Bounds, 1974, p.134)
Analysis of Covariance with a Covariate Correction: an
analysis of covariance which includes a method of reducing
the measurement error in the pretest
Artifacts: "phenomena due to the combined effects of
measurement and design conditions" (Fortune & Hutson, 1984,
p. 197); examples include a negative correlation between
pretest scores
differential rate
and
of
gain
change
scores,
across
an unpredictable
groups, errors of
13
measurement, bias, or reduced or enhanced treatment effects
Assumptions: criteria that must be met for a statistical
test results to be valid
Asymptotic Relative Efficiency:
between two statistical tests
Attenuation: weakening
Bias: a form of measurement error
a measure of efficiency
Bootstrapping: a simulation method whereby data from a
sample are copied many times (perhaps a billion), the
copies are shuffled, and a number of subsamples are drawn;
subsequently, statistical tests are completed on each
subsample, and an interval based on the parameter of
interest is found to estimate the accuracy of the statistic
(Diaconis & Efron, 1983)
Concomitant Variable: the covariate or covariant
Covariance: "a measure of association or relationship
between two variables" (Kenny, 1979, p. 14)
Covariate: a pretest or a pre-measure used to adjust
14
treatment means of a dependent variable in the analysis of
covariance; covariate means are expected to be equal across
groups in a randomized experiment
Dependent Variable:
dependent variable
differences
a posttest or post-score; means of the
are analyzed to ascertain group
Distribution-Free: "implication that the underlying
distribution of the random variable from which the sample
was drawn is either unknown or unspecified" (Seaver, in
preparation, p. 2)
Fallible Variable: a variable containing errors of
measurement; any variable which has been measured with less
than perfect reliability (Porter, 1967); contains an
unobservable true part and unobservable error component
F Value: a test statistic that, with its corresponding
probability, is a measure of group differences; in addition
to the analysis of covariance, is a measure associated with
the analysis of variance and regression analysis
Homogeneity of Regression Slopes: the assumption (for the
parametric ANCOVA) that the within-group regression slopes
for each treatment group are parallel; that there is no
15
interaction between the slopes
Homogeneity of Variance:
parametric ANCOVA) that
groups
the
the asswnption (for the
variances are equal across
Level of Significance: the probability of a Type I error;
alpha
Linearity: the asswnption (for the parametric ANCOVA) that
the relationship between the covariate (pretest)and the
dependent variable (posttest) is linear; that ''an increase
of a specified nwnber of points on the covariate is related
to about the same increase on the dependent variable" (Huck
et al, 1974, p. 144)
Measurement Error: "the discrepancy between the obtained
reliability coefficient and a perfect reliability of 1.00"
(Cohen & Cohen, 1983, p. 68); examples include sampling
error, improper recording or coding, differences in
grouping, error in the system of measurement, invalid
conditions under which a treatment is given, fluctuations
in an individual's scores, outside influences of other
variables
Multiple Analysis Research Methodology: when it is nqt
16
apparent which test will provide the most accurate results,
the methodology of performing several different kinds of
statistical tests and then comparing the results to see if
they converge
Nonequivalent Control Groups: intact groups used as a base
of comparison; groups used in a quasi-experimental design
(one that does not use random assignment) to create the
comparisons (Cook & Campbell, 1979)
Nonlinearity: violation of the (parametric ANCOVA)
assumption of linearity
Nonparametric Test: a statistical method that satisfies
one of the following criteria:
A. data has a nominal scale of measurement
B. data has an ordinal scale of measurement
c. data has an interval or ratio scale of
measurement, and the distribution of the underlying
parent population of the random variable is either
unspecified or unknown
Normality: the assumption (for the parametric ANCOVA) that
the underlying parent population of the dependent variable
being measured is distributed normally; data points are
distributed pictorially like a bell curve
17
Parametric Test: a statistical method whereby the data
have an interval or ratio scale of measurement and the
underlying parent population of the random variable is
assumed to be distributed normally
Power: (1-the probability of committing a Type II error),
or (1-the probability of failing to reject a false null
hypothesis)
Nominal Power: the power level determined by the
researcher in the belief that all statistical
assumptions are met
Actual Power: the true probability of failing to
reject the null hypothesis (when a certain alternative
hypothesis is true) based on an understanding of
particular assumptions that are violated
Probability Interval: a sample interval under the central
part of the frequency distribution of the probabilities
that contains 68% of the sample probabilities
Probability Level: the significance level where a null
hypothesis is rejected
R2: the coefficient of determination; the percent of
variance of the dependent variable that is explained by the
independent variable(s)
18
Randomization: the "use of initial random assignment for
inferring treatment-caused change" (Cook & Campbell, 1979,
p. 6); an assumption necessary for the validity of
parametric statistical tests; the process of allowing
individuals an equal chance of group membership
Rank Transform Analysis of Covariance:
equivalent to the basic ANCOVA; the
the covariate are ranked and a
a nonparametric
dependent measure and
traditional ANCOVA is
completed on the ranks
Reliability: a measure of internal consistency in a
measure, such as a test score
Robust: the condition where a statistic is not affected by
an assumption violation
Robust Regression: a regression technique which, through
iteration procedures and weighting, is used to find a
regression equation that represents the majority of the
data, but is not greatly influenced by outliers
Spurious: false
True Score: the unobservable part of a variable that is
measured without error
19
TyPe I Error: rejecting a null hypothesis when it is true;
alpha; the level of significance
Nominal Probability of a Type I Error: the level of
significance defined by the researcher in the belief
that all assumptions are met; the nominal significance
level
Actual Probability of a Type I Error: the true
probability of a Type I error that is found in
relation to an understanding of particular assumptions
that are violated; the actual significance level
Type II Error: accepting (failing to reject) a null
hypothesis when it is false; beta
Limitations of the Study
Generalization of the results from the analyses of
covariance completed in the current investigation is
limited to the population on which the secondary analyses
were performed. The population includes fourth, fifth, and
sixth grade students enrolled in schools under study which
participated in Bilingual and/or Follow Through programs,
or in neither program. Whether or not
are found in the analyses is not the
methodological study, and the various
statistical techniques performed are
group differences
issue. This is a
models
the basis
used and
of the
20
investigation, not the individual results of each analysis.
Because the study is based on nonequivalent control groups,
which present inherent potential for error, results of the
analyses of covariance cannot be generalized to other
populations, even if that were the intent of the research.
In addition, methodological results of the study are
not generalizable to every other investigation based upon a
nonequivalent control group sampling plan. The purpose of
the research was to provide alternative models for analysis
and introduce an innovative statistical technique to an
area of research that is not completely defined. The
"best" statistical model for one study may not be the
"best" for another, but possibilities must be researched,
so that results of the alternatives best suited for the
sampling plan are
investigators who must
made available for
rely upon intact
answers to their research questions.
Significance of the Study
evaluators and
groups for the
The nonequivalent control group design is a commonly
used research and sampling design, which, because of the
lack of randomization, has the potential of introducing
bias and error into the measurement. While randomization
is preferable whenever and wherever possible, the
population of interest does not always lend itself to
21
random selection/random assignment or even accessible
selection/random assignment (Huitema, 1980).
Because the nonequivalent control group design brings
with it an element of risk that impacts the validity of an
associated statistic, it is crucial for researchers to have
adequate confidence that the statistical models and
procedures they use or would like to see completed provide
the most valid results possible to better enable them to
answer their research questions.
While there is no error-free procedure to measure pre
and post conditions in nonequivalent control groups, the
parametric ANCOVA with a covariate correction is typically
used by behavioral investigators to help reduce bias and
error in the covariate and ultimately provide a more
reliable test statistic. The reliability correction may be
a powerful statistical tool for error reduction, but it is
not the panacea if assumptions underlying the parametric
ANCOVA are not met. For this reason, it is crucial that
alternatives to parametric ANCOVAS be studied. If the
nonparametric ANCOVA is better suited to the study of
interest because it does not require as many assumptions,
it stands to reason that the rank transform ANCOVA with a
covariate correction may prove to be as good, if not even
better.
This study will investigate new and different models
and procedures in the area of nonequivalent control group
22
research: nonparametric analysis of covariance,
nonparametric ANCOVA with a covariate correction, the Monte
Carlo simulation for ANCOVA model selection, and robust
regression. A Monte Carlo simulation will be used to
generate random samples from four intact populations.
Intervals containing sampling statistics will then be
compared to parent population parameters. Although the
assumption of randomization cannot hold for the intact
groups, it will be met for the samples. A significant
contribution to the field of research and statistics is a
comparison of new models by new techniques. While there is
always the possibility that these models and procedures may
have been completed previously
groups, there is no apparent
literature.
with nonequivalent control
reporting of that in the
Chapter Two
REVIEW OF THE RELATED LITERATURE
This chapter will present a summary of literature
pertinent to the following topics: the analysis of
covariance, assumptions of the parametric ANCOVA, the
measurement of change, nonequivalent control groups, the
ANCOVA in nonequivalent control group research, and the
nonparametric ANCOVA.
The Analysis of Covariance
The analysis of covariance is a "statistical technique
based upon the general linear model, and, as such, can be
presented as an extension of either analysis of variance or
regression analysis, or of both" (Wildt & Ahtola, 1978, p.
5). The mathematical additive model appropriate to the
analysis of covariance is:
Yij =µ+a j + ~ (Xij - x .. ) + eij
Verbally, the model explains that the dependent value is
equal to a sum
population mean,
that includes four components: the
the treatment effect, the covariate
effect, and the error component.
The analysis of covariance is used to compare group
means of a dependent variable after they have been adjusted
for differences between the groups on some relevant
23
24
covariate variable (Huck et al., 1974). The covariate is a
pre-measure or pretest, and the dependent variable is a
post-measure or posttest. An adjusted mean is the mean
dependent score that would be expected or predicted for a
group of subjects if the covariate mean for the group were
the same as the grand covariate mean (Huitema, 1980). The
formula for the computation of the adjusted means is
included in Huitema's writings (1980):
Y· adj = Yj - bw ( X· - x .. ) J J
Verbally, this reads that the adjusted treatment mean for
the jth treatment group is equal to the unadjusted
treatment mean for the jth group minus the product of the
pooled within-group regression coefficient and the
difference between the covariate mean for the jth group and
the grand covariate mean. Abeyasekera (1984) writes that
this adjustment is recommended unless the covariate and the
dependent variable are very poorly correlated.
Random assignment in the analysis of covariance is
theoretically supposed to equalize groups, but the groups
never will have exactly the same covariate mean. If groups
have equal covariate means, no adjustment would be needed
on the post-measure (Huck et al. 1974), and an analysis of
variance would be the statistical procedure that should be
employed to test for group mean differences. With the
analysis of covariance, a covariate is included to account
for a part of the variance in the dependent measure
25
(Griffey, 1982). Griffey (1982) goes on to explain that
while the analysis of variance leaves this portion of
variance in the error term, the analysis of covariance
"handles the variance separately, reducing the error term
and increasing the size of the resulting F-ratio'' (p. 548).
There are four main applications of the analysis of
covariance that are listed by Cox and Mccullagh (1982).
The first is to increase precision in randomized
experiments. As mentioned previously, the analysis of
covariance removes some of the variance from the error term
and increases the precision of the measurement. Cochran
(1957) reports that this is the most frequent application
of the analysis of covariance.
Cox and Mccullagh (1982) cite that the second
principal use of the ANCOVA is to adjust for bias in
observational studies, or studies that are nonrandomized.
Two purposes include generalizing from the sample to the
whole population and ensuring that groups are comparable.
While there is no safeguard in the absence of randomization
(Cochran, 1957), the analysis of covariance is used to
adjust for bias in observational studies.
The third principle use of the analysis of covariance
is to adjust for missing values in balanced designs. For
this purpose, Cox and Mccullagh (1982) report that the
analysis of covariance is used purely as a numerical
device, whereby after missing values are replaced by any
26
convenient numbers, covariates are included as an
"indicator vector for each missing value" (p. 548).
The last use of the analysis of covariance that Cox
and Mccullagh (1982) cite is to adjust for historical
controls in clinical trials. A research study that is
similar to a current study, but was completed during a
different, earlier time period, is called a historical
control. The advantages of using historical, as opposed to
concurrent controls, include a reduction in both the number
of subjects that would be required for the new study and
administrative costs (Cox & Mccullagh, 1982). Among the
disadvantages are changes in definitions, standards, and
forms of measurement relating to the area of study; the
validity of the relationship between the two studies; and
additional unobserved concomitant variables.
Assumptions underlying the parametric analysis of
covariance will be discussed in the next section of the
literature review.
Assumptions of the Parametric ANCOVA
Assumptions are specifications in the data and design
that must be met in order for statistical tests to be
valid. In reference to a researcher's position regarding
assumptions, Baldwin, Medley, and MacDougall (1984)
indicate that "in field settings, researchers must rely on
27
statistical techniques that were developed for closely
controlled laboratory experiments. Often these statistical
techniques are sensitive to the violation of assumptions
that occur when applied in field settings to intact groups"
(p. 68). Like practically all other statistical tests, the
parametric analysis of covariance has its own set of
assumptions.
Huitema (1980) presents a thorough listing and
discussion of the assumptions of the parametric ANCOVA.
The eight assumptions he lists include:
1. Randomization.
2. Homogeneity of within-group regressions.
3. Statistical independence of the covariate and
treatment.
4. Fixed covariate values that are error free.
5. Linearity of within-group regressions.
6. Normality of conditional Y scores.
7. Homogeneity of variance of conditional Y scores.
8. Fixed treatment levels.
Specific information regarding each assumption will be
included in the following subsections.
Randomization
The subject of randomization as it pertains to
the current study will also be covered in the literature
28
subsections Nonequivalent Control Groups and The ANCOVA in
Nonequivalent Control Group Research.
The random assignment of subjects to treatments in the
parametric analysis of covariance is essential for a valid
interpretation of F tests and confidence intervals in
experimental studies. The assumption of randomization of
subjects is crucial because it equalizes the groups with
which comparisons are to be made. Overall and Woodward
(1977) write that "in the absence of random assignment,
groups are likely to differ, prior to administration of
experimental treatments" ( p. 588) . It of ten follows that
one group would score higher or lower on an outcome measure
of performance even prior to any experimental treatment
(Magidson, 1977). If there are differences between groups,
there is a statistical association between the treatments
and the covariate (Evans & Anastasio, 1968). Consequently,
comparisons of mean posttest performance for the groups may
reflect not only treatment effects, but also group
differences (Bryk & Weisberg, 1977).
Huitema (1980) also states that randomization is
important because if subjects are randomly assigned to
treatments, it is more likely that the error terms in the
ANCOVA model will be independent. While Huitema does not
list independent error terms as an assumption for the
ANCOVA, Wu (1984) does include it as an assumption, and
describes it as "the error components eij are normally
29
distributed with mean zero and equal error variance across
treatment groups and uncorrelated with each other" (p.
649). This assumption is basic for regression analysis,
and since the ANCOVA is based partially on regression
analysis, the assumption should be acknowledged. According
to Huitema (1980), "the basic issue (with independent error
terms] is deciding whether the subjects within treatment
groups are responding independently of each other. This is
important because dependence can have drastic effects on
the F test" (p. 100).
The assumption of randomization is often coupled with
another assumption which refers to error
Various covariate corrections have
compensate for these joint problems and
in the covariate.
been devised to
will be discussed
along with the assumption described in the subsection Fixed
Covariate Values that are Error Free.
When randomization is not possible, an analysis of
variance completed on the covariate indicates whether or
not the covariate means are equivalent. If the means are
statistically significantly equivalent, the researcher may
contend that even though the assumption of randomization
has not been met, the groups may be considered equal.
30
Homogeneity of Within-Group Regressions
This assumption has also been ref erred to as
homogeneous regression coefficients, the assumption of
common or parallel slopes (Huck et al., 1974). With this
assumption, "it is assumed that the regression slopes
associated with the various treatment groups are the same"
(Huitema, 1980, p. 102). One of the independent components
of the ANCOVA model is the regression coefficient
associated with the covariate. Because the regression
slopes for each group are assumed to be parallel (the lines
do not cross indicating an interaction), a pooled
regression slope representing the between-group regression
is determined. It is the regression coefficient of this
pooled slope that is a component of the ANCOVA model.
When heterogeneous slopes are present, the ANCOVA will
indicate smaller F values "because the
(the denominator of the F ratio) is an
mean square error
overestimate of the
population conditional variance" (Huitema, 1980, p. 105).
With a smaller F value, the chances of a Type I error and
incorrectly found inequality of group means are increased.
Studies (not based on nonequivalent control groups)
have indicated that the parametric ANCOVA is robust to
violations of this assumption provided the group sizes are
equal (Hamilton, 1976; Levy, 1980). Glass et al. (1972)
have reported that "if sample sizes differ, however, the
31
actual Type I error rate can either overestimate or
underestimate the nominal significance level depending on
the relationship between slope discrepancies and sample
size differences" (Olejnik & Algina, 1985, p.71).
Hendrix et al. (1982) indicate that a separate
intercept and slope are traditionally fitted for each group
when this assumption is not met. The procedures associated
with this technique are included in the writings of Baldwin
et al. (1984), but are beyond the scope of this research
study.
Statistical Independence of Covariate and Treatment
This assumption accompanies the assumption of
randomization. As explained earlier, if subjects are not
randomly assigned to treatment groups, but are studied as
intact groups, there is a statistical dependence between
the covariate and the group effect. Huitema (1980)
indicates:
Treatments may produce covariate means that, when
averaged, yield a grand covariate mean that has
no counterpart in reality . the ANCOVA could
be computed, but it is not clear why an
investigator would want to estimate and interpret
adjusted means that refer to levels of X that are
associated with treatments and subjects that are
32
nonexistent. (p. 108)
Another possible problem
covariate is affected by the
reliability of the covariate
that arises when the
treatment concerns the
(Huitema, 1980). Numerous
researchers have written about a covariate correction that
will compensate for this
Porter, 1967; Olejnik &
further discussion on
assumption.
lack of reliability (Lord, 1960;
Porter, 1978). There will be
this subject with the next
Fixed Covariate Values that are Error Free
This assumption indicates that the covariate must be
measured with perfect reliability. According to Porter
(1967), a variable which has been measured with less than
perfect reliability is a fallible variable. A fallible
variable will cause measurement error that will bias
results of the ANCOVA. Porter (1967) explains that a
fallible covariate variable is
an unobservable true part called
unobservable error component.
made up of two components,
the true score, and an
The regression coefficient in the ANCOVA is attenuated
by measurement error, which is caused by lack of perfect
reliability in the pretest. As a result, the regression
coefficient must be corrected for attenuation. Here arises
the rationale underlying the analysis of covariance with a
33
covariate correction. The True-Score ANCOVA (Porter,
1967), which is based on a reliability correction of the
covariate, is one of the most respected and frequently used
covariate-corrected ANCOVAS.
There are several forms of reliability measures.
Alternate-forms reliability, split-half reliability, test-
retest reliability, and the Kuder-Richardson are some of
the more familiar (Tuckman, 1978). When a reliability
measure is not available, Huitema (1980) proposes use of
rXY' the pretest-posttest within-group correlation, where
the pooled within-group sum of deviation crossproducts for
pretest and posttest is divided by the square root of the
product of the pooled within-group sum of squares on the
pretest and the pooled within-group sum of squares on the
posttest. The formula for the pretest-posttest within-
group correlation is:
rxy = Exyw/J(Ex2w)(Ey2w)
The reliability correction estimate is first
multiplied by the difference between each X value and the
group mean. These products are then added to the group mean
before they become the new values for the covariate. The
basic parametric analysis of covariance is then completed
as usual. The model for this correction by group is:
Xadj = Xj + rxy (Xi - Xj)
The degree of reliability is crucial for adequate
covariate correction. While Porter (1967) advocates a
reliability estimate
Eichelberger (1980)
Martin (1973), who
34
of .70
say .80
write of
or greater, Dicostanzo and
or greater, and Marks and
the importance of highly
reliable test forms,
employ reliabilities
correlation between
indicate the investigator should
in excess of .85, especially if the
the dependent variable and the
covariate is .70 or less.
Linearity of Within-Group Regressions
The assumption of linearity requires that the
relationship between the covariate and the dependent
variable be linear for each treatment group. The easiest,
though not the most scientific, method of testing for
linearity is to run scatterplots of X and Y values on the
computer. If data points appear to flow in a straight
line, the researcher may gain satisfaction from this
informal technique that there is a linear relationship
between the covariate and the dependent measure.
If linearity does not exist in ANCOVA, the "reduction
of the total and within-group sums of squares after
adjustment for X will be too small . • • and the utility of
the covariate will be diminished" (Huitema, 1980, p. 116).
If the covariate loses its credibility, the analysis of
variance would be the preferred method of analysis, and the
loss of precision that would have been gained by the
35
analysis of covariance would be lost.
Elashoff (1969) writes that the effect of nonlinearity
is most severe when random assignment is not possible or
when the Y values are not normally distributed. An
interaction of different assumptions can cause artifacts to
arise in the analysis of covariance.
Normality of Conditional Y Scores
Dependent scores conditional on covariate scores are
assumed to be normally distributed for each treatment group
(Huitema, 1980). According to Elashof f ( 1969), an
extension of the normality assumption for Y scores is that
the residuals (error terms) are normally distributed. On
the other hand, the covariate Xs are assumed to be error
free and fixed, so normality of X scores is not a necessary
assumption. Levy (1980) reinforced this assertion by
studying various forms of nonnormal covariate
distributions. He found ANCOVA to be robust to departures
from normality in the covariate.
Normality can be determined in several ways. Stem
-and-leaf plots, boxplots, and normal probability plots
provide visual illustrations of normality. The Shapiro-
Wilks (Conover, 1980), the Cramer von Mises (Stephens,
1974), and the Lilliefors (Conover, 1980) statistics each
provide a numerical representation.
36
There are a number of types of nonnormal
distributions. Conover and Iman (1982) studied (though not
with nonequivalent control groups) distributions of the
dependent variable that were lognormal, exponential,
uniform, and Cauchy. They found the parametric ANCOVA
provided appropriate Type 1 error rates for the exponential
and uniform distributions, but were not robust with
lognormal and Cauchy distributions.
When scores are not normally distributed because of
influential observations, or outliers, in the data set,
means can be distorted. There appears to be a void in the
literature in regard to the effect of outlier-induced
nonnormality on ANCOVA.
Homogeneity of Variance of Conditional Y Scores
This assumption states that the variance of the
conditional Y scores is the same for each treatment group
and that the variance of the conditional Y scores does not
depend on the covariate X (Huitema, 1980). Elashoff (1969)
writes about two types of violations for this assumption.
The first violation has been referred to as
heteroscedasticity. It refers to the overall variance
being the same for each group, but "the variance for Y
(conditional on individual X values) increases as X
increases" (Huitema, 1980, p. 118). The second violation
37
occurs when the within-group variances are different for
each treatment group, but are the same over different
levels of X.
Huitema (1980) indicates that ''conventional tests
of homogeneity of variance can be applied (with appropriate
modification of degrees of freedom) to dependent scores
conditional on the covariate'' (p. 118). While the F-Test
for homogeneity of variances is based upon normality of
distributions, other tests exist that are robust to the
normality violation (Games, Winkler, & Probert, 1972). In
addition, Elashoff (1969) reports that "inequality of
variance independent of X may be detected by comparing the
variances of the estimated residuals across treatments"
(p.395).
Olejnik and Algina (1985) have reported that the
parametric ANCOVA is robust
within-group variances for
sizes. In another study by
to heterogeneous conditional
both equal and unequal sample
Olejnik and Algina (1984),
results indicated that the parametric ANCOVA was robust to
violations of homoscedasticity unless that assumption and
the assumption of normality were both violated. It is
important to remember that neither of these studies was
based upon nonequivalent control groups.
On the other hand, Huitema (1980) summarizes a
potential adverse effect of this assumption violation by
saying:
38
When variance sizes and sample sizes differ, the
F is conservative if the larger variances are
associated with the larger sample sizes and the
smaller variances are associated with the smaller
sample sizes. When the smaller variances are
associated with the larger sample sizes, the bias
is liberal. (p. 121)
Fixed Treatment Levels
For fixed treatment levels, it is assumed that:
The treatment levels included in the experiment
are not selected by randomly sampling the
population of possible treatment levels . . • the
levels selected are the specific levels of
interest to the experimenter, and the
generalization of the results of the experiment
is with respect to these levels. (Huitema, 1980,
p. 121)
If treatment levels are selected randomly and the
design has more that one factor, computation associated
with mixed and random effects models must be completed
(Huitema, 1980).
39
The Measurement of Change
In a very comprehensive review of model selection for
the measurement of change in nonequivalent control groups,
Fortune and Hutson (1984), discuss three primary conditions
that directly or jointly produce artifacts. These
conditions include "1) the effects of outside influences
which change the base of comparison; 2) a priori
differences in the groups being compared; and 3) the
fallibility of the measurement" (p. 197). These conditions
are results of two familiar parametric ANCOVA assumption
violations that were outlined in a previous subsection of
the literature review, Randomization and Fixed Covariate
Values that are Error Free. The authors (1984) indicate
that these conditions accompany the nonequivalent control
group research base, and explain that a number of artifacts
may occur:
1. outside influences may interact with the treatment
effect and "reduce or enhance its visibility" (p. 198).
2. A priori group differences may "result in an
observable differential rate of change of the criterion
measure across groups" (p. 198).
3. The absence of a perfectly reliable covariate
measure "results in the negative correlation between
pretest scores and raw gain score ratios" (p. 198).
4. The interaction of outside influences and a priori
40
group differences may result in an "unpredictable
differential rate of change across groups" (p. 198) due to
correlation differences between the dependent measure and
the covariate across groups.
5. A priori group differences and fallibility of
measurement may interact and inf late or deflate treatment
effects.
6. The interaction of outside influences and
fallibility of measurement may result in "differential
errors of measurement within groups across testing periods
confounding the effects of random error on group
differences" (p. 198).
The conditions and artifacts listed above are
potential "evils" that are very likely to occur in
nonequivalent control group research. For additional
specific information
the subsections of
on nonequivalent control groups, see
the literature review entitled
~N~o~n~e_g_u~i~v~a~l~e~n~t~~C~o~n_t~r~o~l~~-G_r~o_u_p_s and The AN COVA in
Noneguivalent Control Group Research.
Fortune and Hutson (1984) identify over fifty
variations of five generic types of parametric statistical
models that have been devised to make estimates of change
by adjusting or estimating data in some way. The five
classification types of models include:
1. Adjusted gain score models
2. True score models
41
3. Group equating models
4. Growth analysis
5. Structural equations models (p. 200)
One or more of the five different classification types
of models address four purposes for the evaluation of
change that Fortune and Hutson (1984) also delineate. The
purposes and corresponding classification types include:
1. "The identification of individuals who achieved
specific levels of change, such as high-gainers or low-or
no-gainers" (p. 200).
Adjusted gain score models address this purpose.
2. The identification of factors associated with
high gain.
"Models which do not alter the covariant structure of
the data" (p. 200) are most appropriate for addressing che
second purpose. They include the adjusted gain score
models and true score models.
3. "Estimation of the magnitude of change" (p. 200).
Group equating, growth analysis, and structural
equations models address the third purpose.
4. "Comparison of the amount of change across groups
or ascertainment of relative group change" (p.200).
Structural equations models are recommended for the
fourth purpose in order to obtain precise comparisons of
gain.
Table 1 presents a delineation of the four purposes
for the evaluation
classification types.
42
of change and corresponding
43
Table 1
Purposes for the Evaluation of Change and Corresponding
Model Classification l'ypes, as Reported by Fortune & Hutson
(1984)
Model Types
Adjusted Gain Score Models
True Score Models
Group Equating Models
Growth Analysis
Structural Equations Models
Identify Subjects Achieving Specific Levels of Change
x
Identify Factors Associated with High Gain
x
x
Purposes
Estimate Magnitude of Change
x
x
x
Compare the Amount of Change Across Groups
x
44
The first three of the classification types,
especially the first and
form of the analysis
second categories, include some
of covariance. While the second
classification type, the true score models, relates
specifically to one of the procedures proposed in this
study and was discussed in the subsection of the literature
under the assumption Fixed Covariate Values that are Error
Free, some attention must also be given to classification
one, adjusted gain score models. Further information
regarding the other three classifications is found in the
article by Fortune and Hutson (1984).
Cronbach and Furby (1970) describe gain scores as
" 'raw change' or 'raw gain' scores" that are "formed by
subtracting pretest scores from posttest scores'' (p. 68).
:~ose researchers' opinion that gain scores "lead to
i~lacious conclusions, primarily because such scores are
systematically related to any random error of measurement"
(p. 68), is shared by others (Marks & Martin, 1973; Fortune
& Hutson, 1984; Campbell & Erlebacher, 1970; Baldwin et
al., 1984; Werts & Linn, 1970; Linn & Slinde, 1977; Kenny,
1975). Campbell and Erlebacher (1970) even go so far as to
compare gain scores to a "treacherous quicksand" (p. 197)
that should not be recommended for any purpose.
One method of analysis of covariance modeling is based
upon using the difference between the posttest and the
pretest as the dependent measure. This difference score as
45
the dependent measure is subject to the same fallibility as
the basic gain score analysis used to detect differences in
change. An example of basic gain score analysis is the
one-sample paired t-test, which indicates if gain is
significantly different from either zero or another value
determined by the investigator.
Further discussion of gain score analysis in the
literature will be viewed in light of its association with
the analysis of covariance model described in the above
paragraph.
Linn and Slinde (1977) indicate that gain scores, or
difference scores, have several major defects, which
include:
1. The difference score will typically have a
negative correlation with the pretest, and "the magnitude
of the correlation will usually be small in absolute value"
(p. 122). If the correlation between the dependent
variable (in this case, the difference score) and the
covariate is small, it makes little sense to use the
analysis of covariance (Abeyasekera, 1984). Lack of
perfect correlation and the concept of regression toward
the mean are essentially the same (Nesselrode, Stigler, &
Baltes, 1980).
Regression toward the mean is the phenomenon where
large positive difference scores are more likely to be
observed for subjects with low scores on the pre-measure
46
(covariate), and low difference scores are more likely to
be observed for subjects with high scores on the covariate
measure (Linn & Slinde, 1977). Nesselrode et al. (1980)
indicate that, "in terms of the second occasion, the
extreme groups are 'moving' closer to the overall mean" (p.
625) •
2. The second major defect of gain scores is low
reliability. When the correlation between the pre-measure
and the post-measure is at all large, the reliability of
the difference scores is "discouragingly low" (Linn &
Slinde, 1977, p. 123). One way to obtain a high
reliability for a difference score is to have a low
correlation between the pre-measure and the post-measure,
but, here again, results of the analysis of covariance
would be meaningless. Linn and Slinde (1977) elaborate:
An implication of the low reliability of
difference scores is that it is quite risky to
make any important decisions about individuals on
the basis of gain from pre- to posttesting
periods. Even without any real change, it is
possible to find substantial numbers of
individuals with large difference scores due
simply to the low reliability of these scores.
(p. 124)
3. The third major defect of difference scores is
that they have a different measurement scale from the
47
covariate, and the covariate and the dependent measure must
be based on the same scale of measurement.
Problems inherent with difference scores have led a
number of people to seek alternatives, one of which is the
residual score (Linn & Slinde, 1977). Cronbach and Furby
(1970) write that a residual score singles "out individuals
who have gained more (or less) than expected" (p. 74). An
advantage of the residual score over the difference score
is that the "residuals do not give an advantage to persons
with certain values of the pretest scores whereas
difference scores do'' (Linn & Slinde, 1977, p. 125).
Nevertheless, the residual scores, just like the difference
scores, have a low reliability.
Because of the major defects associated with
difference scores, the parametric and nonparametric ANCOVAS
in the current study will include a post-measure as a
dependent measure.
Nonequivalent Control Groups
The experimental design of any research experiment
must include a sampling procedure that specifies the
population to which the results can be generalized
(Huitema, 1980). Cook and Campbell (1979) refer to this
ability to generalize as external validity, and in addition
to population, specify that settings and time periods can
48
also be considered in terms of generalizability. Huitema
(1980) lists three general types of sampling selection
procedures and their corresponding generalizabilities:
1. The first type of sampling selection is
established by random selection and random assignment. In
this procedure, subjects from a defined population are
randomly assigned to different treatments or to a control
group. Results can be generalized to the population from
which the subjects were chosen.
2. The second type of sampling selection procedure is
based upon accessible selection and random assignment. In
this case, subjects are not randomly selected from a
defined population, but are simply accessible to the
experimenter who randomly assigns them to treatment
conditions and applies a statistical procedure. For t:b..i.s
type of sampling, the "experimenter can only state that L1e
results can be generalized to a population of subjects who
have characteristics like those who were included in the
sample" (p. 8). Huitema (1980) goes on to say that the
second design type is "the rule rather than the exception
in most behavioral experiments" (p. 9), as opposed to the
superior, though impractical, first type of sampling design
which is based upon statistical inference.
3. The third type of sampling selection procedure
consists of assignment of treatments to intact groups
(quasi-experiment). There is neither random assignment nor
49
random selection to treatment conditions, and ''the use of
conventional statistical procedures is generally
questionable" (p. 9). As with the second type of sampling
design, the results cannot be generalized to the entire
population, but only to subjects having characteristics
similar to those in the study.
Included in Cook and Campbell's (1979) list of threats
to external validity is "interaction of selection and
treatment". When the subject-to-treatment selection
process is based upon intact groups, the selection process
can interact with the treatment to cause artifacts that not
only include measurement error that will bias the results,
but will limit generalizability.
The nonequivalent control group design is one type of
quasi-experiment where measurements
and a comparison group are completed
treatment (Cook & Campbell, 1979).
of a treatment group
before and after a
The analysis of
covariance is often associated with nonequivalent control
group designs. The ANCOVA is used to indicate differences
between or among group means on the basis of a covariate (a
pre-measure), and then to compare these adjusted means to
see if they are different (Huck et al., 1974).
50
The ANCOVA in Nonequivalent Control Group Research
From the standpoint of the literature, the use of the
basic analysis of covariance in nonequivalent control group
research or even the nonequivalent control group design
itself is controversial. Philosophies on one side indicate
that in behavioral experimentation, there is no substitute
for randomization, and that it is unfortunate that this
rule is so often ignored, for if investigators were willing
to exert themselves, no substitute would be needed (Bock,
1985). Some researchers are not willing to risk the
potential problems associated with using the analysis of
covariance without randomization. Lord (1967) even went so
far as to say that there simply is no logical or
statistical procedure that can be used to make proper
allowances for preexisting group differences. In reference
to the basic parametric ANCOVA, Campbell and Erlebacher
(1970) indicate:
On using analysis of covariance to correct for
pretreatment differences the texts that treat the
issue are either wrong or noncommittal (that is,
fail to specify the direction of bias) and
probably 99% of experts who know of the procedure
would make the error of recommending it.
(p. 204)
Aiken (1981) writes that ''the nature of the variables
51
with which ANCOVA is employed often leads to error in
interpreting research results" (p. 13). In addition, such
artifacts as bias, sampling error, a spurious linear
correlation between the covariate and the treatment,
undetected differences among groups prior to the treatment,
measurement error, low precision from extrapolation due to
group differences, measurement error, an unpredictable
differential rate of change across groups, reduced or
enhanced treatment effects, and absurd conclusions, may be
enough of a potential hazard to scare off any number of
"pure" statisticians.
Other researchers acknowledge the greater strength and
precision that come with randomization, but acknowledge it
is not always possible. Campbell and Erlebacher (1970)
indicate:
Even though "true" experiments in the field
setting are . more "quasi" than those in the
laboratory, (and those in the laboratory more
"quasi" than published reports and statistical
treatments indicate), experiments with randomized
assignment to treatments are greatly to be
preferred where possible. We believe that any
investigator fully attending to the presumptions
he is making in using quasi-experimental designs
will prefer the random assignment of [subjects]
to treatments where this is possible. (p. 205)
52
Numerous researchers write about the many situations
where intact groups appear to be the only reasonable choice
in attempting to answer necessary questions regarding group
differences relating to change or treatment effects.
According to Linn and Slinde (1977), "random assignment is
seemingly impossible in many situations where answers to
questions about treatment effects are sought" (p. 132).
Fortune and Hutson (1984) indicate that, "program
evaluation usually does not allow the control or random
assignment required in experimental studies" (p. 198).
Karabinus (1983) adds that, "seldom is random selection of
children or random assignment of treatment groups feasible"
(p. 841). Baldwin et al. (1984), also, say that "comparing
non-equivalent groups is a persistent problem in
educational research methodology, especially teacher
effectiveness research'' (p. 68). While these problems may
seem unique to educational investigations, Marascuilo and
Dagenois (1979) write that "researchers and evaluators
might find some consolation in learning that the problem
also exists in the studies of public health, epidemiology,
demography, medicine, and related disciplines" (p. 49).
It appears that nonequivalent control group research
is here to stay and must be viewed with an openness to its
potential value. According to Kenny (1975):
Because the internal validity of quasi-
experiments is lower than true experiments, it
53
does not argue against using the judgments of
quasi-experiments. We would all prefer to have
the testimony about an event from a sighted man
over a blind man. But when we have only the
blind man, we would not dismiss his testimony,
especially if he were aware of his biases and had
developed faculties of touch and hearing that the
sighted man could have developed but has
neglected. The difference between the true
experiment and the quasi-experiment is of the
magnitude of the difference between sight and
blindness. We must of ten grope in the darkness
with quasi-experimental designs, but this
blindness both forces us to compensate for biases
and helps us develop a newfound sensitivity to
the structure of the data. (p. 360)
Because the parametric analysis of covariance is one
of the most viable options to measure change in
nonequivalent control group studies, measures must be taken
to determine how the ANCOVA can be used most effectively
despite assumption and design limitations. It must be
acknowledged that random assignment is only one of the
assumptions that must be met for parametric ANCOVA results
to be valid, and randomization is not the panacea for every
potential problem. Linn and Slinde (1977) write that,
"another assumption of the analysis of covariance is that
54
the covariate is measured without error . even with
random assignment, errors of measurement limit the value of
traditional analyses of covariance" (p. 132). In addition,
Evans and Anastasio (1968) indicate that the analysis of
covariance is inappropriate when there is
correlation between the independent variable
factor) and the covariate.
a strong
(grouping
When the assumptions underlying an analysis are
violated, Urquhart (1982) cites three alternatives: "do
nothing; do the familiar analysis, but exercise great care
in interpretation; or modify the familiar analysis to
accommodate the changed assumptions, but preserve the
purpose of the analysis" (p. 651). The following section
will present another alternative to the parametric analysis
of covariance.
The Nonparametric ANCOVA
In any discussion of nonparametric statistics, it is
important to first explain the relationship of the terms
"nonparametric" and "distribution-free". Seaver (in
preparation) writes:
Strictly speaking, the term "nonparametric"
implies an estimation or inference statement that
is not directly concerned with parameters, while
"distribution-free" implies that the underlying
55
distribution
the sample
of the random variable from which
was drawn is either unknown or
unspecified (i.e., all the parameters that
determine the distribution are unknown). (p. 2)
"Nonparametric" and "distribution-free" are terms that
are generally used interchangeably (Mcsweeney & Katz, 1978;
Daniel, 1980; Seaver, in preparation; Royeen, 1986), and
while it is acknowledged here that the two terms are not
synonymous, they will also be used interchangeably in this
investigation. In addition, because the nonparametric
analysis of covariance is based upon ranks, the terminology
"rank analysis of covariance" or "rank transform ANCOVA"
may also be substituted for the more generic terms
"nonparametric" and "distribution-free".
The hypotheses of parametric and nonparametric
statistics differ. When the basic parametric ANCOVA
assumptions are met, both the parametric and nonparametric
ANCOVA test the identical hypothesis of location (Olejnik &
Algina, 1985). Huitema (1980) explains that in the case of
serious assumption
procedures do not test
testing the equality of
tests the hypothesis
violations, the two
the same hypothesis.
adjusted means, the
that the conditional
statistical
"Rather than
rank ANCOVA
population
distributions, which are of unspecified form, are
identical" (p. 256). Huitema (1980) suggests that both
parametric and nonparametric ANCOVAS be performed
simultaneously when
56
there are serious
violations, because even though the parametric
biased, the adjusted means will not.
assumption
F will be
Huitema (1980) describes two specific situations where
rank analysis of covariance should be considered, the first
being when the data is based upon an ordinal (rank) form of
measurement. Because the parametric ANCOVA is suitable for
interval or ratio data, it cannot accurately be used to
measure group differences based on ranks. Daniel (1980)
adds that nonparametric procedures may be used with nominal
(count) data, and that parametric procedures are
inappropriate when the variables of interest are measured
on a nominal scale.
The second situation described as being appropriate
for the rank ANCOVA is one in which the variables of
interest are measured on an interval or ratio scale, but
are distributed in such a way that some of the basic
parametric assumptions are violated (Huitema, 1980). While
the parametric ANCOVA is robust to the violation of the
assumptions of normality and homogeneity of conditional
variances when sample sizes are equal (NOT allowing for
nonequivalent control groups), with unequal sample sizes
and severe parametric ANCOVA assumption violations, the
rank ANCOVA may provide a more accurate measure.
There are certain assumptions that must be met for the
nonparametric ANCOVA to be valid. Olejnik and Algina
57
(1985) indicate:
The distribution-free procedures considered . . .
are less restrictive than parametric analysis of
covariance in that conditional normality, a
linear relationship, and homogeneity of
conditional variance within and between groups
are not assumed. With respect to the linearity
assumption, the nonparametric procedures require
only a monotonic increasing function relating the
pretest and the posttest.
(p. 52)
While assumptions for the rank ANCOVA may be less
restrictive and less frequently emphasized than those
underlying the parametric ANCOVA, assumptions exist, and
they must be considered. In addition to the relationship
between the covariate and the dependent variable being
monotonically related, it is necessary for the degree of
monotonicity to be the same for each treatment population
(Huitema, 1980). The population Spearman correlation
coefficient can be used by applying a homogeneity of
regression slopes test on the ranked Xs and Ys (Huitema,
1980). Another assumption for the ranked ANCOVA is that
the covariate and dependent variable be measured on at
least an ordinal or dichotomous scale of measurement.
Huitema (1980) includes an additional assumption which
requires that the marginal distributions of the covariate
58
be identical. While the Kruskal-Wallis test run on the
covariate will test this assumption, it is not necessary to
do so if random assignment has been employed. The last
assumption is, perhaps, the most noteworthy in relation to
the current investigation. A basic assumption for the rank
ANCOVA is that the "subjects are randomly and independently
assigned to treatment groups" (Huitema, 1980, p. 266).
Because the nonequivalent control group sampling design
obviously negates the possibility for this final assumption
to be met, there is even more reason to want to study
alternatives for its analysis.
A summary of the assumptions for the parametric and
nonparametric ANCOVA is delineated in Table 2.
59
Table 2
Assumptions Underlying the Parametric and Nonparametric
Analysis of Covariance
Model
Assumptions Parametric Nonparametric
Randomization x x Parallel Slopes X
Independence of Covariate and X Treatment
Error Free Fixed Covariate Values X
Linearity between the Covariate X and the Dependent Measure
Monotonic Relation between the X covariate and the Dependent Measure
Equal Degree of Monotonicity for X Each Population
Normality of Conditional Y Scores X (and Error Terms)
Homogeneity of Variance of X Conditional Y Scores
Fixed Treatment Levels X
Covariate and Dependent Measured X on at least an Ordinal or Dichotomous Scale
Covariate and Dependent Measured X on at least an Interval Scale
Identical Marginal Distributions X of the Covariate
60
Five variations of the nonparametric ANCOVA have been
reported by Olejnik and Algina (1985). The first kind of
nonparametric ANCOVA was first presented by Quade (1967).
Quade's distribution-free test "assumes that the
relationship between the pretest and the posttest is the
same across all groups and does not provide a procedure to
test it" (Olejnik & Algina, 1985, p. 57). This
distribution-free ANCOVA uses residuals as the dependent
measure, and even though the test appears to have Type 1
error and power rates similar to other reputable forms of
rank analysis of covariance (Olejnik & Algina, 1985;
Lawson, 1983), it will not be used in this study because of
the high potential for low reliability of residual scores.
A second form of nonparametric ANCOVA is that of Puri
and Sen (1969). While Puri and Sen's solution is based on
the chi-square distribution, Hamilton (1976) reported that
goodness-of-fit tests indicated that it was not consistent
with the chi-square distribution for small sample sizes.
Burnett and Barr's (1977) nonparametric ANCOVA is the
third type of analysis reported by Olejnik and Algina
(1985). This nonparametric analogy of the ANCOVA uses
difference scores as the dependent measure. Because of the
major defects inherent in difference scores, Burnett and
Barr's (1977) rank ANCOVA is not the form of analysis
chosen for the current investigation.
A fourth form of nonparametric ANCOVA is that
61
presented by Shirley (1981). Shirley's distribution-free
method is a general linear model solution which is based on
"the ratio of the adjusted sum of squares for the grouping
factor and the total mean square error for the unadjusted
posttest ranks'' (Olejnik & Algina, 1985, p. 63). Those
researchers (1985) found that Shirley's method is a
conservative test that is not a "reasonable alternative to
the parametric ANCOVA" (p. 68).
Mcsweeney and Porter's (1971) rank transform ANCOVA,
the fifth form of nonparametric ANCOVA, ranks the covariate
and the dependent measure across groups. Once the X and Y
values have been ranked, the basic analysis of covariance
procedure is completed on the ranks. After the analysis is
completed, it is possible to run a secondary analysis to
test for parallel slopes, a process not available for
Quade's, Puri and Sen's, and Shirley's distribution-free
tests. Mcsweeney and Porter's rank transform ANCOVA will
be used in the current study in its basic form and with a
covariate correction.
Monte Carlo studies have been completed (though not
with nonequivalent control groups) to compare the
parametric and nonparametric ANCOVA on various assumption
violations, singly and relative to other violated
assumptions. The two forms of ANCOVA have been compared
in terms of both the number of Type I errors made and the
statistical power.
62
A Type I error is committed when an investigator
rejects a null hypothesis when it is true. On the basis of
the ANCOVA, a Type I error means that equality of adjusted
means exists, but statistical results erroneously indicate
inequality. In simulation studies, nominal and actual
probabilities of a Type I error are compared. The nominal
probability of a Type I error is the level of significance
defined by the researcher in the belief that all
assumptions are met. The actual probability of a Type I
error is the true probability of a Type I error that is
found in relation to an understanding of particular
assumptions that are violated (Glass et al, 1972).
Similarly, statistical power of the parametric and
nonparametric ANCOVA have been compared through Monte Carlo
studies. Power relates to the probability of committing a
Type II error (failing to reject a null hypothesis, when in
fact it is false). As with probabilities of a Type I
error, there are nominal and actual power levels. A
nominal power level is one determined by the researcher in
the belief that all statistical assumptions are met. An
actual power level is the true probability level associated
with failing to reject a false null hypothesis based on an
understanding of particular assumptions that are violated
(Glass et al., 1972).
Simulation studies have been completed that compared
the parametric ANCOVA and the rank transform ANCOVA in
63
reference to parametric assumption violations, but none of
these studies included nonequivalent control groups.
Nevertheless, results of these simulations will be
reported.
Using the basic parametric and nonparametric ANCOVAS,
including the rank transform ANCOVA, Hamilton (1976) tested
the parametric ANCOVA assumption of homogeneity of within-
group regressions. Hamilton indicates:
Parametric ANCOVA maintained larger empirical
power for nearly all of the data situations.
Both parametric and nonparametric techniques
appeared not to be robust when violation of the
parametric assumption of equal slopes was coupled
with unequal group sizes and distributions were
normal. (p. 864)
Hamilton (1976) also reports that the parametric
ANCOVA was more powerful than the nonparametric
alternatives, even when the assumption of parallel slopes
was violated, except for small equal samples (of ten).
on the other hand, Conover and Iman (1982) found that,
with normal and non-normal distributions, the rank
transform ANCOVA was more powerful than the parametric
ANCOVA. Nevertheless, there was some variation in power.
While the rank ANCOVA appeared to be more robust and
powerful with skewed exponential and lognormal
distributions, the parametric ANCOVA was more robust and
64
powerful with both normal distributions and uniform
distributions without tails.
Olejnik and Algina (1984) studied the parametric
ANCOVA and the rank transform ANCOVA with non-normal data
and heteroscedasticity and found a power advantage for the
rank ANCOVA. Their results indicate:
The rank ANCOVA tended to be more powerful than
the parametric ANCOVA for both strengths of
relationship, all sample sizes, and all non-
normal conditional distributions (p. 146) .•.
the rank ANCOVA usually has Type I error rates
near the nominal level, is usually more powerful
than the parametric ANCOVA, and can be
substantially more powerful. (p. 147)
Seaman, Algina, and Olejnik (1985) found somewhat
different results in their study of the rank and parametric
ANCOVAS with distributions differing in skew and/or scale.
Under only a limited number of conditions involving
"distributions that were moderately non-normal
having a larger conditional variance" (p. 366), the rank
transform ANCOVA was shown to be the pref erred procedure in
regard to Type I error probabilities and power, but
otherwise, "the parametric ANCOVA was typically the
procedure of choice both as a test" (p. 345) of equal means
and distributions.
Porter and Mcsweeney (1974) studied the robustness of
65
parametric and rank ANCOVAS in regard to nonlinearity.
Their results indicate that, while Type I error rates for
both forms of analysis were not influenced by nonlinearity
between the covariate and the dependent measure, there were
considerable power advantages for the rank transform
ANCOVA.
As stated previously, none of the aforementioned
studies were completed with a nonequivalent control group
sampling plan. Nevertheless, results of the robustness and
power studies need to be considered in light of parametric
assumption violations and differing forms of analysis. A
summary of the simulation studies reported is included in
Table 3.
66
Table 3
Simulation Studies Comparing Power and Robustness of the
Parametric ANCOVA and the Rank Transform ANCOVA in Relation
to Parametric Assumption Violations
Assumption
Parallel Slopes (Hamilton, 1976)
Linearity (Porter & Mcsweeney, 1974)
Normality (Conover & Iman, 1982)
Normality and Unequal Variances (Olejnik & Algina, 1984)
Normality (Seaman, Algina, & Olejnik, 1985)
Comparisons
Power
Parametric ANCOVA more powerful in most instances
Rank Transform more powerful with small samples (n=lO)
Rank Transform had considerable power advantage
Robustness
Neither model robust with unequal sample sizes and normality
No difference in robustness
Rank Transform more powerful and robust with skewed exponential and lognormal distributions
Parametric more powerful and robust with normal and uniform distributions
Rank Transform more powerful and robust
Parametric more powerful and in most instances
Rank Transform better with distributions that were moderately non-normal and had a larger conditional variance
67
Chapter Summary
Chapter two has included a review of the parametric
analysis of covariance, assumptions of the parametric
ANCOVA, the measurement of change, nonequivalent control
groups, the ANCOVA in nonequivalent control group research,
and the nonparametric ANCOVA. The next chapter will focus
upon the methodology and procedures of the current
investigation.
Chapter Three
METHODS AND PROCEDURES
Overview
The current investigation is a methodological study.
It is based upon the rationale underlying the selection of
the various ANCOVA models and the Monte Carlo simulation
sampling technique, the logically-defined sequence of both
initial data exploration and appropriate analyses, and the
application of methodologically-derived results in response
to research questions. Whether or not group differences
are found is not the issue in the current study.
Results of the current investigation provide a
systematic and theoretical framework of alternative models
that are available for researchers who choose to use the
analysis of covariance, but must work with intact groups of
subjects. Sampling by Monte Carlo simulation was
introduced and serves as a procedure that will assist in
determining model comparison and accuracy.
While nonequivalent control groups provide the basis
of the study, a population consisting of four groups, at
least two of which were predetermined to be different
(nonequivalent), were used to provide the true parameters
to which the Monte Carlo sample findings were compared.
Thus, interpretation of the final results of the study were
68
69
made based on rationale, selection, and application of the
analysis of covariance and accompanying procedures in
relation to known population parameters.
Chapter Three specifically describes the methods and
procedures of the current investigation in light of the
study population, data identification, research design,
analytical procedures, and implementation guidelines.
Study Population
The current research investigation included analyses
on data obtained from the Acuma Indian Tribe of New Mexico.
Subjects included 287 fourth, fifth, and sixth grade
students from six schools. While two of the schools were
involved with a Bilingual education program, two others
were participants in Follow Through, a program based upon
preserving Indian culture. A fifth school included both
Bilingual education and Follow Through as part of its
curriculum, and the sixth school did not participate in
either program. The study population is delineated on
Table 4.
70
Table 4
Sample Size for the Study Population
School
School 1
School 2
Total
School 3
School 4
Total
School 5
School 6
TOTALS
Fourth Fifth
Bilingual
11 9
13 12
24 21
Follow Through
34 35
15 8
49 43
Both Programs
9 7
Neither Program
21 18
103 89
Grade Levels
Sixth
8
16
24
32
13
45
8
18
95
TOTAL
28
41
69
101
36
137
24
57
287
71
Data Identification
Data studied in the current investigation are
comprised of 1981 California Achievement Test (CAT) math
scores, reported in NCEs. Scores were obtained from two
testing periods, a pretest and a posttest. There are no
missing values in the data set; a score is included for
each student for both testing periods.
The posttest scores serve as the dependent measure in
the analysis, and the pretest serves as the covariate.
Four classifications of program types represent the
independent variable, or grouping factor.
Research Design
Design on Groups
The grouping factor upon which the current
investigation is based is school type. Four types of
schools served as the independent variable for the study.
School types included participants in a Bilingual education
program, a Follow Through program, both Bilingual and
Follow Through programs, and neither program. The design
on groups is delineated on Table 5.
72
Table 5
Design on Groups
Group
* * * * * * * * * * * Bilingual * Follow * Both * No Program * * * * * * * * Through * Programs * * * * * * * * * * * * * * * * * * * * * * * post test * post test * post test * post test * * * * * * * score * score * score * score * * * * * * * * * * *
Dependent Measure: Posttest Scores
Independent Measure: Grouping Factor
Covariate: Pretest Scores
73
Design on Comparisons
In addition to the design on groups, there is a
research design on comparisons. The four ANCOVA models
that were analyzed in the current investigation include:
(1) the parametric ANCOVA, (2) the parametric ANCOVA with
a covariate correction, (3) the rank transform ANCOVA, and
(4) the rank transform ANCOVA with a covariate correction.
Comparisons of the four ANCOVA models were divided
procedurally into three steps. Initially, population data
exploration was completed, and population parameters were
obtained for each of the four models. Model comparisons
were made based on these parent population parameters.
Secondly, samples were generated from the population data
set, and probability levels were obtained on each of the
samples. Subsequent comparisons of four Monte Carlo sample
probability intervals were made initially to each other
and, ultimately, to the true population parameters.
Primary and secondary comparisons to be included in
the initial phase of the current study are delineated by
the matrix in Table 6.
74
Table 6
Design on Comparisons Matrix for Population Parameters
Note.
Note.
ANCOVAS p PCC
p
PCC
R
RCC
p: primary comparison;
ANCOVAS: (P) Parametric
p
R
p
s
RCC
s
p
p
s: secondary comparison.
ANCOVA, (PCC) Parametric
ANCOVA with a Covariate Correction, (R) Rank Transform
ANCOVA, and (RCC) Rank Transform ANCOVA with a Covariate
Correction.
77
As with population comparisons, the six sample
comparisons outlined in Table 7 are delineated on two
levels. Four primary comparisons of probability intervals
of the four models have been made in response to general
research question 4. There were also two informal
secondary comparisons that considered differences and
similarities, between probability intervals of other
combinations of the ANCOVAS. The third stage of
comparisons, sample probability intervals in relation to
true population parameters, is outlined on Table 8.
78
Table 8
Design on Comparisons Matrix for Sample Probability
Intervals in Relation to True Population Parameters
ANCOVAS
SP
SPCC
SR
SRCC
p
p
PCC
s
p
R
s
s
p
RCC
s
s
s
p
Note. p: primary comparison; s: secondary comparison.
Note. ANCOVAS: (P) Parametric ANCOVA, (PCC) Parametric
ANCOVA with a Covariate Correction, (R) Rank Transform
ANCOVA, (RCC) Rank Transform ANCOVA with a Covariate
Correction, (SP) Sample Parametric ANCOVA, (SPCC) Sample
Parametric ANCOVA with a Covariate Correction, (SR) Sample
Rank Transform ANCOVA, and (SRCC) Sample Rank Transform
ANCOVA with a Covariate Correction.
79
There are four primary comparisons between the
sample probability intervals and the true population
probability levels. Each of sample intervals has been
compared to the parameter of its population model
counterpart, in response to general research question 4.
While these four comparisons are the most crucial in light
of the purpose and significance of the current
investigation, six informal secondary comparisons have also
been made.
The following subsection will further describe the
different analysis of covariance models and other analyses
that were completed in the current investigation.
Analytical Procedures
This subsection will describe the analytical
procedures that have been completed in the current
investigation. Those procedures have been divided into
five stages that were completed in the order of their
presentation.
Variable Production
Before initial data exploration and subsequent
ANCOVAS were completed, data points for four new variables
were computed: covariate-corrected X values and ranked Y,
X, and covariate-corrected X values.
80
Covariate-corrected X values were first derived. For
the current investigation, rxyr the pretest posttest
within-group correlation coefficient, was used as the
reliability estimate. After the reliability estimate was
determined, it was then multiplied by the difference
between each X value and the group mean. These products
were then added to the group mean before they became the
values for a reliability-corrected covariate. The formula
for the covariate correction by group reads:
Xadj = Xj + rxy (Xi - Xj)
In order to complete the rank transform ANCOVA and the
rank ANCOVA with a covariate correction, it was necessary
to rank the X, Y, and covariate-corrected X values. Three
new variables of ranked data were produced on computer with
SAS (Statistical Analysis System User's Guide, 1982).
Subsequent analyses were then completed on SAS, SPSSX
(Statistical Package for the Social Sciences, 1986), or
Number Cruncher (Number Cruncher Statistical System, 1986).
Data Exploration
The initial step in studying a data set is to perform
numerous preliminary analyses on the data, with the intent
of determining its characteristics. For the current
investigation, procedures were completed to determine
whether or not assumptions underlying the various ANCOVAS
81
had been met.
Assumptions underlying the parametric analysis of
covariance were delineated and described in the literature
review. Each of the assumptions has been addressed, and,
if appropriate, procedures have been included to test for
them.
For the parent population, the assumption of
randomization of samples is not an issue. Because the
samples were randomly generated, the assumption of
randomization was also met for sample analyses.
It is important to test for the second assumption,
homogeneity of within-group regressions. Tests for this
second assumption were completed along with the analyses of
covariance.
The third assumption, statistical independence of the
covariate and treatment, and the fourth assumption, fixed
covariate values that are error free, are assumptions that
often accompany the assumption of randomization. For the
parent population of the current investigation, the third
assumption cannot be met, since it relates specifically to
randomly assigning subjects to treatment groups. With the
parent population, subjects are studied as intact groups,
which causes a statistical dependence between the covariate
and the group effect. Despite random generation of the
samples, this statistical dependence has been conveyed to
the samples, due to the makeup of the parent population
82
from which they are drawn.
Assumption four, fixed covariate values that are error
free, has been accounted for in the parent population by
the inclusion of a reliability correction on two of the
four ANCOVA models. Because the population X values of two
of the models have been corrected for error by a
reliability coefficient before sample generation, this
correction carries over to the samples.
The assumption of linearity between the covariate and
the dependent measure has been determined for the
population by computer-produced scatterplots. Pairwise
Pearson and Spearman correlations have also been completed
to determine the degree of the relationship between the
covariate and the dependent measure.
The sixth assumption, normality of the conditional Y
scores, has been determined for the population. Included
are boxplots, visual representations of the distributions
which indicate lack or presence of normality. In addition,
a goodness-of-fit test for normality has been completed on
the population y values.
Homogeneity of variance of conditional Y scores has
been tested for the population by the Bartlett-Box
univariate homogeneity of variance test.
Because the treatment levels included in the study
were not selected through random assignment, the last
parametric assumption, fixed-treatment levels, is not an
83
issue.
In addition to assumptions for the parametric ANCOVA,
there are several assumptions that have been addressed· for
the rank transform ANCOVA. The scale of measurement is
interval data, which meets the nonparametric assumption
that the data be measured on at least an ordinal or
dichotomous scale. The assumption of randomization has
already been addressed in relation to parametric
assumptions. The degree of monotonicity between the
covariate and the dependent variable has been determined
for the population by a homogeneity of regression slopes
test on the Spearman correlations for the ranked Xs and Ys.
The Kruskal-Wallis test was completed on the population
covariate to ascertain whether or not the marginal
distributions were identical.
After the data exploration was completed, results were
charted and logged in relation to assumption violations and
pertinent analyses of covariance. Before any determination
of the best ANCOVA model relative to assumption violations
was prescribed, all forms of the analysis of covariance
were completed on both the population and the samples.
The ANCOVAS on the Population
The first step after data exploration is to assess the
assumptions for violations. For research studies,
84
assumption diagnosis is completed prior to the assignment
of a particular statistical procedure to a data set.
However, in the current investigation, that was not the
case. All assumptions were studied, and all analyses of
covariance were completed. From a review of the
assumptions, it should be possible to determine whether or
not a particular form of parametric and/or nonparametric
ANCOVA should be recommended, but based on the purpose of
the current study, that determination was not made until
both the data exploration and the ANCOVAS were completed.
There are four models of the analysis of covariance
that were analyzed in
the population and
ANCOVA models include:
the current
subsequently
1. Parametric ANCOVA
investigation, first on
on the samples. Those
2. Parametric ANCOVA with a Covariate Correction
3. Rank ANCOVA
4. Rank ANCOVA with a Covariate Correction
Results of the four ANCOVAS completed on the parent
population serve as measures of the true parameters to
which the sample ANCOVAS have been compared.
Robust Regression on the Population
Robust regression analysis (Montgomery & Peck, 1982)
was completed with the parametric and covariate-corrected
85
models to determine whether outlier-induced nonnormality
impacted the regression estimates. Through four weighted
iteration procedures completed with each model, it was
possible to delineate the probability level that
corresponds to the F-like value of the regression equation
that best represents the majority of the data.
In addition, from the weights obtained in the fourth
iteration, it was possible to identify outliers. Outliers
were then deleted from the data set and ANCOVAS were
completed for each of the two parametric models. Results
were compared to findings of the original analyses.
Monte Carlo Simulation Samples and ANCOVAS
In order to complete the four ANCOVAS on samples, a
table of 100 random numbers was first generated by BASIC,
and served as the seeds for data generation. SPSSX was
implemented to generate the 100 random samples with
replacement from the parent population. One-third of the
observations from each of the four program types was
selected for each sample, with a total n of 95 for each
sample. Both parametric ANCOVAS, and both rank transform
ANCOVAS were then completed on each of the 100 random
samples.
After the sample analyses of covariance were
completed, results of all models were studied and
86
comparisons were made.
Assumption Diagnosis and ANCOVA Comparisons
While assumption diagnosis should be completed prior
to the assignment of a particular statistical procedure to
a data set, that was not the case in the current
investigation. As indicated earlier, that process is not
necessary in light of the purpose for the study.
After the data exploration and the ANCOVAS were
completed on the population, assumptions and reliabilities
were reviewed, and results of the ANCOVAS were compared.
Based on these factors, it was possible to informally
determine whether or not a particular parametric and/or
nonparametric ANCOVA model should be recommended. Final
determination of the "best" model was not made until the
sampling was completed.
The results of the parametric and rank transform
ANCOVA and those ANCOVAS with a covariate correction were
compared with each other in light of the research questions
and primary and secondary comparisons (see Table 6).
Specific results compared include: (1) F values and
corresponding probability levels; (2) between-group
regression equations; (3) parametric and nonparametric
assumptions; (4) means and adjusted means; and (5) the
covariate.
87
The frequency distributions of the probabilities
associated with the F values obtained through the Monte
Carlo sampling were analyzed and graphed. The interval
under the central part of the frequency distribution
containing 68% of the sample probabilities was then
identified. The intervals of the four ANCOVAS on the
samples have been compared to each other and to the true
population parameters (see Tables 7 and 8).
To determine the "best" model, four criteria were
assessed:
1. Assumption violations of the parent population
indicate whether a parametric or nonparametric model should
be prescribed.
2. The strength of the reliability coefficient shows
whether or not a covariate correction reduces error and
improves the estimate.
3. The tightest probability interval (one indicating
the greatest convergence of the sample probability levels)
serves to delineate the most stable probability level and
the preferred model.
4. Probability intervals corresponding to true parent
probability levels helped to identify the most accurate
model.
5. Results of robust regression indicated whether
outlier-induced nonnormality has an effect on the
regression estimates and impacts model selection~
88
Conclusions have been drawn, and recommendations have
then been made based on the statistical results. The next
subsection will explain the implementation guidelines for
the study.
89
Implementation Guidelines
The specific guidelines for the current investigation
can be divided into seven stages: (1) covariate correction
and ranked data generation (2) data exploration of the
population, (3) data analysis of the population, (4) sample
generation, (5) analysis of the samples, (6) data
assessment, and (7) conclusions and recommendations.
1. Covariate Correction and Ranked Data Generation
A. Determine the covariate correction and define the
covariate-corrected X values.
B. Rank the X, Y, and covariate-corrected X data and
define the ranked pre, post, and covariate-corrected pre
variables.
2. Data Exploration of the Population
A. Test, record or plot the following parametric
assumptions on the population in relation to the ANCOVA
models:
1) Linearity.
2) Normality of the conditional Y scores.
3) Homogeneity of variance of conditional Y
scores.
4) Homogeneity of within-group regressions.
B. Test, record or plot the following nonparametric
assumptions on the population:
90
1) Degree of monotonicity.
2) Equality of marginal distributions.
3. Data Analysis of the Population
A. For each of the ANCOVA models:
1) Complete ANOVAS on the covariate.
2) Complete Pearson and Spearman correlations.
B. Complete the following ANCOVAS:
1) Parametric ANCOVA.
2) Parametric ANCOVA with a Covariate
correction.
3) Rank Transform ANCOVA.
4) Rank Transform ANCOVA with a covariate
correction.
c. complete robust regression for the parametric
and covariate-corrected parametric ANCOVA.
4. Data Generation
A. Generate 100 random numbers to serve as seeds for
data generation.
B. Generate 100 random samples from the parent
population.
5. Analysis of Samples
A. Complete the two parametric and two nonparametric
ANCOVAS on the samples.
6. Data Assessment
A. Population
1) Examine results of the data exploration in
91
light of assumption violations and the appropriate
corresponding analyses of covariance.
2) Examine and compare the results of the
parametric ~COVA, nonparametric ~COVA, and those
with a covariate correction in relation to pertinent
research questions.
3) Determine the strength of the reliability
coefficient.
4) Compare the probability levels of the
parametric, nonparametric, and covariate-corrected
~COVAS with one another.
5) Assess the results of the robust regressions.
B. Samples
1) Determine the parametric and nonparametric
~COVA sample intervals (and half intervals) under
the central part of the frequency distribution of
probabilities under which 68% of the area lies.
2) Compare the parametric and nonparametric
intervals with each other.
c. Overall
1) Assess assumption violations in relation to
parametric and nonparametric ~COVAS.
2) Assess the strength of the reliability
coefficient to determine if error reduction is gained
from its use as a covariate correction.
3) Determine the tightest interval (the one with
92
the greatest convergence of the sample probability
levels) in order to assess the stability of the
probability level.
4) Compare the probability intervals of the
sample ANCOVAS with the population probability
levels, with the intent of determining the interval
most closely corresponding to the parent population
parameter.
5) Determine whether or not the robust
regression delineates a preferred model.
6) Determine the "best" ANCOVA model.
7. Conclusions and Recommendations
A. Upon completion of the previous procedures,
tests, and comparisons, determine conclusions based on the
results of the analys·es and procedures, and give
recommendations for the further study of nonequivalent
control group research.
Chapter Four
RESULTS
overview
The current chapter will present results of the
present investigation in light of each of the specific and
general research questions. While the study ultimately
seeks to answer five specific research questions leading to
a sixth focal question, data exploration on the population
will be examined, results specific to the four models will
be presented, Monte Carlo samples will be discussed, robust
regression will be examined, and a number of general
questions comparing models will be addressed and answered
subsequent to those major questions underlying the main
purpose of the research. The specific research questions
that will be ultimately addressed are as follows:
1. How do the results of data exploration on the
population assist in delineating a preferred ANCOVA model?
2. Is the reliability correction on the covariate
strong enough for error reduction?
3. Which sample model shows the tightest
probability interval?
4. Are population parameters contained in the
sample probability intervals?
5. Do the robust regressions indicate a preferred
model?
93
94
6. In light of assumption violations, the
reliability of the covariate correction, probability
interval size, true parent population parameters, and the
results of robust regression, what is the "best" overall
model for this nonequivalent control group investigation?
Population Results
Data Exploration on the Population
Results of assumption diagnosis for parametric and
nonparametric ANCOVAS will be reported in this section.
Linearity
The assumption of linearity between the covariate and
the dependent variable for each school program type was
visually assessed by scatterplot. Because of differences
in data for each of the four models due to covariate
correction and ranking, it
linearity for each model.
was necessary to assess
In all instances, the scatterplots showed a strong or
moderate linear relationship between the covariate and the
dependent variable over all models. Scatterplots are
included in Figures 1 through 8.
To provide a cross-validation for the scatterplot
assessment, Pearson correlations were completed to
95
determine the degree of the relationship between the
covariate and the dependent measure. The Pearson r for
each school program type across all models was strong
enough to indicate linearity.
Table 9.
Results are included in
Spearman correlations were also run to compensate for
outliers in the data set (see Figure 9). Because of
ranking, the Spearman correlations for each program type
were equal across models. Spearman correlations for Follow
Through and Bilingual were slightly lower than the Pearson
and slightly higher for groups with both programs and
neither program. Although there were differences between
the Pearson and Spearman correlations, all were similar
enough to consistently indicate a linear relationship
between the covariate and the dependent measure for all
program types and across all ANCOVA models. Spearman
correlations are included on Table 9.
96
'·' u.. u.o so.• u.1 -.1.1 st.• ••. a "·' 10.1 11.0 i. ----·----·----·----·----·----·----·----·----·----· ----·----·----·----· ----·----· ----·----·--:··----· i
i i ! I 1----------------------------------------------------------------------·---------------------------1 i . . .. !:· I i . · .. · ;:: .. ·· I 1 : ' • • 1----------------------=-----=:,: ... :: ...... ::.: ________ : ................ _____________________________ I . . . . i . . = • = • i= • ·: I I .. : ·: I I· . . : . I • • • • • . •• I ·: ..... 1 ...... !.~-·--! .. : ................. : .... : .. ___ ........................................................................... _. ___ _ 1.o •·• 11.1 u.1 u.• u.o st.• n.a H.1 "·' as.6
1J,O
u.o
sr.o
••.a
u.o
n.o
n.o
11.0
•.o 1.0
u.o
11.0
u.o
u.o
u.o
n.1
u.o
11.0
•.o 1.0
,. .. ........ !:! ........... !!;! ........ _!!;! ........... !!:! ........... !!;! ........... !!;! __ ... _!!;! __ ._!~;! .......... !!;! .......... !!;! __ • i ·' . I I I I · · I 1-------------------------·---------------:·---------:.:. _______ ·----------------------------1 i .. i
u.1
., .. Jt.4
n.o
u.o
11 ••
u.1
u.1
Jt.4
n.o
I . . I i . . . . . . i i----·----·-=-:··------=------:·:··-------;-------------------------------------1
I . . I I· . : · : · I ': o·····--;:;··· .... ;;:1····-;i:i·--·-·;;:;····-·;;:ir·-···;1:;··-···;;:i·--·--:i:1··•··;::;·-•·-i;:a
Zl.I
lO.I
n.1
'·' 1.0
.... 10.1
n.a
'·' 1.0
FIGURE 1. Scatterplots of the dependent and the covariate
for the parametric ANCOVA model for the Follow Through and
Bilingual groups.
u.o
5l.O
u.o
J7.0
ll.O
H.O
ll.O
tr.a
u.o
7.0
u.o
•l.'
u.o
t.J.o
10.0
lJ.S
u.o
tO.t
•.o
97
; •.. ! !; ! ...•.. !!; ! ...... !! ;! ...•.. ! ! ; ! ---· .. ! ! ; ! ...... !! ; ! ·-· •.. ! ! ; ! ...•.. !~; ! ··-·· .! !; ! ...•.. !!; ! ... : i l
I . . I I- -----------------------------------------------------------------------=--------=--------------1 I . I i i ! ! !--------------------------------------·-····----:-·-;-----;·------------------------------------1 i . i t l L . I ,·---··-·--··---·----·---·-·--·----·----·----·----·----·----·---.. ---·----·----·----·---·----·----· .o u.s zo.o U.S J1.0 H.S U.O 47.S U.O H.t u.6
.•... !;! ......... !!;? ....... !!;! .......... ~!;! .......... !!:! ........... !!:! ........... !!:~---·--!~:! ............ !!;! ............ !?:!. .... . ; .1 .: ! ! l l l I . . . · · · I j ..... ··-·· .. ·-· -------··-· ----·--;-· _______ : ·:·-------------·--·-··:·: .... -------------------------1 i . . . I t . . . i I . .. . I . . . . . i ----------------------···--:··----;--·----·-·-:--·-----;---------------------------------------i I· I ! . I ':a·· -·-·10 :;---··-io: ;-·-·-·;o:;··-·-·;;:i---·-·;;: ;··-·-·;;:;·--·--;i~ ;-··•··;;:;··-·-·;; :; ---·-·;;:a
11.0
n.o
u.o
u.o
u.o
ir.o
u.1
n.o
u.o
7.0
••.a
u.s
H.O
o.s
u.o
"·' JO.O
u.s
n.o
10.s
•.o
FIGURE 2. Scatterplots of the dependent and the covariate
for the parametric ANCOVA model for the groups with both
programs and neither program.
98
u.•r.oo
51. lHOO
J1 .6'400
.t.0'400
". 77'00
S.l u.• n.o lO.• JI.I •7.Z SS.• ••• o 7Z.• 10.1 i. ---- ·----·----·----. ----. ----· ----. ----· ----·----·----·----. ----. --- -· ----. ----. ----. ----. --;-·----. , ! 1 I I 1- ----------------------------------------------·------------·---------·------· ----------------·----1 i . . . !:' i i : . 0
z z: • • i I--------------------=---!-=---=:,= .. :: ____ =::.; .......... : ............................................ ----------------1 i . . : : . i: •• : i
11.61400
SI. JJ400
S1.6HOO
u.ouoo
Jl.41600
J1. 77'00
lS.1hQO
U.4'400
s.zuoo
I . : I . . . . l; ____ i: ___ ,:_;_: __ ~_;_:_: ____ , ____ ; __ :_, __ :_, ____ :: __ , ____ , ____ , ____ , ____ , ____ , ____ , ____ , ___ , ____ , l Lo •·• 11.1 u.z u.• u.o s1.• ''·' u.z u.• u.6
11.1)400
s.zuoo
u.usso ....... ?;! ......... !!:! ...... -!!:~ ......... !!1! ......... ?!:! .......... !!;!. .......... ?!;! ........ !!:! .......... !!;!_ ... _!!t! ...... . • .i SI. USSD
H.UUO
•Z.lZHO
31 .•OSSO
l l I I I · · I 1-------------------------·--------------:----------=-=---------·----------------------------1 i . .. I
n.nno
u.uuo
"·""' H. t17SO
J1.tOHO
lt.ltUO
lO.tlUO
1t.utSO
10.n1so
s.ouso
I . . . · . ·. . I I . . . . I ,-------·------:-·;··-------:-·--·-;·:·---·----;----------------------------·----------------,
i : . i I· . : · . I , :c;-·-·--·;:;---·-·1;:1·····;;:;····--;;:;··-·--,;:o··-·-·;;:;--•·-;;:i·--·--;i:;·--·-·;;:;·--·-·i;:,
U.ltJSO
10.uuo
u .... ,,
10.uno
s.ouso
FIGURE 3. Scatterplots of the dependent and the covariate
for the covariate-corrected parametric ANCOVA model for the
Follow Through and Bilingual groups.
99
11.1 u.J u.1 za.s JJ.1 Jt.J u.1 \O.J u.1 a1.J u.1u10
ti.l.60710
u.0110
J6. n110
u.00110
Z1.IS1t0
lJ.101'0
u.u110
n.n110
i ·----·----·----·----·----·----·----·----·----·----· ----·----. ----. ----·----·----·----·----·----· ----: i i i I : I 1-------------------------------------------------------------------------:----------------------1 l I I I l l 1------------------------------------------------;---;------:-------·------····-·-········-······I i . I I I I. . I •:o····--1~:;····--,a:o··-···;;:;····-·;;:a······;;:;··-·-·;;:o··-·--;;:;---··-;;:o······;;:;··-·--;;:1
U.1'110
..... ,,. 66.0110
... ,.,,. so.1111a
u.ao110
,,,.,,,, u.10111
1t.nr10
""°'" "·""'
6J.Ptt60
\J.OOHO
.. r.61060
U.IZ060
J1.UHO
16.0JOtO
, .. .,,.. 1S.Z60IO
.... ,60
....... !;! .......... !!;! ....... _!!;! ........... !!;! ........... !~;! ........... !!;! ........... !!;! ........ !!;! ........... !!;! .......... !!i! ....... , i ·! . , I I I . . I 1--------------··-·-·--·----------:------·-·--:··-·-----··-·-·--··:·-········-----·········-·····-1 i . . i I . I l _________ : __________ !_ ••••• :.:: _____ ::=-----------·---=-----·-···-·-····-······-·····-···-······I . . . i . · · . I I· I f . I ; :o······;o:;-····;a:;--···-;a:;··-···;;:;·····-;;:;······;;:;······;;:;····-·;i:;·-····;;:;·····-;;: 1
u.nuo
U.60060
n.oouo
"·""' 6Z.Z1UO
J6.U060
Jt.UHO
16.0JOaO
lO.UHO
.... ,.,
FIGURE 4. Scatterplots of the dependent and the covariate
for the covariate-corrected parametric ANCOVA model for the
groups with both programs and neither program.
u1.oo
ZOJ.60
ht.OD
120.zo
•••• o
Jt.10
•.oo
l1t.OO
lH.00
llS .oo
,, •• oo
111.00
tu.oo
11'.00
•o.oo
.,.oo u.oo
t.oo
100
u.u u.u n.u 101.U tzt.H 1u.u ut.u zu.1' zu.o' 111.n ...... -- ·----... ---... -·-......................................... --- ....... --. ---- ...... -.... -............... -- ....... -............ -· -.......... -- ............................................. . . ·l. I .. . I
. . = I i----------------------·-----------r=--------:--------------; ____ !__ • ..!-----:----=-------:--------1 I .. ' . . . I l i l-----------=---------=--·----=--------------:-=---=--------------------------------------------1 i . · = · . . • • I ! = . I !::,. ·:: . . I 1: ;o ·· • ··;o :ia·· ·-·;;: 10·· • --i;: ;a··•·;;; :10··•-;;;: oo·-··;;;:;o···-;oo: ;o···-zii: ;o···-;;;:io-··;i;: to
.• -!!;!! .......... !!;!? ........... !!;!? .......... -!~!;?! ........ !!!;!?. ...... !!!;!! ......... !!!;!? .......... !!!;!? ......... !!!;!! .......... !!!;!! ..... j d . I I I I . . : · . . I 1---------------:-------------------------------:--·--------------:----------------------------1 I . i I . . I ------------;--:------------:·-------·---------------------·-----------------------1 I . . I . I • i I . . . I 1·. . . : I 1:;0····-;0:10···-·;;:10·····;;:;;--·-;;;:;o··•·;u:oa-··;;;:;a--··;ao:;o···-iii:;o·-··i;;:ia·-··;;;:10
ll1 .oo
lSt.lO
lJ1.40
lOJ.60
,,, •• o
ue.oo
uo.11
.... o
u.10
•.oo
11'.00
1u.oo
us.oo
ttl.00
111.00
U6.00
IH.00
u.oo
u.oo
•.oo
FIGURE 5. Scatterplots of the dependent and the covariate
for the rank AN COVA. model for the Follow Through and
Bilingual groups.
llO. tO
191.U
17S.70
1U.U
1 so.10
101. J1
IS.tO
tJ.4'
•1.00
lll.00
lU.SO
.!11.00
lOS. SO
110 • .lO
1h.SO
1 u.oa
,OJ.SO
11.00
Sl.SO
u.oo
101
zs .• , sL02 1t.S1 to1.n 121.01 1u.u 111.11 zos.u zu.u '"·" ; • ·-....... - ---·---- • -· ... ·----. ----· .. --- • - -- • ·---- .................................. - - - -· ... - ....... -- .. ·--. - .............................. ·-: i
I . I i i i i 1----------------------------------------------------------------·---------------=----------------! i i I I I. ------------------------------------------------------=-------------------------------------------·1' . . . . I . I i i I I .. . . .•-·-··----·----·----·---·----·----·----·-·--·----·----·--···----·--····---·----···--·---··---·----· n.oo 11.11 u.10 at.as 1u.•o n•.n us.to no.u zu.10 u1.u zu.to
....... !!;!! ........... !?i!! ......... !!;!! ........ !~!s~! ........ !!~;?! ........ !?!;!! ........ !!!;!! ...... !!!:!! .......... !!!;!! ......... !!!;!!_ •• . . = ..... :;
i I . f 1----------------------------------:-·:-------- . ----------------------=-=--------------------------1 i i I . . . I I---------------------------------·-----------------------------------·--------------------1 I . . I I· I i . i i: ;o ·-·-·;o:;; ·-·-·;;: ~-- ._.i;: i;--·-;;;: ;;;·· ··;:;: ;; --·-;;; :~o--• ·;o;: ;;···-;;o: ;o ···z;i:;; --··;i;: &a
u,.,o US.OS
uo.•o
n1.u
111.10
UJ.H
110.IO
101.n
IS.10
u.o
41.00
ru.oo
'"·so u1.oo
ros. so
110.00
,,,. sa
Ut.00
UJ.SO
11.00
u.so
JP.DO
FIGURE 6. Scatterplots of the dependent and the covariate
for the rank ANCOVA model for the groups with both programs
and neither program.
102
ll1.00 u.u u.•s 11.ls 101.u ut.u '"·" 1u.u 2u.rs us.as 271.is : ......... -·----·----· ----. ----·---i ·----·----. ----· .. --·----· ----. ----·----·----· ----. ---.. ---· ·· :; : i
zs•. 10
111.00
lit.JO
I . . . : : I j-------------------·--------··r-··--: ......... : __ : __ ~ ___ j __ :_; ___ :--~: .. :.:. .. :------1 i ·· . I I I • I • • • I ! • . • • • • •
l--··:··-----: ........ .: .. ;····=--------------i-: ... :;-----------------------------------------1 I · : · . . I i . . i ,. • • • I • • • • .. I ••ZS4 l• • • 1: so--·-·io :ao·· ·-·;;:;a--·--i ;:;o·-·-;;;:;o·-··;;;: oa···-;;;:;o··•·;oo:;o··•·zii :ti·-•-;;;:zo-··zi;: to
l )1 •• o
lOJ. tO
1 l't.ZO
uo.10
•s.10
U.40
u.ro
10.00
IJ1.6Q
IQS,fQ
17•.ZO
hi.SO
110.10
tJ.10
to.oo
zn.oo ; • -... !! :!? ........... !!: !? ......... !~:!! ....... ! ~! ;?! .... , .. !!! :!? ....... !?!:!! ....... !!! :!? ....... !!!;!! ....... !!!:!! ...... ! !!:!? ··: i
l I I I I I ., . . . . ., ............................... : ................................................................ ; .... ·---·----·--·-·:··-------------------------·-·-.
Ht.O
U1.t0
10.n
1u.10
111.00
'"·" U1.t0
.... ,, U6.10
I I I I l---=--------·-·:·-=--------·----------:---:------------------:-----------------------1 I I .. • I I I r I . . . . z: ;o·· ··-io:io·· ·-·;;: ;;·-·-·;;:;o·· ·-;;;:;;;·-·-;;;: oo-•-;;z:io·-·-;oo:;o····;zi: ;o···-i;1:;0·-·-iii: 10
t Jt.lS
u.ts
,. •• o
n.os
1.SO
Ut.U
ut.ro
... ,, "·'' It.OS
FIGURE 7. Scatterplots of the dependent and the covariate
for the covariate-corrected rank AN COVA model for the
Follow Through and Bilingual groups.
103
zu.oo .• __ ! ? ;! ! ....... ?! ;~!.- ..... ! ! ;!! ....... ! ~ ! ; ! ! .... ... ! ! ! ; ~! ....... ! ?! : ! ! ....... !! ! : !! ....... !~~ ;! ! ....... !! !; ~! ....... ! ! ! ;!! .. -· I I . . ·1
I I . I 1----------------------------------------------------------------=---------------------------------1 I I i . i 1---------------------------------------------------------------------------------------1 . . . . l . I I I i i .. . . , s:oo···--ii:;;-··-·;;:;o·-·--i;:o;···-;;;:;o···-;;;:;;-·-;;;:;o···-;;o:;;···-;;;:io--·-;;;:;;-·-i;;:10
Z-,J.U
ll1. JO
1'1.H
, , •• ,a
114.U
1 U.to
10t."
ar.zo
u.u
4l.SO
zu.oo
zu.o
Ut. JO
,. ... , 176.60
1u.u
,,, ·'' 10•.u
IPolO
... ., u.so
us.oo i•--!!;!L.-.!!;!! ••• _!!;!! ••• _!!!;!! __ •• !!!;!L.-!?!;!!._._!!!;!L • .!!?;!! ••• _!!:;!L.:!~~::~:~:i
j . . . . I I . . . I 1----------------------------------·--·----------------------------------------------------------1 i . . I i . . . i !---------------------------------!---------------------------------!--------------------! I · . l 1. I I I . . . 1: ;o·· ·-·io:;;--··-;;: ;o··· --i;:i;--··;;;:;a····;;;:;;--·-;;;:;a··· ·;a;:;;·-· ·z;o:;o··•·;;;:;;--·-;;;: &o
ZU.60
lH.lO
zot.10
Ut.00
1 H.tO
1u.zo
11.to
U.40
Jt.oo
us.oo
IH,60
U•.10
ZOt,IO
US.40
'''·°' ,,. ... 111.10
11.10
Jt.00
FIGURE 8. Scatterplots of the dependent and the covariate
for the covariate-corrected rank AN COVA model for the
groups with both programs and neither program.
104
Table 9
Pearson and Spearman Correlations between the Covariate and
the Dependent Measure by Program Type
Program Type
Follow Through
Bilingual
Both
Neither
Follow Through
Bilingual
Both Programs
Neither Program
Model
Par & Paree
Pearson Correlations
r=.78760
r=.84509
r=.90188
r=.76153
Spearman Correlations
r=.79550
r=.71441
r=.91712
r=.71226
Rank
.76211
.80156
.89284
.68295
Rank CC
.76539
.80380
.89304
.67314
(Equivalent
correlations
across all
four models)
105
Normality
To visually determine normality of the dependent
measure, boxplots were completed for each program type on
both population and ranked Y scores. In addition, a
goodness-of-fit test was completed to give a statistical
representation of normality.
Normality cannot be assumed for the Y scores of the
current investigation. Boxplots show both mild and extreme
outliers (see Figure 9) for both population and ranked
data. A goodness-of-fit test statistic completed on the
population data, however, indicates that normality can be
assumed. Because boxplots clearly indicate there are
outliers and nonnormal distributions, robust regression was
completed to determine whether or not the parametric
ANCOVAS were impacted by the outliers. Table 10 provides
results of the goodness-of-fit test.
106
98 0
E2 E x E2 I
E2 I 02 I x x I I x --+-- --+--I I I I I I I I I * I I I --+-- I I I I * I
I I --+-- --+-- I I I * I I I I --+--I I I * I I I --+-- --+-- I I
I I x I I I 0 x I I
1 x x 0
287 x x x I I x I I I --+-- --+--I I I I I I I I I I I I I I I * I I I --+-- I I I I * I
I I I I I I I I I --+-- I I I I I I I I --+-- I I I * I I I I I I I I I I I --+--I I I * I I I I I I I x I I I I I I --+-- --+-- I
I I I I I 0 I I I 0 I
2.5 x x x
FIGURE 9. Boxplots of Y scores of program types (Follow
Through, Bilingual, both program, and neither program)
according to population and ranked data.
107
Table 10
Goodness-of-Fit Test on Normality of Y Scores
Program Type Statistic
Population Data
Follow Through W2=.000005287
Bilingual W2=.00004785
Both Programs W2=.00333715
Neither Program W2=.0000306247
Note. * E > .OS.
Probability
Level
p>.15 * p>.15 * p>.15 * p>.15 *
108
Homogeneity of Variances
Homogeneity of variance of Y scores was assessed by
the Bartlett-Box F test for both population and ranked
data. In both instances, the F test indicated that
homogeneity of variance can be assumed. However, because
normality is an assumption required for the Bartlett-Box F
test, and normality cannot be assumed for the data, results
of the F test may not be valid. Table 11 includes results
of the univariate homogeneity of variance tests.
Homogeneity of Within-Group Regressions
Parallel slopes were tested, and the assumption was
found to hold for all four models. Table 12 indicates the
F values and corresponding probabilities for parallel slope
testing.
109
Table 11
Bartlett-Box Homogeneity of Variance Tests on Y Scores
Data
Population
Ranked Data
Note. * E > .05.
F Value
F=l.03304
F= .37381
Probability Degrees of
Level Freedom
p=.377 * 3, 63502
p=.722 * 3, 63502
110
Table 12
Homogeneity of Within-Group Regressions
Model
Parametric
Parametric cc Rank
Rank cc
Note. * E > .05.
F Values
F=.690
F=.690
F=.244
F=.288
Probability
Level
p=.559 *
p=.559 * p=.866 * p=.834 *
111
Monotonicity
While linearity is not an assumption required for the
ranked 'ANCOVA, monotonicity between the covariate and the
dependent measure must be assumed. An analysis of the
scatterplots indicates the assumption holds (see Figures 1
through 8.)
In addition, results of a homogeneity of regression
slopes test on the Spearman correlations for the ranked Xs
and Ys indicated that the degree of monotonicity was
equivalent for the covariate and the dependent measure,
f(3, 282) = .244, E > .05.
Equality of Marginal Distributions
To assess equality of the marginal distributions of
the covariate, a Kruskal-Wallis was completed. A
significant probability level, £=.0000, for each of the
four analyses indicates that the marginal distributions are
not equal. Consequently, this nonparametric assumption
does not hold. Tables 13 to 16 include the Kruskal-Wallis
Tables for the four analyses.
112
Table 13
Kruskal-Wallis Table to Assess Equality of Marginal
Distributions and Differences in the Parametric X Scores
CASES
287
MEAN RANK
130.60
125.76
162.15
190.64
CHI-SQUARE
26.0513
Note. * E < .05.
CASES
137
69
24
57
TYPE = 1
TYPE = 2
TYPE = 3
TYPE = 4
287 TOTAL
FOLLOW THROUGH
BILINGUAL
BOTH
NEITHER
SIGNIFICANCE
0.0000
CORRECTED FOR TIES
CHI-SQUARE SIGNIFICANCE
26.0762 0.0000 *
Table 14
Kruskal-Wallis Table
113
to Assess Equality of Marginal
Distributions and Differences in the Covariate-Corrected
Parametric X Scores
CASES
287
MEAN RANK
129.02
121.71
165.42
197.96
CHI-SQUARE
35.1366
Note. * E <. 0 5 •
CASES
137
69
24
57
TYPE = 1
TYPE = 2
TYPE = 3
TYPE = 4
287 TOTAL
FOLLOW THROUGH
BILINGUAL
BOTH
NEITHER
SIGNIFICANCE
0.0000
CORRECTED FOR TIES
CHI-SQUARE SIGNIFICANCE
35.1475 0.0000 *
Table 15
Kruskal-Wallis Table
114
to Assess Equality in Marginal
Distributions and Differences in the Rank X Scores
CASES
287
MEAN RANK
130.60
125.76
162.15
190.64
CHI-SQUARE
26.0513
Note. * E. < • 0 5 .
CASES
137
69
24
57
TYPE = 1
TYPE = 2
TYPE = 3
TYPE = 4
287 TOTAL
FOLLOW THROUGH
BILINGUAL
BOTH
NEITHER
SIGNIFICANCE
0.0000
CORRECTED FOR TIES
CHI-SQUARE SIGNIFICANCE
26.0762 0.0000 *
115
Table 16
Kruskal-Wallis Table to Assess Equality of Marginal
Distributions and Differences in the Covariate-Corrected
Rank X Scores
CASES
287
MEAN RANK
129.02
121.71
165.42
197.96
CHI-SQUARE
35.1366
Note. * E < .OS.
CASES
137
69
24
57
TYPE = 1
TYPE = 2
TYPE = 3
TYPE = 4
287 TOTAL
SIGNIFICANCE
0.0000
FOLLOW THROUGH
BILINGUAL
BOTH
NEITHER
CORRECTED FOR TIES
CHI-SQUARE SIGNIFICANCE
35.1475 0.0000 *
116
While two-sample testing does not protect the overall
comparison level for more
Kolmogorov-Smirnov goodness
than two groups, a two-sample
of fit test was completed
pairwise between each group of each model to check for
differences in distributions. Testing of the pretests of
all four 'ANCOVA models revealed differences between
distributions of the group without either program and both
the Bilingual and Follow Through groups (.000 < E < .001).
Data Analysis on the Population
Covariate Correction
To assist in error reduction in the covariate, a
reliability correction was completed on the covariate X
values. Selected for the reliability correction was the
pretest posttest within-group correlation coefficient,
where the pooled within-group sum of the deviation
crossproducts for the pretest and the posttest are divided
by the square root of the product of the pooled within-
group sum of squares on the pretest and the pooled within-
group sum of squares on the posttest. The reliability
correction estimate was first multiplied by the difference
between each X value and the group mean. These products
were then added to the group mean before they became new
values for the covariate. The basic parametric 'ANCOVA was
then repeated with covariate-corrected X values. The
117
reliability correction used in this analysis was r=.83.
ANCOVAS on Pre Measures
Analyses of variance completed on the covariate of all
four models indicated group differences, £=.0000. Results
cross-validated those of the four Kruskal- Wallis tests
reported earlier. It is safe to assume that differences in
pre-measures exist, and that an analysis of covariance,
rather than an analysis of variance, should be used to
assess group differences. Analysis of variance tables for
premeasures of each model are included in Tables 17 to 20.
118
Table 17
Analysis of Variance Table to Assess Differences in the
Parametric X Scores
source of
Variation
WITHIN CELLS
CONSTANT
TYPE
SS
82496.488
234342.861
7823.652
Note. *E. < • 0 5.
DF
283
MS
291.507
1 234342.861
3 2607.884
F Sig of F
803.901 .000
8.946 .000 *
119
Table 18
Analysis of Variance Table to Assess Differences in the
Covariate-Corrected Parametric X Scores
Source of
Variation
WITHIN CELLS
CONSTANT
TYPE
SS
56831. 817
234342.603
7823.651
Note. *E < • 0 5 •
DF
283
MS
200.819
F Sig of F
1 234342.603 1166.934 .ooo 3 2607.884 12.986 .000 *
120
Table 19
Analysis of Variance Table to Assess Differences in the
Rank X Scores
Source of
Variation
WITHIN CELLS
CONSTANT
TYPE
SS
1788645.490
5951232.000
179441.510
Note. *E < .05.
DF
283
MS
6320.302
F Sig of F
1 5951232.000 941.606 .000
3 59813.837 9.464 .000 *
121
Table 20
Analysis of Variance Table to Assess Differences in the
covariate-Corrected Rank X Values
Source of
Variation
WITHIN CELLS
CONSTANT
TYPE
SS
1727336.400
5951232.000
242021.100
Note. *E < .OS.
DF
283
1
3
MS
6103.662
5951232.000
80673.700
F Sig of F
975.026 .000
13.217 .000 *
122
Pearson and Spearman Correlations
Pearson correlations were completed among the
dependent measure, the independent measure (grouping
factor), and the covariate for each of the four models.
Pearson correlations between the pre- and post-measures
ranged from .7962 to .8303. There was not a strong
correlation between the grouping factor (program type) and
the pre- and post-measures.
Spearman correlations were included to compensate for
the lack of normality of the Y scores. Spearman
correlations between the pre- and post-measures ranged from
.7962 to .7997. As with the Pearson, there was not a
strong correlation between the grouping factor (program
type) and the pre- and post-measures with the Spearman.
Both Pearson and Spearman correlations are included in
Table 21.
123
Table 21
Pearson and Spearman Correlations
Pre
Post
Ccpre
Post
Rpre
Post
ccrpre
Post
Correlation
Pearson
Parametric
Post Type Post
.8308 .2671 Pre .7962
.2762 Post
Covariate-Corrected Parametric
Post
.8334
Post
.7962
Post
.7997
Type
.3157
.2762
Type
.2742
.2765
Rank
Cc pre
Post
Rpre
Post
Post
.7997
Post
.7962
Covariate-Corrected Rank
Type
.3150
.2765
Post
Ccrpre .7997
Post
Spearman
Type
.2401
.2391
Type
.2739
.2391
Type
.2401
.2391
Type
.2739
.2391
124
The ANCOVAS on the Population
The Parametric ANCOVA.
The parametric analysis of covariance model indicates
that there is a difference among program types, E=.039.
With a researcher-set alpha level of E=.05, it is safe to
say that, according to the parametric ANCOVA, the null
hypothesis of group (program type) equality can be
rejected, and that significant group differences do exist.
Table 22 includes the ANCOVA table for the parametric
model.
The analysis of covariance provides an adjustment of
the dependent measure means per group, which allows an
adjusted comparison of group change. The means for the
Follow Through and Bilingual groups were slightly increased
through adjustment, and the means for groups with both
programs and neither program were reduced substantially,
especially the group having neither program. Table 23
delineates the observed and the adjusted dependent measure
means overall and per program type.
also included.
Covariate means are
An undefined pairwise post hoc multiple comparison
procedure was supplied by SAS to assess where differences
in program types lie. While results indicated that the
group with both programs was significantly different from
both Follow Through and Bilingual, it was not specified
whether a comparison-wise or an overall error rate was
125
assumed. The Bilingual group and the group with neither
program were close to being significantly different. Table
24 provides the matrix that delineates differences in
program types.
126
Table 22
Parametric Analysis of Covariance Table
Source of variation SS OF
WITHIN CELLS 28414.724 282
MS
100.761
F Sig of F
Regression
CONSTANT
TYPE
Note. * E < .as.
56506.088
10516.555
852.143
1 56506.088 560.791 .000
1 10516.555 104.371 .000
3 284.048 2.819 .039 *
127
Table 23
Observed and Adjusted Means for the Parametric Analysis of
Covariance
Cell Type Observed Mean SD Adjusted Mean
Dependent Measure
1 Follow Through 32.708 17.431 35.014
2 Bilingual 30.478 16.005 33.605
3 Both Programs 42.708 15.052 39.839
4 Neither Program 45.316 19.350 37.196
Population 35.512 18.174
covariate
1 Follow Through 25.788 18.149
2 Bilingual 24.797 16.177
3 Both Programs 32.042 16.050
4 Neither Program 38.386 15.800
Population 28.575 17.771
128
Table 24
Matrix Delineating Specific Group Differences by Post Hoc
Analysis for the Parametric Model
Program Types
Follow Through
Bilingual
Both
Note. * l2 < • 0 5.
Bilingual
.3425
Both
.0315 *
.0096 *
Neither
.1851
.0543
.2821
129
The Covariate-Corrected ANCOVA.
A covariate correction on the X scores provides a
parametric analysis of covariance model which gives quite
different results from the parametric model described
subsequent to this analysis. The covariate-corrected
ANCOVA indicates that there is no difference in Y scores
among different program types, £=.227. Table 25 provides
the ANCOVA table for the covariate-corrected analysis of
covariance.
While the observed means have remained the same in
this analysis as with the first model, adjusted Y scores
are quite different because of covariate adjustment. With
the covariate correction, adjusted means have increased for
the Follow Through and Bilingual program types and
decreased for the group with both programs and the group
with neither program, to the point where there is no longer
a statistically significant difference in means. Table 26
delineates observed and adjusted dependent measure means
per program type. Covariate means are also included.
Because significant group differences were not found,
it was not necessary to perform a post hoc multiple
comparison procedure.
130
Table 25
Covariate-corrected Parametric Analysis of Covariance Table
Source of Variation SS DF
WITHIN CELLS 28414.723 282
MS
100.761
F Sig of F
Regression
CONSTANT
TYPE
Note. * J2 > • 0 5 .
56506.089
2759.995
439.659
1 56506.089 560.791 .000
1 2759.995 27.391 .000
3 146.553 1.454 .227 *
131
Table 26
Observed and Adjusted Means for the Covariate-Corrected
Parametric Analysis of Covariance
Cell Type Observed Mean SD Adjusted Mean
Dependent Measure
1 Follow Through 32.708 17.431 35.487
2 Bilingual 30.478 16.005 34.245
3 Both Programs 42.708 15.052 39.252
4 Neither Program 45.316 19.350 35.533
Population 35.512 18.174
Covariate
1 Follow Through 25.788 15.064
2 Bilingual 24.797 13.427
3 Both Programs 32.042 13.322
4 Neither Program 38.386 13.114
Population 28.575 15.036
132
The Rank ANCOVA.
After the X and the Y scores were both individually
ranked, an analysis of covariance was performed. Results
indicate a statistically significant difference among
program types, £=.010. Table 27 provides the analysis of
covariance table for the rank ANCOVA.
For this analysis, observed means represent the group
means of the ranked Y scores. The adjusted means for the
ranked
types
Y scores have increased substantially for program
Follow Through and Bilingual, and decreased
substantially for the other two program types. Table 28
specifies observed and adjusted dependent measure means for
the rank analysis of covariance. Covariate means are also
included.
Unspecified pairwise post hoc multiple comparison
procedures supplied by SAS indicated that the Bilingual
group was different from both the group with both programs
and the group with neither program.
Through was different from the group
In addition, Follow
with both programs.
Table 29 delineates post hoc comparisons between groups.
133
Table 27
Rank Analysis of Covariance Table
Source of Variation SS DF
WITHIN CELLS 692455.319 282
MS
2455.515
F Sig of F
Regression
CONSTANT
TYPE
1064360.171
71867.477
28152.879
Note. * E < .05.
1 1064360.17 433.457 .ooo 1 71867.477 29.268 .000
3 9384.293 3.822 .010 *
134
Table 28
Observed and Adjusted Means of the Rank Analysis of
Covariance
Cell Type Observed Mean SD Adjusted Mean
Dependent Measure
1 Follow Through 131.347 82.182 141. 682
2 Bilingual 119.109 74.273 133.178
3 Both Program 185.021 74.406 171. 023
4 Neither Program 187.272 77.439 151. 293
Population 144.000 82.965
Covariate
1 Follow Through 130.602 83.924
2 Bilingual 125.761 78.534
3 Both Programs 162.146 79.727
4 Neither Program 190.640 68.814
Population 144.000 82.954
135
Table 29
Matrix Delineating Specific Group Differences by Post Hoc
Analysis for the Rank Model
Program Types
Follow Through
Bilingual
Both
Note. * E < .05.
Bilingual
.2462
Both
.0082 *
.0015 *
Neither
.2376
.0497 *
.1042
136
The Covariate-Corrected Rank ANCOVA.
For this analysis, the covariate-corrected X scores
were ranked, and the analysis of covariance was performed
on those ranked scores along with the ranked Y scores. The
covariate-corrected rank analysis of covariance indicates
near group differences, E=.069. With a researcher-selected
alpha level of E=.05, statistically significant group
differences are close but do not exist. Table 30 provides
the analysis of covariance table for the covariate-
corrected rank analysis of covariance.
While adjusted means for this analysis appear to
indicate quite different program types, statistically they
are not close enough to be significantly different. Table
31 delineates observed and adjusted dependent measure means
for the covariate-corrected rank analysis of covariance.
covariate means are also included.
Even though ANCOVA results were not statistically
significant, because of near significance, pairwise post
hoc tests supplied by SAS were acknowledged. Results
indicated that the group with both programs was different
from both Follow Through and Bilingual. The group with
both program types and the group with neither program were
close to being significantly different. Table 32 provides
the matrix that delineates differences in program types.
137
Table 30
Covariate-Corrected Rank Analysis of Covariance Table
Source of Variation SS
WITHIN CELLS 692021.548
Regression 1064793.942
CONSTANT
TYPE
Note. * E > .05.
61806.622
17576.879
DF MS
282 2453.977
1 1064793.94
1 61806.622
3 5858.960
F Sig of F
433.905 .000
25.186 .000
2.388 .069 *
138
Table 31
Observed and Adjusted Dependent Measure Means for the
covariate-Corrected Rank Analysis of Variance
Cell Type Observed Mean SD Adjusted Mean
Dependent Measure
1 Follow Through 131.347 82.182 143.107
2 Bilingual 119.109 74.273 136.609
3 Both Programs 185.021 74.406 168.206
4 Neither Program 187.272 77.439 144.902
Population 144.000 82.965
covariate
1 Follow Through 129.022 82.677
2 Bilingual 121.710 78.236
3 Both Programs 165.417 78.744
4 Neither Program 197.965 65.313
Population 144.000 82.981
139
Table 32
Matrix Delineating Specific Group Differences by Post Hoc
Analysis for the Covariate-Corrected Rank Model
Program Types
Follow Through
Bilingual
Both
Note. * E < .05.
Bilingual
.3754
Both
.0238 *
.0081 *
Neither
.8274
.3744
.0554
140
Robust Regression
The Parametric Analysis of Covariance.
To determine the impact of outlier-induced normality,
robust regression was completed. After four iterations,
the weights stabilized, and regression estimates were
determined for the full model, which includes the covariate
and the three qualitative grouping variables, and for the
reduced model, which includes only the covariate. A robust
F-like statistic of 6.01 (3, 282) and a probability level
of E .0006 were obtained.
Subsequently, 19 outliers identified by the fourth
weighting were deleted from the population data set.
Relative to group size, the greatest proportion of outliers
were deleted from the group with neither program type. Six
(4%) outliers were deleted from the Follow Through group,
six (9%) from the Bilingual group, and seven (13%) from the
group with neither program type. The parametric ANCOVA was
completed on the outlier-deleted data. Results indicated
an F value of 3.87 (3, 263) and a probability of E=.0098.
The ANCOVA Table for the parametric ANCOVA with
observations deleted is on Table 33. Because the
subsequent ANCOVAS were completed on SAS, the ANCOVA table
differs from those reported earlier. The probability level
of interest is that associated with the variable program
type under the category TYPE III SS.
The adjusted means for the analysis are found on Table
141
34. Adjusted means appear to be similar for the Follow
Through and Bilingual groups, and higher for the group with
both program types.
Pairwise post hoc multiple comparison procedures
supplied by SAS indicated that the group with both program
types was different from the Follow Through and the
Bilingual groups. In addition, the group with neither
program was different from the Bilingual group. Table 35
delineates post hoc comparisons between groups.
142
Table 33
ANCOVA Table for the Parametric Model with Outliers Deleted
SOURCE DF
MODEL 4
ERROR 263
CORRECTED TOTAL
SOURCE
SCHOOL
PREMATH
SOURCE
SCHOOL
PREMATH
DF
3
1
DF
3
1
Note. *E < .as.
SUM OF SQUARES
60991.30225159
18199.42536035
79190.72761194
TYPE I SS
8419.55473117
52571. 74752042
TYPE III SS
803.08955516
52571. 74752042
MEAN SQUARE
15247.82556290
69.19933597
F VALUE
40.56
759.71
F VALUE
3.87
759.71
F VALUE
220.35
PR > F
.0001
PR > F
.0001
.0001
PR > F
.0098 *
.0001
143
Table 34
Adjusted Dependent Measure Means for the Parametric ANCOVA
with Outliers Deleted
Cell Type Adjusted Mean
Dependent Measure
1 Follow Through 34.5465851
2 Bilingual 33.4658800
3 Both Programs 39.6967733
4 Neither Program 36.6864870
144
Table 35
Matrix Delineating Specific Group Differences by Post Hoc
Analysis for the Parametric ANCOVA with Outliers Deleted
Program Types
Follow Through
Bilingual
Both
Note. * E < .as.
Bilingual
.3981
Both
.0059 *
.0021 *
Neither
.1353
.0492 *
.1478
145
The Covariate-Corrected Parametric Analysis of
Covariance.
An F-like robust statistic of 4.93 (3, 282) with a
probability level of E = .0023 were obtained from the
robust regression after four iterations. Subsequently, 13
outliers identified by the fourth weighting were deleted
from the original population data set. As with the
parametric ANCOVA, relative to group size, the greatest
percentage of concentration of the outliers was in the
group with neither program type. Six (4%) outliers were
deleted from the Follow Through group, three (4%) from the
Bilingual group, one (4%) from the group with both program
types, and three (5%) from the group with neither program.
The covariate-corrected parametric ANCOVA was completed on
the outlier-deleted data. Results indicated an F value of
3.34 (3, 269) and a probability of E=.0198. The ANCOVA
table for the covariate-corrected parametric ANCOVA with
outliers deleted is on Table 36.
The adjusted means for the analysis are found on Table
37. There appears to be a discrepancy in adjusted means
between the group with both program types and the other
three groups.
Pairwise post hoc multiple comparison procedures
supplied by SAS indicated that the group with both program
types was different from each of the other three groups.
Table 38 delineates post hoc comparisons between groups.
146
Table 36
ANCOVA Table for the Covariate-Corrected Parametric Model
with Outliers Deleted
SOURCE DF
MODEL 4
ERROR 269
CORRECTED TOTAL
SOURCE DF
SCHOOL 3
CCPREMAT 1
SOURCE DF
SCHOOL 3
CCPREMAT 1
Note. *E. < .05.
SUM OF SQUARES
45363.77068857
24611.99573479
69975.76642336
TYPE I SS
11874.87615269
33488.89453588
TYPE III SS
917.27490649
33488.89453588
MEAN SQUARE
11340.94267214
91.49440794
F VALUE
43.26
366.02
F VALUE
3.34
366.02
F VALUE
123.95
PR > F
.0001
PR > F
.0001
.0001
PR > F
.0198 *
.0001
147
Table 37
Adjusted Dependent Measure Means for the Covariate-
Corrected Parametric ANCOVA with Outliers Deleted
Cell
1
2
3
4
Type
Follow Through
Bilingual
Both Programs
Neither Program
Adjusted Mean
Dependent Measure
33.6731691
31.8910625
39.2323457
34.9937180
148
Table 38
Matrix Delineating Specific Group Differences by Post Hoc
Analysis for the Covariate-Corrected Parametric ANCOVA with
Outliers Deleted
Program Types
Follow Through
Bilingual
Both
Note. * E < .05.
Bilingual
.2183
Both
.0121 *
.0021 *
Neither
·• 4321
.1006
.0777
149
Sample Results
Monte Carlo Samples and the ANCOVA
In order to complete the four ANCOVAS on 100 samples,
a table of 100 random numbers was generated by BASIC (see
Appendix A). These 100 random numbers served as the seeds
for data generation. One hundred random samples with
replacement were generated from the parent population by
SPSSX. Sampling consisted of 100 different selections of
one-third of the observations in each program type, with
each sample containing 95 observations. Both parametric
and nonparametric ANCOVAS were then completed on each of
the 100 samples.
Probability levels for each of the 100 samples over
each of the four ANCOVA models were logged and graphed. An
interval containing 68% of the sample probability levels
(typically plus and minus one standard deviation from the
mean) was determined for each of the four models. Figure
10 depicts each of the probability intervals. End points,
interval size, interval probability level mean, and
population parameters are defined.
.010
.039 p
p
150
PARAMETRIC
0--------------------------0 .107 .667
COVARIATE-CORRECTED PARAMETRIC
.227 p
0-----------------------0 .236 .752
RANK
0-----------------------0 .059 .577
.069 p
o-------o
COVARIATE-CORRECTED RANK
.ooo .169
0 .1 .2 .3 .4 • 5 .6 .7 .a
Figure 10. Probability level intervals and true parent
population probability levels (P) for each of the
four ANCOVA models.
151
Probability Level Interval Description
As indicated on Figure 10, the interval for the
parametric ANCOVA is the widest interval. It ranges from
.107 to .667, an interval size of .560. One-half the
interval is equal to one standard deviation from the
interval mean. One-half of the interval equals .285. The
interval probability level mean is .392, and the population
parameter for the parametric ANCOVA is .039. Figure 11
displays a graph of the frequency distribution of the
probability levels for the parametric ANCOVA.
The interval for the covariate-corrected parametric
ANCOVA ranges from .236 to .752, giving an interval size of
.516. As with the parametric ANCOVA, one-half the interval
is equal to one standard deviation from the interval mean.
One-half the interval is .258. The interval probability
level mean is .494, and the population parameter for the
covariate-corrected parametric ANCOVA is .227. Figure 12
displays a graph of the frequency distribution of the
probability levels for the covariate-corrected parametric
ANCOVA.
152
Parametric ANCOVA rID1'B = .560
.667
i = .382 tl•
.107
0 2 J 5 I 1 I 9 10 u
FREQUENCY
Figure 11. Frequency distribution of the sample
probability levels for the parametric ANCOVA.
153
Covariate-Corrected Parametric ANCOVA JfIDTB = .616
.752
i = .494 tl•
.238
population p-value = .2272
Figure 12. Frequency
FREQUENCY
distribution of the sample
probability levels for the covariate-corrected parametric
ANCOVA.
154
The rank ANCOVA interval ranges from .059 to .577,
with an interval size of .517, very close to that of the
covariate-corrected parametric ANCOVA. Like the parametric
ANCOVAS, one-half the interval is equal to one standard
deviation from the interval mean. One-half the interval is
equal to .259. The interval probability level mean is
.318, and the population parameter for the rank ANCOVA is
.010. Figure 13 displays a graph of the frequency
distribution of the probability levels for the rank ANCOVA.
The covariate-corrected rank ANCOVA interval, the
smallest interval,
of .169. Unlike
interval is not
ranges from O to .169, an interval size
the other four models, one-half of the
equal to one standard from the interval
mean; one standard deviation left of the interval mean is
-.073, and there are obviously no probability levels less
than 0. While one standard deviation equals .121, one half
the interval equals .085. The interval mean probability
level is .048, and the population parameter is .069.
Figure 14 displays a graph of the frequency distribution of
the probability levels for the covariate-corrected rank
ANCOVA.
155
Rank ANCOVA 'fl'IDTH= .518
o.m
.577
X = .318 tis
.059
population p-valu11 = . 0100
FREQUENCY
Figure 13. Frequency distribution of the sample
probability levels for the rank ANCOVA.
156
Covariate-Corrected Rank ANCOVA
.189
population p-vatue_ = • 0692 T x = .048 l
o.o
0.!175 0.!125 0.175 0.1125 0.775 0. 72!5 0.&75 0.625 0.575
Figure 14. Frequency
'ffIDTll =. 169
FREQUENCY
distribution of the sample
probability levels for the covariate-corrected rank ANCOVA.
157
General Questions
To gain a thorough understanding of how specific
ANCOVA models operate in the current investigation, four
general questions are addressed and answered. The initial
comparison is made between the parametric ANCOVA and the
parametric ANCOVA with a covariate correction. Next the
rank transform ANCOVA is compared to the rank transform
ANCOVA with a covariate correction. Thirdly, comparisons
are made among the parametric and the rank analyses. Final
comparisons are then made among sample intervals and
population parameters. The general questions are stated,
with responses following.
1. How do the Parametric ANCOVA and the Parametric ANCOVA
with a Covariate Correction compare?
Probabilities of a Type I error are different for each
of the two models. At a researcher-set alpha level of
p=.05, the parametric model rejects the null hypothesis,
p=.039, but the covariate-corrected parametric model fails
to reject the null, p=.227. There is a substantial
discrepancy between the two probabilities of a Type I
error.
Adjusted means are different for the parametric and
covariate-corrected parametric ANCOVAS (see Tables 22 and
25). With covariate correction, there is a .473 increase
158
for Follow Through and a .640 increase for Bilingual. On
the other hand, there is a .587 decrease for the group with
both program types and a 1.663 decrease for the group with
neither program. While adjusted means appear to be only
slightly different with the covariate-correction, the
adjustment is substantial enough to indicate group equality
for the different program type means.
The between-group regression equations for the two
models are somewhat different. It is difficult to make
comparisons between the two regression equations because
most of the coefficients have been determined as blue or
zero. The X'X matrix was deemed singular by SAS, and a
generalized inverse was employed to solve the normal
equations. The regression estimates represent only one of
many possible solutions to the normal equations. Only the
covariate estimate (the final coefficient in each equation)
in not biased. The other coefficients do not estimate the
parameter, but are blue for some linear combination of
parameters or are zero. The regression equations for the
parametric and the covariate-corrected parametric,
respectively, are as follows:
Y=l3.54687651 - 2.18172439 Xl - 3.5911415 X2 +
2.64319886 X3 + .82761793
Y= 7.04000472 - .04627070 Xl - 1.2876714 X2 +
3.71860361 X3 + .99713004
Each of the covariate coefficients serves to slightly
159
increase the expected value for the dependent when all
other variables are held constant.
The assumption of homogeneous slopes holds for each
model (see Table 12), and is addressed by the identical F
statistics. Linearity can be assumed for both forms of
analysis (see Figures 1 through 4). Pearson and Spearman
correlation coefficients between the covariate and the
dependent measure are the same for each model, and indicate
linearity (see Table 9).
Normality of the X scores is not an assumption
necessary for the validity of the ANCOVA. Nevertheless,the
covariate-corrected X scores were assessed for normality
and were found to be nonnormal. Because the distribution
was not normal before correction, it remained nonnormal
afterward. However, the covariate correction did decrease
the variance of the covariate overall and for each program
type (see Tables 23 and 26). Marginal distributions of the
covariate are unequal, with or without the covariate
correction (see Tables 13 and 14).
2. With the parent population, how does the rank
transform ANCOVA compare to the rank transform ANCOVA with
a covariate correction?
Probabilities of a Type I error are slightly different
for each of the two models. At a researcher-set alpha
level of 2=.05, the rank model rejects the null hypothesis,
160
E=.010, but the covariate-corrected rank model is only
close to being significant, E=.069. While the discrepancy
between the two Type I errors is not great, with the
covariate-corrected rank model, its range falls from the
rejection into the acceptance region.
Adjusted ranked means are quite different for the rank
and the covariate-corrected rank models (see Tables 28 and
31). With the ranked covariate-correction, there is a
1.425 increase for Follow Through and a 3.431 increase for
Bilingual. On the other hand, there is a 2.817 decrease
for the group with both program types and a 6.391 decrease
for the group with neither program. Rank means have been
adjusted substantially enough by covariate-correction to
indicate only near significant group differences, p=.069,
by program types instead of clear group differences,
E=.010.
The between-group regression equations for the two
models are somewhat different, but as with the parametric
models, represent only one of many possible solutions to
the normal equations. The regression equations for the
rank and the covariate-corrected rank, respectively, are as
follows:
Y=40.21110412 - 9.61150649 Xl 18.11490994 X2 +
19.72970274 X3 + .77140451
Y=31.84275362 1.79564264 Xl - 8.29294941 X2 +
23.30366947 X3 + .78513497
161
Each of the covariate coefficients (the final
coefficient in each equation) serves to
the expected value for the dependent
variables are held constant.
slightly increase
when all other
The assumption of homogenous slopes holds for each
model (see Table 12), and linearity can be assumed (see
Figures 5 through 8). Pearson and Spearman correlation
coefficients between the covariate and the dependent
measure by each program type indicate moderate to strong
linearity (see Table 9).
As with the parametric models, the covariate
correction serves to slightly decrease the variance of the
covariate overall and for each program type (see Tables 28
and 31). Marginal distributions of the covariate are
unequal, with or without the covariate correction (see
Tables 15 and 16).
3. With the parent population, how do the rank
transformation procedures compare with those of the
parametric ANCOVAS ?
Probabilities of a Type I error are different for each
of the four models. At a researcher-set alpha level of
£=.05, the parametric and rank models reject the null
hypothesis, £=.039 and £=.010, respectively. On the other
hand, the covariate-corrected parametric and covariate-
corrected rank models fail to reject the null of
162
inequality, E=.227 and E=.069, respectively. While the
probability levels for the parametric, rank, and covariate-
corrected rank models are similar, there is a large
discrepancy between those probability levels and that of
the covariate-corrected parametric model.
Because of the difference in metric, observed and
adjusted means for the two parametric and the two rank
analyses cannot be compared. The between-group regression
equations for the four models are different, with slightly
smaller coefficients for the covariates. Assumption
diagnosis is consistent across all four models.
4. With samEles generated by Monte Carlo simulation, how
do both the Earametric and the nonEarametric ANCOVAS
comEare?
There are discrepancies in the sizes of the intervals
containing 68% of the sample probability levels for the
four ANCOVA models. The intervals for the parametric,
covariate-corrected parametric, and rank ANCOVAS are all
quite similar, with interval width being .560, .516, and
.518, respectively. The intervals for the covariate-
corrected parametric ANCOVA and the rank ANCOVA are almost
equal in width. However, the interval for the covariate-
corrected rank ANCOVA is substantially tighter than the
other three intervals. Its interval width is .169.
While the widths of the first three ANCOVA models are
163
similar in width, they are not similar in location. The
probability level interval means, differing for the
parametric, covariate-corrected, and rank ANCOVAS, are
.392, .494, and .318, respectively. As was true for
population values, .039 and .010, the parametric and rank
ANCOVA probability level interval means are closer than
those of other models. The covariate-corrected parametric
population parameter, .227, is the highest level, as is the
interval mean for that model. The probability level
interval mean for the covariate-corrected rank ANCOVA is
.048, by far the lowest interval mean. The population
parameter, .069, is quite close to the interval mean. In
addition, the probability interval for the covariate-
corrected rank ANCOVA is the only one of the four intervals
that contains the population parameter. Table 39
summarizes the population parameters, means for the
probability level intervals, interval range and width, and
the location of the population parameter in relation to the
interval.
164
Table 39
Summary of Population and Sample Probability Levels and
Intervals for the Four ANCOVA Models
Model
PAR
PARCC
RANK
RANK CC
Population
Parameter
.039
.227
.010
.069
Interval
Means
.392
.494
.318
.048
Interval
Range &
Width
.107 - .667
.560
.236 - .752
.516
.059 - .577
.518
.000 - .169
.169
Location of
Population
Parameter
outside
interval
outside
interval
outside
interval
inside
interval
Note. ANCOVAS: (PAR) Parametric ANCOVA, (PARCC) Covariate-
Corrected Parametric ANCOVA, (RANK) Rank ANCOVA, and
(RANKCC) Covariate-Corrected Rank ANCOVA.
165
Research Questions
The research questions upon which the current
investigation has been based will now be addressed, with
the intention of determining a theoretical basis for a
"best" ANCOVA model.
1. How do the results of data exploration on the
population assist in delineating a preferred ANCOVA model?
There are nine assumptions that must be met in order
for the parametric analysis of covariance to be valid.
With nonequivalent control group research, three out of the
nine assumptions are automatically
assumptions include randomization,
not met. Those three
independence of the
covariate and the treatment, and error free fixed covariate
values. Of the remaining six assumptions, one is not an
issue in the current investigation, that of fixed treatment
levels. Four of the other five assumptions have been met
in the current study. They include parallel slopes (see
Table 12); linearity between the covariate and the
dependent measure (see Figures 1 through 8 and Table 9);
homogeneity of variance of the conditional Y scores (see
Table 11); and the covariate and dependent being measured
on at least an interval scale. The remaining assumption,
normality of the conditional Y scores, has not been met.
Mild and extreme outliers impact the distribution and
166
prevent the assumption of normality from being met (see
Figure 9 and Table 10). With four assumptions not being
met for the parametric ANCOVA, that form of analysis is
questionable.
To aid in error reduction in the covariate and
compensate for randomization and independence of the
covariate and the treatment, the covariate-corrected
parametric ANCOVA was completed. This model serves to
correct for those three assumptions not being met with the
parametric ANCOVA. Nevertheless, normality is an
assumption that is not met in the current investigation and
would therefore invalidate the results of any parametric . ANCOVA.
There are five assumptions that must be met for the
rank ANCOVA to be valid. As discussed previously,
randomization has not been met for this or any model.
Three of the four remaining assumptions have been met.
They include a monotonic relation between the covariate and
the dependent (see Figures 1 through 8), an equal degree of
monotonicity for each population, and the covariate and
dependent being measured on at least an ordinal or
dichotomous scale. The last assumption, identical marginal
distributions of the covariate, has not been met in the
current study (see Tables 13 through 16). Because this
assumption does not hold, the results of the rank ANCOVA is
questionable.
167
To compensate for the lack of randomization, a
covariate-corrected rank ANCOVA was completed.
Nevertheless, the failure to assume identical marginal
distributions of the covariate shadows any rank ANCOVA.
Because basic assumptions have not been met in the
current investigation for both the parametric and rank
ANCOVAS, the decision to choose a single "best" analysis is
not clear cut. Both covariate-corrected analyses do
compensate for failure to meet some of the
with model selection that is based
assumptions, so
upon assumption
violations, either the
the covariate-corrected
covariate-corrected parametric or
rank ANCOVA should be pref erred
over the parametric and rank analyses.
The literature does not discuss failure to meet the
assumptions of normality of conditional Y scores and
identical marginal distributions of the covariate in
nonequivalent group research, so a sound basis upon which
selection of either covariate-corrected analysis can be
made does not yet exist. Because ranking compensates for
the nonnormal Y scores, the covariate-corrected rank model
may be more precise than the covariate-corrected parametric
model. However, because there appears to be no alternative
to aid in error reduction due to marginal distributions of
the covariate being unequal, the covariate-corrected rank
ANCOVA is not a panacea.
Table 40 summarizes the parametric and rank
168
assumptions, and specifies whether or not each is met in
the current investigation.
169
Table 40
Analysis of the Assumptions Underlying the Parametric and
Rank Analysis of Covariance
Assumptions
Randomization
Parallel Slopes
Independence of Covariate and and Treatment
Error Free Fixed Covariate Values
Linearity between the Covariate and the Dependent Measure
Monotonic Relation between the Covariate and the Dependent
Equal Degree of Monotonicity for Each Population
Normality of Conditional Y Scores (and Error Terms)
Homogeneity of Variance of Conditional Y Scores
Fixed Treatment Levels
Covariate and Dependent Measured on at least an Ordinal or Dichotomous Scale
Covariate and Dependent Measured on at least an Interval Scale
Identical Marginal Distributions of the Covariate
Par
x
x
x
x
x
x
x
x
x
Non
x
x
x
x
x
Model
Assumption Met
no
yes
no
no
yes
yes
yes
no
yes
n/a
yes
yes
no
170
2. Is the reliability correction on the covariate strong
enough for error reduction?
A pretest posttest within-group correlation
coefficient of .83 was used as the reliability correction
on the covariate. A correlation between .70 and .90
indicates a high positive linear relationship between the
two variables being measured (Hinkle, Wiersma, & Jurs,
1979). Although some researchers (Marks & Martin, 1973)
write of the importance of highly reliable test forms and
advocate reliabilities in excess of .85, other researchers
(Porter, 1967, and Dicostanzo & Eichelberger, 1980)
indicate that a minimum reliability of less than .85 is a
powerful enough estimate.
a reliability estimate
respectively.
Those researchers advocate using
of at least .70 and .80,
Because the reliability correction used in the current
investigation indicates a high positive linear relationship
between the covariate and the dependent measure, it is safe
to assume that it provides adequate covariate correction
and is effective in error reduction. As a result, either
the covariate-corrected parametric or the covariate-
corrected rank ANCOVA would be preferred models over the
parametric or rank ANCOVA.
171
3. Which sample model shows the tightest probability
interval?
The covariate-corrected rank ANCOVA shows the tightest
probability interval. Its width is .169, as compared to
intervals of .560, .516, and .518 for the other models (see
Table 32). The tightest interval serves to delineate the
model with the greatest convergence of probability levels,
and, theoretically, is the most stable model.
4. Are population parameters contained in the sample
probability intervals?
The sample probability interval of the covariate
-corrected rank ANCOVA is the only interval of the four
ANCOVA models that contains the true population parameter
(see Table 33). In addition, the covariate-corrected rank
model is the only one where the population parameter, .069,
and the interval mean, .048, are close in value.
5. Do the robust regressions indicate a preferred model?
Robust regression indicated that outliers in the data
set impacted the regression equations for both the
parametric and the covariate-corrected parametric models.
Subsequent ANCOVAS completed with outliers identified by
weighting and removed from the data set indicated
differences in probability levels for both initial and
subsequent ANCOVAS. For the original parametric ANCOVA,
172
the probability level was E = .039, and for the outlier-
deleted analysis it was E = .0059, indicating a very large
discrepancy between analyses. For the original covariate-
corrected parametric ANCOVA, the probability level was E =
.2272, and for the outlier-deleted analysis was E = .0489,
indicating a
analyses.
difference in significance between the
These findings clearly indicate that outlier-induced
nonnormality has impacted the validity of both parametric
models. Therefore, feasibility of either parametric test
is questionable for best model selection.
6. In light of assumption violations, the reliability of
the covariate correction, probability interval size, true
parent POEulation parameters, and the results of robust
regression, what is the "best" overall model for this
noneguivalent control group research?
As indicated in response to research questions one and
two, assumption violations and the reliability of the
covariate both indicate that a covariate-corrected ANCOVA
should be implemented. Research question five questions
the selection of both parametric ANCOVAS. Relative to
questions one, two, and five, research questions three and
four provide conclusive evidence to assist in delineating
the "best" model, the covariate-corrected rank analysis of
covariance.
173
summary
This chapter has presented the results of the current
investigation in light of data exploration of the
population, four analysis of covariance models completed on
the population and Monte Carlo samples, model comparison,
and five specific research questions leading to a sixth
focal question that serves to delineate a "best" ANCOVA
model. Chapter five will present a discussion of the
findings in
research.
relation to nonequivalent control group
Chapter Five
DISCUSSION
The current methodological investigation sought to
systematically and theoretically identify a best analysis
of covariance model to assess change in this nonequivalent
control group study. Parent population parameters were
delineated and Monte Carlo sample probability level
intervals were determined for each of four analysis of
covariance models. In light of population assumption
violations, reliability of the covariate correction, the
size/stability of the probability intervals, true parent
population parameters, and robust regression, the
covariate-corrected rank analysis of covariance was clearly
delineated as the best model.
Because of intrinsic bias and error that accompany
nonequivalent control group studies, investigators who must
rely on intact groups to answer their research questions
need to be made aware of new and innovative techniques for
analysis. While there has been no reporting in the
literature of the covariate-corrected rank analysis of
covariance or of Monte Carlo simulation in relation to
model selection and the ANCOVA for nonequivalent control
groups, results of the current study now provide a viable
and timely option. Nonequivalent control group research
can be a sticky situation, but, for lack of alternative
174
175
methodologies suitable for the
remains a frequently utilized
study of intact groups, it
research design. Coupled
analysis of covariance and its with the temperamental
numerous underlying assumptions,
stickier. Nevertheless, given
the sticky becomes even
intact groups without
possible randomization, an integration of the nonequivalent
control group research design and the ANCOVA produces a
viable means of analysis. Under such tenuous conditions,
it is crucial for researchers to provide the least biased
estimates possible to address their research questions.
Results of the current study offer such a strategy.
Issues Relating to the Study
The current investigation provides not only new
methods of analysis, the covariate-corrected rank ANCOVA,
robust regression, and Monte Carlo simulation and ANCOVA
model selection for nonequivalent control groups, but also
presents a systematic process for determining the best
model for nonequivalent control group research. As
indicated previously, the exact model specification for
this study was based upon population assumptions, the
reliability of the covariate correction, the size/stability
of the probability level intervals, true parent population
parameters, and the results of robust regression.
Obviously, each research study has its own set of
176
questions, variables, and data distributions. Assumption
violations on the parent population of one study will not
be entirely the same for another investigation. Due to
assumption violations, a parametric or rank ANCOVA may be
clearly indicated, or neither may be obvious. Reliability
estimates may differ both in type and in power. While a low
estimate would negate use of a covariate-corrected model, a
high reliability would indicate covariate correction should
be employed. Sample probability level interval widths may
blatantly differ, or intervals may all be of similar size.
Parent population parameters may or may not be contained in
any or all intervals. Robust regression estimates will
differ depending on outliers in the data set, the number of
iterations necessary for stability, and the derived
weights. Sometimes it is apparent that observations in the
data set have had no impact on regression estimates. Other
times it is obvious that influential observations have
impacted the regression equation. Each of the criteria
must be considered and assessed before model specification
can be defined. In any event, the covariate-corrected rank
ANCOVA may not be the best model in every situation, but
its inclusion as a model option is crucial in light of its
relation to criteria assessed in this study and the
proposed methodological schemata.
one hundred samples were obtained and analyzed
through Monte Carlo simulation. Of the four models
177
studied, only the covariate-corrected rank ANCOVA indicated
stability of the probability levels through convergence of
the probability levels in the model intervals. As
differences in varied research studies produce differences
in results, so may be the case with the number of generated
samples. While fewer samples than 100 are not recommended
by this investigator, there is always a possibility that a
larger number of samples may indicate differences in
probability level interval width and inclusion of true
parent parameters within the intervals. Results of the
current study must be tempered with the realization that
changes in design may produce changes in findings.
Further Research
It is crucial for results based upon new models and
new techniques to be replicated in further investigations.
Perhaps the covariate-corrected rank ANCOVA may arise
consistently as the model of choice for nonequivalent
control group research. Perhaps it may seldom reappear.
Nevertheless, additional studies based upon the four ANCOVA
models must be completed and results compared.
Assumptions underlying the parametric and rank
analyses of covariance should be reported more loudly in
the literature, and alternative methods of assessment
should be studied in order to better indicate violations.
178
Analyses in addition to the Kruskal-Wallis need to be
considered to assess equality of marginal distributions of
the covariate; the Birnbaum-Hall Test for three independent
samples could be adjusted to measure more than three
groups, and
studied.
the two-sided k-Sample Smirnov Test could be
Additional analyses to measure monotonicity
the covariate and the dependent variable may be between
researched. While several nonparametric tests to determine
normality exist (e.g., the Shapiro-Wilks, the Lilliefors,
and the Cramer von Mises), they may become unwieldy or
inaccurate with large samples. Alternatives to assess
normality of distributions should continuously be
investigated, along with specific types of nonnormality
(e.g., outliers, skew, kurtosis, etc.) in reference to Type
I error and power. All assumptions need to be studied in
light of nonequivalent control group research, and not just
in a research setting where randomization was possible.
The necessity for further research using Monte Carlo
simulation and model selection for the analysis of
covariance for nonequivalent control groups is obvious.
The statistical sampling technique provides a sound
theoretical basis for methodological decision making. In
addition to generating 100 samples through Monte Carlo
simulation, studies using 200 to 1000 samples may provide
even more precise estimates and should be completed.
Results comparing findings of 100 and 200 samples, for
179
example, may serve to indicate that 100 is adequate, or
that 200 may be even better.
In addition, bootstrapping, a robust subsampling
technique (Efron, 1979 & 1982; Efron & Gong, 1983;
Freedman, 1984; Freedman & Peters, 1984; and Peters &
Freedman, 1984), could be addressed and completed in
relation to the analysis of covariance and the inclusion of
a known parent population. To subsample with
bootstrapping, a large sample is first drawn from a
population, that sample data is copied many times (perhaps
a billion), and the copied data is then shuffled.
Subsequently, bootstrapped subsamples are be generated from
that large shuffled data set, analyses completed, and
intervals constructed based on a statistic of interest.
Interval comparisons of a statistic are then compared to
both sample and population parameters. Bootstrapped
intervals could be compared to intervals obtained through
Monte Carlo simulation to assess differences in the two
sampling techniques.
Robust regression is another area which should be
studied in relation to the ANCOVA. When results of robust
regression differ substantially from least squares
findings, weights should be examined and the model
specification reassessed. Some researchers believe that
robust regression should be used routinely when doing
regression analysis. Because regression analysis is a
180
statistical procedure that may be used to obtain ANCOVA
results, robust regression procedures should be completed
more regularly with nonequivalent control group studies,
especially those with suspected outliers in the data set.
As with the current investigation, the results of robust
regression may serve to more fully indicate the impact of
outlier-induced nonnormality on selection of a best
analysis of covariance model.
181
Sununary
A systematic and theoretical approach to model
specification for the analysis of covariance with
nonequivalent control groups has been delineated in the
current investigation. Chapter Five included a discussion
of the findings, issues relating to the study, and
recommendations for further research.
LITERATURE CITED
Abeyasekera, s. (1984). The desirability of covariance adjustments. Applied Statistics, 11(1), 33-37.
Aiken, L. (1981). Analysis of covariance or partial correlation: What is the question? Educational Research Quarterly, ~(2), 13-16.
Baldwin, L., Medley, D., & MacDougall, M. (1984). A comparison of analysis of covariance to within-class regression in the analysis of non-equivalent groups. Journal of Experimental Education, 52, 68-76.
Birch, J., & Myers, R. (1982). Robust Analysis of Covariance. Biometrics, 38, 699-713.
Bock, R. (1985). Multivariate statistical methods in behavioral research. Chicago: Scientific Software, Inc.
Bryk, A., & Weisberg, H. (1977). Use of the nonequivalent control group design when subjects are growing. Psychological Bulletin, 84(5), 950-962.
Buckland, s. T. (1984). Monte Carlo confidence intervals. Biometrics, 40, 811-817.
Burnett, T., & Barr, D. (1977). A nonparametric analogy of analysis of covariance. Educational and Psychological Measurement, 37, 341-348.
Campbell, D., & Erlebacher, A. (1970). How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful. In J. Hellmuth (Ed.), Compensatory education: A national debate. New York: Brunner/Mazel.
Cochran, W. (1957). Analysis of covariance: Its nature and uses. Biometrics, 261-281.
Cohen, J., & Cohen, P. (1983). Applied multiple regression correlation analysis for the behavioral sciences. LEA: Hillsdale, New Jersey.
Conover, w. (1980). Practical nonparametric statistics. New York: Wiley.
Conover, w., & Iman, R. (1982). Analysis of covariance using the rank transformation. Biometrics, 38, 715-724.
182
183
Cook, T., & Campbell, D. (1979). Quasi-experimentation. Chicago: Rand McNally College Publishing Co.
Cox, D., & Mccullagh, P. (1982). Some aspects of analysis of covariance. Biometrics, 2§_, 541-561.
Cronbach, L., & Furby, L. (1970). How we should measure "change"--or should we? Psychological Bulletin, 74(1), 68-80.
Daniel, w. (1978). Applied nonparametrics statistics. Boston: Houghton Mifflin.
Dicostanzo, J., & Eichelberger, R. (1980). Reporting ANCOVA results in evaluation settings. Evaluation Review, !{4), 419-450.
Diaconis, P., & Efron, B. (1983). Computer-intensive methods in statistics. Scientific American, 248, 116 -130.
Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, z, 1-26.
Efron, B. (1982). The jackknife, the bootstrap and other resampling plans. Philadelphia: Society for Industrial and Applied Mathematics.
Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37(1), 36-48. ~-
Elashoff, J. (1969). Analysis of covariance: A delicate instrument. American Educational Research Journal, i(3), 383-401.
Evans, s., & Anastasio, E. (1968). Misuse of analysis of covariance when treatment effect and covariate are confounded. Psychological Bulletin, 69(4), 225-234.
Fortune, J., & Hutson, B. (1984). Selecting models for measuring change when true experimental conditions do not exist. Journal of Educational Research, 77(4), 197 -206.
Freedman, D. (1984). On bootstrapping two-stage least-squares estimates in stationary linear models. The Annals of Statistics, 12(3), 827-842.
Freedman, D., & Peters, s. (1984). Bootstrapping a regression equation: Some empirical results. Journal of the American Statistical Association, 79(385), 97-106.
184
Games, P., Winkler, H., & Probert, D. (1972). Robust tests for homogeneity of variance. Educational and Psychological Measurement, 32, 887-909.
Glass, G., Peckham, P., & Sanders, J. (1972). Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research, 42(3), 237-288.
Griffey, D. (1982). Alternate approaches to analysis of covariance: Non-simultaneous and simultaneous inference. Research Quarterly for Exercise and Sport, 21(1), 20-26.
Hamilton, B. (1976). A Monte Carlo test of the robustness of parametric and nonparametric analysis of covariance against unequal regression slopes. Journal of the American Statistical Association, 71(356), 864-869.
Hendrix, L., Carter, M., & Scott, D. (1982). Covariance analyses with heterogeneity of slopes in fixed models. Biometrics, 1.§., 641-650.
Hinkle,D., Wiersma, W., & Jurs, s. (1979). Applied statistics for the behavioral sciences. Chicago: Rand McNally.
Hollander, M., & Wolfe, D. (1973). Nonparametric statistical methods. New York: Wiley.
Huck, s., Cormier, w., & Bounds, W. (1974). Reading statistics and research. New York: Harper and Row.
Huitema, B. (1980). The analysis of covariance and alternatives. New York: Wiley.
Karabinus, R. (1983). Nonequivalent group designs. Evaluation Review, 1(6). 841-849.
Kenny, D. (1975). A quasi-experimental approach to assessing treatment effects in the nonequivalent control group design. Psychological Bulletin, 82(3), 345-362.
Kenny, D. (1979). Correlation and causality. New York: Wiley.
Lawson, A. (1983). Rank analysis of covariance: Alternative approaches. The Statistician, 32, 331-337.
185
Levy, K. (1980). A Monte Carlo study of analysis of covariance under violations of the assumptions of normality and equal regression slopes. Educational and Psychological Measurement, 40, 835-840.
Linn, R., & Slinde, J. (1971). The determination of the significance of change between pre- and posttesting periods. Review of Educational Research, 47(1), 121-150.
Locascio, J., & Cordray, D. (1983). A re-analysis of Lord's paradox. Educational and Psychological Measurement, 43, 115-126.
Lord, F. (1960). Large sample covariance analysis when the control variable is fallible. American Statistical Association Journal, 55, 309-321.
Lord, F. (1962). Elementary models for measuring change. In c. w. Harris (Ed.), Problems in measuring change. Madison: The University of Wisconsin Press.
Lord, F. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 68, 304-305.
Lord, F. (1969). Statistical adjustments when comparing 'preexisting groups. Psychological Bulletin, 72(5),
336-337.
Magidson, J. (1977). Toward a causal model approach for adjusting for preexisting differences in the nonequivalent control group situation. Evaluation Quarterly, 1(3), 399-420.
Marascuilo, L., & Dagenais, F. (1979). Measuring differences among non-randomized groups: An epidemiological model for identifying successful school programs. Journal of Experimental Education, 48, 49-59.
Marks, E., & Martin, c. (1973). Further comments relating to the measurement of change. American Educational Research Journal, 10(3), 179-191.
Mcsweeney, M., & Katz, B. (1978). Nonparametric statistics: Use and non-use. Perceptual and Motor Skills, !(3), 1023-1032.
Mcsweeney, M., & Porter, A. (1971). Small sample properties of nonparametric index of response and rank analysis of covariance. Occasional paper no. 16, Office of Research Consultation, Michigan State University, East Lansing, Michigan.
186
Montgomery, D., & Peck., E. (1982). Introduction to linear regression analysis. New York: John Wiley & Sons.
Nesselroade, J., Stigler, S., & Baltes, P. (1980). Regression toward the mean and the study of change. Psychological Bulletin, ~(3), 622-637.
Number cruncher statistical system (1986). Kaysville, Utah: Dr. Jerry L. Hintze.
Olejnik, s., & Algina, J. (1985). A review of nonparametric alternatives to analysis of covariance. Evaluation Review, ~(1), 51-83.
Olejnik, S., & Algina, J. (1984). Parametric ANCOVA and the rank transform ANCOVA when the data are conditionally non-normal and heteroscedastic. Journal of Educational Statistics, ~(2), 129-149.
Olejnik, s. & Porter, A. (1978). Estimating treatment effects and precision for quasi-experiments assuming differential group and individual growth patterns. Paper presented at the Annual Meeting of the American Educational Research Association.
overall, J., & Woodward, J. (1977). Nonrandom assignment and the analysis of covariance. Psychological Bulletin, 84(3), 588-594.
Peters, S., & Freedman, D. (1984). Some notes on the bootstrap in regression problems. Journal of Business & Economic Statistics, l(4), 406-409.
Porter, A. (1967). The effects of using fallible variables in the analysis of covariance. Unpublished Ph.D. dissertation, Madison: University of Wisconsin.
Porter, A., & Mcsweeney, M. (1974). Comparison of three common experimental designs to improve statistical power when data violate parametric assumptions. Presented at the meeting of the American Educational Research Association.
Puri, M., & Sen, P. (1969). Analysis of covariance based on general rank scores. The Annals of Mathematical Statistics, 40 ( 2), 610.-618.
Quade, D. (1982). Nonparametric analysis of covariance by matching. Biometrics, 38, 597-611.
187
Richards, J. (1975). A simulation study of the use of change measures to compare educational programs. American Educational Research Journal, 12(3), 299-311.
Royeen, c. (1986). An exploration of parametric versus nonparametric statistics in occupational therapy clinical research. Unpublished Ph.D. dissertation. Blacksburg: Virginia Polytechnic Institute and State University.
Seaman, s., Algina, J., & Olejnik, s. (1985). Type I error probabilities and power of the rank and parametric ANCOVA procedures. Journal of Educational Statistics, 10(4), 345-367.
Seaver, w. (In Preparation). Nonparametric statistics. In c. Royeen (Ed.), Issues in clinical research: An analysis for occupational therapy, Thoroughfare, New Jersey: Slack, Inc.
Shirley, E. (1981). A distribution-free method for analysis of covariance based on ranked data. Applied Statistics, lQ(2), 158-162.
Statistical analysis system user's guide. (1982). Cary, North Carolina: SAS Institute, Inc.
Statistical package for the social sciences. (1986). New York: McGraw-Hill.
Stephens, M. (1974). EDF Statistics for Goodness of Fit and and Some Comparisons. Journal of the American Statistical Association, 69(347), 730-737.
Tuckman, B. (1978). Conducting educational research. New York: Harcourt Brace Jovanovich, Inc.
Urquhart, N. (1982). Adjustment in covariance when one factor affects the covariate. Biometrics, 38, 651-660.
Werts, c., & Linn, R. (1970). A General Linear Model for studying growth. Psychological Bulletin, 1.1(1), 17-22.
Wildt, A., & Ahtola, o. (1978). Analysis of covariance. Beverly Hills: Sage Publications.
Wu, Y. (1984). The effects of heterogeneous regression slopes on the robustness of two test statistics in the analysis of covariance. Educational and Psychological Measurement, 44, 647-663.
Appendix A
100 Random Numbers for Monte Carlo Sample Generation
188
189
Table A-1
100 Random Numbers for Monte Carlo Sample Generation
.73792050 .73568250 .75429860 .8502446 .54452330 .2028372
.94194570 .76996930 .58735110 .9183478 .51343150 .1514593
.40503930 .82880850 .94147310 .6390627 .87704350 .8583526
.76617090 .67497930 .48926450 .1117775 .98563260 .3273853
.84967350 .32766960 .69659890 .3685585 .45393710 .5815293
.98506840 .60300330 .70104200 .2388963 .06692255 .4455758
.15892030 .15847420 .67855040 .7496083 .08007455 .1452555
.71747390 .88503510 .16720250 .6536471 .13297280 .0665167
.58882210 .74059480 .06670988 .9315654 .24489690 .2687084
.24335540 .36866980 .28246050 .3593869 .61121260 .5954205
.86124180 .09594315 .23284260 .4866341 .16929030 .4767294
.42568910 .15247920 .68860060 .6182615 .15270390 .7666010
.71073750 .20834490 .66297140 .6442457 .10246130 .2091987
.38233390 .57038670 .32135270 .8105811 .05401647 .3778487
.68143750 .63291480 .12925210 .7754329 .86881190 .1804108
.40790640 .42604220 .70726730 .5441949 .92910300 .2748059
.17391560 .35543150 .11184820 .1272050
Appendix B
Sample Probability Levels from Four ~COVA Models
190
191
Table B-1
Sample Probability Levels from Four ANCOVA Models
Sample Parametric Covariate- Rank Covariate-corrected Corrected Parametric Rank
001 .238 .507 .221 .017 002 .687 .631 .392 .001 003 .658 .868 .521 .006 004 .159 .257 .084 .014 005 .634 .689 .359 .029 006 .246 .429 .096 .002 007 .672 .967 .536 .017 008 .860 .786 .750 .070 009 .662 .286 .473 .343 010 .485 .245 .304 .011 011 .773 .668 .651 .123 012 .000 .001 .000 .000 013 .696 .936 .845 .011 014 .214 .468 .069 .026 015 .169 .511 .139 .008 016 .106 .254 .084 .030 017 .018 .080 .041 .003 018 .115 .351 .087 .000 019 .299 .150 .319 .107 020 .549 .924 .417 .005 021 .034 .106 .012 .000 022 .869 .664 .538 .072 023 .927 .722 .720 .045 024 .131 .290 .172 .027 025 .427 .740 .257 .007 026 .809 .634 .533 .006 027 .639 .669 .237 .000 028 .317 .704 .616 .022 029 .011 .049 .001 .001 030 .468 .413 .218 .051 031 .086 .190 .013 .001 032 .058 .241 .015 .000 033 .270 .458 .222 .077 034 .092 .276 .068 .017 035 .881 .545 .836 .070 036 .035 .149 .014 .001 037 .666 .320 .363 .896 038 .134 .174 .793 .016 039 .987 .758 .921 .103 040 .574 .729 .375 .033 041 .489 .522 .360 .001
192
042 .066 .172 .076 .000 043 .408 .824 .229 .001 044 .350 .548 .088 .014 045 .024 .092 .008 .000 046 .722 .827 .498 .508 047 .563 .759 .234 .001 048 .528 .846 .379 .048 049 .873 .780 .762 .010 050 .654 .695 .551 .028 051 .963 .773 .784 .168 052 .365 .561 .141 .102 053 .196 .072 .243 .349 054 .840 .676 .772 .011 055 .748 .482 .802 .219 056 .754 .862 .672 .007 057 .104 .283 .058 .000 058 .284 .525 .207 .001 059 .049 .114 .027 .012 060 .319 .615 .348 .010 061 .319 .720 .279 .002 062 .185 .574 .251 .001 063 .325 .462 .327 .027 064 .531 .333 .486 .019 065 .773 .863 .575 .006 066 .218 .371 .210 .004 067 .479 .325 .253 .018 068 .075 .194 .050 .006 069 .271 .628 .209 .001 070 .939 .987 .906 .080 071 .025 .121 .020 .000 072 .449 .613 .283 .061 073 .094 .318 .062 .001 074 .115 .447 .178 .009 075 .337 .651 .143 .029 076 .576 .798 .385 .014 077 .454 .816 .341 .000 078 .336 .698 .150 .000 079 .066 .144 .051 .002 080 .223 .553 .084 .000 081 .586 .858 .532 .001 082 .054 .188 .067 .001 083 .209 .517 .356 .000 084 .861 .673 .832 .157 085 .431 .459 .542 .063 086 .094 .383 .064 .004 087 .206 .274 .105 .000 088 .628 .563 .517 .003 089 .071 .229 .025 .000 090 .108 .308 .089 .004 091 .083 .372 .170 .002 092 .629 .442 .802 .426 093 .002 .031 .003 .000
193
094 .231 .103 .262 .067 095 .167 .320 .047 .001 096 .422 .766 .171 .003 097 .551 .651 .509 .017 098 .290 .513 .180 .001 099 .129 .412 .325 .004 100 .711 .795 .359 .002
The two page vita has been removed from the scanned
document. Page 1 of 2
The two page vita has been removed from the scanned
document. Page 2 of 2
