View
14
Download
0
Category
Tags:
Preview:
DESCRIPTION
Specifying the Conceptual and Operational Models and the Research Questions that Follow. Mark W. Lipsey Vanderbilt University. IES/NCER Summer Research Training Institute, 2010. Focus on randomized controlled trials. - PowerPoint PPT Presentation
Citation preview
Specifying the Conceptual and Operational Models and the Research Questions that Follow
Mark W. LipseyVanderbilt University
IES/NCER Summer Research Training Institute, 2010
Focus on randomized controlled trials
Purpose of the Summer Training Institute: Increasing capacity to develop and conduct rigorous evaluations of the effectiveness of education interventions
Caveat: “Rigorous evaluations” are not appropriate for every intervention or every research project involving an intervention They require special resources (funding,
amenable circumstances, expertise, time) They can produce misleading or uninformative
results if not done well The preconditions for making them meaningful
may not be met.
Critical preconditions for rigorous evaluation A well-specified, fully developed
intervention with useful scope basis in theory and prior research identified target population specification of intended outcomes/effects “theory of change” explication of what it does
and why it should have the intended effects for the intended population
operators’ manual: complete instructions for implementing
ready-to-go materials, training procedures, software, etc.
Critical preconditions for rigorous evaluation (continued) A plausible rationale that the intervention is
needed; reason to believe it has advantages over what’s currently proven and available
Clarity about the relevant counterfactual– what it is supposed to be better than
Demonstrated “implementability”– can be implemented well enough in practice to plausibly have effects
Some evidence that it can produce the intended effects albeit short of standards for rigorous evaluation
Critical preconditions for rigorous evaluation (continued) Amenable research sites and
circumstances: cooperative schools, teachers, parents, and
administrators willing to participate student sample appropriate in terms of
representativeness and size for showing educationally meaningful effects
access to students (e.g., for testing), records, classrooms (e.g., for observations)
IES funding categories Goal 2 (intervention development) for
advancing intervention concepts to the point where rigorous evaluation of its effects may be justified
Goal 3 (efficacy studies) for determining whether an intervention can produce worthwhile effects; RCT evaluations preferred.
Goal 4 (effectiveness studies) for investigating the effects of an intervention implemented under realistic conditions at scale; RCT evaluations preferred.
Specifying the theory of change embodied in the intervention1. Nature of the need addressed
what and for whom (e.g., 2nd grade students who don’t read well)
why (e.g., poor decoding skills, limited vocabulary)
where the issues addressed fit in the developmental progression (e.g., prerequisites to fluency and comprehension, assumes concepts of print)
rationale/evidence supporting these specific intervention targets at this particular time
Specifying the theory of change2. How the intervention addresses the need and
why it should work content: what the student should know or be able
to do; why this meets the need pedagogy: instructional techniques and methods to
be used; why appropriate delivery system: how the intervention will arrange
to deliver the instructionMost important: What aspects of the above are
different from the counterfactual conditionWhat are the key factors or core ingredients most
essential and distinctive to the intervention
Logic models as theory schematics
4 year old pre-K
children
Exposed to intervention
Positive attitudes to
school
Improved pre-literacy
skills
Learn appropriate
school behavior
Increased school
readiness
Greater cognitive gains in K
TargetPopulation Intervention Proximal Outcomes Distal Outcomes
Mapping variables onto the intervention theory: Sample characteristics
4 year old pre-K
children
Exposed to intervention
Positive attitudes to
school
Improved pre-literacy
skills
Learn appropriate
school behavior
Increased school
readiness
Greater cognitive gains in K
Sample descriptors:basic demographics diagnostic, need/eligibility identificationnuisance factors (for variance control)
Potential moderators:setting, contextpersonal and family characteristicsprior experience
Mapping variables onto the intervention theory: Intervention characteristics
4 year old pre-K
children
Exposed to intervention
Positive attitudes to
school
Improved pre-literacy
skills
Learn appropriate
school behavior
Increased school
readiness
Greater cognitive gains in K
Independent variable:T vs. C experimental condition
Generic fidelity:T and C exposure to the generic aspects of the intervention (type, amount, quality)
Specific fidelity:T and C(?) exposure to distinctive aspects of the intervention (type, amount, quality)
Potential moderators:characteristics of personnelintervention setting, context e.g., class size
Mapping variables onto the intervention theory: Intervention outcomes
4 year old pre-K
children
Exposed to intervention
Positive attitudes to
school
Improved pre-literacy
skills
Learn appropriate
school behavior
Increased school
readiness
Greater cognitive gains in K
Focal dependent variables:pretests (pre-intervention)posttests (at end of intervention)follow-ups (lagged after end of intervention
Other dependent variables:construct controls– related DVs not expected to be affectedside effects– unplanned positive or negative outcomesmediators– DVs on causal pathways from intervention to other DVs
Main relationships of (possible) interest
Causal relationship between IV and DVs (effects of causes); tested as T-C differences
Duration of effects post-intervention; growth trajectories
Moderator relationships; ATIs (aptitude-Tx interactions): differential T effects for different subgroups; tested as T x M interactions or T-C differences between subgroups
Mediator relationships: stepwise causal relationship with effect on one DV causing effect on another; tested via Baron & Kenny (1986), SEM type techniques.
Formulation of the research questions Organized around key variables and
relationships Specific with regard to the nature of the
variables and relationships Supported with a rationale for why the
question is important to answer Connected to real-world education issues What works, for whom, under what
circumstances, how, and why?
Describing and Quantifying Outcomes
Mark W. LipseyVanderbilt University
IES/NCER Summer Research Training Institute, 2010
Outcome constructs to measureIdentifying the relevant outcome constructs follows from the theory development and other considerations covered in the earlier session What: proximal/mediating and distal outcomes When: temporal status– baseline, immediate
outcome, longer term outcomes What else:
possible positive or negative side effects construct control outcomes not targeted for
change
Aligning the outcome constructs and measures with the intervention and policy objectives
Instruction
Assessment
Policy relevant outcomes(e.g., state achievement standards)
Alignment of instructional tasks with the assessment tasks
Identical
Analogous(near transfer)
Generalized(far transfer)
Instructional tasks,activities, content
Basic psychometric issuesValidity (typically correlation with established
measures or subgroup differences)Reliability (typically internal consistency or test-
retest correlation) standardized measures of established validity
and reliability researcher developed measures with validity
and reliability demonstrated in prior research new measures with validity and/or reliability to
be investigated in present study
Sensitivity to change: Achievement effect sizes from 124 randomized education studies
Type of Outcome Measure
Mean EffectSize
Number of Measures
Standardized test, broad .04 103Standardized test, narrow .28 426Focal topic test, mastery test .40 300
Data from which measurement sensitivity can be inferred
Observed effects from other intervention studies using the measure
Mean effect sizes and their standard deviations from meta-analysis
Longitudinal research and descriptive research showing change over time or differences between relevant criterion groups
Archival data allowing ad hoc analysis of, e.g., change over time, differences between groups
Pilot data on change over time or group differences with the measure
Variance control and measurement sensitivity
Variance control via procedural consistency and statistical control usingcovariates for e.g., pre-intervention individual differences and differences in testing procedures or conditions
Issues related to multiple outcome measures
Correlated measures: overlap and efficiency
Subtest
Factor Loadings
Pre-KPretest
Pre-KPosttest
KindergartenFollow-up
Letter Word IdentificationQuantitative ConceptsApplied ProblemsPicture VocabularyOral ComprehensionStory Recall
.60
.82
.82
.75
.82
.53
.69
.82
.80
.76
.79
.55
.73
.78
.75
.67
.74
.61
Factor Analysis of Preschool Outcome Variables
Correlated change may be even more relevant
Subtest
Factor LoadingsPre toPost
Post toFollow-up
Pre toFollow-up
Basic School Skills Letter Word Identification Quantitative Concepts Applied Problems
Complex Language Picture Vocabulary Oral Comprehension Story Recall
.74 -.19
.66 .14
.54 .08
.09 .77 .16 .75-.08 .37
.73 -.06
.70 .06
.47 .16
.14 .48 .17 .72-.16 .68
.79 -.15
.74 .13
.40 .41
-.04 .74 .13 .69-.01 .37
Factor Analysis of Gain Scores for Pre-K Outcomes
Handling multiple correlated outcome measures Pruning– try to avoid measures that have
high conceptual overlap and are likely to have relatively large intercorrelations
Procedural– organize assessment and data collection to combine where possible for efficiency
Analytic create composite variables to use in the analysis use multivariate techniques like MANOVA to
examine omnibus effects as context for univariate effects
use latent variable analysis, e.g., in SEM
IES Guidelines on multiple significance testsSchochet, P.Z. (2008). Technical methods report: Guidelines for
multiple testing in impact evaluations. IES/NCEE 2008-4108.http://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=NCEE20084018
Delineate separate outcome domains in the study protocol. Define confirmatory and exploratory analysis prior to data analysis Specify which subgroups will be part of the confirmatory analysis and
which will be part of the exploratory analysis Design the evaluation to have sufficient statistical power for examining
effects for all prespecified confirmatory analyses For domain-specific confirmatory analysis, conduct hypothesis testing
for domain outcomes as a group Multiplicity adjustments are not required for exploratory analysis Qualify confirmatory and exploratory analysis findings in the study
report
Practicality and appropriateness to the circumstances
Feasibility– time and resources required Respondent burden– minimize demands,
provide incentives/compensation Developmental appropriateness– consider
not only age but performance level, possible ceiling and floor effect
For follow-up beyond one school year, may need measures designed for a broad age span to maintain comparability
May need to tailor measures or assessment procedures for special populations (disabilities, English language learners)
Recommended