Upload
phamhanh
View
224
Download
1
Embed Size (px)
Citation preview
CENTER FOR
RESPONSE TO INTERVENTION IN EARLY CHILDHOOD
Preschool RTI Summit, Measurement Strand
Tamaya Hyatt, September 27, 2011
Alternative Designs for
Evaluating RTI Models
Charles R. Greenwood
Juniper Gardens Children’s Project,
University of Kansas
Guiding Issue for Today’s
Breakout
The success of the RTI approach is predicated on
use of evidence-based practice
Any RTI approach is a package of evidence-
based components
Screening and progress monitoring measures
Practices supporting multiple tiers of support
Fidelity of implementation guiding program
improvement
Others (i.e., coordination strategies, PD)
So, how do we create the supporting evidence?
Guiding Issue for Today’s
Breakout
How do we create the evidence?
For the component practices?
For the entire model?
Hopefully, this discussion will contribute to a
deeper understanding of the issues,
challenges, and study designs best suited for
generating the evidence we need
Briefly, What Exactly is RTI?
RTI is an early intervening approach using evidence-based practices to prevent the need for special education services
Use of an evidence-based Tier 1, core curriculum for all students
Frequent universal screening of all children (students)
Identification of those children not making expected progress
Providing these children more intensive Tiers of support (Tier 2, Tier 3), each with supporting evidence.
Based on short-term progress, decisions to keep or change a child’s level of intervention are made
What are the Implications?
Each RTI component practice needs research evidence
Efficacy of the Tier 1, core curriculum
Efficacy of the Tier 2 and 3 interventions
Validity/utility of the measurement system and its decision making benchmarks
The fully operating RTI model needs research evidence
Efficacy of the full model
Efficacy of the coordination, professional development, and implementation support systems
How do we handle dynamic RTI interventions
where children may change intervention at any
time during a year based on progress?
By definition, RTI services are intentionally
dynamic, school teams or teachers make
intervention change decisions
This may violate the assumptions in some quasi-
experimental designs, for example the
Regression Discontinuity Design)
Presents challenges to attribution of causal
effects that include variable intervention changes
and different exposures (dosage)
How should we handle repeatedly
measured progress monitoring data –
growth trajectories?
Time series, repeatedly measured progress
data are typical in RTI research and present
some challenges to assumptions and
interpretation
Single case designs are highly appropriate when
the unit of study is the individual child’s progress
repeatedly measured (AB being the simplest)
Growth curve analyses are appropriate when the
unit of analysis is multiple children repeatedly
measured
How do we handle multiple layers of
covarying measures (e.g., mastery,
fluency, fidelity)
RTI research typically involves multiple
collection of multiple measures (e.g., dependent
measures, and covariates like fidelity of
implementation, time in treatment, etc.)
Research questions typically focus on how do
covariates affect change in the dependent measure
SCD – graphing the data in the same figure to
display covaration
GCA – testing whether or not covariates
significantly affect the observed trajectories
Some RTI Research and
Development Target Goals
New screening and progress monitoring measurement development, or improving existing
New technique development
Refine and replicate a promising intervention implemented to fidelity
Evaluate the efficacy of a single promising, feasible intervention (e.g., Tier 1, 2, or 3 intervention)
Evaluate the efficacy of a fully operating RTI model
RTI Measurement Development
Design Challenges
Multi-year screening of all children requires measures brief enough that they are feasible to implement by staff Screening measures must have criterion validity, classification
accuracy, and predictive utility
Progress monitoring measures must be 1. Sensitive to individual children’s growth over time
2. Have end-of-year benchmarks
3. Specified rates of progress
4. Local norms
5. Sensitive to intervention effects as evidenced by teacher planning and changes in interventions over time
Mastery monitoring measures require a specified skill sequence be assessed and pass/fail benchmarks at relevant ages or grade levels.
Criterion Validity
McConnell, S., Rodriguez, M., Schmitt, B. & Clayton, K. (2011, February). Early Childhood Response to Intervention: Using Second-Generation IGDIs. Symposium presented at annual meetings of the National Association of School Psychologists, San Francisco CA
Average IGDI at Final Status = 30.43 • Variance 34.95 (p<.001) substantial
variability across children in IGDI final
status.
Overall IGDI Growth
IGDI (Picture Naming)
Growth
IGDI (Picture Naming)
Final
Status
Picture
Naming
Wave 1
Picture
Naming
Wave 2
Picture
Naming
Wave 3
Model Fit: χ2(2)=9.47; RMSEA=.079(.034-.132); CFI=.99; TLI=.99
Average IGDI linear trajectory
across children = 3.16 • Children gained an average of 3.16
on IGDI at each Wave.
• Variance 8.89 (p<.01) substantial
variability across children in IGDI
final status.
At Wave 3 children that are
higher on IGDI are gaining
at a lower rate than those
lower on IGDI.
Screening Predictive Utility
Normative Trajectories Greenwood, C. R., Walker, D., & Buzhardt, J. (2010). The Early
Communication Indicator (ECI) for Infants and Toddlers: Early Head Start Growth Norms from Two States. Journal of Early Intervention, 32(5), 310-334.
Classification Accuracy Wilson, S. B., & Lonigan, C. J. (2010). Identifying preschool
children at risk of later reading difficulties: Evaluation of two emergent literacy screening tools. Journal of Learning Disabilities, 43(1), 62-76.
Odds of Reaching Future Goals Good, R. H., Kaminski, R., Powell-Smith, K., McConnell, S.,
Rodriguez, M., & Greenwood, C. R. (2011, February ). Developing benchmark goals and cut-points for risk: Alternative approaches. Paper presented at the Pacific Coast Research Conference, San Diego.
New Technique Development
Design Challenge
Iterative development of an intervention with evidence of implementation feasibility and pilot student outcome
Step 1: Initial development and tryout
Step 2: Refine and replicate a promising intervention implemented to fidelity
Step 3: Evaluate the efficacy of this single developed and feasible intervention (e.g., Tier 1, 2, or 3 intervention)
Step 1: Initial Development
and Tryout
Students are the unit of analysis and single case research designs can be used for causal demonstrations of effects
Student participants are identified by screening measures
Measures of mastery and fluency used to measure progress over time
Design Considerations If teaching new skills (language and early literacy), will
your design display impacts at the lesson or unit level of instruction as well as overall at the end of instruction?
Step 1: Initial Development
and Tryout
Design Considerations (continued)
If teaching new skills (i.e., language and early
literacy), will your design display impacts at the
lesson or unit level of instruction as well as over
the sequence of instruction?
Step 1: Initial Development
and Tryout
Design Considerations (continued)
Will repeated measurement prove to be reactive?
1. Will children show signs of over-testing?
2. Will children show how growth with repeated testing in
the absence of instructional intervention (baseline)?
Step 1: Initial Development
and Tryout
Design Considerations (continued)
This is what you would like to see!
Other Potential Baseline Effects with
Repeated Testing
Repeated
Testing
Effect?
Repeated
Testing
Fatigue?
Step 2: Refine/Replicate When
Implemented to Fidelity
The external validity of promising initial study
findings are extended by replications where
Teachers are implementers and achieving
findings as promising as those achieved with
researcher/developers as implementers
Implementers and student participants are from
other settings and programs, and also achieving
promising results
Student participants represent diverse
subpopulations
Step 2: Refine/Replicate When
Implemented to Fidelity
Vocabulary and Comprehension Tier 2 Listening
Station Intervention
1. Developed in Ohio, Replicated in KS 2010-11
2. Different implementers and different sample and
subpopulations represented
The Vocabulary and
Comprehension Tier 2 Intervention
Setting Arrangement: Table and seats with some
degree of privacy
Listening Station: MP3 player with a splitter
allowing up to 3 students and an aide to listen
1. Auto track: Narrator and Guide
Book Series: If Elephants Could Fly
1. Content Taught: Scope and Sequence
2. 9 Books in the Series, 1 Book per Week
Vocabulary and Comprehension
Intervention Replication Study Samples
State
Classrooms
Children
Non-White
Non- English
Home
Language
Individual
Education
Program
Ohio 3 9 8 0 0
Kansas 3 9 8 5 3
Total 6 18 16 5 3
KS Sample
w/ELL and
IEP
Student Identification and
Risk Status at Start
State Measure M Range
Ohio Vocabulary IGDI (Max = 15) 7.2 of 15 5-11
Peabody Picture Vocabulary Test
(PPVT) (M = 100, SD = 15)
84.3 78-96
Clinical Evaluation of Language
Fundamentals (CELF) (M = 100, SD =
15)
86.4 73-94
Kansas Vocabulary IGDI (Max = 15) 7.3 of 15 5-13
Peabody Picture Vocabulary Test
(PPVT) (M = 100, SD = 15)
86.9 73-107
Clinical Evaluation of Language
Fundamentals (CELF) (M = 100, SD =
15)
72.6 50-102
Note. IGDI = Individual Growth and Development Indicator
Ohio and Kansas’ Effect
Sizes
Mastery
Outcome
Ohio
Kansas
Vocabulary 1.62 1.12
Comprehension 0.67 0.46
d = standard mean difference for single case
research
Step 3: Evaluate Efficacy of
this Intervention
Design Considerations
RCT with Early Head Start Home Visitors as
implementers of a single Tier 2intervention
Buzhardt, J., Greenwood, C. R., Walker, D., Anderson, R.,
Howard, W. J., & Carta, J. J. (2011). Effects of web-based
support on Early Head Start home visitors’ use of evidence-
based intervention decision making and growth in children’s
expressive communication. NHSA Dialog: A Research-to-
Practice Journal for the Early Childhood Field, 13(3), 121-146.
Unique RTI Randomized Control
Trial Design Challenges
How does the nature of the “RTI intervention” define cluster units (i.e., teachers/ classrooms, schools, children) in the study design? Issue: Clustering can lead to incorrect inferences about
treatment effects
How do we treat the “Rolling” enrollment of children to Tiered interventions over time? Issue: Age at start will vary.
How do we handle covariates? Issue: Need to rule out potential confounds
Unique RTI Randomized Control
Trial Design Challenges
How do we handle Tier intervention
assignments that are fluid over time?
Issue: Dosage and time in treatment varies
based on child progress
How should we treat repeatedly measured
progress monitoring data?
Issue: Growth curve analysis
RCT Study Highlights
Purpose – To assess the efficacy of a Tier 2 naturalistic language intervention delivered in the home to parents by home visitors who coach and monitor the parents implementation with their children under two conditions:
A = RTI screening, progress monitoring, decision making, and language promoting strategies implemented in the home by parents
B = A + Making Online Decisions (MOD) Support System
Wait-list RTC
Home visitors randomized within programs
Children nested under home visitors
Participants –
Early Head Start: 5 Programs and 659 children in all
Home Visitors (N = 48, 26 and 22 by group A and B respectively)
Children identified below benchmark (N = 124)
Study Highlights
Unit of Treatment – because treatment was
guided by home visitors, they were
randomized to the two conditions, not children
Measures
Repeatedly measured Early Communication skills
allowing examination of children’s growth over
time
Fidelity of implementation for home visitors and
parents
Web-based Support for
Intervention Decision Making
To improve the performance of HVs and parents when implementing an RTI approach to improve children’s growth in language
Analytic Strategy
Because this was a randomized design and
the dependent variable was children’s
language growth trajectories we, used
univariate CGA
Individual children’s growth is considered in terms
of slope and intercept
It handles missing data
It supports the use of independent variables and
covariates (IFSP status, Age at Eligibility)
Analytic Strategy
Because children were screened into the study
at different times, each child’s language data
was converted to a time scale in terms of
months before and after onset of the Tier 2
intervention.
This enabled use of a twice-piece CGA with
the intercept centered at the last time point
prior to start of the intervention (time = 0)
Level 1 CGA Findings
Progress for All
Below Benchmark
Children Before
Treatment
Progress for All
Eligible Children
After Treatment Mean Intercept at Time Point Prior
to Eligibility and Enrollment
Level 2 CGA Findings
Rate of Progress
for Both Groups
Children Before
Treatment
Rate of Progress
for Both Groups
Children After
Treatment
X2 Comparison Test for Nested
Models Table 3. Best Fitting Two-Piece ECI Total Communication Growth Model.
Deviance Number of Decrease In
Models Statistic Parameters Deviance X2 df p
Level 1 4589.898403 10
Level 2-Age at Eligibility 4498.382414 13 91.515989 94.66 6 0.0001
Level 2-Age at Eligibility +
IFSP 4491.090308 16 7.292106 7.29 3 0.062
Level 2-Age at Eligibility +
IFSP + Comparison Groups 4481.568057 19 9.522251 9.52 3 0.023
Note. Age and IFSP Interaction effects were not significant
Effects of Treatment with
Covariates Included Earlier in
the Growth Model
How does the nature of a fully operational
RTI Model define cluster units for
randomization (cluster randomization)?
Tier 3 intervention provided to students by
parents at home (Randomize parents)
Tier 2 intervention provided children by
parents by coaching from a home visitor
(Randomize home visitors)
A full RTI model serving all children in a
classroom or a preschool (Randomize
classrooms or schools)
Multi-Level Growth Models
Represent Clustering
Classroom Level and Child Level Example of
Growth in 1 IGDI
Classroom Level and Child Level Example of
Growth in 2 IGDIs
Program Level and Child Level Example
Preschool Level, Child Level Example
Basic Latent Growth Curve Model for
Progress Monitoring Data
Picture
Naming Status
Picture
Naming
Wave 1
Picture
Naming
Wave 2
Picture
Naming
Wave 3
Model Fit: χ2(4) = 10.23; RMSEA = .049; CFI = .99; TLI = .99
Picture
Naming Growth
1.0
2.0 3.0
Growth in Picture Naming
Children gained an average of 3.16 on Picture
Naming at each Wave (Variance 8.89 [p<.01])
Children averaged 30.4 at Final Picture Naming
Status (Variance 34.95 [p<.001]) at year’s end
At Wave 3 children that are higher on Picture
Naming were gaining at a lower rate than those
starting lower on Picture Naming
Growth in Sound ID
Children gained an average of 3.5 on Sound ID
at each Wave (Variance 8.89 [p<.01])
Children averaged 13.7 on Final Sound ID
Status (variance 21.6 [p<.001]) at year’s end
At Wave 3 children that are higher on Sound ID
were gaining at a higher rate than those lower
on Sound ID.
Multi-level Growth in Picture Naming at the
Child and Classroom Levels
CHILD LEVEL
CLASSROOM LEVEL
Linear Slope model: WITHIN
• Slope is -5.74 (p < .001)
• Intercept variance = 43.92 (p < .001)
• Slope variance = 3.97 (p < .001)
• All free parameters significant.
IGDI
Growth
IGDI
Inter.
IGDI
Growth
IGDI
Inter. Linear Slope model: BETWEEN
• Slope is -3.72 (p < .001)
• Intercept variance = 16.75 (p < .001)
• Slope variance = 1.11 (p = .01)
• All free parameters significant.
• INT mean = 27.29; SLP mean = 2.13
Model Fit: χ2(6) = 12.05; RMSEA = .039; CFI = .99; TLI = .99
Multi-level Growth Curve Model for Two
IGDIs at the Classroom and Child Levels
χ2 (22, n = 659) =
123.623,
p < .001,
RMSEA = .084,
NNFI = .921,
CFI = .959.
Multi-Level IGDI Growth with
Two IGDIs and Two Predictors
χ2 (22, n = 659) = 123.623, p < .001, RMSEA = .084, NNFI = .921, CFI = .959.
Table 4. Descriptive Statistics for Level-2 Analyses
Level 1 Descriptive Statistics
Variable N Mean SD Min Max
ECI Total Communication 16677 11.3 9.4 0 55
Time Group-Mean Centered at 36 Mos. 16677 -14.4 9.4 -32 6
Time Squared 16677 295.4 282.8 0 1024
Level 2 Descriptive Statistics
Variable N Mean SD Min Max
Program 2 5180 0.04 0.20 0 1
Program 3 5180 0.04 0.20 0 1
Program 4 5180 0.03 0.18 0 1
Program 5 5180 0.02 0.14 0 1
Program 6 5180 0.02 0.14 0 1
Program 7 5180 0.03 0.17 0 1
Program 8 5180 0.06 0.23 0 1
Program 9 5180 0.05 0.21 0 1
Program 10 5180 0.04 0.20 0 1
Program 11 5180 0.09 0.28 0 1
Program 12 5180 0.06 0.24 0 1
Program 13 5180 0.05 0.22 0 1
Program 14 5180 0.03 0.17 0 1
Program 15* 5180 0.00 0.00 0 0
Program 16 5180 0.01 0.08 0 1
Program 17 5180 0.02 0.13 0 1
Program 18 5180 0.02 0.13 0 1
Program 22 5180 0.05 0.22 0 1
Program 23 5180 0.02 0.14 0 1
Program 24 5180 0.04 0.20 0 1
Program 25 5180 0.07 0.25 0 1
Program 26 5180 0.06 0.24 0 1
Program 27 5180 0.07 0.25 0 1
Program 29 5180 0.01 0.10 0 1
3-Level Nonlinear Growth Model
(Program, Home Visitor, Children) Table 2. Descriptive Statistics for Level-3 Analyses
Level 1 Descriptive Statistics
Variable N Mean SD Min Max
ECI Total Communication 16677 11.3 9.4 0 55
Time Mean Centered at 36 Mos. 16677 -14.4 9.4 -32 6
Time Squared 16677 295.4 282.8 0 1024
Level 2 Descriptive Statistics
Variable N Mean SD Min Max
-- -- -- -- -- --
Level 3 Descriptive Statistics
Variable N Mean SD Min Max
State (0 = State 1, 1 = State 2) 26 - - 0 1
%Children with IFSPs 26 8.4 5.8 0.0 20.1
%Male Children 26 50.5 6.5 40.1 69.8
%NonEnglish 26 8.0 13.7 0.0 57.5
%Data by Certified Assessors 26 59.6 33.9 0.0 100.0
%ECI Score Outliers 26 2.1 6.6 0.0 34.0
%ECIs at 6-Minutes Duration 26 97.2 4.2 83.8 100.0
%ECIs with Agreement Checks 26 2.8 4.4 0.0 20.8
Note. -- = No variables included at level-2.
Summary/Conclusion
Preschool RTI needs a generation of research
seeking to reach a level of effectiveness
greater than business as usual
This work challenges to study designs that are
predicated on the nuances in the RTI
approach and evidence-based practices
Solutions to some of these issues (certainly
not all!) were illustrated