Upload
esmond-wright
View
212
Download
0
Tags:
Embed Size (px)
Citation preview
Universal Screening of Academics and Behavior in an RTI Framework
A collaboration of Vanderbilt University and the University of Kansas
Funded by U.S. Department of Education. Office of Special Education Programs, Judy Shanley, Project Officer
Award No. H324U010004
Daryl Mellard
April 1-2, 2008Virginia’s RTI Institute
Fredericksburg, VA
Acknowledgements from previous presentations• Marcia Invernizzi, U. of Virginia,
November 2007 presentation at Roanoke
• Hugh Catts, U. of Kansas, April 2006 presentation at NRCLD National SEA conference on RTI, KCMO (available at http://nrcld.org/sea/index.html)
A Little Overview• General principles
about screening
• Early reading screening
• Behavioral screening
Screening Component in an RTI Framework1. Academic and behavioral prediction
2. Measures that are quick, low cost, repeatable, critical (predictive) skills, minimal administration training
3. Question: Student at-risk?
4. Affirmative answer: More attention (assessment/ intervention) to class or student
5. Criteria: Criterion benchmark or normative
Screening Accuracy
Three influences:• Base rate.
• Diagnosticity.
• Values of those setting the cutoff (criterion) score (i.e., resource priorities).
Constructing Screening Measures
• Match the curricular demands facing students
• Proximity of the screening test to the criterion performance (Closer should yield higher accuracy.)
• Should include multiple, related indicators
Screening AccuracyScreening Accuracy
Particular attention is given to the Particular attention is given to the accuracy of screening instrumentsaccuracy of screening instruments
Errors in identification can be costlyErrors in identification can be costly - over identification- over identification - under identification- under identification Accuracy typically quantified within a Accuracy typically quantified within a
clinical decision making modelclinical decision making model
Clinical Decision Making Clinical Decision Making ModelModel
At risk Not at risk
Norm
al
RD
Ou
tcom
e
True Positive
a
False Negativ
e
bTrue
Negative
d
False Positive
c
Screen
Sensitivity
a / a + b
Specificity
d / c + d
Positive Predictive
Power
a / a + c
Negative Predictive
Power
d / b + d
Accuracy of screening is determined by Accuracy of screening is determined by
…… How well your instrument separates How well your instrument separates
those who eventually will have a those who eventually will have a problem from those who will notproblem from those who will not
What you choose as a cut-off scoreWhat you choose as a cut-off score
The Ultimate ScreenThe Ultimate Screen
TP100
FN0
FP0
TN100
More Typical ScreenMore Typical Screen
TP80
FN20
FP20
TN80
More Typical ScreenMore Typical Screen
TP90
FN5
FP35
TN70
ROC CurveROC Curvehttp://www.anaesthetist.com/mnm/stats/roc/http://www.anaesthetist.com/mnm/stats/roc/
What to Measure?What to Measure?
What is the criterion?What is the criterion? Reading comprehension involves a Reading comprehension involves a
mixture of complex abilities mixture of complex abilities Role of each changes over time Role of each changes over time
Part 2:Early Reading Part 2:Early Reading ScreeningScreening
Predictive Predictive indicatorsindicators
Criterion Criterion measuresmeasures
Remember: Time Remember: Time interval is a big interval is a big influenceinfluence
What to Measure?What to Measure?
Variables related to early reading Variables related to early reading - letter knowledge- letter knowledge- phonological awareness- phonological awareness- rapid naming- rapid naming- vocabulary and grammar- vocabulary and grammar
- reading itself (non-word or word reading)- reading itself (non-word or word reading)
What to Measure?What to Measure?
Variables related to later reading Variables related to later reading - word reading- word reading
- oral reading fluency- oral reading fluency - vocabulary and grammar- vocabulary and grammar - text comprehension- text comprehension
Reading Screening MeasuresHugh Catts, April 2006, http://nrcld.org/sea/index.html Texas Primary Reading Inventory (Foorman et al.,
1998). Dynamic assessment model (O’Connor & Jenkins,
1999). Catts, Fey, Zhang, & Tomblin (2001). Dynamic Indicators of Basic Early Literacy Skills
(DIBELS). Phonological Awareness Literacy Screening (Invernizzi,
Juel, Swank & Meier, 1997). CBM tools.
Early Screening ToolsEarly Screening Tools
Comprehensive Test of Phonological ProcessingComprehensive Test of Phonological Processing Test of Phonological AwarenessTest of Phonological Awareness Test of Early Reading AbilityTest of Early Reading Ability
All correlated with reading outcomes (moderate All correlated with reading outcomes (moderate range), but little data on sensitivity and specificityrange), but little data on sensitivity and specificity
Texas Primary Reading InventoryTexas Primary Reading Inventory(Foorman et al., 1998- www.tpri.org)(Foorman et al., 1998- www.tpri.org)
Designed to be used by teachers to Designed to be used by teachers to identify children at risk for RD and to identify children at risk for RD and to further evaluate their strengths and further evaluate their strengths and weaknesses in reading-related skillsweaknesses in reading-related skills
5 screens for K-25 screens for K-2ndnd grade grade Designed to hold false negatives to a Designed to hold false negatives to a
minimumminimum Includes an inventory of secondary Includes an inventory of secondary
measures to help rule out false positivesmeasures to help rule out false positives
TPRI (1998)TPRI (1998)K (Dec) predicting end of 1stK (Dec) predicting end of 1st
At risk Not at risk
Norm
al
RD
Ou
tcom
e 92 5
181143
Screen (shorten version)
Sensitivity
94.8%
Specificity
55.9%
Positive Predictive
Power
39.1%
Negative Predictive
Power
97.3%
Base rate 23%
Risk rate 55.8%
Dynamic Indicators of Basic Dynamic Indicators of Basic Early Literacy Skills (DIBELS)Early Literacy Skills (DIBELS)
Standardized and readily availableStandardized and readily available www.dibels.uoregon.edu www.dibels.uoregon.edu www.aimsweb.comwww.aimsweb.com Developed to monitor progress and inform Developed to monitor progress and inform
instructioninstruction
DIBELSDIBELSK (Fall) predicting end of 1st K (Fall) predicting end of 1st
At risk Not at risk
Norm
al
RD
Ou
tcom
e 8577 1824
122589345
Screen ( Initial sound fluency, Letter name fluency)
Sensitivity
82.5%
Specificity
56.7%
Positive Predictive
Power
47.9%
Negative Predictive
Power
87.0%
Base rate
32.5%
Risk rate
56.0%
CBM ToolsCBM Tools
Letter-Name FluencyLetter-Name Fluency Letter-Sound FluencyLetter-Sound Fluency Initial-Sound FluencyInitial-Sound Fluency Phoneme Segmentation FluencyPhoneme Segmentation Fluency Nonword Reading FluencyNonword Reading Fluency Oral Reading FluencyOral Reading Fluency Oral Retell FluencyOral Retell Fluency Maze FluencyMaze Fluency
CBM ToolsCBM Tools
Assessments given 3 or more times a Assessments given 3 or more times a year to evaluate growth in reading year to evaluate growth in reading (meeting benchmarks) (meeting benchmarks)
Each can be considered a screening Each can be considered a screening opportunityopportunity
O’Connor & Jenkins (1999)O’Connor & Jenkins (1999)
Large battery of preliteracy skillsLarge battery of preliteracy skills- - rapid letter namingrapid letter naming (# of letters named from (# of letters named from
random list in 1 min)random list in 1 min)
- - syllable deletionsyllable deletion (say “baseball” without “ball”) (say “baseball” without “ball”)
- - segmenting phonemessegmenting phonemes (tell me how many (tell me how many
sounds in “saw”)sounds in “saw”)
- - phoneme repetition phoneme repetition (say “p” “I” “f” )(say “p” “I” “f” )
Chose cut-off scores to maximize sensitivityChose cut-off scores to maximize sensitivity
O’Connor & Jenkins (1999)O’Connor & Jenkins (1999)K (K (Nov) predicting April of 1stNov) predicting April of 1st
At risk Not at risk
Norm
al
RD
Ou
tcom
e 15 0
19223
Screen (phonem seg, RLN, deletion)
Sensitivity
100%
Specificity
89.3%
Positive Predictive
Power
39.5%
Negative Predictive
Power
100%
Base rate
6.5%
Risk rate
16.5%
TPRI (1998)TPRI (1998)11stst (Oct) predicting end of 1st (Oct) predicting end of 1st
At risk Not at risk
Norm
al
RD
Ou
tcom
e 111 8
305175
Screen (letter-sound, blending, word reading)
Sensitivity
93.3%
Specificity
63.5%
Positive Predictive
Power
38.8%
Negative Predictive
Power
97.4%
Base rate
19.9%
Risk rate
47.7%
DIBELSDIBELSBeginning 1Beginning 1stst NWF predicting end 1st ORF NWF predicting end 1st ORF
At risk Not at risk
Norm
al
RD
Ou
tcom
e 7477 2956
165445067
Sensitivity
71.7%
Specificity
76.6%
Positive Predictive
Power
59.6%
Negative Predictive
Power
84.8%
Risk rate
39.1%
Base rate
32.6%
Dynamic AssessmentDynamic Assessment May have advantage over static assessmentMay have advantage over static assessment Measurement of ability over time in order to Measurement of ability over time in order to
monitor progress monitor progress Measurement of learners’ potential over the Measurement of learners’ potential over the
short termshort term Assessor actively intervenes during the Assessor actively intervenes during the
course of the assessment with the goal of course of the assessment with the goal of intentionally inducing changes in the intentionally inducing changes in the learner's current level of performance.learner's current level of performance.
““Mini-assessment” of response to Mini-assessment” of response to interventionintervention
O’Connor & Jenkins (1999)O’Connor & Jenkins (1999)Oct 1Oct 1stst predicting April 1 predicting April 1stst
At risk Not at risk
Norm
al
RD
Ou
tcom
e 11 0
17826
Screen (phoneme seg, RLN, phoneme repetition)
Sensitivity
100%
Specificity
87.3%
Positive Predictive
Power
29.7%
Negative Predictive
Power
100%
Base rate
5.1%
Risk rate
17.2%
O’Connor & Jenkins (1999)O’Connor & Jenkins (1999)
Dynamic AssessmentDynamic Assessment- phoneme segmentation- phoneme segmentation- used Elkonin boxes to progressively - used Elkonin boxes to progressively
teach segmentation of a set of test items teach segmentation of a set of test items wordswords
- score based on the number of trials needed - score based on the number of trials needed to master the taskto master the task
O’Conner & Jenkins (1999)O’Conner & Jenkins (1999)Oct 1Oct 1stst predicting April 1 predicting April 1stst (dynamic) (dynamic)
At risk Not at risk
Norm
al
RD
Ou
tcom
e 10 1
1959
Screen
Sensitivity
90.9%
Specificity
95.6%
Positive Predictive
Power
52.6%
Negative Predictive
Power
99.5%
Base rate
5.1%
Risk rate
8.8%
Compton, Fuchs, Fuchs, Compton, Fuchs, Fuchs, & Bryant (2006)& Bryant (2006)
Screened in 1Screened in 1stst (Oct) predicting end of 2 (Oct) predicting end of 2nd nd
MeasuresMeasures
- CTOPP Sound Matching- CTOPP Sound Matching
- CTOPP Rapid Digit Naming- CTOPP Rapid Digit Naming
- WJPB-R Oral Vocabulary- WJPB-R Oral Vocabulary
- Word Identification Fluency (WIF)- Word Identification Fluency (WIF) Initial level, 5-week slopeInitial level, 5-week slope
Grade 1Grade 1Word-Identification Word-Identification
FluencyFluency
Teacher: Teacher: Read Read these wordsthese words..
Time: 1 minute.Time: 1 minute.
twotwo
forfor
comecome
becausebecause
lastlast
fromfrom ......
Compton et al. (2006)Compton et al. (2006)11stst (Oct) predicting end of 2 (Oct) predicting end of 2ndnd
At risk Not at risk
Norm
al
RD
Ou
tcom
e 35 2
15514
Screen (includes WIF level & slope – CTA)
Sensitivity
94.6%
Specificity
91.7%
Positive Predictive
Power
71.4%
Negative Predictive
Power
98.7%
Beyond First gradeBeyond First grade
Most common screening for Tier 2 has been Most common screening for Tier 2 has been oral reading fluency (ORF)oral reading fluency (ORF)
ORF strongly correlated with 3ORF strongly correlated with 3rdrd grade state grade state assessmentsassessments
Strong correlations do not necessarily Strong correlations do not necessarily translate into high sensitivity and specificitytranslate into high sensitivity and specificity
Measurement of level and slope may help Measurement of level and slope may help (e.g., dual discrepancy)(e.g., dual discrepancy) Must deal with potential scaling problemsMust deal with potential scaling problems
What does research tell us What does research tell us about screening?about screening?
Can identify children at risk for reading problems Can identify children at risk for reading problems Can be done as early as the fall of kindergartenCan be done as early as the fall of kindergarten Need to choose measures carefullyNeed to choose measures carefully Must match measures to curriculumMust match measures to curriculum - letter naming- letter naming - phonological awareness- phonological awareness - word reading- word reading - text reading- text reading Must not forget about other factors related to Must not forget about other factors related to
comprehensioncomprehension - oral language - oral language
What does research tell us What does research tell us about screening?about screening?
False positive rates are high and efforts need to False positive rates are high and efforts need to be in place to limit the cost of over prediction be in place to limit the cost of over prediction
Progress monitoring within a RTI framework may Progress monitoring within a RTI framework may serve this purposeserve this purpose
Need to equate forms to scaleNeed to equate forms to scale Dynamic assessment has potentialDynamic assessment has potential
Screening for Possible Reading RiskGrade CBM Probe Cut-off
Kindergarten Letter Sound Fluency < 10 letters/minute
Grade 1 Word IdentificationFluency
< 15 words on list/minute
Grade 2 Passage Reading Fluency
< 15 words in text/minute
Grade 3 Passage Reading Fluency
< 50 words in text/minute
Grade 4 Maze Fluency < 10 Maze replacements/2.5 minutes
Grade 5 Maze Fluency < 15 Maze replacements/2.5 minutes
Grade 6 Maze Fluency < 20 Maze replacements/2.5 minutes
Note: These figures may change pending additional RTI research.
Tier 1–Primary Prevention:Confirming Risk Status With PM• At the end of 5-8 weeks, student risk
status is confirmed or disconfirmed.
Grade
InadequateReading
Slope
Inadequate Math
ComputationSlope
Inadequate Math
Concepts and Applications
Slope
Kindergarten < 1 (LSF) < 0.20 < 0.20
Grade 1 < 1.8 (WIF) < 0.25 < 0.30
Grade 2 < 1 (PRF) < 0.20 < 0.30
Grade 3 < 0.75 (PRF)
< 0.20 < 0.50
Grade 4 < 0.25 (Maze)
< 0.50 < 0.50
Grade 5 < 0.25 (Maze)
< 0.50 < 0.50
Grade 6 < 0.25 (Maze)
< 0.50 < 0.50
Note: These figures may change pending additional RTI research.
Dr. Marcia InvernizziUniversity of Virginia
Universal Literacy Screening:
First Steps Toward Prevention & Intervention
Universal Literacy Screening
Screen routinely Fall – Mid-year – Spring
Why? Ensures that students who need additional
support do not go too long before receiving additional instruction/intervention.
Helps identify the “point of entry” into the tiers of RtI intervention & the kinds of supports needed.
Monitors student growth over time
Phonological Awareness Literacy Screening
The state-provided screening tool for Virginia’s EIRI Consists of three instruments:
PALS-PreK (for preschoolers) PALS-K (for students in kindergarten) PALS 1-3 (for students in grades 1-3)
Measures young children’s knowledge of important literacy fundamentals.
45
PALS Instrument Content AreasPALS-PreK PALS-K PALS 1-3
Print and Word Awareness X
Nursery Rhyme Awareness X
Name Writing X
Rhyme Awareness X X
Beginning Sound Awareness X X
Alphabet Knowledge X X X
Letter Sounds X X X
Concept of Word X X
Blending X X
Sound-to-letter X X
Spelling X X
Word Recognition in isolation X X
Fluency X
Universal Literacy Screening
Purpose #1: Identification of children in need of further
assessment and/or intervention
Solution: PALS class reports available on the PALS
website: http://pals.virginia.edu
Universal Literacy Screening
Purpose #2: Provision of feedback about how a class is
performing so that classroom-based curriculum or instructional issues can be identified as soon as possible.
Solution: PALS class reports available on the PALS
website: http://pals.virginia.edu
49
Universal Literacy Screening
Purpose # 3: Diagnosis of children who may have had a poor
testing experience. Solution:
PALS individual student reports available on the PALS website: http://pals.virginia.edu
Online Score Entry & Reporting
Activities for TeachersPALS Web site Instructional resources
http://pals.virginia.edu
Fitzpatrick, J.. (1997). Phonemic Awareness. Cyprus, CA: Creative Teaching Press. (p. 30)
Please contact the PALS office if you have any questions or concerns!
1-888-UVA-PALS [email protected]
ReferencesReferences Catts, H.W., Fey, M.E., Zhang, X., & Tomblin, J.B. (2001). Catts, H.W., Fey, M.E., Zhang, X., & Tomblin, J.B. (2001).
Estimating risk for future reading difficulties in kindergarten Estimating risk for future reading difficulties in kindergarten children: A research-based model and its clinical implications. children: A research-based model and its clinical implications. Language, Speech, and Hearing Services in the Schools, 32, Language, Speech, and Hearing Services in the Schools, 32, 38-50.38-50.
Compton, D.L., Fuchs, D., Fuchs, L.S., & Bryant, J.D. (2006). Compton, D.L., Fuchs, D., Fuchs, L.S., & Bryant, J.D. (2006). Selecting at-risk readers in first grade for early intervention: A Selecting at-risk readers in first grade for early intervention: A two-year longitudinal study of decision rules and procedures. two-year longitudinal study of decision rules and procedures. Journal of Educational Psychology, Journal of Educational Psychology, 98, 394-40998, 394-409..
Foorman, B.R., Fletcher, J.M., Frances, D.J., Carlson, C.D., Chen, D., Foorman, B.R., Fletcher, J.M., Frances, D.J., Carlson, C.D., Chen, D., Mouzaki, A., Schatschneider, C., Wristers, K., & Taylor, R. (1998). Mouzaki, A., Schatschneider, C., Wristers, K., & Taylor, R. (1998). Technical Report Texas Primary Reading Inventory Technical (1998 Technical Report Texas Primary Reading Inventory Technical (1998 Edition). Edition). Houston, TX: Center for Academic and Reading Skills and Houston, TX: Center for Academic and Reading Skills and University of Houston. University of Houston.
O’Connor, R.E., & Jenkins, J.R. (1999). Prediction of reading O’Connor, R.E., & Jenkins, J.R. (1999). Prediction of reading disabilities in kindergarten and first grade. disabilities in kindergarten and first grade. Scientific Studies of Scientific Studies of Reading, 3, Reading, 3, 159-197. 159-197.
Part 3: Behavioral ScreeningPart 3: Behavioral Screening
Challenges in Challenges in screening behaviorscreening behavior
Recognize age Recognize age influences on influences on predictive and predictive and criterion datacriterion data
Alternative Alternative proceduresprocedures
Challenges in Behavioral Screeningvan Lier, Verhulst & Crijnen (2003)
• Screening was originally for detecting the presence or absence of highly specific medical conditions that could be detected in a benign presymptomatic stage and for which treatment was available.
• Disruptive behaviors lack this specificity and unitary underlying conditions; multi-factorial
• Lack a well-delineated onset affecting valid detection
Other Challenges• Referral by regular classroom teachers
is regarded as the most vulnerable to bias due to differing expectations (Hersh & Walker, 1983; Lloyd et al., 1991)
• Referral peak for students with behavioral problems occurs in grade 9 (academic problems grades 2 and 3)
Social-environmental Domains of Behavioral InfluenceMulti-factorial influences across
developmental periodsFamilyPeer groupSchool and NeighborhoodIncorporate into a multiple gating
procedure (e.g., Loeber, 1990; Lochman, 1995)
Behavioral DimensionsInternalizing
• Depression
• Phobias
• Social isolation
• Peer isolation
Externalizing
• ADHD
• Oppositional defiant disorder
• Conduct disorder
Extra-Ordinary ChallengesChallenges
• Stressful life events
• Exposure to community violence
• Maltreatment
• Poverty
• Divorce
• Maternal mental illness
Generalization
• Developmental consistency
• Gender consistency
• Social group consistency
Family Survey Items (Multiple item composite score)• History of maternal
mental illness• High maternal anxiety• Rigid parental
perspectives (attitudes, beliefs, values)
• Frequency of positive parental interactions, esp. with mother
• Stressful life events• Large family size• Minimal maternal
education level• Head of household
in unskilled occupations
• Disadvantaged minority status
• Single parenthood
Academic PredictorsUse an percentile
index of the number of --
• absences• school changes• failed courses• grades repeated• discipline referralsGrade x school level
data
Student Self-Ratings as Predictors
The degree to which the student perceives -- • School related
Staff cares about the students. Racial tension in school between staff and
students. Sense of being treated fairly.
• Peer related Work hard in school. Discuss schoolwork/intellectual topics. Cheat on tests. Pressure to use drugs.
Instruments for Multi-gating (1)Inattention/Overactivity with Aggression (IOWA
Conners) 10 items for teacher rating1. Fidgeting2. Hums and makes odd noises3. Excitable, impulsive4. Inattentive, easily distracted5. Fails to finish things, short attention span6. Quarrelsome7. Acts “smart”8. Temper outbursts (explosive/unpredictable)9. Defiant10. Uncooperative
Multi-gating screening (2)• Conners Abbreviated Symptom
Questionnaire (CASQ)
• 10 items• Inattention
• Overactivity
• Impulsivity
Results for consideration1.5 SD on the IOWA Conners were
judged as situationally disruptive in the classroom rather than pervasively disruptive
2.0 SD on the IOWA Conners and CASQ more conservative; students in most need of services
Systematic Screening for Behavior Disorders • Walker and Severson, 1990• Items on externalizing and internalizing factors• 1st gate: Classroom teacher nomination of 3
students• 2nd gate: Teacher rating of 3 highest on
external and internal factors; best for behavior problems
• 3rd gate: School staff assess on 2 measures of school adjustment with direct observations
• Critical Events Index: 33 External & internal behavior problems; Least expensive screen
Supplemental measures• Academic Engaged Time: 2, 20 minute
classroom observations; % engaged
• Peer Social Behavior: 2, 20 minute playground observations; level, quality, and distribution of behavior
School Social Behavior Scale (Merrill, 1993)Two broad domains
1. Social competence: interpersonal skills, self-management skills and academic skills
2. Anti-social behavior: hostile-irritable, violation of school rules, disruptions of school activities
• Teacher or staff student ratings; 65 items
• Not so great on internal dimension
Revised Behavior Problem Checklist (Quay & Peterson, 1987)• Teacher rating scale; ages 5 to 18
• Lack of representative national norms
• Spanish translation
• 89 items; 20 minutes
Psychosocial Constraints Severson, H.H., Walker, H.M., Hope-Doolittle, J.,
Kratochwill, T.R. & Gresham, F.M. (2007). Proactive, early screening to detect behaviorally at-risk students. Journal of School Psychology, 45, 193-223.
Sameroff. A.J., & Rosenblum, K.L. (2006). Annals of New York Academy of Science, 1094, 116-124.
Sameroff, A.J., Peck, S.C., and Eccles. J. S. (2004). Development and psychopathology, 16, 873-896.
Phelan, P., Yu, H.C., & Davidson, A.L. (1994). Navigating the psychosocial pressures of adolescence: The voices and experiences of high school youth. American Educational Research Journal, 31 (2), 415-447.
School-wide Information System• Horner and Sugai
• Web-based software system
• Recording, entering, ordering, and reporting Office Detention Referrals
• http://SWIS.org
Thank YouOn the web @ nrcld.org
Daryl [email protected]