View
221
Download
0
Category
Tags:
Preview:
Citation preview
Using Local Norms to Using Local Norms to Support RtI PracticesSupport RtI Practices
NASP NASP
20102010
AbstractThe purpose of this presentation is to show how local norms can support RtI practices in a large, urban school district.
Topics discussed will include: Rationale and creation of local norms, The potential benefits and limitations of their use How the assessment of relative risk can improve
the efficiency of resource allocation in the context of an RtI framework
RtIRtI RtI is the practice of (1) providing high quality RtI is the practice of (1) providing high quality
instruction/intervention instruction/intervention MATCHED TO MATCHED TO STUDENT NEEDSTUDENT NEED and (2) using learning rate and (2) using learning rate over time and level of performance to (3) make over time and level of performance to (3) make important educational decisions. These important educational decisions. These components of RtI are essential to the components of RtI are essential to the development of a successful RtI development of a successful RtI implementation strategy.implementation strategy.¹¹
Matching students to instruction cannot be Matching students to instruction cannot be done when you define this based on students done when you define this based on students other than the ones in front of you.other than the ones in front of you.
1 Batsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J., Prasse, D., Reschly, D., 1 Batsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J., Prasse, D., Reschly, D., Schrag, J., & Tilly, D. (2005). Schrag, J., & Tilly, D. (2005). Response to Intervention: PolicyConsiderations and Response to Intervention: PolicyConsiderations and Implementation. Alexandria, VA: National Association of State Directors of Special Implementation. Alexandria, VA: National Association of State Directors of Special Education.Education.
RtI TriangleRtI Triangle While RtI models typically use triangles to While RtI models typically use triangles to
illustrate the “ideal” distribution of students to illustrate the “ideal” distribution of students to resources, reality often looks very different.resources, reality often looks very different.
It is common in many at-risk schools to see an It is common in many at-risk schools to see an inversion of where the resources are allocated.inversion of where the resources are allocated.
It is also common to see a hole or a gap at tier 2.It is also common to see a hole or a gap at tier 2. Often the initial distribution of students into the Often the initial distribution of students into the
tiers is based on their proficiency status.tiers is based on their proficiency status.Tier 3_____________________5%
Tier 2__________________15%
Tier 1_____________80%
50%___________Tier 3
10% ___________Tier 2
40% __________Tier 1
A Look at the Status QuoA Look at the Status Quo Impact of Lawsuit and Building CoordinationImpact of Lawsuit and Building Coordination Results have been an increase in the number of total initial referrals Results have been an increase in the number of total initial referrals Psychologist responsibilities are centered on building coordination Psychologist responsibilities are centered on building coordination
and evaluations, leaving less time for pre-referral activitiesand evaluations, leaving less time for pre-referral activities
4000
4500
5000
5500
6000
6500
2003-2004 2004-2005 2005-2006 2006-2007 2007-2008 2008-2009
School Year
To
tal
Init
ial
Ref
erra
ls
Impact on School ResourcesImpact on School Resources Most students referred (whether or not the Most students referred (whether or not the
referral is appropriate) for an evaluation referral is appropriate) for an evaluation ultimately require formal testingultimately require formal testing
Costs of Special Ed ReferralsCosts of Special Ed Referrals A full evaluation can be valued as high as A full evaluation can be valued as high as
$3000 (Mirkin & Potter, 1983)$3000 (Mirkin & Potter, 1983) Also consider the costs of special educational Also consider the costs of special educational
programming programming The shift to RtI will largely impact the role The shift to RtI will largely impact the role
of the school psychologistof the school psychologist
VanDerHeyden, A., Joseph, C., and Gilbertson, D. (2007)VanDerHeyden, A., Joseph, C., and Gilbertson, D. (2007)
Where Does This Put Us and Where Does This Put Us and Where Do We Want to GoWhere Do We Want to Go
A large urban district, with limited financial and A large urban district, with limited financial and instructional resourcesinstructional resources
Necessity to identify at-risk students and provide Necessity to identify at-risk students and provide services to improve overall district performanceservices to improve overall district performance
An increasingly inefficient evaluation process for An increasingly inefficient evaluation process for special educationspecial education
Limited and inconsistent success in Limited and inconsistent success in implementing RtI using national normsimplementing RtI using national norms
Using Local Norms may address these issues…Using Local Norms may address these issues…
What is the Goal?What is the Goal? The students’ skills have not changed just The students’ skills have not changed just
because we look at their data differentlybecause we look at their data differently But with limited resources we can channel But with limited resources we can channel
them in ways that can increase the systems them in ways that can increase the systems capacity to be effective.capacity to be effective.
Proficiency is still the destination. Using local Proficiency is still the destination. Using local data to support RtI practices lets us get their data to support RtI practices lets us get their with the car we own, not the jet we wish we with the car we own, not the jet we wish we had.had.
Reading FirstReading First
DIBELSDIBELS Training and technical supportTraining and technical support The OSPS Data SystemThe OSPS Data System The use of available national cut scores to The use of available national cut scores to
allocate local district resourcesallocate local district resourcesdibels.uoregon.edu
© University of Oregon Center on Teaching and Learning. All rights reserved.
Grade Level Spring DataGrade Level Spring Data
When using National norm data When using National norm data
Tier 3: 12 %
Tier 2: 55 %
Tier 1: 33 %
Creating Local NormsCreating Local Norms
=100*percentrank(A:A,A2,2)=100*percentrank(A:A,A2,2)
Better than sliced bread!
Grade Level Spring DataGrade Level Spring Data
When using Local norm data When using Local norm data
Tier 3: 9 %
Tier 2: 24 %
Tier 1: 67 %
Determine Your ApproachDetermine Your Approach
Before attacking the data, identify…Before attacking the data, identify… How many children can be effectively How many children can be effectively
serviced (school/district)? serviced (school/district)? Does our core service need to change?Does our core service need to change? What will be the cut-offs for which tier?What will be the cut-offs for which tier?
RtI tiers defined by percentilesRtI tiers defined by percentiles 80-15-580-15-5 20-40-6020-40-60
Pros of Local NormsPros of Local Norms
Decreases bias in decision makingDecreases bias in decision making Increases ability to match instruction Increases ability to match instruction
to student needto student need Maintains proficiency expectationsMaintains proficiency expectations Illuminates identifiable patterns of Illuminates identifiable patterns of
performance and changes over timeperformance and changes over time
Stewart, L., & Kaminski, R. (2002) Stewart, L., & Kaminski, R. (2002) Stewart, L. & Silberglitt, B. (2008)Stewart, L. & Silberglitt, B. (2008)
Cons of Local NormsCons of Local Norms Can be misinterpretedCan be misinterpreted Does not define acceptable Does not define acceptable
performanceperformance Norms are not diagnostic in isolationNorms are not diagnostic in isolation Must adhere to appropriate testing Must adhere to appropriate testing
standards (be careful to ensure integrity standards (be careful to ensure integrity of administration)of administration)
Stewart, L., & Kaminski, R. (2002) Stewart, L., & Kaminski, R. (2002) Stewart, L. & Silberglitt, B. (2008)Stewart, L. & Silberglitt, B. (2008)
Not Lowering the BarNot Lowering the Bar Proficiency status and response are two different Proficiency status and response are two different
questions.questions. Proficiency tells us if a student has reached a Proficiency tells us if a student has reached a
destination. For WI it is officially assessed in the fall destination. For WI it is officially assessed in the fall with results unavailable till 3/4with results unavailable till 3/4 thth of the school year is of the school year is over.over.
Response to local normative data tells us if a Response to local normative data tells us if a student is on the right track, and continued student is on the right track, and continued monitoring will estimate when they’ll reach their monitoring will estimate when they’ll reach their destination.destination.
A student’s response to their education is evident A student’s response to their education is evident by using local data to look at growth.by using local data to look at growth.
Further Use of Local Norms
Can be used to predict outcomes such as: Ending benchmark risk status Proficiency on state assessments Graduation rate
Can use receiver operator characteristic curves to predict binary outcomes
Creating Cut Scores for Risk on Creating Cut Scores for Risk on Binary OutcomesBinary Outcomes
Receiver Operating Receiver Operating Characteristic Curve Characteristic Curve (ROC Curve) (ROC Curve) A graphical plot of A graphical plot of
sensitivity vs. 1-sensitivity vs. 1-specificity as the specificity as the threshold point is threshold point is varied.varied.
What Does a ROC Curve Do?What Does a ROC Curve Do?
Plots sensitivity (y-axis) and 1-specificity Plots sensitivity (y-axis) and 1-specificity (x-axis) to determine accuracy of (x-axis) to determine accuracy of predictionprediction
In choosing a cut score, factors of In choosing a cut score, factors of sensitivity sensitivity andand specificity specificity need to be need to be understood. understood.
SensitivitySensitivity
Sensitivity is the proportion of true positives that Sensitivity is the proportion of true positives that were correctly classified as positive given the were correctly classified as positive given the threshold or cut score used to establish it.threshold or cut score used to establish it.
In terms of sickness it is the proportion of people In terms of sickness it is the proportion of people that actually are sick that were correctly that actually are sick that were correctly classified as being sick (strep)classified as being sick (strep)
In proficiency terms, it is the proportion of In proficiency terms, it is the proportion of students that actually are not proficient that were students that actually are not proficient that were correctly classified as at-risk.correctly classified as at-risk.
SpecificitySpecificity Specificity is the proportion of true negatives that Specificity is the proportion of true negatives that
were correctly classified as negative.were correctly classified as negative. In terms of sickness it is the proportion of people In terms of sickness it is the proportion of people
that are healthy that were correctly classified as that are healthy that were correctly classified as healthy.healthy.
In terms of proficiency it is the proportion of In terms of proficiency it is the proportion of students that are proficient that were correctly students that are proficient that were correctly classified as proficient.classified as proficient.
1-specificity is the proportion of students that are 1-specificity is the proportion of students that are incorrectly classified as being at-risk when in incorrectly classified as being at-risk when in fact they are proficient.fact they are proficient.
The ROC CurveThe ROC Curve The dashed red The dashed red
line represents line represents test B in terms of test B in terms of diagnostic utility. diagnostic utility. As the threshold As the threshold varies (cut varies (cut scores) the test scores) the test does no better does no better than random at than random at correct correct classificationclassification
Test A has more Test A has more utility and utility and
Area Under the CurveArea Under the Curve
Test Result Variable(s):Winter ORF Score
Area Std. ErrorAsymptotic
Sig.
Asymptotic 95% Confidence Interval
Lower Bound
Upper Bound
.950 .014 .000 .922 .978
Coordinates of the CurveCoordinates of the CurveTest Result Variable(s):Winter ORF Score (Predicting Spring Percentrank)
Positive if Less Than or Equal To Sensitivity 1 - Specificity
5.50 .381 .011
6.50 .476 .022
7.50 .571 .027
8.50 .714 .049
9.50 .762 .076
10.50 .810 .097
11.50 .881 .114
12.50 .905 .130
13.50 .905 .146
14.50 .929 .168
Using ROC Curves to Produce Local Cuts Based on
State Test Scores
Winter/Spring AUCWinter/Spring AUC
Lower Bound
Upper Bound
Winter ORF
.829 .013 .000 .802 .855
Spring ORF .857 .012 .000 .833 .881
Area Under the Curve
Test Result Variable(s) Area Std. Errora
Asymptotic
Sig.b
Asymptotic 95%
Winter 1Winter 1stst Grade ORF Grade ORF
Positive if Less Than or Equal
Toa Sensitivity1 -
Specificity
15.50 .643 .16916.50 .669 .18117.50 .693 .19618.50 .716 .21919.50 .736 .24420.50 .755 .26521.50 .780 .28922.50 .791 .30523.50 .801 .31724.50 .814 .339
Coordinates of the Curve
Spring 1Spring 1stst Grade ORF Grade ORF
Positive if Less Than Sensitivity 1 - Specificity
38.50 .779 .21439.50 .788 .21840.50 .792 .22641.50 .799 .23342.50 .807 .24443.50 .814 .25744.50 .829 .27145.17 .838 .28545.67 .838 .28746.50 .844 .316
Coordinates of the Curve
Test Result Variable(s):Spring ORF
DIBELS Oral Reading Fluency Risk Indicators
Local and National Norms Fall Winter Spring Grade Level/
Risk Category Local National Local National Local National
1 At Risk N/A N/A ≤ 10 ≤ 11 ≤ 24 ≤ 19 1 Some Risk N/A N/A 11 – 16 12 – 18 25 – 37 20 – 39 1 Low Risk N/A N/A > 16 > 18 >37 >39 2 At Risk ≤ 21 ≤ 28 ≤ 30 ≤ 49 ≤ 50 ≤ 69 2 Some Risk 22 – 31 29 – 44 31 – 55 50 – 70 51 – 73 70 – 89 2 Low Risk > 31 > 44 > 55 > 70 > 73 > 89 3 At Risk ≤ 44 ≤ 54 ≤ 53 ≤ 63 ≤ 67 ≤ 79 3 Some Risk 45 - 58 55 – 75 54 – 70 64 – 87 68 – 89 80 – 109 3 Low Risk > 58 > 75 > 70 > 87 > 89 > 109 4 At Risk ≤ 55 ≤ 68 ≤ 67 ≤ 82 ≤ 74 ≤ 95 4 Some Risk 56 – 67 69 – 88 68 – 82 83 – 102 75 – 97 96 – 117 4 Low Risk > 68 > 88 > 82 > 102 > 97 > 117 5 At Risk ≤ 76 ≤ 87 ≤ 76 ≤ 93 ≤ 88 ≤ 102 5 Some Risk 77 – 93 88 – 100 77 – 97 94 – 108 89 – 110 103 – 123 5 Low Risk > 93 > 100 > 97 > 108 > 110 > 123
Making Data Actionable
Use the data to determine: How instruction can be matched to student
need Whether or not the core instruction is effective Goals Progress monitoring frequency When the goal will be achieved by their
current rate of growth Whether or not interventions are effective
Potential benefits to using local Potential benefits to using local Norms within RtINorms within RtI
• Local data takes into consideration the Local data takes into consideration the instructional environment of the studentinstructional environment of the student
• Appropriate use of local data could help improve Appropriate use of local data could help improve overall instruction by making assessment more overall instruction by making assessment more meaningful meaningful
• Improved “hit rate”Improved “hit rate”
• Creating your own triangle, instead of using Creating your own triangle, instead of using someone else’s rectanglesomeone else’s rectangle
Potential Outcomes and UsesPotential Outcomes and UsesExpanded role of school psychologistsExpanded role of school psychologists
• Shift in the types of assessments towards Shift in the types of assessments towards screening, progress monitoring and interventionsscreening, progress monitoring and interventions
• This shift will result in more direct involvement in This shift will result in more direct involvement in student learningstudent learning
• Helping schools make the connections between Helping schools make the connections between assessment and instructional decision makingassessment and instructional decision making
• New roles may develop in the areas of data New roles may develop in the areas of data analysis and could include such things as whole analysis and could include such things as whole school data collection, management, and school data collection, management, and interpretationinterpretation
• Training other school personnelTraining other school personnel in the in the understanding and interpretation of local dataunderstanding and interpretation of local data
For more information:For more information:
www2.milwaukee.k12.wi.us/rti www2.milwaukee.k12.wi.us/rti Margaret Peters, Margaret Peters, kreulm@milwaukee.k12.wi.us Kelly Witz, Kelly Witz, witzka@milwaukee.k12.wi.us Robert Latterman, Robert Latterman, latterrl@milwaukee.k12.wi.us Marc Sanders, Marc Sanders, sandermc@milwaukee.k12.wi.us Steve Smith, Steve Smith, smithsl2@milwaukee.k12.wi.us
Handouts are available for download on the Handouts are available for download on the NASP websiteNASP website
ReferencesBatsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J., Prasse, D., Reschly, D., Schrag, J., &
Tilly, D. (2005). Response to Intervention: PolicyConsiderations and Implementation. Alexandria, VA: National Association of State Directors of Special Education.
Good, R. H., III, & Kaminski, R. A. (1996). Assessment for instructional decisions: Toward a proactive/prevention model of decision-making for early literacy skills. School Psychology Quarterly, 11(4), 326- 336.
Good, R. H., Wallin, J., Simmons, D. C., Kame’enui, E. J., & Kaminski, R. A. (2002). System-wide Percentile Ranks for DIBELS Benchmark Assessment (Technical Report 9). Eugene, OR: University of Oregon.
Hintze, J., & Silberglitt, B. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review, 34(3), 372-386.
Shinn, M. (1998). Advanced applications of Curriculum-Based Measurement. New York, NY US: Guilford Press.
Stewart, L., & Kaminski, R. (2002). Best practices in developing local norms for academic problem solving. Best Practices in School Psychology IV (Vol. 1, Vol. 2) (pp. 737-752). Washington, DC US: National Association of School Psychologists.
Stewart, L., & Silberglitt, B. (2008). Best practices in developing academic local norms. Best Practices in School Psychology V (Vol. 2) (pp. 225-242). Washington, DC US: National Association of School Psychologists.
VanDerHeyden, A., Joseph, C., and Gilbertson, D. (2007) A multi-year evaluation of the effects of a Response to Intervention (RtI) model on identification of children for special education. Journal of School Psychology, 45, 225-256..
Recommended