Upload
hayden-maher
View
214
Download
2
Tags:
Embed Size (px)
Citation preview
Making Data Driven Decisions:Cut points, Curve Analysis and Odd Balls
Laura Lent
IU 13
Pennsylvania
District-Wide RtI Development
• Essential Question: How do you use data to facilitate systemic paradigm shifts?
District Demographics
• 3200 students total• 5 Elementary Buildings, 2 Secondary• 4 K-4 buildings ranging from 110 to 500
students• 1 5-6 building of 630 students• 25-35% FRL• <15% minority• Equivalent of 2.5 full-time school
psychologists
PSSA Scores: Reading
• Warning=(W), School Improvement 1=SI 1
Grade 2006 2007 2008 2009
3 75 80 85 81
4 70 74 72 75
5 59 66 (W) 60 (SI 1) 62 (SI 1)
6 69 73 70 72
Intensive Reform Focus
• 5th/6th grade building the “identified patient” due to School Improvement status.
• Building has 5 teacher teams of a pair of 5th grade teachers with a pair of 6th grade teachers.
• Two 5th grade classrooms allocated as “ELM” or Essentials of Literacy and Math.
• Each team has a learning support teacher assigned to it who does inclusion during social studies and science.
• No universal screening. No consistent reading instructional practices.
Data Collection
• Use of CBMs for Universal Screening and Progress Monitoring including:
• Early Literacy, Early Numeracy, MAZE, R-CBM, Math Applications and Single Digit Computation.
• K Screening: Use of Individual Growth and Development Indicators (IGDIs)-Picture Naming and Rhyming and Early Numeracy in August to form heterogeneous classes.
Benchmark Data Analysis
• Move from DIBELS to AIMSweb allowed for determination of local benchmark criteria to be used in the reporting system.
• Question: How do we set benchmarks that are sensitive and specific to PSSA performance?
ROC Curve Analysis
• Receiver Operating Characteristic (ROC)
• Statistical Evaluation Process to identify benchmarks by identifying cut scores that are the most sensitive without sacrificing specificity.
• Sensitivity: Fewest False Negatives
• Specificity: Fewest False Positives
Scatterplot Interpretation of Screening Results
False Positives/
“Happy Surprises”
Adapted from Silberglitt (2009).
True Negatives
True Positives False Negatives/
“Unhappy Surprises”
PS
SA
P
rofi
cien
t
R-CBM
Low Risk
Benefits of ROC Curve Analysis
• Re-set Benchmarks for Greater Classification Accuracy.
• Minimize the “word caller” hysteria.
• Fringe Benefits: “Odd Ball” or Outlier Score Investigation.
Acknowledgement
Thanks to Dr. Edward Shapiro and Dr. Gini Hampton, Center for Promoting Research to Practice, Lehigh University, for sharing the proceeding slides of ROC curve analysis for this district.
ROC Curve Analysis
• R-CBM District Benchmarks- 25th %tile
Grade Score - WCPM
3 80
4 104
5 114
6 130
Probability Outcomes based on AIMSweb (25th %tile) Spring Benchmarks
• Grade 4 = 104
Adv/Proficient
Basic/Below Basic Total
RCBM BMK=104 Low Risk Count 172 28 200% within RCBM BMK=104
86.0% 14.0% 100.0%
% of Total 68.3% 11.1% 79.4%Some Risk Count 15 20 35
% within RCBM BMK=104
42.9% 57.1% 100.0%
% of Total 6.0% 7.9% 13.9%At Risk Count 1 16 17
% within RCBM BMK=104
5.9% 94.1% 100.0%
% of Total .4% 6.3% 6.7%Total Count 188 64 252
% within RCBM BMK=104
74.6% 25.4% 100.0%
% of Total 74.6% 25.4% 100.0%
Probability Outcomes based on Reset R-CBM Spring Benchmarks
• Grade 4 = 117 Reading PSSA Level
Adv/Prof Basic
Below Basic Total
RCBM ROC B=117
Low Risk Count 143 18 0 161
% within RCBM ROC B=117 88.8% 11.2% .0% 100.0%
% of Total 56.7% 7.1% .0% 63.9%
Some/At Risk
Count 45 27 19 91
% within RCBM ROC B=117
49.5% 29.7% 20.9% 100.0%
% of Total 17.9% 10.7% 7.5% 36.1%
Total Count 188 45 19 252
% within RCBM ROC B=117 74.6% 17.9% 7.5% 100.0%
% of Total 74.6% 17.9% 7.5% 100.0%
Probability Outcomes based on AIMSweb (25th %tile) Spring Benchmarks
• Grade 5 = 114
Adv/Proficient
Basic/Below Basic Total
RCBM BMK=114 Low Risk Count 167 72 239% within RCBM BMK=114
69.9% 30.1% 100.0%
% of Total 58.0% 25.0% 83.0%Some Risk Count 7 20 27
% within RCBM BMK=114
25.9% 74.1% 100.0%
% of Total 2.4% 6.9% 9.4%At Risk Count 2 20 22
% within RCBM BMK=114
9.1% 90.9% 100.0%
% of Total .7% 6.9% 7.6%Total Count 176 112 288
% within RCBM BMK=114
61.1% 38.9% 100.0%
% of Total 61.1% 38.9% 100.0%
Probability Outcomes based on Reset R-CBM Spring Benchmarks
• Grade 5 = 140 Reading PSSA Level
Adv/Prof Basic
Below Basic Total
RCBM Bmk=140 Gr 5
Low Risk Count 135 29 7 171
% within RCBM Bmk=140 Gr 5
78.9% 17.0% 4.1% 100.0%
% of Total 46.9% 10.1% 2.4% 59.4%
Some/At Risk
Count 41 40 36 117
% within RCBM Bmk=140 Gr 5
35.0% 34.2% 30.8% 100.0%
% of Total 14.2% 13.9% 12.5% 40.6%
Total Count 176 69 43 288
% within RCBM Bmk=140 Gr 5
61.1% 24.0% 14.9% 100.0%
% of Total 61.1% 24.0% 14.9% 100.0%
Probability Outcomes based on AIMSweb (25th %tile) Spring Benchmarks
• Grade 6 = 130
Adv/Proficient
Basic/Below Basic Total
RCBM BMK=130 Low Risk Count 176 39 215
% within RCBM BMK=130
81.9% 18.1% 100.0%
% of Total 66.2% 14.7% 80.8%Some Risk Count 11 25 36
% within RCBM BMK=130
30.6% 69.4% 100.0%
% of Total 4.1% 9.4% 13.5%At Risk Count 0 15 15
% within RCBM BMK=130
.0% 100.0% 100.0%
% of Total .0% 5.6% 5.6%Total Count 187 79 266
% within RCBM BMK=130
70.3% 29.7% 100.0%
% of Total 70.3% 29.7% 100.0%
Probability Outcomes based on Reset R-CBM Spring Benchmarks
• Grade 6 = 154
Adv/ProficientBasic/Below
Basic TotalRCBM Bmk=154 Gr 6 Low Risk Count 144 23 167
% within RCBM Bmk=154 Gr 6 86.2% 13.8% 100.0%
% of Total 54.1% 8.6% 62.8%
Some/At Risk Count 43 56 99
% within RCBM Bmk=154 Gr 6 43.4% 56.6% 100.0%
% of Total 16.2% 21.1% 37.2%
Total Count 187 79 266
% within RCBM Bmk=154 Gr 6 70.3% 29.7% 100.0%
% of Total70.3% 29.7% 100.0%
ROC Curve Analysis to Reset Cut Points
• Reset Spring Targets: R-CBM Proficient
Grade Spring R-CBMScore
Spring R-CBM
Percentile3 93 17th
4 117 37th
5 140 42nd
6 154 39th
What to do about those odd balls?
To investigate the scores that were false positives and false negatives or “happy surprises” and “unhappy surprises” the following variables were examined:
• Current and historical PSSA performance• Demographic characteristics including
ELL, Economic Disadvantage (ED), IEP status, gender and teacher assignment.
False Positive Results
• Students who were “at risk” or below the 10th percentile on the spring R-CBM but scored Proficient or above on PSSA.
Grade 3 4 5 6
Number of Scores
0 1 2 0
“Extreme” False Negative Results
or Very Unhappy Surprises
• Students who were “low risk” on spring RCBM and Below Basic on PSSA.
Grade 3 4 5 6
Number of False Positives
4 1 17 6
Who are these students?
• 5th Grade (n=17):– 11 Male, 6 Female– 7 with IEPs, 4 ED, 1 ELL– 3 scored Proficient in 4th grade– Teacher distribution: all had at least 1, three teachers
held 3 or more.• 6th Grade (n=6):
– 5 Female, 1 Male– 1 with IEP, 1 ELL– None scored Proficient in 5th grade
“Near”False Negative Results
• Students who scored “low risk” on RCBM and Basic on PSSA.
Grade 3 4 5 6
Number of False Positive Scores
13 27 55 33
Who are these kids?
• Grade 3:– Evenly distributed across schools (n=3, n=5, n=5)– 3 students ED, 3 students IEP, 1 ELL
• Grade 4:– Evenly distributed across schools (n=7, n=10, n=8)– 5 students “retainees”, 5 ED
• *12/23 were students who previously scored Proficient!
• 5th Grade:
• 12/55 retentions, 12 ED (1 overlap), 2 IEP
• 31 fell from Proficient to Basic
• 2 fell from Advanced to Basic
• *33/47 or 70% were students who previously scored Proficient!
• 6th Grade:
• 6 were retainees, 5/ED, 4/IEP, 0 ELL
• 11/33 or 33% fell from Proficient to Basic!
Teacher Distribution: 5th Grade
• Total # False Negatives =72
• Of the 12 teachers, “F” had the most false positives outside of the combined “ELM” classes.
T A B C D E F G H ELM
Total#FalseNeg.
3 6 5 4 3 10 5 5 22
I
5
J
4
Teacher Distribution: 6th Grade
• Total Number of False Negatives=33
T A B C D E F G H I#
False
Neg
1 6 2 2 4 8 4 5 2
Local Impact of “Odd Ball” Analysis
• Consensus-Building: • Administration conceded that major infrastructure and
implementation changes were needed. Staff agreed.
• Infrastructure:• 5/6 teams were restructured into 5th and 6th only teams. Weaker
teams paired with stronger and 5th grade classrooms moved closer to the office.
• Implementation:• 6 full days professional development on core reading instruction.• Administrators attend all inservice trainings and all team meetings.• R-CBM targets re-set for 09-10 to attempt to better identify students
who need targeted intervention. • Targeted intervention provided daily for 45 minutes by all classroom
teachers and interventionists. Groups matched to identified need.
Questions for Further Analysis
• Do the False Negatives fail to identify those with comprehension problems or fail to identify those who are failing to receive adequate instruction?
• Would MAZE identify the same sample of False Negatives as “at risk”?
• Would a diagnostic or benchmark measure based on grade level standards have identified these students as “at risk”?