Setting Performance Standards
Grades 5-7 NJ ASK
NJDOE
Riverside Publishing
May 17, 2006
2
What do we mean by “Standard Setting”?• Content standards: specify the content that
should be learned by students – given implementation of the state standards and an opportunity to learn.
• Performance standards: specify the precise level of performance required on an assessment to achieve some category or outcome. How much must students know and be able to do?
3
Proficiency Levels on NJ ASK
AdvancedProficient
PartiallyProficient Proficient
ProficientCut Score
AdvancedCut Score
100 - 199200 - 249
250 - 300
4
NJ ASK Grades 5-7 Standard Setting
Phase 1 20 panelists on the Math Grades 5-7 & LAL
Grade 7 panels 19 panelists on the LAL Grades 5-6 panels Used a Modified Angoff procedure to
calculate cut scores in a 2-day workshop Phase 2
Representatives from Phase 1 and state and local policy makers reviewed results
Examined consistency across grades and subjects
5
The Angoff Procedure• Research-based procedure used since
the early 1970s• The most commonly used standard
setting method — used in many other state testing programs as well as on certification tests
• Has undergone many modifications over the years and is often referred to as the Modified Angoff or Extended Angoff procedure
6
Angoff Procedure (con’t.)• Original Angoff procedure asked
panelists to examine each item on a test and determine whether a student who was just barely Proficient would answer it correctly
• A common modification is to ask “If we had 100 barely Proficient students in the room, how many of them would answer this item correctly?”
7
The Angoff Task
Review each item For MC items, determine how many out
of 100 just barely Proficient and just barely Advanced Proficient students should answer each item correctly
For OE items, determine the average score a just barely student at each performance level should receive
8
Validity Considerations The process was well-planned – see
standard setting proposal. All policy, operational and technical
decisions were reasonable and well-documented.
The judges were satisfied with the process and its results – see the evaluation results.
9
The Final Decision on Cut Scores – Reminder
Legitimate authority should have all the information they want or need to make an informed decision
At a minimum it should include: Overall results – Phase 1 and Phase 2 Variance in judgments and in measurement Impact on subpopulations in the state Impact on accountability Documentation of the process (with as
much detail as is necessary)
10
Ratings Cut scores are determined over three
rounds of ratings Ratings were made independently After each set of ratings, panelists
saw their cut scores and the average cut scores
There was a discussion between each round
11
Calculating the Cut Score Consider the MC ratings as probabilities We sum the probabilities to obtain a cut score for
the MC items and add the average scores for the OE items to get the total cut score
Item RatingPossible1 .85 12 .55 13 .40 14 1.5 3
Total= .85 + .55 + .40 + 1.5 = 3.3 points out of a possible 6 points
12
“Just Barely” Performance The idea of just barely performance
is key to the Angoff procedure We spent time talking about what it
means to be just barely Proficient and Advanced Proficient
Our discussions were driven by the NJ ASK LAL and Math Grades 5-7 Performance Level Descriptors (PLDs)
13
Phase 2 Meeting Policymakers and stakeholders gathered
to review the results of Phase 1 in light of NJ ASK program goals.
Reviewed results from Phase 1 along with standard deviation of judgments and impact data.
Compared results of Grades 3, 4 & 8 LAL and Math with Grades 5-7 LAL and Math and discussed the need for consistency across grades and content areas.
14
Phase 2 Results: Final Cut Score Recommendations Committee produced cut score recommendations for
NJDOE and the State Board:
ProficientAdvanced Proficient Proficient
Advanced Proficient
Grade 5 16 30 18 30Grade 6 20 32 17 31Grade 7 21 34 13 26
LAL Math
15
Evidence the Results Were Technically Sound
Survey Results (see handout) Standard deviation of judgments in each round:
ProficientAdvanced Proficient Proficient
Advanced Proficient
Round1 2.6 2.1 3.2 3.12 2.2 2.4 2.6 2.53 1.6 1.7 0.4 0.4
1 2.5 3.3 2.0 2.42 2.9 4.4 1.6 0.23 0.6 1.4 0.7 0.9
1 2.2 2.0 2.6 2.62 3.5 3.4 2.0 2.33 2.4 2.2 0.8 1.1
SD of Judgments
Grade 5
Grade 6
Grade 7
LAL Math
SD of Judgments
16
Distribution of Students by SubgroupLAL Grade 5
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 8.1 6.7 9.6 11.3 1.8 2.0 19.0 10.0 1.0 0.6 2.0
P 70.7 70.3 71.3 77.4 59.6 62.4 73.0 76.5 50.2 42.0 60.2
PP 21.2 23.0 19.1 11.3 38.6 35.6 8.0 13.5 48.8 57.4 37.8
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
17
Distribution of Students by SubgroupLAL Grade 6
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 7.4 5.8 9.1 10.7 1.0 1.7 15.0 8.9 0.9 1.0 1.3
P 56.5 55.1 58.2 66.2 41.2 42.1 62.4 63.0 32.1 22.2 40.9
PP 36.1 39.1 32.7 23.1 57.8 56.2 22.6 28.1 67.0 76.8 57.8
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
18
Distribution of Students by SubgroupLAL Grade 7
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 5.7 3.9 7.5 8.1 1.5 1.2 12.2 6.9 0.6 0.5 1.2
P 56.8 54.3 59.5 67.3 39.9 41.8 69.2 63.9 31.0 18.8 38.4
PP 37.5 41.8 33.0 24.6 58.6 57.0 18.6 29.2 68.4 80.7 60.4
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
19
Distribution of Students by SubgroupMath Grade 5
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 22.6 24.0 21.2 29.4 7.7 10.2 48.9 26.7 7.0 7.6 9.5
P 56.0 53.5 58.7 59.1 51.6 54.8 45.1 59.2 44.4 40.9 51.4
PP 21.4 22.5 20.1 11.5 40.7 35.0 6.0 14.1 48.6 51.5 39.1
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
20
Distribution of Students by SubgroupMath Grade 6
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 13.4 14.4 12.6 18.1 3.3 4.3 32.2 16.1 2.5 3.0 3.8
P 53.0 51.6 54.6 62.5 36.6 41.5 54.4 59.0 28.7 25.5 38.0
PP 33.6 34.0 32.8 19.4 60.1 54.2 13.4 24.9 68.8 71.5 58.2
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
21
Distribution of Students by SubgroupMath Grade 7
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 11.8 13.5 10.2 16.7 2.2 2.7 30.6 14.4 1.4 2.1 2.5
P 47.6 46.5 48.9 56.9 31.1 37.2 52.2 53.7 22.0 24.7 32.5
PP 40.6 40.0 40.9 26.4 66.7 60.1 17.2 31.9 76.6 73.2 65.0
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
22
LALPhase 2 Results
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Per
cen
t in
Lev
el
AP
P
PP
AP 4.4 4.4 8.1 7.4 5.7 8.0
P 78.8 77.2 70.7 56.5 56.8 64.3
PP 16.7 18.4 21.2 36.1 37.5 27.7
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Final Recommended CutscoresLAL Grades 5-7 (with LAL Grades 3,4 &8)
23
MathPhase 2 Results
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Per
cen
t in
Lev
el
AP
P
PP
AP 28.8 31.7 22.6 13.4 11.8 18.7
P 53.7 48.5 56.0 53.0 47.6 43.7
PP 17.5 19.8 21.4 33.6 40.6 37.6
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Final Recommended CutscoresMath Grades 5-7 (with Math Grades 3,4 &8)
24
Key Statistics from Surveys