Upload
charity-stafford
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Presentation at the Fall 2011 Meeting of theMichigan Educational Research Association
2
Identifying MME Cut Scores
3
University 2-Year Institution 2-Year Institution
Central Michigan University Alpena Community College Mid Michigan Community College
Eastern Michigan University Delta College Monroe County Community College
Ferris State University Glen Oaks Community College Montcalm Community College
Grand Valley State University Gogebic Community College Mott Community College
Michigan Technological University
Grand Rapids Community College
Muskegon Community College
Michigan State University Henry Ford Community College North Central Michigan College
Oakland University Jackson Community College Northwestern Community College
Northern Michigan University Kalamazoo Valley Community College
Oakland Community College
Saginaw Valley State University
Kellogg Community College Schoolcraft College
The University of Michigan-Ann Arbor
Kirtland Community College Southwestern Michigan College
University of Michigan-Dearborn
Lake Michigan College St. Clair County Community College
University of Michigan-Flint Lansing Community College Washtenaw Community College
Wayne State University Macomb Community College West Shore Community College
Western Michigan University
4
MME content area College courses usedMathematics College Algebra.
Reading
Courses identified by 4-year universities.
Reading-heavy courses such as entry-level literature, history, philosophy, or psychology for 2-year universities.
Science
Courses identified by 4-year universities.
Entry level biology, chemistry, physics, or geology for 2-year universities.
Social Studies
Courses identified by 4-year universities.
Entry level history, geography, or economics for 2-year universities.
5
Grades were put on a numeric scale from 0-4 0 = F 1 = D 2 = C 3 = B 4 = A Not used
o AU, AWF, DR, R, RA, FR, T, TR, X Coded as 3.0
o P, CR Coded as 0.0
o IN, N, NC, NE, NS, W, WF, WP, WX, and U
6
MME Content
AreaSample
Size
Percent B or
higher
Course Grade MME Score
Mean SD Mean SD
Math 6,286 47.0 2.49 1.18 1112.2 13.2
Reading 37,952 54.9 2.64 1.23 1117.2 24.6
Social Studies
39,721 54.4 2.63 1.22 1135.4 26.3
Science 15,608 50.0 2.54 1.19 1123.5 23.5
7
MME Content Area
Course Type Number of Students
Mathematics College Algebra 6567
Reading
Literature 456
American History 1731
Other History 3010
Psychology 16231
Sociology 8236
Political Science 6114
Philosophy 1869
Other 2517
8
MME Subject Area Course Type Number of Students
Science
Biology/Life Science 8355
General Chemistry 5807
Physics 535
Other 1483
Social Studies
American History 1734
Other History 3006
Psychology 16230
Sociology 8231
Geography 612
Political Science 6108
Economics 3498
Other 2361
9
Students receiving an A Students receiving a B or better Students receiving a C or better
Students receiving a B or better in 4-year universities
Students receiving a B or better in 2-year institutions
10
Logistic Regression (LR)o Identify score that gives a 50% probability of achieving an Ao Identify score that gives a 50% probability of achieving a B or
bettero Identify score that gives a 50% probability of achieving a C or
better Signal Detection Theory (SDT)
o Identify scores that maximize the proportion receiving consistent classifications from MME to college grades• i.e., both proficient/advanced and receiving a A/B/C or better• i.e., both not proficient/partially proficient and receiving a A-/B-/C- or
worseo Equivalent to LR under mild monotonicity assumptions
Selected SDT as the preferred method because of its purpose (maximizing consistent classification)
11
Where•success is obtaining an A/B/C or better•e is the base of the natural logarithm•β0 is the intercept of the logistic regression•β1 is the slope of the logistic regression•x is the MME score
12
13
14
15
16
Basic Idea Set the MME cut score to…
Maximize the number of students in the Consistent cells Minimize the number of students in the Inconsistent cells
Maximize consistent classification from MME to first-year college grades
MME(unknown cut score)
Freshman Grade (known cut score)
B- or Lower B or Higher
College Ready Inconsistent Consistent
Not College Ready Consistent Inconsistent
17
0 0.2 0.4 0.6 0.8 1 1.2
MM
E Co
nte
nt
Are
a Sc
ore
Grade in First Related Freshman Credit Bearing Course
Known Cut Score
B- or lower B or higher
18
0 0.2 0.4 0.6 0.8 1 1.2
MM
E Co
nte
nt
Are
a Sc
ore
Grade in First Related Freshman Credit Bearing Course
Consistently Classified
Inconsistently Classified
Known Cut Score
Unknown Cut Score
B- or lower B or higher
82.8%consistent classification
19
0 0.2 0.4 0.6 0.8 1 1.2
MM
E Co
nte
nt
Are
a Sc
ore
Grade in First Related Freshman Credit Bearing Course
Consistently Classified
Inconsistently Classified
Known Cut Score
Unknown Cut Score
B- or lower B or higher
82.8%consistent classificationAdjust the unknown cut
score to maximize consistent classification
20
0 0.2 0.4 0.6 0.8 1 1.2
MM
E Co
nte
nt
Are
a Sc
ore
Grade in First Related Freshman Credit Bearing Course
Consistently Classified
Inconsistently Classified
Known Cut Score
Unknown Cut Score
B- or lower B or higher
88.6%consistent classification
21
0 0.2 0.4 0.6 0.8 1 1.2
MM
E Co
nte
nt
Are
a Sc
ore
Grade in First Related Freshman Credit Bearing Course
Consistently Classified
Inconsistently Classified
Known Cut Score
Unknown Cut Score
B- or lower B or higher
90.45%consistent classification
22
Analyses treating grades of A as the success criterion produced unusable results (i.e., the highest possible MME scale scores
Analyses treating grades of C as the success criterion produced unusable results (i.e., MME scale scores below chance level)
Analyses treating 4-year and 2-year institutions did produce different cut scores, but they were within measurement error of each other
Used analyses based on all institutions and grades of B or better to produce MME cut scores
Used probability of success of 33% and 67% to set the “partially proficient” and “advanced” cut scores
SDT and LR produced very similar results Used SDT because it was the preferred methodology
23
Content Area
Classification
Consistency
Partially Proficient Cut Score
Proficient Cut Score
Advanced Cut Score
Mathematics 65% 1093 1116 1138
Reading 63% 1081 1108 1141
Science 67% 1106 1126 1144
Social Studies 63% 1097 1129 1158
24
Identifying MEAP Cut Scores
25
Cohort
Grade
3 4 5 6 7 8 9 10 11 12 13
1 - - - - - 05-06 06-07 07-08 08-09 09-10 10-11
2 - - - - 05-06 06-07 07-08 08-09 09-10 10-11 -
3 - - - 05-06 06-07 07-08 08-09 09-10 10-11 - -
4 - - 05-06 06-07 07-08 08-09 09-10 10-11 - - -
5 - 05-06 06-07 07-08 08-09 09-10 10-11 - - - -
6 05-06 06-07 07-08 08-09 09-10 10-11 - - - - -
7 06-07 07-08 08-09 09-10 10-11 - - - - - -
8 07-08 08-09 09-10 10-11 - - - - - - -
9 08-09 09-10 10-11 - - - - - - - -
10 09-10 10-11 - - - - - - - - -
26
Logistic Regression (LR)o Identify score that gives a 50% probability of achieving proficiency
on a later-grade test (i.e., MME or MEAP) Signal Detection Theory (SDT)
o Identify scores that maximize the proportion receiving consistent classifications from one grade to a later grade• i.e., proficient/advanced on both tests• i.e., not proficient/partially proficient on both tests
o Equivalent to LR under mild monotonicity assumptions Equipercentile Cohort Matching (ECM)
o Identify scores that give the same percentage of students proficient/advanced on both tests
Selected SDT as the preferred method because of its purpose (maximizing consistent classification)
However, SDT and LR are susceptible to regression away from the mean
27
Same as for identifying MME cut scores Criterion for success is proficiency on a later grade test
rather than getting a B or better in a related college course
28
Each dot represents a plot of test scores in grade 8 and grade 11 for a single student
29
Grade 11: ProficientGrade 11: Not proficient
30
Grade 8: ProficientGrade 11: Not proficient
Grade 8: ProficientGrade 11: Proficient
Grade 8: Not proficientGrade 11: Not proficient
Grade 8: Not ProficientGrade 11: Proficient
31
32
33
The more links in the chain, the greater the effects of regression
Original plan for Math and Readingo Link grade 11 MME to college gradeso Link grade 8 MEAP to grade 11 MMEo Link grade 7 MEAP to grade 8 MEAPo Link grade 6 MEAP to grade 7 MEAPo Link grade 5 MEAP to grade 6 MEAPo Link grade 4 MEAP to grade 5 MEAPo Link grade 3 MEAP to grade 4 MEAP
Original plan results in 7 links by the time the grade 3 cut is set
Original plan results in inflated cut scores in lower grades
34
Revised plan for Math and Reading For Grade 3, 4, 5, 6
o Link grade 11 MME to college gradeso Link grade 7 MEAP to grade 11 MMEo Link grade 3, 4, 5, or 6 MEAP to grade 7 MME
For Grade 7, 8o Link grade 11 MME to college gradeso Link grade 7 or 8 MEAP to grade 11 MME
Results in a maximum of three links for any one grade
35
No evidence of regression away from the mean in identifying MEAP “proficient” cut scoreso Looking for a consistently lower percentage of students
proficient as one goes down in gradeso Used SDT to identify MEAP “proficient” cut scores
Evidence of regression away from the mean in identifying MEAP “partially proficient” and “advanced” cut scoreso Increasingly smaller percentages of students in the “Not
proficient” and “Advanced” categories as one goes down in grade
o Used ECM instead to identify MEAP “Not Proficient” and “Advanced” cut scores
36
No evidence of regression away from the mean in identifying MEAP “proficient” cut scoreso Looking for a consistently lower percentage of students
proficient as one goes down in gradeso Used SDT to identify MEAP “proficient” cut scores
Evidence of regression away from the mean in identifying MEAP “partially proficient” and “advanced” cut scoreso Increasingly smaller percentages of students in the “Not
proficient” and “Advanced” categories as one goes down in grade
o Used ECM instead to identify MEAP “Not Proficient” and “Advanced” cut scores
37
Classification Consistency Rates for MEAP Cut Scores in Mathematics
GradeCut Score
Partially Proficient
Proficient Advanced
8 83% 86% 95%
7 81% 84% 95%
6 82% 83% 96%
5 81% 84% 95%
4 80% 82% 94%
3 77% 80% 95%
38
Classification Consistency Rates for MEAP Cut Scores in Reading
GradeCut Score
Partially Proficient
Proficient Advanced
8 83% 78% 87%
7 86% 76% 85%
6 85% 74% 83%
5 88% 75% 84%
4 80% 82% 94%
3 80% 72% 86%
39
Classification Consistency Rates for MEAP Cut Scores in Science
GradeCut Score
Partially Proficient
Proficient Advanced
8 80% 84% 92%
5 76% 82% 92%
40
Classification Consistency Rates for MEAP Cut Scores in Science
GradeCut Score
Partially Proficient
Proficient Advanced
9 85% 81% 91%
6 81% 77% 91%
41
Creating Mini-Cuts for PLC Calculations in Reading and Mathematics
42
43
44
45
46
Low Mid High Low High Low Mid High MidLow M I I SI SI SI SI SI SIMid D M I I SI SI SI SI SIHigh D D M I I SI SI SI SILow SD D D M I I SI SI SIHigh SD SD D D M I I SI SILow SD SD SD D D M I I SIMid SD SD SD SD D D M I IHigh SD SD SD SD SD D D M I
Advanced Mid SD SD SD SD SD SD D D M
PartiallyProficient
Proficient
Year X Grade Y MEAP
Performance Level
Year X+1 Grade Y+1 MEAP Performance Level
ProficientPartially
ProficientNot
Proficient
NotProficient
Adv
47
New Versus Old Cut Scores
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Mathematics, Grade 11
Old Cut Scores
New Cut Scores
48
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Mathematics, Grade 8
Old Cut Scores
New Cut Scores
49
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Mathematics, Grade 7
Old Cut Scores
New Cut Scores
50
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Mathematics, Grade 6
Old Cut Scores
New Cut Scores
51
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Mathematics, Grade 5
Old Cut Scores
New Cut Scores
52
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Mathematics, Grade 4
Old Cut Scores
New Cut Scores
53
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Mathematics, Grade 3
Old Cut Scores
New Cut Scores
54
55
New Versus Old Cut Scores
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Reading, Grade 11
Old Cut Scores
New Cut Scores
56
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Reading, Grade 8
Old Cut Scores
New Cut Scores
57
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Reading, Grade 7
Old Cut Scores
New Cut Scores
58
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Reading, Grade 6
Old Cut Scores
New Cut Scores
59
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Reading, Grade 5
Old Cut Scores
New Cut Scores
60
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Reading, Grade 4
Old Cut Scores
New Cut Scores
61
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Reading, Grade 3
Old Cut Scores
New Cut Scores
62
63
New Versus Old Cut Scores
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Science, Grade 11
Old Cut Scores
New Cut Scores
64
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Science, Grade 8
Old Cut Scores
New Cut Scores
65
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Science, Grade 5
Old Cut Scores
New Cut Scores
66
67
New Versus Old Cut Scores
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Social Studies, Grade 11
Old Cut Scores
New Cut Scores
68
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Social Studies, Grade 9
Old Cut Scores
New Cut Scores
69
0102030405060708090
100
07-08 08-09 09-10 10-11
Perc
ent P
rofic
ient
School Year
Social Studies, Grade 6
Old Cut Scores
New Cut Scores
70
71
Joseph A. Martineauo Executive DirectoroBureau of Assessment & AccountabilityoMichigan Department of Education
[email protected] 517-241-4710