Presentation at the Fall 2011 Meeting of the Michigan Educational Research Association

Presentation at the Fall 2011 Meeting of theMichigan Educational Research Association

2

Identifying MME Cut Scores

3

University 2-Year Institution 2-Year Institution

Central Michigan University Alpena Community College Mid Michigan Community College

Eastern Michigan University Delta College Monroe County Community College

Ferris State University Glen Oaks Community College Montcalm Community College

Grand Valley State University Gogebic Community College Mott Community College

Michigan Technological University

Grand Rapids Community College

Muskegon Community College

Michigan State University Henry Ford Community College North Central Michigan College

Oakland University Jackson Community College Northwestern Community College

Northern Michigan University Kalamazoo Valley Community College

Oakland Community College

Saginaw Valley State University

Kellogg Community College Schoolcraft College

The University of Michigan-Ann Arbor

Kirtland Community College Southwestern Michigan College

University of Michigan-Dearborn

Lake Michigan College St. Clair County Community College

University of Michigan-Flint Lansing Community College Washtenaw Community College

Wayne State University Macomb Community College West Shore Community College

Western Michigan University

4

MME content area College courses usedMathematics College Algebra.

Reading

Courses identified by 4-year universities.

Reading-heavy courses such as entry-level literature, history, philosophy, or psychology for 2-year universities.

Science

Courses identified by 4-year universities.

Entry level biology, chemistry, physics, or geology for 2-year universities.

Social Studies

Courses identified by 4-year universities.

Entry level history, geography, or economics for 2-year universities.

5

Grades were put on a numeric scale from 0-4 0 = F 1 = D 2 = C 3 = B 4 = A Not used

o AU, AWF, DR, R, RA, FR, T, TR, X Coded as 3.0

o P, CR Coded as 0.0

o IN, N, NC, NE, NS, W, WF, WP, WX, and U

6

MME Content

AreaSample

Size

Percent B or

higher

Course Grade MME Score

Mean SD Mean SD

Math 6,286 47.0 2.49 1.18 1112.2 13.2

Reading 37,952 54.9 2.64 1.23 1117.2 24.6

Social Studies

39,721 54.4 2.63 1.22 1135.4 26.3

Science 15,608 50.0 2.54 1.19 1123.5 23.5

7

MME Content Area

Course Type Number of Students

Mathematics College Algebra 6567

Reading

Literature 456

American History 1731

Other History 3010

Psychology 16231

Sociology 8236

Political Science 6114

Philosophy 1869

Other 2517

8

MME Subject Area Course Type Number of Students

Science

Biology/Life Science 8355

General Chemistry 5807

Physics 535

Other 1483

Social Studies

American History 1734

Other History 3006

Psychology 16230

Sociology 8231

Geography 612

Political Science 6108

Economics 3498

Other 2361

9

Students receiving an A Students receiving a B or better Students receiving a C or better

Students receiving a B or better in 4-year universities

Students receiving a B or better in 2-year institutions

10

Logistic Regression (LR)o Identify score that gives a 50% probability of achieving an Ao Identify score that gives a 50% probability of achieving a B or

bettero Identify score that gives a 50% probability of achieving a C or

better Signal Detection Theory (SDT)

o Identify scores that maximize the proportion receiving consistent classifications from MME to college grades• i.e., both proficient/advanced and receiving a A/B/C or better• i.e., both not proficient/partially proficient and receiving a A-/B-/C- or

worseo Equivalent to LR under mild monotonicity assumptions

Selected SDT as the preferred method because of its purpose (maximizing consistent classification)

11

Where•success is obtaining an A/B/C or better•e is the base of the natural logarithm•β0 is the intercept of the logistic regression•β1 is the slope of the logistic regression•x is the MME score

12

13

14

15

16

Basic Idea Set the MME cut score to…

Maximize the number of students in the Consistent cells Minimize the number of students in the Inconsistent cells

Maximize consistent classification from MME to first-year college grades

MME(unknown cut score)

Freshman Grade (known cut score)

B- or Lower B or Higher

College Ready Inconsistent Consistent

Not College Ready Consistent Inconsistent

17

0 0.2 0.4 0.6 0.8 1 1.2

MM

E Co

nte

nt

Are

a Sc

ore

Grade in First Related Freshman Credit Bearing Course

Known Cut Score

B- or lower B or higher

18

0 0.2 0.4 0.6 0.8 1 1.2

MM

E Co

nte

nt

Are

a Sc

ore

Grade in First Related Freshman Credit Bearing Course

Consistently Classified

Inconsistently Classified

Known Cut Score

Unknown Cut Score

B- or lower B or higher

82.8%consistent classification

19

0 0.2 0.4 0.6 0.8 1 1.2

MM

E Co

nte

nt

Are

a Sc

ore

Grade in First Related Freshman Credit Bearing Course

Consistently Classified

Inconsistently Classified

Known Cut Score

Unknown Cut Score

B- or lower B or higher

82.8%consistent classificationAdjust the unknown cut

score to maximize consistent classification

20

0 0.2 0.4 0.6 0.8 1 1.2

MM

E Co

nte

nt

Are

a Sc

ore

Grade in First Related Freshman Credit Bearing Course

Consistently Classified

Inconsistently Classified

Known Cut Score

Unknown Cut Score

B- or lower B or higher

88.6%consistent classification

21

0 0.2 0.4 0.6 0.8 1 1.2

MM

E Co

nte

nt

Are

a Sc

ore

Grade in First Related Freshman Credit Bearing Course

Consistently Classified

Inconsistently Classified

Known Cut Score

Unknown Cut Score

B- or lower B or higher

90.45%consistent classification

22

Analyses treating grades of A as the success criterion produced unusable results (i.e., the highest possible MME scale scores

Analyses treating grades of C as the success criterion produced unusable results (i.e., MME scale scores below chance level)

Analyses treating 4-year and 2-year institutions did produce different cut scores, but they were within measurement error of each other

Used analyses based on all institutions and grades of B or better to produce MME cut scores

Used probability of success of 33% and 67% to set the “partially proficient” and “advanced” cut scores

SDT and LR produced very similar results Used SDT because it was the preferred methodology

23

Content Area

Classification

Consistency

Partially Proficient Cut Score

Proficient Cut Score

Advanced Cut Score

Mathematics 65% 1093 1116 1138

Reading 63% 1081 1108 1141

Science 67% 1106 1126 1144

Social Studies 63% 1097 1129 1158

24

Identifying MEAP Cut Scores

25

Cohort

Grade

3 4 5 6 7 8 9 10 11 12 13

1 - - - - - 05-06 06-07 07-08 08-09 09-10 10-11

2 - - - - 05-06 06-07 07-08 08-09 09-10 10-11 -

3 - - - 05-06 06-07 07-08 08-09 09-10 10-11 - -

4 - - 05-06 06-07 07-08 08-09 09-10 10-11 - - -

5 - 05-06 06-07 07-08 08-09 09-10 10-11 - - - -

6 05-06 06-07 07-08 08-09 09-10 10-11 - - - - -

7 06-07 07-08 08-09 09-10 10-11 - - - - - -

8 07-08 08-09 09-10 10-11 - - - - - - -

9 08-09 09-10 10-11 - - - - - - - -

10 09-10 10-11 - - - - - - - - -

26

Logistic Regression (LR)o Identify score that gives a 50% probability of achieving proficiency

on a later-grade test (i.e., MME or MEAP) Signal Detection Theory (SDT)

o Identify scores that maximize the proportion receiving consistent classifications from one grade to a later grade• i.e., proficient/advanced on both tests• i.e., not proficient/partially proficient on both tests

o Equivalent to LR under mild monotonicity assumptions Equipercentile Cohort Matching (ECM)

o Identify scores that give the same percentage of students proficient/advanced on both tests

Selected SDT as the preferred method because of its purpose (maximizing consistent classification)

However, SDT and LR are susceptible to regression away from the mean

27

Same as for identifying MME cut scores Criterion for success is proficiency on a later grade test

rather than getting a B or better in a related college course

28

Each dot represents a plot of test scores in grade 8 and grade 11 for a single student

29

Grade 11: ProficientGrade 11: Not proficient

30

Grade 8: ProficientGrade 11: Not proficient

Grade 8: ProficientGrade 11: Proficient

Grade 8: Not proficientGrade 11: Not proficient

Grade 8: Not ProficientGrade 11: Proficient

31

32

33

The more links in the chain, the greater the effects of regression

Original plan for Math and Readingo Link grade 11 MME to college gradeso Link grade 8 MEAP to grade 11 MMEo Link grade 7 MEAP to grade 8 MEAPo Link grade 6 MEAP to grade 7 MEAPo Link grade 5 MEAP to grade 6 MEAPo Link grade 4 MEAP to grade 5 MEAPo Link grade 3 MEAP to grade 4 MEAP

Original plan results in 7 links by the time the grade 3 cut is set

Original plan results in inflated cut scores in lower grades

34

Revised plan for Math and Reading For Grade 3, 4, 5, 6

o Link grade 11 MME to college gradeso Link grade 7 MEAP to grade 11 MMEo Link grade 3, 4, 5, or 6 MEAP to grade 7 MME

For Grade 7, 8o Link grade 11 MME to college gradeso Link grade 7 or 8 MEAP to grade 11 MME

Results in a maximum of three links for any one grade

35

No evidence of regression away from the mean in identifying MEAP “proficient” cut scoreso Looking for a consistently lower percentage of students

proficient as one goes down in gradeso Used SDT to identify MEAP “proficient” cut scores

Evidence of regression away from the mean in identifying MEAP “partially proficient” and “advanced” cut scoreso Increasingly smaller percentages of students in the “Not

proficient” and “Advanced” categories as one goes down in grade

o Used ECM instead to identify MEAP “Not Proficient” and “Advanced” cut scores

36

No evidence of regression away from the mean in identifying MEAP “proficient” cut scoreso Looking for a consistently lower percentage of students

proficient as one goes down in gradeso Used SDT to identify MEAP “proficient” cut scores

Evidence of regression away from the mean in identifying MEAP “partially proficient” and “advanced” cut scoreso Increasingly smaller percentages of students in the “Not

proficient” and “Advanced” categories as one goes down in grade

o Used ECM instead to identify MEAP “Not Proficient” and “Advanced” cut scores

37

Classification Consistency Rates for MEAP Cut Scores in Mathematics

GradeCut Score

Partially Proficient

Proficient Advanced

8 83% 86% 95%

7 81% 84% 95%

6 82% 83% 96%

5 81% 84% 95%

4 80% 82% 94%

3 77% 80% 95%

38

Classification Consistency Rates for MEAP Cut Scores in Reading

GradeCut Score

Partially Proficient

Proficient Advanced

8 83% 78% 87%

7 86% 76% 85%

6 85% 74% 83%

5 88% 75% 84%

4 80% 82% 94%

3 80% 72% 86%

39

Classification Consistency Rates for MEAP Cut Scores in Science

GradeCut Score

Partially Proficient

Proficient Advanced

8 80% 84% 92%

5 76% 82% 92%

40

Classification Consistency Rates for MEAP Cut Scores in Science

GradeCut Score

Partially Proficient

Proficient Advanced

9 85% 81% 91%

6 81% 77% 91%

41

Creating Mini-Cuts for PLC Calculations in Reading and Mathematics

42

43

44

45

46

Low Mid High Low High Low Mid High MidLow M I I SI SI SI SI SI SIMid D M I I SI SI SI SI SIHigh D D M I I SI SI SI SILow SD D D M I I SI SI SIHigh SD SD D D M I I SI SILow SD SD SD D D M I I SIMid SD SD SD SD D D M I IHigh SD SD SD SD SD D D M I

Advanced Mid SD SD SD SD SD SD D D M

PartiallyProficient

Proficient

Year X Grade Y MEAP

Performance Level

Year X+1 Grade Y+1 MEAP Performance Level

ProficientPartially

ProficientNot

Proficient

NotProficient

Adv

47

New Versus Old Cut Scores

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Mathematics, Grade 11

Old Cut Scores

New Cut Scores

48

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Mathematics, Grade 8

Old Cut Scores

New Cut Scores

49

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Mathematics, Grade 7

Old Cut Scores

New Cut Scores

50

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Mathematics, Grade 6

Old Cut Scores

New Cut Scores

51

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Mathematics, Grade 5

Old Cut Scores

New Cut Scores

52

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Mathematics, Grade 4

Old Cut Scores

New Cut Scores

53

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Mathematics, Grade 3

Old Cut Scores

New Cut Scores

54

55

New Versus Old Cut Scores

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Reading, Grade 11

Old Cut Scores

New Cut Scores

56

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Reading, Grade 8

Old Cut Scores

New Cut Scores

57

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Reading, Grade 7

Old Cut Scores

New Cut Scores

58

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Reading, Grade 6

Old Cut Scores

New Cut Scores

59

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Reading, Grade 5

Old Cut Scores

New Cut Scores

60

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Reading, Grade 4

Old Cut Scores

New Cut Scores

61

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Reading, Grade 3

Old Cut Scores

New Cut Scores

62

63

New Versus Old Cut Scores

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Science, Grade 11

Old Cut Scores

New Cut Scores

64

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Science, Grade 8

Old Cut Scores

New Cut Scores

65

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Science, Grade 5

Old Cut Scores

New Cut Scores

66

67

New Versus Old Cut Scores

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Social Studies, Grade 11

Old Cut Scores

New Cut Scores

68

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Social Studies, Grade 9

Old Cut Scores

New Cut Scores

69

0102030405060708090

100

07-08 08-09 09-10 10-11

Perc

ent P

rofic

ient

School Year

Social Studies, Grade 6

Old Cut Scores

New Cut Scores

70

71

Joseph A. Martineauo Executive DirectoroBureau of Assessment & AccountabilityoMichigan Department of Education

omartineauj@michigan.govo 517-241-4710