Predicting Success

Predicting Success

Kris Campbell and Dan KernlerElgin Community College

Elgin, IL

Want to follow along on your device? Download the presentation: http://tinyurl.com/AMATYC-S178

Outline

• Part 1: Background• Part 2: Initial question – Do placement scores,

ACT Math sub-scores, and HS GPA predict success in Intermediate Algebra?

• Part 3: Do they predict success at the transfer level?

• Part 4: What effect does high school coursework have on placement and success?

Part 1: Background

Elgin Community College (ECC)

• 18,088 students (Fall 2013)• 69% part-time• Average age is 28

White, non-Hispanic, 54.2%

International, 0.8%

Hispanic, 32.5%Asian / Pacific Islander,

6.3%

Black, non-Hispanic, 6.0%

American Indian/Alaskan, 0.2%

Background – Curriculum

Mth090Pre-Algebra

Mth099

CombinedBasic &

Intermediate Algebra

Mth096Basic

Algebra

Mth098Intermediate

AlgebraMth097Geometry

Transfer Level Math Courses

Background – Curriculum

Developmental Math Courses

Mth101Quantitative

Literacy

Mth102Gen. Ed.Statistics

Mth110Math for Elem.

Teach. I

Mth112College Algebra

Mth113Math for Elem.

Teaching II

Mth114Trigonometry Mth126

Calc. for Business

Mth120Statistics I

Mth125Finite Math

Mth133Calculus I

Background – Placement

• COMPASS Exam• Written by ACT• Calculators allowed• Can be retaken once• Can place students

up through Calculus I• 4 Domains• Branching

Pre-Algebra

College Algebra

Trigonometry

AlgebraStart

College Ready

ArithmeticPre-Algebra

BasicAlgebra

0-2021-43

44-100

ArithmeticPre-Algebra

BasicAlgebra

College Ready

BasicAlgebra

Int. Algebra

College Ready

0-32

38-54

55-100

33-37 Combined Basic & Int. Algebra

Background – Placement

Pre-Algebra Algebra College Algebra

Part 2: Initial Question

Background – Initial Data

• New students, all recent high school graduates• Fall 2008, 2009, and 2010 only• Valid placement• Highest COMPASS score only• Initially focused on Intermediate Algebra only• Initial analysis considered only three districts

due to an extreme weighting system in the fourth

Method

• Binomial logistic regression using COMPASS score as predictor for course success (C or better)

• Add ACT Math sub-score and HS Cumulative GPA to regression model

Results

• COMPASS score was a statistically significant predictor.

• High school GPA significantly and dramatically increased the accuracy of the prediction of success.

• ACT Math sub-score did not have a unique contribution to the model.

COMPASS only (Algebra domain)

Results

COMPASS alone is a significant predictor (p<.001). Each additional point on the exam increases the odds of success by 1.08.

ResultsCOMPASS, GPA, andACT Math sub-score

HS GPA is a significant predictor (p<.001), while the ACT Math sub-score is not (p=.901). Each additional point increase in GPA increases the odds of success by nearly 10. (Roughly double the effect of COMPASS once the scale is taken into consideration.)

ResultsResidualsRecall: residual = observed – predicted

High COMPASS score and GPA, but were not successful.

Low COMPASS score and GPA, but were successful.

Results

COMPASS only Cox & Snell R2 = 0.041 (Algebra domain) Nagelkerke R Square = 0.055

COMPASS, GPA, and Cox & Snell R2 = 0.260ACT Math sub-score Nagelkerke R Square = 0.351

Pseudo R2 Recall: R2 is the percent of variation in success explained by the model.

1.5 2 2.5 3 3.5 438

40

42

44

46

48

50

52

54

High School Cumulative GPA

COM

PASS

Alg

ebra

Sco

re

Success by COMPASS

49% success

57% success

66% success

69% successC or better

D, W, or F

1.5 2 2.5 3 3.5 438

40

42

44

46

48

50

52

54

High School Cumulative GPA

COM

PASS

Alg

ebra

Sco

re

Success by GPA

10% 30% 53% 81% 92%

C or better

D, W, or F

C or Better Distribution

22.5

33.5

4

0%10%20%30%40%50%60%70%80%90%

100%

38

43

47

51

Cumulative High School GPA

Perc

ent C

or b

etter

COMPASS Algebra

C or Better Distribution

22.5

33.5

4

0%10%20%30%40%50%60%70%80%90%

100%

38

43

47

5114%

26%

50%

75%

100%

0%

17%

48%

81%89%

17%

42%

59%

90% 88%

42%

67%

79%90%

Cumulative High School GPA

Perc

ent C

or b

etter

COMPASS Algebra

C or Better Distribution

22.5

33.5

4

0%10%20%30%40%50%60%70%80%90%

100%

38

43

47

51

75%

100%81%

89%90% 88%79%

90%

Cumulative High School GPA

Perc

ent C

or b

etter

COMPASS Algebra

85% C or better

39% of the population

B or Better Distribution

22.5

33.5

4

0%10%20%30%40%50%60%70%80%90%

38

43

47

51

0%

9%

29%

38%

58%

0%0%

28%

38%

78%

17%

25% 18%

57%

88%

25%27%

71% 70%

Cumulative High School GPA

Perc

ent B

or b

etter

COMPASS Algebra

B or Better Distribution

22.5

33.5

4

0%10%20%30%40%50%60%70%80%90%

38

43

47

51

58%

78%

88%70%

Cumulative High School GPA

Perc

ent B

or b

etter

COMPASS Algebra

72% A/B

12% of the population

C or Better Distribution

22.5

33.5

4

0%10%20%30%40%50%60%70%80%90%

100%

38

43

47

5114%

26%

50%

75%

100%

0%

17%

48%

81%89%

17%

42%

59%

90% 88%

42%

67%

79%90%

Cumulative High School GPA

Perc

ent C

or b

etter

COMPASS Algebra

C or Better Distribution

22.5

33.5

4

0%10%20%30%40%50%60%70%80%90%

100%

38

43

47

5114%

26%

0%

17%17%

42%42%

Cumulative High School GPA

Perc

ent C

or b

etter

COMPASS Algebra

Part 3: Transfer Level Success

Transfer Level Success

HS GPA is a clear predictor of success in developmental courses.

What about at the transfer level?

Transfer Level Success

• Consider only introductory transfer level courses:– Mth101 Quantitative Literacy– Mth102 General Education Statistics– Mth110 Math for Elementary Teachers I– Mth112 College Algebra

• Focus on students with a COMPASS Algebra score only.

• Reported GPA in database (some are weighted)

COMPASS Algebra Scor

e

HS Cumulative GPA

Success by COMPASS & HS GPA

C or better

D, W, or F

1.5 2.0 2.5 3.0 3.5 4.0

Intermediate Algebra and Introductory Transfer Level Courses

Success by COMPASS & HS GPA

Success Rates (C or better)

COMPASS Algebra Scor

e

25% 30% 63% 83%

0% 64% 59% 78%

0% 33% 44% 57% 82%

50% 69% 79% 79%

17% 38% 63% 88% 83%

0% 29% 47% 79% 82%

10% 26% 48% 61% 81%

HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0

C or better

D, W, or F

Intermediate Algebra and Introductory Transfer Level Courses

Success by COMPASS & HS GPA

Success Rates (C or better)

COMPASS Algebra Scor

e

62-65 25% 30% 63% 83%Trans

fer Level

58-61 0% 64% 59% 78%

54-57 0% 33% 44% 57% 82%

50-53 50% 69% 79% 79%

Inter. Algebra

46-49 17% 38% 63% 88% 83%

42-45 0% 29% 47% 79% 82%

38-41 10% 26% 48% 61% 81%

HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0

C or better

D, W, or F

Intermediate Algebra and Introductory Transfer Level Courses

COMPASS CSEM

• COMPASS Reference Manual reports a conditional standard error of measurement (CSEM)

• Can be interpreted similarly to confidence intervals

• Measures the difference between actual score and expected score if the test were repeated

• Depends on the score and test (hence conditional)

COMPASS CSEM

COMPASS CSEM

COMPASS CSEM

Success Rates (C or better)

COMPASS Algebra Scor

e

62-65 25% 30% 63% 83%Trans

fer Level

58-61 0% 64% 59% 78%

54-57 0% 33% 44% 57% 82%

50-53 50% 69% 79% 79%

Inter. Algebra

46-49 17% 38% 63% 88% 83%

42-45 0% 29% 47% 79% 82%

38-41 10% 26% 48% 61% 81%

HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0

C or better

D, W, or F

Success Rates (C or better)

COMPASS Algebra Scor

e

62-65Trans

fer Level

58-61

54-57

50-53

Inter. Algebra

46-49

42-45

38-41

HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0

C or better

D, W, or F

+10

-10

GPA: 3.6COMPASS: 50

COMPASS CSEM

COMPASS CSEM

Success Rates (C or better)

COMPASS Algebra Scor

e

62-65 25% 30% 63% 83%Trans

fer Level

58-61 0% 64% 59% 78%

54-57 0% 33% 44% 57% 82%

50-53 50% 69% 79% 79%

Inter. Algebra

46-49 17% 38% 63% 88% 83%

42-45 0% 29% 47% 79% 82%

38-41 10% 26% 48% 61% 81%

HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0

C or better

D, W, or F

Adjusting Placement

• Should we consider high school GPA as a mitigating factor in placement?– A student who places into Intermediate Algebra

but has a high HS GPA might do fine in an introductory transfer level course.

– Based on similar analysis, a student who places into Basic Algebra but has a high HS GPA might do fine in Combined Basic and Intermediate Algebra.

BasicAlgebra

Int. Algebra

College Ready

0-32

38-54

55-100

33-37 Combined Basic &Int. Algebra

ArithmeticPre-Algebra

BasicAlgebra

0-2021-43

44-100

College Ready

Background – Placement

Pre-Algebra Algebra College Algebra

Basic Algebra

Intermediate Algebra

College Ready

0-32

38-54

55-100

33-37 Comb. Basic & Int. Algebra

Adjusting Placement – Proposal

Cumulative HS GPA2.5 3.02.0 3.5

COM

PASS

Alg

ebra

Sco

re

College Ready

Combined Basic & Int. Algebra

Two principles:• Redefine “College Ready”• Accelerate completion of the

developmental sequence

Adjusting Placement – Fall 2014

* Assuming C or better both semesters of a second-year HS algebra course.

38 - 54

1.5 2.0 2.5 3.0 3.5

High School Cumulative GPA

College Ready(Eligible for Mth101, Mth102, Mth110, or Mth112)

COM

PASS

Alg

ebra

Sco

re 55-100

33 - 37

0 - 32

Intermediate Algebra (Mth098)

Combined Basic and Int. Algebra (Mth099)

Basic Algebra (Mth096)

College Ready*

Strengths and Weaknesses

• Strengths– Research-based– Uses multiple measures– Should improve success and retention

• Weaknesses– Complex – not as transparent to students as a single

score– Could encourage more student complaints– Still does not include factors that have been shown

to be significant predictors

Concerns & Complications

• Not all districts report on a 4.0 scale.• One district only reports weights, and has

weights up to 6.5 on a 4.0 scale.• How should current students be treated?• Should returning students be under the same

placement criteria?• What concerns would you have?

Part 4: High School Coursework

HS Coursework Questions

• Does taking a math course senior year predict placement?

• Does taking a math course senior year predict success?

No Yes0%

10%20%30%40%50%60%70%80%90% 83%

55%

17%

45%

Not College ReadyCollege Ready

4th Year Math Class

Results

Does a fourth year math class impact college readiness?

p < 0.0001n = 1112

Note: For those who placed via COMPASS, the comparison is 10% vs 28%.

No Yes0%5%

10%15%20%25%30%35%40%45%50%

17%

8%

42%

23%19% 21%21%

47%

3 Levels Below2 Levels Below1 Level BelowCollege Ready

4th Year Math Class

Results

Does a fourth year math class impact college placement ?

p < 0.0001n = 1085

Results

• Does a fourth year math class impact success in the first college math course?

No Yes0%

10%20%30%40%50%60%70%80%

42%33%

58%67%

D, W, or FC or better

4th Year Math Class

p =0.006n = 990

Results

• Does a fourth year math class impact success?

Pre-Algebra

Basic A

lgebra

Int. Alg.

Combined

Quant. L

it.Sta

ts

College

Alg.0%

20%

40%

60%

80%

No senior Yes senior

Pre-Algebra

Basic A

lgebra

Int. Alg.

Combined

Quant. L

it.Sta

ts

College

Alg.0%

20%

40%

60%

80%

No senior Yes senior

Results

• Does a fourth year math class impact success?

p = 0.05n = 55

p = 0.03n = 34

p = 0.06n = 3193

Further Research

• Should we investigate other high school variables?– Success in 4th year course– Highest math course– Grade in that highest math course

• Does our new placement procedure predict success as we expect?

• How do we compare with other colleges in our state, or versus state where a 4th year math class is required?

Discussion

• Questions?• What is your placement procedure?• Do you use multiple methods?

Didn’t like the presentation? Here’s a picture of some cute kids!

Thanks for going to California and missing Halloween, Papa. Your

presentation better have been good!

Buzzzzzzz!!

Questions?

Want more information ora copy of this presentation?

dkernler@elgin.edukcampbell@elgin.edu