Tweaking the pilotA Case Study from DVMT 100 at Frostburg State University

Dr. Megan E. Bradley

DVMT 100• 3 credits, does not count toward graduation

or GPA*

• Must take if need MATH 102 (College Algebra) or MATH 106 (Algebra with Calculus – Business majors)

• About 450 students per year• 1078% increase since inception in 1985• No budget increase

Course Issues• Failure rate with gender gap in DVMT 100:

• 41% failure rate overall• 44% rate for males; 35% rate for females

• Failure rate in next math course:COURSE DVMT DWF RATE

NON-DVMT DWF RATE

DIFFERENCE

MATH 102 56% 39% 16%

MATH 106 43% 33% 11%

Course Issues• Staffing issues

• Relied solely on undergraduate students to teach

• Course Drift• Delivery: ½ sections all face-to-face (f2f); other

½ all computer lab• Different textbook, syllabus, point system• No system for checking reliability of grading

What we did• Emporium Model done right

• 1 large lecture• Rest of time in lab

• Hired new staff member to serve as lead instructor• Undergraduates became ULAs, shifting role to lab

assistant

• Added material to help students in next math class

Pilot – spring ‘11• Traditional lecture

• all face-to-face (f2f) classes• no online work• taught by trained undergraduates• point system for course grade• 1 final exam but could have earned other points with previous

assignments to make final exam not have much weight• Redesign

• Lecture 1x/week by instructor & lab 4x/week with trained Undergraduate Learning Assistants (ULAs) using ASAlgebra by Plato

• 3 modules & corresponding exams• Mastery learning – retake exams until passed

• Pass course by passing all 3 modules with 80% or higher• Extra credit for attending and doing online homework & evaluates

assessment• Pass/Fail rates• Scores on “core questions”

• Questions that show up on the redesign module exams & the final exams for the traditional sections

• Focus groups

Pilot results• Pass/fail

• Historical failure rate: : 41%

• Redesign failure rate: 47.2% which was significantly worse than…

• Traditional failure rate: 22.6%

• Males failed more than females

Pilot results• Core questions

• Difficult to use final grades due to different grading systems

• Considering all core questions, a one-way ANOVA of Type of Classroom (2: Redesign versus traditional) by Core Qs (All) was significant, F = 37.429, p = .000, eta2 = .327.

• Redesign students (X = 87.98%) performed significantly better than traditional students (X = 63.14%).

Pilot results• Core questions

• Below is a breakdown of core questions per module, • Students from the redesign section scored significantly

higher than traditional sections for all three modules:• M1: Redesign (X = 86.20%) > traditional (X=83.66%) • M2: Redesign (X = 84.90%) > traditional (X=74.07%) • M3: Redesign (X = 90.85%) > traditional (X=59.05%)

Pilot results• Regression indicated which of course

activities significantly related to student grade on core questions.• Attendance: correlated but weak• Online homework: correlated but weak• Online evaluates: strongly correlated

• Homework & evaluates: needed 80% to pass and move on• Evaluates: Often only had 4 questions so needed

to get perfect score.

Additional results• We examined students’ time on task and

when they were using software.• Reviewed focus group suggestions.• Compared student performance on certain

items in traditional sections.• Created hypotheses and tested them out as

best as we could.• Reassessed the team

Issues & tweaks1. Students compared DVMT

100 sections.

2. Redesign students did not effectively use their lab time wisely.

3. Redesign students did not have enough deadlines – 1x/module, night before exam.

4. Students fell behind next module while retaking previous module exam.

1. Fall 2011 – full implementation.

2. Changed lab to 2x/wk and used technology to block other sites.

3. Created several deadlines with last deadline before test review day.

4. Added retake week after Mod1.

Issues & tweaks1. The grading system in the

redesign confused students.

2. Redesign students found and exploited a loophole about retaking modules next semester.

3. Lab assistants were scattered across different labs.

4. No pedagogy to address gender gap.

1. Revised to be based on weights that required and rewarded important course aspects.

2. Modified retaking of modules.

3. Assigned lab assistants.

4. Created Train Your Brain Program

Issues & tweaks1. Failure rate on first

version of module exam was very poor:

• Mod1 = 27% passed• Mod2 = 20% passed• Mod3 = 17% passed

1. Implemented PreModule Exam

• Earn 85% or higher – no need to take Module exam

• Reward studying & doing well

•Fall 2011 pass/fail rate• 20.3% failure rate overall• 19.7% rate for males; 21.3% rate for females• Gender analyses NOT statistically significant.

Full implementation results

• Remember this?• Failure rate with gender gap:

• 41% failure rate overall• 44% rate for males; 35% rate for females• Pilot redesign failure rate: 47.2%

W FX F NC P0

102030405060708090

GRADE

GRADE

Why Stop There?

assessment• Score on pre-test in next math class

• Non-DVMT• Redesign DVMT• Traditional DVMT

Pre-Test in Next Math Course

Math Level Student Type Pre-Test Score

ML 3, 4  Eligible for Pre-Cal or Calculus 63%

ML 2   Eligible for College Algebra Repeating the Course 67%

ML 2  Eligible for College AlgebraFirst Time Students 62%

ML 1   DVMT 100 Redesign Graduates 60%

ML 1 Traditional DVMT 100 Graduates 51%

Why Stop There?

assessment• Final grades in next math class

• Who passed with a C or higher?

ALL S

TUDEN

TS

NON-DVMT

TRAD

ITIONAL

DVMT

DVMT R

EDES

IGN

0%10%20%30%40%50%60%70%80%

0.56

% PASSED

% PASSED BY GROUP

ALL S

TUDEN

TS

NON-DVMT

TRAD

ITIONAL

DVMT

DVMT R

EDES

IGN

0%10%20%30%40%50%60%70%80%

56%

35%

% PASSED

% PASSED BY GROUP

ALL S

TUDEN

TS

NON-DVMT

TRAD

ITIONAL

DVMT

DVMT R

EDES

IGN

0%10%20%30%40%50%60%70%80%

56%

35%24%

% PASSED

% PASSED BY GROUP

ALL S

TUDEN

TS

NON-DVMT

TRAD

ITIONAL

DVMT

DVMT R

EDES

IGN

0%10%20%30%40%50%60%70%80%

56%

35%24%

41%

% PASSED

% PASSED BY GROUP

HIST

ORIC

AL N

ON-D

VMT

NON-

DVMT

HIST

ORIC

AL D

VMT

DVMT

RED

ESIG

N

0%20%40%60%80%

100%

0.380.56

FAILURE RATE MATH 102

FAILURE RATE FOR MATH 102

HIST

ORIC

AL N

ON-D

VMT

NON-

DVMT

HIST

ORIC

AL D

VMT

DVMT

RED

ESIG

N

0%20%40%60%80%

100%

38%53% 56%

42%

FAILURE RATE MATH 102

FAILURE RATE FOR MATH 102

Impact of changes• Deadlines = large % students completed

deadlines• Weights & lab changes = better attendance

and time on task• Train Your Brain = no gender gap, better

performance overall

Impact of changes1. Failure rate on first

version of module exam was very poor:

• Mod1 = 27% passed• Mod2 = 20% passed• Mod3 = 17% passed

1. PreModule Exam results• Module 1

• Premod: 36% passed• Version 1: 72% passed

• Module 2• Premod: 16.2% passed• Version 1: 60% passed

• Module 3• Premod: 18.3% passed• Version 1: 53% passed

Overall recommendations

• Look, look, look.• Add structure.• Improve based on evidence (from pilot, from

other redesigns, from published research)• Add psychology

• Provide incentives• Spacing effect• Practice effect• Mastery learning

Your reward