2014-15 Early Mathematics Placement Tool Program Evaluation

Mark J. Schroeder Daniel J. Adams Sonya K. Sedivy

James A. Wollack

UW Center for Placement Testing

1 BACKGROUND

Beginning with the 2008-2009 academic year, the Early Mathematics Placement Tool

(EMPT) became available to Wisconsin high school mathematics teachers and their students. A

small group of teachers participated during Fall 2008, and the entire program was made live, for

any teacher, in January 2009. The program is free-of-charge for all Wisconsin participants and

provides teachers with the option for a paper-and-pencil and an online version, with a choice of

two different forms. Teachers receive feedback letters which indicate the mathematical

preparedness of students and an online resource guide lists the mathematics requirements of

majors at UW campuses. Teacher recruitment during the pilot year was primarily word-of-mouth

with staff also presenting at the annual meeting of the Wisconsin Mathematics Council in Green

Lake.

During the 2014-2015 academic year, the structure of the EMPT remained the same as in

previous years: (1) The program still uses retired items from the University of Wisconsin

Mathematics Placement Test (MPT) to indicate to teachers, students, and parents how prepared

the student is for undergraduate mathematics at UW campuses and (2) an online resource guide

outlines the mathematics requirements needed for all undergraduate majors at UW four- and

two-year campuses and the Wisconsin Technical Colleges. Recruitment during 2014-2015 was

expanded beyond word-of-mouth to include outreach through continued contact and marketing

oriented toward mathematics faculty at all Wisconsin private colleges; high school counselor

outreach during the Wisconsin School Counselors Association annual meeting; submissions to

the Department of Public Instruction’s e-mail blast, as well as the Wisconsin Mathematics

Council conference agenda; and, continued participation in the annual Wisconsin Mathematics

Council meeting. Since the pilot year, the EMPT has gained acknowledgement under different

websites as a reliable resource (e.g. UW Help, the Wisconsin DPI website and the Wisconsin

Ide@s website) through the work of our continued outreach. To elaborate, the Ide@s website is

a searchable database of high quality education resources for Wisconsin’s PK-16 educators to

access for integration into curriculum and classroom instruction. Our program receives credit as

the creator of the EMPT every time the resources are viewed on the Ide@s website.

Furthermore, the EMPT also continued it’s partnership with UW-La Crosse and their College

Readiness Math MOOC program. The MOOC program is designed to prepare traditional and

non-traditional students for credit-bearing mathematics courses and uses the EMPT to evaluate

the progress of students after they complete a series of voluntary mathematics-based short

courses.

After two years the success of the EMPT slowed, we experienced some growth in the 2014-

2015 school year, similar to what we experienced in the first four years. The increased

percentages compared to the previous year are 5.7% in the total number of participating schools,

6.3% of teachers, 18.4% of classrooms, and 3% of students. The changes in the numbers of

schools, teachers, classrooms, and students from the 2008-2009, through the 2014-2015 school

years are illustrated in Figure 1.

FIGURE 1. Trends of total number of participating schools, teachers, students, and classrooms over the last seven years.

The number of participants who took the EMPT as sophomores, juniors, and seniors is shown

in Figure 2. The percentage of students who took the EMPT at each grade level has been highly

consistent from the 2008-2009 school year to the 2014-2015 school year. It is interesting to note

that although the program was originally developed with high school juniors in mind, fewer than

half the participants are juniors. High school teachers have elected to use the EMPT with large

numbers of sophomores and seniors. Approximately 82.9% of the students took the EMPT using

the computer-based version and 18.1% took the EMPT using the paper-based version.

FIGURE 2. Percentage of students taking the EMPT by grade level in 2014-15.

The number of EMPT participating students with different curricula background is

summarized in Figure 3.

FIGURE 3. Number of participating students by curricula background for previous seven years.

The general pattern of EMPT usage has remained similar within different curricula. In the

seven years the EMPT has been in existence, over 60% of participants have been enrolled in

either Algebra II or Pre-calculus. Pre-calculus was the most common course of enrollment

(36.1% of students in 2014-15). Algebra II was the second most common course of enrollment

(31.2% of students in 2014-15). The other curricular choices remained fairly stable at low

percentages as compared to Algebra II and Pre-calculus.

Lastly, the number of teachers and students who used the EMPT for pre-post testing was

investigated. Pre-post testing was defined as having taken the EMPT on more than one occasion

with at least 45 days in between the two test administrations. It was found that 57 teachers and

1,534 students used the EMPT in a pre-post manner. These numbers were very similar to the

number of teachers and students who used the EMPT as a pretest-posttest last year.

2 EVALUATIONS

In terms of evaluating the performance of the EMPT program in the 2014-2015 academic

year, we have two main types of evaluations. First and foremost, we quantified the impact of the

EMPT program on students’ improvement of their 2015 MPT placement levels via both

descriptive statistics and statistical tests. Second, we communicated with high school teachers

through several conferences. The EMPT program staffed an information table and gave

presentations on the program at annual meetings for the Wisconsin Math Council (WMC).

Additionally, the EMPT staff exhibited at an information table at the Wisconsin School

Counselors Association (WSCA). Our attendance at these different events will be continued in

our outreach efforts. Furthermore, the EMPT staff also continue the partnership with UW-La

Crosse’s College Readiness Math MOOC program. The communication we have undertaken

with instructors across the state has been extremely valuable in searching for ways to improve

the program and we plan on continuing our outreach efforts to gather feedback and make the

appropriate adjustments.

2.1 Findings via Statistical Analysis

For the first part of the evaluation, we tracked the students who participated in the EMPT as

juniors in the 2013-2014 academic year and followed them until they took the actual MPT in

spring, 2015. Table 1 uses the 9-point EMPT scale (shown in Appendix A) to show the change

in placement level that resulted over the students’ senior year.

TABLE 1: Comparison of placement levels. Note the bold black numbers along the diagonal represent individuals whose

MPT matched their EMPT levels.

2015 MPT

Levels

2013-2014 EMPT Levels

1 2 3 4 5 6 7 8 9 Total Avg.

Level

1 13 1 4 2 5 0 0 0 0 25 2.40

2 18 14 10 6 6 2 0 0 0 56 2.54

3 37 13 43 28 15 3 1 1 0 141 2.92

4 20 19 75 35 51 13 7 10 4 240 3.94

5 13 19 58 33 61 37 16 14 14 265 4.66

6 1 5 17 19 39 29 8 8 10 136 5.27

7 1 0 4 7 17 17 7 10 6 69 5.96

8 0 0 7 10 21 26 7 31 35 137 6.82

9 2 0 7 11 48 42 34 76 230 450 7.72

Total 105 71 225 151 263 169 80 150 299 1513 5.49

Avg. Level 3.20 3.79 4.41 4.93 5.79 6.68 7.12 7.75 8.49 6.12

Of the 1,513 juniors who completed the EMPT in 2013-2014 and the MPT in 2015, 47.8%

saw their placement levels increase (compared to 52.1% in 2013-14): 18.4% improved one level,

and 29.4% of students increased at least two math levels. Among the 1,214 students who scored

below Level 9 on the EMPT (and therefore were able to improve on the MPT), 58.9% improved

their placements. Of the equivalent group from last year’s evaluation, 62.4% improved their

placements, again indicating a high degree of consistency with previous results. In limited cases,

there were students who placed into Level 1 or 2 on the EMPT, but then Level 8 or 9 on the

MPT. Conversely, 21.6% of the students received lower placement levels on the MPT than on

the EMPT: 11.9% of the students dropped one level on the MPT while 9.7% dropped at least

two levels. Among the 1,445 students who scored above Level 1 on the EMPT (for whom it was

possible to receive a lower MPT placement), 23.6% saw their placements decrease. Again, this

percentage was comparable to the percentage from last year, where 22.7% of students from last

year saw their placements decrease. Finally, 30.6% of the students received the same placement

level on the MPT as they received on the EMPT.

On average, students performed much better on the MPT than on the EMPT. The MPT saw a

dramatic decrease in the percentage of students scoring at the developmental level.

Approximately 11.63% of the students scored at Levels 1 or 2 on the EMPT during their junior

year, compared with only 5.35% on the MPT, a decrease of 54.0%. Similarly, 50.5% more

students placed into Level 9 on the MPT than on the EMPT (29.74% compared with 19.76%).

The last row of Table 1 shows the average placement level on the MPT for students scoring at

each of the nine different placement levels on the EMPT. For example, of the students who

placed into Level 1 on the EMPT, the average MPT placement level was 3.20, an increase of

2.20 placement levels. For students placing into EMPT Level 2, the average MPT placement

level was 3.79, an increase of 1.79 levels. In contrast, the average MPT placement level for

students earning a Level 6 on the EMPT was 6.68, an increase of just 0.68 placement levels.

These data show very clearly the EMPT has the largest impact for students in the lowest levels.

Because EMPT is specifically intended to reduce the amount of remediation and serve as a

wake-up call to those who are ill-prepared for college-level math, these results strongly suggest

that the program is working as intended.

Finally, the last column of Table 1 shows the average EMPT placement levels for students

placing into each of the nine levels on the MPT. These data also show an upward trend (i.e.,

students placing higher on the EMPT tend to place higher on the MPT). Because the achieved

MPT level is less than the average EMPT level for MPT Levels 1, 2, and 3, this suggests that

students earning MPT placements of 3 or less tended, on average, to score somewhat higher on

the EMPT. This result, though counter-intuitive, is expected at the low end of the scale, where it

is difficult to score lower than the MPT. As an example, students who placed into Level 1 on the

MPT cannot possibly have scored lower on the EMPT. To the extent that any individuals

received an EMPT score other than Level 1, such differences will necessarily involve EMPT

scores above 1, thereby raising the average EMPT level above 1. This is referred to as a floor

effect. A similar situation, known as a ceiling effect, occurs at the upper end of the scale, where

students receiving an MPT level 9 were unable to receive EMPT scores that were higher. As a

result of these floor and ceiling effects, the data in the last column should be interpreted with

caution. The most interesting piece of information from that column is the overall average EMPT

level for all students, 5.49. When compared with the average MPT Level of 6.12, we see that the

typical EMPT participant increased 0.63 placement levels over the course of their senior year. In

comparison, the average EMPT participant from last year’s evaluation increased by 0.82

placement levels from their junior to senior year.

Another interesting pattern relates to the impact of EMPT for different curricula. As part of

the EMPT, instructors are asked to indicate the curriculum being taught in each class. Table 2

shows the change in average placement level (EMPT vs. MPT) for students, broken down by

curricular area. Note that the curricula reflect the courses the students completed as juniors, but

do not capture what type of course, if any, the students completed as seniors. Sample sizes for

each of these tests are also included in Table 2, denoted as N. When sample sizes were

sufficiently large (i.e., N ≥ 20), the change was evaluated using a two-tailed matched-pair t-test.

Results that are statistically significant at α = .05 level are identified with an asterisk.

According to Table 2, most of the participating students showed significant growth from the

2013-14 EMPT to the 2015 MPT, usually between about half a level or one full level. For the

courses with a sufficient sample size, students enrolled in the Trigonometry curriculum

demonstrated the largest growth, with MPT scores increasing an average of 1.06 levels. When

we take into consideration only the courses with sufficient sample size, the curricula showing the

least gains in scores was Pre-calculus. As expected, students with higher levels of high school

math (EMPT) tended to place higher on the MPT. Students taking Pre-calculus or Trigonometry

as juniors earned higher MPT placements than other students.

TABLE 2: Comparison of 2012-13 EMPT and 2014 MPT average levels for students with different curricula

backgrounds.

High School Curricula

Background

2013-14 EMPT

Average Level

2015 MPT

Average Level

Changes from 13-14

EMPT to 15 MPT

College Preparatory

Math (CPM) (N=4) 4.50 5.25 0.75#

Core Plus (N=10) 3.10 4.30 1.20#

Integrated Math (N=19) 7.42 7.79 0.37#

Pre-Algebra (N=6) 5.00 5.50 0.50#

Algebra I (N=28) 5.11 5.46 0.35

Geometry (N=19) 2.68 3.79 1.11#

Transition to College

Math (N=10) 5.20 7.00 1.80#

Algebra II (N=448) 3.94 4.72 0.76*

Trigonometry (N=136) 5.98 7.05 1.07*

Pre-calculus (N=654) 6.74 7.04 0.30*

Calculus (N=16) 6.81 7.75 0.94#

Statistics (N=13) 6.08 6.77 0.69#

Other Math (N=150) 4.45 5.58 1.13*

Total (N=1513) 5.49 6.12 0.63*

Note * p <.05; # represents that the matched-pair t test is not conducted due to small sample size.

Next, we compared the 2015 MPT levels for those students who participated in the 2013-14

EMPT with those who did not participate in the EMPT to further examine the impact of the

EMPT on the MPT scores. Table 3 shows the distributions of MPT levels for the 1,513 students

who participated in the EMPT as high school juniors and went on to take the MPT versus the

22,829 incoming UW students who did not participate in the EMPT program as high school

juniors.

TABLE 3: Comparison of MPT placement levels for students who took EMPT with those who did not

Actual 2015 MPT

Placement Level Participated

in 2013-14 EMPT

Did Not Participate

in 2012-13 EMPT

1 1.7% 5.1%

2 3.7% 6.8%

3 9.3% 11.8%

4 15.5% 16.7%

5 17.5% 15.5%

6 9.0% 9.8%

7 4.6% 3.4%

8 9.1% 7.6%

9 29.7% 23.3%

Table 3 indicates that the percentage of the students placing into Levels 1 or 2 (which

correspond to developmental math levels) was 61.2% lower among those participating in the

EMPT (5.4%) than for those not participating (13.9%). Also, the percentage of students placing

into Calculus (Level 9) was 27.5% higher for the EMPT participants than for non-participants

(29.7% of students who took the EMPT vs. 23.3% of students who did not take the EMPT). This

difference is larger compared to last year where the percentage of students placing into Calculus

was 19.1% higher for EMPT participants than for non-participants. Overall, the average

placement level for EMPT participants was 6.12, compared with 5.52 for non-participants. This

average difference of 0.60 levels is both statistically significant (i.e., non-zero; t = 9.56, df =

1,746, p ≤ 0.001) and sufficiently large as to be quite meaningful.

While the EMPT was created to give high school juniors a preview of what skills are required

on the MPT and to help them realize the benefits of taking a math class as a senior, the EMPT

has been used by teachers for other purposes as well. There are a large number of seniors

already enrolled in a math course who took the EMPT in 2014-15 and then went on to take the

MPT in 2015. Table 4 below shows comparisons between how these seniors scored on the

EMPT and the MPT. On average, this group of students improved 0.28 placement levels on the

MPT compared to their score on the EMPT (5.71 compared to 5.43). The average MPT score

gain for the seniors who took the EMPT during the 2014-15 year (0.28) was 0.35 placement

levels lower than the average score gain for the juniors who took the EMPT during the 2014-15

school year (0.63). This result was expected, because students who took the EMPT during their

junior year would have had a chance to receive more math instruction that could lead to a score

increase. We would not expect as much of a score gain for students taking the EMPT as seniors

because there is little difference in their math instruction between when they take the EMPT and

the MPT. The average score gain of 0.28 for the 2014-15 seniors who took the 2015 MPT could

be due to students performing more optimally under higher stakes testing conditions of the MPT.

Or, alternatively, we could be seeing a practice effect where students perform better on the MPT

after being exposed to a similar test and having a more clear idea of what to expect. It is

important to note that while a primary goal of the EMPT is to increase the number of juniors who

go on to take senior-level math, there will inevitably still be some students who opt out of math

their senior year. To the extent that these students are included in Table 1, the average increase

from EMPT scores to MPT scores for the juniors may be suppressed. The students represented

in Table 4 all took senior-level math, whereas students represented in Table 1 may or may not

have gone on to take another math course, thus any comparisons between these two groups may

not be directly parallel.

TABLE 4: Comparison of placement levels. Note the bold black numbers along the diagonal represent individuals whose

MPT scores matched their EMPT level.

2015 MPT

Levels

2014-15 EMPT Levels

1 2 3 4 5 6 7 8 9 Total Avg.

Level

1 9 5 4 2 0 2 0 0 0 22 2.32

2 22 5 13 4 7 2 1 0 0 54 2.61

3 16 12 26 21 10 3 0 0 1 89 3.13

4 29 13 46 30 65 26 4 8 3 224 4.08

5 18 12 42 45 76 42 6 22 19 282 4.85

6 3 8 14 9 34 34 8 30 13 153 5.78

7 1 1 3 5 14 13 5 13 16 71 6.52

8 1 1 3 2 9 13 9 25 23 86 7.13

9 1 1 2 3 19 17 15 54 130 242 7.95

Total 100 58 153 121 234 152 48 152 205 1223 5.43

Avg. Level 3.49 4.05 4.24 4.55 5.25 5.80 7.04 7.23 8.18 5.71

3 FURTHER DIRECTIONS

By all accounts, the 2014-2015 EMPT was largely successful. However, we are intent on

continuing to make the EMPT even better as we move forward. For starters, we worked

diligently to improve the appearance of the EMPT website, while including further information

with the inclusion of past program evaluations. Also, we have revamped our PowerPoint

presentation that is designed to walk teachers through the registration process and the student

statistical output information. Finally, the new registration system that provides instructors with

NetIDs for security in log in processes has been streamlined and shown significant

improvements in ease of access. In addition to these already-implemented improvements, we are

also working on the following:

• Expand the exposure of the EMPT program to school counselors and sponsors in

Wisconsin, not just limited to math teachers.

• Add related resource links in our website, such as DPI (Wisconsin Department of Public

Instruction).

• Invite Wisconsin private colleges to participate.

• Develop more questions for multiple forms of the EMPT.

• Continue our outreach efforts to raise awareness of the program.

• Work with the UW Math Placement Test Development Committee to evaluate the

alignment of the EMPT to the Common Core Mathematics Standards.

• Improve the tracking of students to (a) identify the mathematics courses taken (or not

taken) by EMPT participants during their senior year, and (b) evaluate the differential

success of EMPT participants and non-participants in their college math courses, so as to

further tease apart the improved math effect from the practice test effect.

• Explore the sale of the EMPT to other states.

• Explore the use of different delivery software that will improve the cohesion of the

EMPT and MPT programs, will facilitate development of additional EMPT forms, and

will make the computer-based EMPT experience more comparable to that of the

computer-based MPT.

• Conversations began with UW-System to increase our outreach and the functionality of

the EMPT program.

Appendix A:

DESCRIPTION OF EARLY MATHEMATICS PLACEMENT TOOL LEVELS

Level 1

Students have a weakness in basic mathematics and are greatly

disadvantaged in all programs. They start the mathematics course sequence

at the lowest level and will pay for credits which do not count towards their

degree

Level 2

Level 3

Students have minimal algebra skills and are substantially disadvantaged. On

some campuses, students at this level may start with a course which does not

carry degree credit but does have a course fee.

Level 4

Level 5

Students have basic algebra skills and are usually ready for a college algebra

course. Further strengthening of these algebra skills is required for most

university programs

Level 6

Level 7 Students have the algebra skills to take courses at the pre-calculus level, but

need to develop better algebra and trigonometry skills to take calculus

Level 8

Level 9 Students have the skills required for the calculus sequence