Using Multiple Measures to Make Math Placement Decisions ......underrepresented minority students were being disproportionately placed into remedial courses (Perry, Bahr, Rosin, &

1

Using Multiple Measures to Make Math Placement Decisions:

Implications for Access and Success in Community Colleges

Federick Ngo

Will Kwon

University of Southern California

July 2014

[A more recent version of this manuscript is forthcoming in Research in Higher Education]

Corresponding Author:

Federick Ngo

Rossier School of Education, University of Southern California

3470 Trousdale Parkway, Waite Phillips Hall WPH 503C

Los Angeles, CA 90089

Email: [email protected]

Phone: 510-326-6037

mailto:[email protected]

USING MULTIPLE MEASURES TO MAKE MATH PLACEMENT DECISIONS 2

Fax: 213-740-3889

Abstract

Community college students are often placed in developmental math courses based on the results

of a single placement test. However, concerns about accurate placement have recently led states

and colleges across the country to consider using other measures to inform placement decisions.

While the relationships between college outcomes and such measures as high school GPA, prior

math achievement, and noncognitive measures are well-known, there is little research that

examines whether using these measures for course placement improves placement decisions. We

provide evidence from California, where community colleges are required to use multiple

measures, and examine whether this practice increases access and success in college-level

courses. Using data from the Los Angeles Community College District, we find that students

who were placed into higher-level math due to multiple measures (e.g., GPA and prior math

background) performed no differently from their higher-scoring peers in terms of passing rates

and long-term credit completion. The findings suggest that community colleges can improve

placement accuracy in developmental math and increase access to higher-level courses by

considering multiple measures of student preparedness in their placement rules.


Using Multiple Measures to Make Math Placement Decisions:

Implications for Access and Success in Community Colleges

An examination of math assessment and course placement in community colleges shows

that many students are deemed unprepared for the demands of college-level work. It is estimated

that over 60 percent of community college students nationally are placed in at least one

postsecondary remedial or developmental course upon entry (NCPPHE & SREB, 2010; Bailey,

2009).1 Although developmental courses can serve as necessary and helpful stepping-stones to

college success, they can also delay access to critical gateway courses necessary for degree

attainment or transfer to four-year colleges. This is of concern because recent descriptive

research shows that only a small proportion of students placed in lower levels of developmental

math sequences enroll in and pass the subsequent math courses needed to attain an associate’s

degree or transfer (Bailey, Jeong, & Cho, 2010; Fong, Melguizo, Bos, & Prather, 2013). Given

that students placed in developmental math sequences also incur substantial costs in the form of

time and money (Melguizo, Hagedorn, & Cypers, 2008), it is critical to accurately assess and

place students into the courses where they are most likely to succeed while not unnecessarily

extending their time towards degree completion or transfer.

Placement tests are commonly used in community colleges across the country to make

these initial course placement decisions (Hughes & Scott-Clayton, 2011). While practices vary

by state and even at the local college level, an increasing number of states have mandated

placement testing and the use of common assessments, seeing placement policies as a potential

lever for increasing student success (Collins, 2008). At the same time, studies have provided

evidence that placement tests have low predictive validity and are only weakly correlated with

1 The terms remedial, developmental, basic skills, and preparatory are often used interchangeably in reference to the set of courses that precede college-level courses. We prefer to use the term developmental.


students’ college outcomes, such as college GPA or credit completion (Armstrong, 2000), and

that as many as one-quarter of community college students may be severely misassigned to their

math courses by placement tests (Scott-Clayton, Crosta, & Belfield, 2012). These same studies

suggest that using other measures, such as information from high school transcripts, may be

more accurate for placing students than using placement tests alone.

Amidst these concerns, several states have revised policies to incorporate the use of

multiple measures in their assessment and placement policies for developmental math (Burdman,

2012). North Carolina, for example, has developed a customized placement assessment that

includes gathering information from multiple measures, such as high school grades and

noncognitive measures (Burdman, 2012). The Texas Success Initiative (TSI) includes revised

assessment and cut score standards, and includes the recommendation that additional multiple

measures such as high school GPA, work hours, or noncognitive measures be considered in

conjunction with assessment test scores (Burdman, 2012; Texas Higher Education Coordinating

Board (THECB), 2012). Connecticut’s SB-40 and Florida’s Senate Bill 1720 have proposed

similar policies to incorporate multiple measures.

While existing studies have shown that measures such as high school GPA or course

completion are predictive of college outcomes (Belfield & Crosta, 2012; Scott-Clayton, 2012),

there is little evidence that using these measures to make placement decisions is an effective

practice in terms of access and success for community college students. This study addresses this

research gap. We draw upon a statewide placement policy for community colleges to identify

measures that are useful for assigning students to developmental math courses. Two research

questions frame our analysis: 1) Does using multiple measures increase access to higher-level

math courses, particularly for groups disproportionately impacted by remediation? 2) How do


students who are placed using these additional measures into a higher-level math course

perform in comparison to their peers? We show that two measures in particular – high school

GPA and information about prior math course-taking and achievement – can increase access to

higher-level math courses and ensure that students are successful in those courses.

The evidence comes from California, which has required community colleges to use

multiple measures to make course placement determinations since the early 1990s (CCCCO,

2011). This policy shift occurred after advocacy groups challenged the accuracy of placement

tests and fairness of using tests alone to make placement decisions, based on evidence that

underrepresented minority students were being disproportionately placed into remedial courses

(Perry, Bahr, Rosin, & Woodward, 2010). The revised state policy prohibited the practice of

using a single assessment instrument and instead promoted the use of multiple measures, with

the goals of mitigating the disproportionate impact of remediation on underrepresented minority

students and increasing access to college-level courses.2 However, whether the students that

benefit from this policy are successful in these higher-level courses remains yet to be seen.

In this study, we examine the extent to which using multiple measures for course

placement achieves the dual goals of access and success. We present evidence from the Los

Angeles Community College District (LACCD), the largest community college district in

California, and one of the largest in the country. During the matriculation process in LACCD,

students provide additional information regarding their educational background or college plans

in addition to taking a math placement test. In most of the LACCD colleges, this multiple

measure information is used to determine whether students should receive points in addition to

their placement test score, which can sometimes result in a student being placed into the next

higher-level course. We call this a multiple measure boost. Using district-level administrative

2 Details of the policy are provided in the Appendix.


and transcript data from 2005-2008, we examine the impact of the multiple measure boost on

access and success in developmental math. Individual college policies in LACCD allow us to

focus on two measures in particular: prior math achievement and high school GPA, each of

which is used singularly some LACCD colleges. These measures have been predicted but not

proven to more accurately assign students to courses than placement tests alone (Scott-Clayton,

Crosta, & Belfield, 2012).

We begin with a review of the literature on measures that are commonly used to identify

college readiness. Modern conceptions of validation provide the framework that we use to

examine the usefulness of multiple measures for making placement decisions. Following this

theoretical discussion, we describe the data and the implementation of the multiple measures

policy in the LACCD, and provide descriptive evidence addressing the first question of access to

higher-level courses. Our findings indicate that while using multiple measures does increase

access to higher-level courses, the racial composition of courses remains largely unchanged. We

then use multivariate linear regression to compare the outcomes of students who received a

multiple measure boost into a higher-level course with those of their higher-scoring peers. We

find that students who received a multiple measure boost based on prior math course-taking or

high school GPA performed no differently from their peers in terms of course passing rates as

well as longer-term credit completion. We conclude by discussing the implications of our

findings for assessment and placement policies in developmental math.

Literature Review: Identifying Readiness for Developmental Math

Absent alignment between the K-12 and higher education systems, community colleges

need some means of identifying students' preparedness for college-level work. However, with

neither a common definition of college readiness nor a common approach to remediation, a


variety of measures are utilized to identify student skill level and college preparedness (Conley,

2007; Merisotis & Phipps, 2000; Porter & Polikoff, 2012). These measures often include

standardized placement test scores and information from high school transcripts, as well as

information gleaned from student interviews with counselors.

An important task for researchers has been to identify and validate measures that are

predictive of college success. Validation has generally involved testing a group of subjects for a

certain construct, and then comparing them with results obtained at some point in the future,

such as college persistence, grades, or completion of a college credential (AERA, APA, &

NCME, 1999; Kane, 2006). This provides an indicator of predictive validity, which is the ability

of a measure to predict future outcomes given present information. Here, we review the literature

on the predictive validity of common measures used to identify readiness for college-level work.

Placement Tests

Standardized placement tests are the most common instruments that community colleges

use to assess students and deem them college-ready or place them in developmental math courses

(Burdman, 2012; Hughes & Scott-Clayton, 2011). These placement tests, many of which are

now computerized, can be less time-consuming and resource-intensive than interviews or

reviews of individual applications and transcripts (Hughes & Scott-Clayton, 2011). The

computerized format can also enable colleges to assess many students and provide course

placement results more quickly. There is considerable variation in the types of tests used across

colleges, but ACCUPLACER and COMPASS, two commercially produced tests, are among the

most common (Hughes & Scott-Clayton, 2011).

Commercially-produced tests, such as ACCUPLACER, generally provide predictive

validity estimates for their products (e.g., Mattern & Packman, 2009). In addition, individual


colleges are advised to conduct validations within their own settings and with respect to their

uses of the assessments (Kane, 2006). However, in an examination of validation practices across

the U.S., Fulton (2012) found that colleges vary in terms of how they validate their placement

tests, with only a handful of states or college systems having validation requirements.

Research studies have provided some evidence that placement tests have low predictive

validity, finding weak correlations between placement tests and students’ course passing rates

and college grades (Armstrong, 2000; Belfield & Crosta, 2012; Jenkins, Jaggars, & Roksa, 2009;

Medhanie et al., 2012; Scott-Clayton, 2012). For example, after investigating the predictive

validity of placement tests across the Virginia Community College System, Jenkins et al. (2009)

found only weak correlations between placement test scores and student pass rates for both

developmental and college-level courses. These findings may reflect the fact that college

readiness is a function of several academic and non-academic factors that placement tests do not

adequately capture (Karp & Bork, 2012). In fact, Belfield and Crosta (2012) found that the

positive but weak association between placement test scores and college GPA disappeared after

controlling for high school GPA, suggesting that high school information may offer more useful

measures for course placement.

High School Information

While standardized placements tests are the most common instruments that community

colleges use to assess and place students in developmental math courses, there is growing

interest in incorporating high school information into the placement decision. High school

transcripts can provide information about academic competence, effort, and college-readiness

that placement tests do not measure. For example, high school grades have been found to better

predict student achievement in college than typical admissions tests do (Geiser & Santelices,


2007; Geiser & Studley, 2003), and this relationship may be even more pronounced in

institutions with lower selectivity and academic achievement (Sawyer, 2013). This may stem

from the ability of report card grades to assess competencies associated with students' self-

control, which can help students study, complete homework, and have successful classroom

behaviors (Duckworth, Quinn, & Tsukayama, 2012).

In the community college setting, measures of prior math course-taking, such as the

number of high school math courses, grades in high school math courses, and highest level of

math taken have been found to be better predictors of achievement than placement test score

alone (Lewallen, 1994). Adelman (2006) demonstrated that a composite of student performance

(i.e., GPA or class rank and course-taking), what he referred to as students' “academic

resources,” can be useful information for identifying readiness for college-level work and can be

highly predictive of college success. DesJardins and Lindsay (2007) confirmed these findings in

subsequent analyses. Similar work in California demonstrates that scores on the California High

School Exit Exam and high school transcript information are also predictive of math readiness

(Jaffe, 2012; Long Beach Promise Group (LBPG), 2013). This type of evidence has led some

community colleges to partner with local school districts and experiment with using high school

information in developmental course placement (Fain, 2003; LBPG, 2013).

Hesitation to use high school background information for placement purposes may be due

to concerns about the consistency of these measures. High school graduation, for example, is not

widely accepted as evidence of college readiness because of the wide variability in the quality of

high school experiences (Sommerville & Yi, 2002). Also, there is no common metric or meaning

across all high schools in regards to student performance and course-taking (Porter & Polikoff,

2012). Grades and summative assessments from high school vary both in rigor and breadth of


content, making them more difficult for colleges to use systematically as college readiness

indicators (Maruyama, 2012).

Nonetheless, the empirical evidence described above suggests that certain combinations

of measures may be the strongest predictors of college performance (Adelman, 2006; DesJardins

& Lindsay, 2007). For example, Belfield and Crosta (2012), finding that such measures as prior

math background in conjunction with high school GPA are strongly associated with college

outcomes, hypothesized that “the optimal decision rule may be to combine information from a

placement test with a high school transcript,” (p. 4). Similarly, Noble and Sawyer (2004) argued

that test scores, high school grades, and other measures could be used jointly to identify students

who are ready for college-level work.

Noncognitive Measures

Research in educational psychology further suggests that an array of factors beyond

cognitive intelligence and skills are predictive of college success and future outcomes

(Duckworth, Peterson, Matthews, & Kelly, 2007; Heckman, Stixrud, & Urzua, 2006; Sedlacek,

2004). Sedlacek (2004), for example, argues that noncognitive measures of adjustment,

motivation, and perception are strong predictors of success, particularly for under-represented

minority students. In a longitudinal study of community college students, Porchea, Allen,

Robbins, and Phelps (2010) found an integration of psychosocial, academic, situational, and

socio-demographic factors to be predictive of persistence and attainment, with motivation being

among the strongest predictors of future achievement. This may be due to the ability of these

variables to capture the effect of unobserved student characteristics associated with success, such

as the importance of college to a student, preference and perseverance towards long-term goals,

effort, and self-control (Duckworth et al., 2007; Duckworth et al., 2012; Sedlacek, 2004).


Given these findings, there is increasing interest in and advocacy for using noncognitive

measures for course placement, which may provide colleges with a vital source of holistic

student information (Boylan, 2009; Hodara, Jaggars, & Karp, 2012). The ACT and ETS, for

example, have developed noncognitive assessments such as the ACT ENGAGE assessments and

the ETS Personal Potential Index (ACT, Inc., 2012; ETS, 2013), which identify noncognitive

attributes associated with student success in college and are predictive of student performance

and persistence (Allen, Robbins, Casillas, & Oh, 2008; Robbins, Allen, Casillas, Peterson, & Le,

2006). In practice however, very few institutions use noncognitive measures for placement

purposes (Gerlaugh, Thompson, Boylan, & Davis, 2007; Hughes & Scott-Clayton, 2011). This

may be due to faculty perceptions that self-reported student information is inaccurate or

irrelevant (Melguizo, Kosiewicz, Prather, & Bos, forthcoming), or to the lack of evidence about

their ability to improve placement decisions.

Using Multiple Measures for Course Placement

This scan of the literature reveals that while researchers have identified cognitive and

noncognitive measures that are strongly associated with and predictive of student outcomes,

there is relatively scant evidence showing that using these measures to make course placement

decisions would be beneficial. This is an important distinction because even though there may be

a strong positive correlation between a measure such as high school GPA and passing the course

in which a student enrolled (i.e., predictive validity), we cannot conclude that the same

relationship would hold if that student was placed into a course under a decision rule that

incorporated GPA as a placement measure.

Scott-Clayton et al. (2012) examined both district- and state-wide community college

data and estimated that placement using high school GPA instead of tests would significantly


reduce the rate of severe placement errors in both developmental math and English courses.

Aside from these prediction-based estimates, the only empirical evidence on actual placement

decisions has come from institutional research, such as one experimental study that utilized a

randomized design to determine the impact of different placement schemes. Marwick (2004)

found that Latino students in one community college who were placed into higher-level courses

due to the use of multiple measures (high school preparation and prior math coursework)

achieved equal and sometimes greater outcomes than when only placement test scores were

considered. Another report of an on-going study by the Long Beach Promise Group (2013)

shows that students who were placed in courses via a “predictive placement” scheme based on

high school grades instead of test scores spent less time in developmental courses and were more

likely to complete college-level English and math courses.

Overall, there is limited use of multiple measures during assessment and placement for

developmental math, and this may stem from a lack of evidence about their ability to improve

placement decisions. Furthermore, qualitative research has found that faculty and staff often do

not feel supported in the identification and validation of measures that can be incorporated into

placement rules, while others perceive measures besides test scores to be insignificant (Melguizo

et al., forthcoming). Given the numerous studies demonstrating the predictive validity of these

other measures, it is important to gather evidence on the usefulness of measures for making

course placement decisions. This involves a process of validation, which is described next.

Conceptual Framework

Validation

The multiple measures mandate in California provides a unique opportunity to validate

placement criteria in terms of their usefulness for making course placement decisions. This


approach is in line with modern conceptions of validation, which emphasize not just accurate

predictions, but actual success (Kane, 2006). From this perspective, the validity of a measure such as

a placement test is based on the decisions or proposed decisions made using the test (AERA et al., 1999;

Kane, 2006; Sawyer, 2007). A validation argument considers the goals and uses of a measure to be

more important than its predictive properties, and emphasizes the examination of outcomes that

result from proposed uses (Kane, 2006). Therefore, in seeking to justify the use of a measure, it

is necessary to demonstrate that the positive consequences of use outweigh any negative

consequences. If the intended goals are achieved, then policies can be considered as successes; if

goals are not achieved, then polices would be considered as failures (Kane, 2006).

The measures used to make course placement decisions in developmental math would

thus be evaluated in terms of student outcomes – placement and success in the highest-level

course possible (Kane, 2006), and the frequency with which these accurate placements occur

(Sawyer, 1996; 2007; Scott-Clayton, 2012). Following this validation approach, measures used

for placement would be considered helpful if they place students in a level of coursework where

they are likely to be successful, and harmful if students are placed in a level where they are

unlikely to be successful. We next expand this validation argument to consider the use of

multiple measures in conjunction with test scores to make course placement decisions.

Placement Decisions

Assume that a math assessment enables us to make inferences about the academic

preparation of a math student. Students who receive low scores have low academic preparation

and students with high scores have high academic preparation. A typical placement policy would

sort students into various math levels based on scores from this math assessment.

For a simple model, let:

SL = Student with low academic preparation CL = Low-level course


SH = Student with high academic preparation CH = High-level course.

Let P be the probability of successfully passing the course, such that P(SLCL ) ≥ P(SLCH )

and P(SHCL ) ≥ P(SHCH ); the probability of passing a low-level course is greater than the

probability of passing a high-level course, for both types of students. Additionally, P(SHCL ) ≥

P(SLCL) and P(SHCH ) ≥ P(SLCH); the probability of passing a given course is higher for a high

academic preparation student than for a low academic preparation student. Transitivity should

predict that P(SHCL ) ≥ P(SLCH), and as result, there are only two possible monotonic

distributions:

P(SLCH) ≤ P(SLCL) ≤ P(SHCH) ≤ P(SHCL) (1)

P(SLCH) ≤ P(SHCH) ≤ P(SLCL) ≤ P(SHCL) (2)

If the raw assessment test score correctly places students in the appropriate math courses

(i.e., cutoff scores are correct), every low academic preparation student should be placed into the

low-level course and every high academic preparation student should be placed into the high-

level course. The placements (SHCL) and (SLCH) should not occur.

Placement Using Multiple Measures

Including multiple measures can be thought of as increasing collateral information, which

should improve the accuracy of placement decisions (van der Linden, 1998). Consider a decision

in which other relevant information from multiple measures is included and students can earn

additional points which are added to the raw test score. In some cases, students identified as low

academic preparation by the raw test score may be placed higher if the total score with additional

points surpasses the cutoff score. This multiple measure boost thus places the low academic

preparation student into the higher-level course, making SLCH possible. The boosted students

would have had among the highest scores on the placement test had they remained in the lower


level. As a result of the multiple measure boost, they are now the lowest-scoring students in the

higher-level course.

The question of interest is whether the boosted students are equally likely to succeed

when compared with other students in the higher-level course despite having lower raw

placement test scores. Following the approach to validation suggested by Kane (2006), the

multiple measure boost can be considered as helpful if boosted students are at least as likely to

pass the higher-level course as their comparable peers. Should the boost be helpful, then there is

an increase in placement accuracy.3 The boost is harmful if the boosted students are less likely to

pass the high-level course than their peers. In this case, the student would be better served if

placed in the lower-level course. Empirically, the comparison of probabilities is between

P(SLCH) and P(SHCH), where the boosted student is compared with other non-boosted students in

the high-level course. The multiple measure boost can be considered as helpful if P(SLCH) ≈

P(SHCH) or harmful if P(SLCH) < P(SHCH).4 We use this validation argument to proceed with our

analysis of student outcomes in the Los Angeles Community College District (LACCD), a

context where multiple measures are used in conjunction with test scores to inform placement

decisions in developmental math.

Setting: Multiple Measures in the LACCD

The LACCD is composed of nine community colleges serving nearly 250,000 students

annually, making it the largest community college district in California and one of the largest in

the country. According to our calculations, roughly 80 percent of students entering the LACCD

each year are placed in developmental math courses. In most of the colleges, the developmental

math sequence is comprised of four courses and includes arithmetic, pre-algebra, algebra, and

3 Unobservable factors such as easiness of grading or grade inflation at the classroom level could make it possible for boosted students to have a

higher probability of passing the higher-level course than the lower-level course: P(SLCH) > P(SLCL). 4 Unobservables factors such as diligence/effort could make it possible for the boosted students to have a greater probability of passing the high-level course than more academically-prepared students: P(SLCH) > P(SHCH).


intermediate algebra. This means an entering student can be placed several levels below college-

level, extending time towards degree or certificate attainment.

According to college policies, students seeking to enroll in degree-applicable or transfer-

level math courses in one of the LACCD colleges must take an assessment test to determine

course placement.5 The LACCD colleges have opted to use the ACCUPLACER, COMPASS, or

Mathematics Diagnostic Testing Program (MDTP) to assess and place students. The

ACCUPLACER and COMPASS are computer-adaptive standardized tests developed by College

Board and ACT, respectively. The MDTP, a joint project of the California State University and

the University of California, is a set of math diagnostics designed to measure student readiness

for mathematics. During the period of this study, 2005-2008, five of the LACCD colleges used

the ACCUPLACER, two of the colleges used COMPASS, and two colleges used the MDTP to

make course placement decisions.

Using Multiple Measures

Revisions to the California Code of Regulations in the early 1990s prohibited community

colleges from using single assessment instruments to place students in remedial courses. The

intent was to mitigate disproportionate impact on access to college-level courses for

underrepresented student populations through the use of multiple measures (see Appendix for a

more in-depth overview of the policy). In addition to standardized test scores, multiple measures

can include measures of a student’s prior academic achievement and other noncognitive

attributes, such as educational goals or motivation. The regulations do not formalize a specific

statewide assessment and placement process, so colleges are afforded autonomy in determining

which measures to consider, so long as the measures are not discriminatory (e.g., based on race,

5 There is a “challenge” process in which students can waive pre-requisites if they provide adequate evidence of their math preparation. Our data suggest that less than 5% of enrolled students complete this process.


ethnicity, or gender). Some manuals provide guidance on how to appropriately select and

validate measures at the institutional level (CCCCO, 1998; CCCCO, 2011; Lagunoff, Michaels,

Morris, & Yeagley, 2012), but the devolved autonomy has resulted in considerable variation in

the multiple measures utilized across the LACCD (Melguizo et al., forthcoming).

The information can be gathered through writing samples, performance-based

assessments, surveys and questionnaires, student self-evaluations, counseling interviews during

the enrollment period, or other processes (CCCCO, 2011; Melguizo et al., forthcoming). Most

often, information is collected through a survey taken before or after the assessment test and

points are rewarded or even deducted for various responses. These are combined with the

student’s placement test score and result in a final score used to make a course placement

recommendation based on each college’s set of cutoff scores. Table 1 shows the multiple

measures used to supplement student placement test scores in eight of the nine LACCD colleges

for which multiple measures information was available.

[Insert Table 1. Multiple measures used for math placement, about here]

As Table 1 shows, each college has also chosen to utilize a different combination of

measures. For example, Colleges B and G award a varying amount of points for college plans,

high school GPA, and previous math courses taken. Furthermore, while most of the schools add

multiple measure points to the test score, two schools in LACCD subtract points for selected

responses. College F gives points for what we term college plans (which include the number of

units a student plans to take and the number of hours they plan to work while taking college

classes), and the degree to which college or math is important to the student (which we classify

as motivation), an example of a noncognitive measure. It also deducts points if the student is a

returning student but has not been enrolled for several years. It is important to note that at no


time during the assessment process are students made aware of the college’s cut scores or the

formula used for placement.

Given these assessment and placement rules in the LACCD, the addition or subtraction of

multiple measure points can sometimes be the determining factor in course placement. The

multiple measure points awarded can be enough to place students into a higher-level course or

place them into a lower-level course. As described earlier, students are considered to have

received a multiple measure boost if the additional multiple measure points placed the students in

a math course one level higher than they otherwise would have been by raw test score alone.

Although there are two colleges that use multiple measure information to subtract points and

drop students down into a lower-level course, this does not happen frequently enough to warrant

further investigation.6

Data

We obtained the data used for the study through a restricted-use agreement with the

LACCD. We examined the assessment and enrollment information for all first-time students who

took a placement test between the 2005/06 and 2007/08 academic years. Transcripts provided

outcome data through the spring of 2012, which resulted in seven years of outcome data

available for the 05/06 cohort, six years for the 06/07 cohort, and five years for the 07/08 cohort.

For the access analysis, we restrict the sample to seven out of nine LACCD colleges: College C

was not included because it did not have information on multiple measures during the period of

the study; College G was also not included because it used multiple measures in conjunction

with multiple test score cutoffs in a way that made it non-comparable with the other colleges.7

6 In College J, only 27 out of 4,303 students earned negative multiple measure points, and of those, only 2 were placed in a lower-level course as

a result of point deductions. 7 For analysis of the use of multiple measures and multiple cutoffs in College G, see Author (2014).


The full sample of assessed students for these seven colleges between 2005 and 2008 includes

44,228 students.

The rich assessment data enable us to identify each student whose raw test score was

below the cutoff score at the institution in which they took the placement test, but whose

multiple measure points resulted in an adjusted test score that was above the cutoff score.

Students who met these criteria were coded as having received the multiple measure boost. This

enabled us to determine the total number of students who received a multiple measure boost in

each college between 2005 and 2008, as well as examine the number of boosted students by

college and level of developmental math.

Multiple Measures and Access to Higher-Level Courses

The first set of findings examines the usefulness of multiple measures in increasing

access to higher-level math courses. Table 2 shows the percentage of students boosted into a

higher-level course due to the multiple measure reward structure at seven LACCD colleges for

which multiple measure boosts could be determined. Overall, only 4.23 percent of all students in

this sample were boosted to the next level course between 2005/06 and 2007/08 academic years.

That is, although their raw test score would have placed them in the lower course, the addition of

multiple measure points caused them to surpass the cutoff score and be placed into a higher-level

course. Although the percentages vary by college, very few students overall are moved to higher-

level courses.

[Insert Table 2. Students receiving multiple measure boosts, about here]

One explicit goal of the Title 5 revisions was to mitigate disproportionate impact on the

number of underrepresented minority students being placed into remediation. To examine

disproportionate impact, we calculated math placement rates for each racial subgroup using the


adjusted test scores including multiple measure points (i.e., sample means). We then simulated

counterfactual course placements for each student by using unadjusted test scores without

multiple measure points. Placement rates are provided for two colleges, A and H, which are also

the subject of our multivariate analyses described in the next section. These two colleges each

use one type of additional measure for course placement: prior math course-taking in College A

and self-reported high school GPA in College H.

We present the disproportionate impact results in two ways. First we looked at the overall

course placements by race. Then, we show the distribution of students by race within each level

of developmental math. Comparing the actual placements with the simulated counterfactual

placements both ways enabled us to determine the extent to which the use of multiple measures

mitigated disproportionate impact of remediation by racial subgroup. Table 3 shows the results

of the simulated placements without multiple measure points and actual placement with multiple

measure points by level of developmental math for Latino and African-American students. Table

4 shows placement by racial subgroups within pre-algebra.

[Insert Tables 3 & 4 about here]

The results indicate that the use of multiple measures as currently operationalized in these

LACCD colleges only marginally increases the number of underrepresented minority students

being placed in higher levels of math. For example, in Table 3 we see that under College A's

multiple measure policy, about 3.4 percent fewer African-American and 2.4 percent fewer Latino

students were referred to arithmetic, the lowest-level course in the developmental math sequence.

There was also a 1.5 percent increase in the number of Latino students being placed in

Intermediate Algebra, the highest-level course in the developmental math sequence.


Although the use of multiple measure points increased access to higher-level courses for

African-American and Latino students, the results in Table 4 show that the overall racial

composition of math classes remains largely unchanged, with no statistical difference even at the

10 percent level. We only present the distribution of students by racial subgroups within pre-

algebra, but the results are similar for all math levels (these results are available in the

Appendix). This evidence suggests that despite the current use of multiple measures in the

LACCD colleges, there continues to be disproportionate impact in assignment to remediation.

Multiple Measures and Student Success

While this descriptive analysis offers some insight into the efficacy of multiple measures

in increasing access to higher-level math courses, one of the goals of the multiple measures

policy, it is also important for community colleges to design and use assessment and placement

policies that promote student success. Students should be placed into courses where they are

likely to succeed given their level of college readiness and math skills. To estimate the

association between multiple measures and student success outcomes, we used linear probability

regression models to compare the outcomes of students who were boosted into a higher-level

course due to added multiple measure points with students whose higher test scores placed them

directly into the same course. The short-range outcome of interest is a dichotomous variable

indicating whether or not the student passed the first enrolled math course with a C or better (the

one in which the student was placed). Scott-Clayton et al. (2012) noted the potential controversy

of using earning a C as an outcome since developmental educators and policy-makers may think

of getting C as a mediocre achievement. However, since students in the LACCD who earn a C

are considered as having completed the prerequisite and can move on to the next course, we

believe that earning a C is an appropriate short-term outcome for examining placement accuracy


in this context. The transcript data also allow us to examine two important longer-term outcomes

for community college students—total of number of degree-applicable units completed and total

number of transfer-level units completed. Degree-applicable units are those which can be applied

towards an associate’s degree, and transfer-level units are those which would be accepted at a

California four-year university.

The linear probability regression model is:

yi = α + β1BOOSTi + β2MMPOINTSi + β3TESTi + γX’i + εi

where yi is the outcome of interest. The treatment variable of interest is BOOSTi, a dichotomous

variable indicating whether or not the student received multiple measure points that resulted in a

boost to the next highest level math course. MMPOINTSi is the number of multiple measure

points a student received, and TESTi is the student’s normalized test score by test type, which

allows for comparison across math levels within each college. The normalized score for each

student provides an indicator of the student’s ability relative to other students who took the same

test. We also include dummy variables indicating placement subtest and math placement level,

which serve as a control for any variation that may be related to the different placement tests

used for each level or in each college.8 Finally, X’i is a vector of student information including

age, race, sex, language spoken at home, and assessment cohort. Including these background

variables enables us to obtain a more precise estimate of the relationship between the multiple

measure boost and the outcomes of interest.

Two Focus Colleges

We focus on the effect of the multiple measure boost in two LACCD colleges: College A,

which awards multiple measure points based solely on a student’s prior math background, and

8 The ACCUPLACER, for example, has different subtests such as Arithmetic or Elementary Algebra. Colleges use different subtest scores to make placement decisions.


College H, which awards multiple measure points based solely on a student’s self-reported high

school GPA. Since the multiple measure boost is determined by a single measure in addition to

the placement test score, we can determine the effectiveness of that specific measure in

increasing placement accuracy. College A awards one point for each of the following prior math

background measures: the highest level of math previously taken with a grade of C or better (+1

for trigonometry or higher), the number of years of math taken in high school (+1 for three years

or more), the length of time since math was last taken (+1 if less than one year), and whether or

not the student has taken algebra (+1). Students who take the placement test (ACCUPLACER) at

College A can score a maximum of 120 points and earn a maximum of four multiple measure

points. College H awards two additional points for a high school GPA in the B to B- range, and

four additional points for a high school GPA in the A to A- range. Students who take the

placement test at College D can score between 40 and 50 points depending on the last subtest

that they take (MDTP). We will also discuss results from pooled analyses with two additional

colleges (D and E), but choose to highlight College A and H because they offer the largest ranges

of additional multiple measure points and the possibility of examining long-term outcomes.

Comparison Groups

We run two linear probability regression models for each of the colleges. First, we

compare boosted students to other students whose test scores are in a narrow bandwidth around

their own. In the second model, we include all students within a given course level. To illustrate

this, consider College A, for which the cut score for placement in pre-algebra is 35 on the

ACCUPLACER Arithmetic subtest (AR). Students who attain a score of 35 and above are placed

in pre-algebra (three levels below transfer) while students scoring below 35 are placed in

arithmetic (four levels below transfer). The multiple measure boost could have pushed a student


from arithmetic to pre-algebra if the addition of multiple measure points pushed the adjusted

ACCUPLACER score (raw score + multiple measure points) to 35 or above. For these boosted

students, the range of raw AR scores is 31≤ARr≤34.9 with a maximum of four multiple measure

points. Their resulting adjusted AR score is 35≤ARa≤38.9. In the first regression model

(Around), we compare the boosted students with 35≤ARa≤38.9 to the non-boosted students

whose raw AR tests scores were in the range 35≤ARr≤38.9. In the second regression model

(Entire), we compare the boosted students to the entire range of students in the same course

level. In College A, students can get placed into pre-algebra with a score of 35≤ARa


enrolled and enrolled students. In both colleges, there is no statistical difference in either of the

groups. This is the desired result since bias would be introduced if students enrolled or attempted

the placed math course based on receiving a multiple measure boost. There are several

significant differences in terms of demographic characteristics between the students who enroll

in their placed math course versus those who do not, as well as between those who are boosted

versus those who are not. However, there appears to be no relationship between receiving a

multiple measure boost and enrolling in a math course.

[Insert Table 5, Boosted and enrolled students, about here]

We therefore proceed with enrolled students only, providing an estimate of the treatment

on the treated effect. The sample includes 5,279 out of 8,323 students in College A, or about 63

percent of the original sample, and 7,575 out of 10,349 students, about 73 percent of the original

sample, in College H. These students enrolled within a year of assessment and attempted the

math course in which they were placed. The one year time window is intended to allow sufficient

time for students to enroll in courses as well as to limit the change in the mathematical

knowledge of the students since the assessment. Students who did not comply with the course

placement were also excluded since the goal of the analysis is to determine placement accuracy.

Overall, fewer than 5 percent of students enrolled in a course other than the one in which they

were placed.9 Rather than assigning a value of zero for passing the attempted math course for

those who never enrolled and calculating the intent to treat effect, we reasoned that students

being unaware of the placement process and whether or not a boost was received would help to

limit selection bias. Students are not informed of the placement criteria or placement rules of the

college. After the placement test, the student simply receives a summary of their score and a

9 This figure includes students who took higher-level courses, which is possible if students challenge their placement and receive permission to enroll in a higher-level course, and students who chose to take lower-level courses.


course recommendation. Importantly, the student does not know if placement into a particular

level was the result of a multiple measure point boost, thus limiting the possibility that students

would have exploited the system to attain additional points. This is in line with qualitative

research highlighting the fact that community college students generally feel uninformed and

unaware of community college assessment and placement policies (Venezia, Bracco, & Nodine,

2010).

Main Findings

A summary of our linear probability regression estimates are presented in Tables 6 and 7.

In these summarized tables, we show differences in outcomes between boosted students and their

higher-scoring peers.10

Two sets of regression results are provided for each outcome. In the

“Around” columns, we restricted the analytical sample to those students within a narrow

bandwidth of test scores around the boosted students who took the same subtest.11

In the “Entire”

columns, we included all students within the course level who took the same subtest. Overall, we

observe strong positive relationships between test scores and student outcomes, and between the

number of multiple measure points and student outcomes, suggesting that both are strongly

predictive of student achievement. We hereafter focus our results on the multiple measure boost.

Prior math. Column 1 of Table 6 shows that, all else constant, lower-scoring students in

College A who received a multiple measure boost (based on prior math) that placed them in a

higher-level course performed no differently from their similar-scoring peers in terms of passing

the first math course they enrolled in. However, when comparing boosted students to all other

students in the same math level, we do observe a statistically significant decrease in the

probability of passing the first math class, by about 8 percentage points (p


be expected since the highest-scoring students in the same math level can have test scores that

are 30 points higher than the lowest-scoring students in the level. In terms of long-term

outcomes, columns 3-6 indicate that there are no statistically significant differences in the total

number of degree-applicable and transfer-level credits that boosted students completed through

spring 2012. In other words, students who received a multiple measure boost had the same

degree and transfer progress as their higher-scoring peers.

[Insert Table 6. Regression results, College A, about here]

We also pooled students from Colleges A and D together, since both utilized measures of

prior math to determine the multiple measure boost.12

Results from these pooled estimates are

similar to those from College A alone – students who received a multiple measure boost were no

less likely to pass the math course in which they were placed. We do not show the pooled results

here because estimates of long-term outcomes may be more attributable to differences between

colleges and placement policies than to the multiple measure boost.13

These results are available

in the Appendix.

High school GPA. Students in College H are awarded a maximum of four multiple

measure points based on their self-reported high school GPA. The results in Table 7 indicate

that, with respect to the linear probability of passing the first enrolled math course and eventual

credit accumulation, most of the differences between boosted students and their peers were not

statistically significant. In Column 1, for example, we observe that boosted students performed

no differently from peers with test scores around the placement cutoff. Surprisingly, when

comparing boosted students to all other students in the same math level, students who were

placed into higher level courses as a result of a multiple measure boost actually had a 6.16

12 We thank the anonymous reviewers for this suggestion. The full pooled results for College A and D are available in the Appendix. 13 For example, both Colleges D and E assign students to what we call “extended” algebra courses, which extends the developmental math sequence by an additional semester.


percent higher likelihood of passing the course than their higher-scoring peers (p


instruction or grading in order to meet the needs of students, which would bias the estimate of

the relationship between multiple measure boosts and student outcomes.

To address these concerns, we make the assumption that instructors do not adjust their

practices much in response to student academic preparation, and this is reasonable given that a

relatively small percentage of students in any given course would have received the boost (about

4.2 percent of students on average for seven of nine LACCD colleges). Furthermore, the non-

significant estimates we obtained in this analysis are consistent across nearly all levels of

developmental math, as well as in both colleges, which suggests that it is unlikely that cut scores

are incorrect across the board or that instructor practices are systematically forgiving to

students.14

Finally, our analytic approach consisted of two models – one in which we compared

boosted students to students with similar placement test scores, and another in which we

compared boosted students to all other students in their level within a given institution. These

results demonstrated that boosted students were not only as successful as the students within a

similar score range, but in most cases also as successful as students within the entire level. Given

that students at a given level of developmental math can have differences as large as 30 points on

their placement tests scores, we remain confident that our estimates of the effect of receiving a

multiple measure boost are consistent and reliable.

Discussion

Increasing Placement Accuracy

This evidence from the LACCD is timely given the changing landscape of placement

testing for developmental education. As mentioned in the introduction, several states, such as

Connecticut, Florida, North Carolina, and Texas, have already or are in the process of revising

developmental education assessment and placement policies to incorporate multiple measures.

14 Analyses by level of developmental are available from the authors upon request.


However, aside from predictive validity estimates, there are few sources of evidence from actual

placement policies. The findings of this study are important because there is limited

understanding of what measures can be validly used to make course placement decisions. In

addition, qualitative research with community college faculty and staff has shown that

practitioners do not feel supported in measure selection and validation, and that they sometimes

perceive measures to be insignificant (Melguizo et al., forthcoming). Even though coordinating

entities such as the CCCCO provide guides for multiple measure use, there is limited evidence

and validation of measures that can be used to inform placement decisions.

Our analysis of LACCD data provides validation for two specific measures – prior math

background and high school GPA. Even though these measures are known to be predictive of

college outcomes, current conceptions of validation highlight the need to examine actual

outcomes in contexts where measures are used to make placement decisions (Kane, 2006). The

results suggest that community colleges can increase placement accuracy by using multiple

measure information in conjunction with placement test scores. Evidence from the LACCD

colleges demonstrates that those students who were placed into higher-level developmental math

courses using multiple measures performed no differently from their higher-scoring peers. Since

these students were given the opportunity to take a higher-level course and performed at least as

well as their higher-scoring peers, these students were more accurately placed than they would

have been by placement test scores alone.

One implication of the findings for the colleges which used measures of prior math is that

these measures can supplement test scores but not necessarily replace them. We observed

improvements in placement accuracy for students who scored around the placement cutoff, but

these students did not match or outperform their higher-scoring peers. This suggests that


although measures of prior math can increase placement accuracy, they should probably be used

in conjunction with placement test scores.

Like other studies, we also found that high school GPA is highly predictive of college

persistence and success. However, the finding that students who received a multiple measure

boost based on GPA outperformed the entire range of students in the same level suggests that

GPA may be a very useful measure for making placement decisions, and underscores the role of

effort and self-control in college achievement (Duckworth et al., 2012). Further research should

examine the extent to which making placement decisions based solely on GPA, either self-

reported or obtained directly from high school transcripts, can lead to even greater improvements

in placement accuracy. Of course, colleges would need to take caution so as not to induce

students to game the system by reporting higher or lower GPAs.

Overall, the findings indicate that these two measures can be systematically used to

improve course placement decisions. Using them in conjunction with test scores can increase

placement accuracy and may be, as Belfield and Crosta (2012) suggest, closer to the optimal

decision rule for placement in developmental math.

Promoting Access and Success

This examination of community college assessment and placement policies also

highlights the underlying tension between the goals of access and success when making

placement decisions. Indeed, promoting progression versus maintaining standards is one of the

“opposing forces” that community colleges often operate under (Jaggars & Hodara, 2013; Perin,

2006). Community colleges have the responsibility to place students in courses in which they are

most likely to succeed given their math skills while simultaneously promoting progression

towards completion and attainment.


The results of the study demonstrate that multiple measures can be utilized to achieve

both of these goals. Based on this evidence from the LACCD colleges, students who received a

multiple measure boost based on prior math and high school GPA took higher-level courses and

succeeded in them at rates no different from their higher-scoring peers. Using these additional

student background measures in conjunction with test scores to make course placement decisions

may therefore achieve the goals of increasing access and ensuring student success.

Nonetheless, while boosted students are just as likely to be successful as their peers, our

analyses also show that the goals of mitigating disproportionate impact in remediation are not

being fully realized. The California state policy explicitly states that assessment practices should

not result in disproportionate impact on any underrepresented minority group. Our simulated

placements with and without multiple measures show that the use of these particular multiple

measures only marginally increased access to higher level math courses for African-American

and Latino students. Community colleges should therefore continue to explore other ways to

improve assessment and placement such that disproportionate placement in remediation is

mitigated while the likelihood of student success is maximized.

Some research shows that noncognitive measures may be useful for identifying college

readiness and promoting access and success in college, particularly for underrepresented

minority students (Sedlacek, 2004). The validity of these measures has yet to be explored in the

context of developmental education. Even though some of the LACCD colleges' placement rules

include noncognitive measures such as college plans, educational goals, availability of social

supports, and motivation, these are often used in conjunction with other measures. We were thus

unable to identify the singular effect of using these to make placement decisions. In addition,

colleges that used these measures also weighted multiple measures in such a way that, relative to


Colleges A and H, very few students received a multiple measure boost. Further research should

focus on validating other cognitive and noncognitive measures that can be useful for identifying

incoming student readiness, specifically those that increase access to higher-level courses for

underrepresented student populations.

Conclusion

The multiple measures policy in California provides an opportunity to validate measures

in terms of their usefulness for course placement. The results of this study indicate that students

who were placed into higher-level courses using information from multiple measures, in this case

high school GPA and prior math course-taking, performed no differently from their peers who

earned higher test scores. This suggests that community colleges can systematically improve

placement accuracy by using student background information in addition to assessment data to

make initial course placement decisions. Such policies would increase access to higher-level

math without decreasing students’ chances of success in the first math course in which they

enroll or eventual credit accumulation. We do recognize that these data may be unavailable for

non-traditional students or international students whom community colleges serve in substantial

numbers. Further research should be done to identify and validate a broader range of measures

that can improve placement accuracy for all types of students. Still, using multiple sources of

information about incoming community college students not only increases access to higher-

level math courses but can also ensure that students are placed at a level where they are likely to

be successful. This can ultimately promote equity and efficiency in the assessment and

placement process, accelerate college completion, and reduce the financial and academic burdens

of postsecondary remediation.


References

ACT, Inc. (2012). ENGAGE College user’s guide. Iowa City, IA: ACT, Inc.

Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through

college. Washington, D.C.: Office of Vocational and Adult Education, U.S. Department

of Education.

Allen, J., Robbins, S. B., Casillas, A., & Oh, I. S. (2008). Third-year college retention and

transfer: Effects of academic performance, motivation, and social connectedness.

Research in Higher Education, 49(7), 647-664.

American Educational Research Association, American Psychological Association, & National

Council on Measurement in Education. (1999). Standards for educational and

psychological testing. Washington, DC: AERA, APA, & NCME.

Armstrong, W. B. (2000). The association among student success in courses, placement test

scores, student background data, and instructor grading practices. Community College

Journal of Research and Practice, 24(8), 681-695.

Author. (2014).

Bailey, T. (2009). Challenge and opportunity: Rethinking the role and function of developmental

education in community college. New Directions for Community Colleges, 145, 11-30.

Bailey, T., Jeong, D. W., & Cho, S. W. (2010). Referral, enrollment, and completion in

developmental education sequences in community colleges. Economics of Education

Review, 29(2), 255-270.

Belfield, C. R., & Crosta, P. M. (2012). Predicting success in college: The importance of

placement tests and high school transcripts (CCRC Working Paper No. 42). New York,

NY: Community College Research Center.


Boylan, H. (2009). Targeted intervention for developmental education students (TIDES). Journal

of Developmental Education, 32(3), 14-23.

Brewer, D., & Smith, J. (2008). A framework for understanding educational governance: The

case of California. Education Finance and Policy, 3(1), 20-40.

Burdman, P. (2012). Where to begin? The evolving role of placement exams for students starting

college. Boston, MA: Jobs for the Future.

Cage, M. C. (1991, June 21). Cal. Community Colleges System Agrees to Change Role of

Testing. The Chronicle of Higher Education, pp. A21.

California Code of Regulations, Title 5, §§ 58106, 55002, 55201, 55510-55532

California Community Colleges Chancellor's Office. (1998). Multiple measures and other

sorrows: A guide for using student assessment information with or instead of test scores.

Sacramento, CA: Author. Retrieved from http://extranet.cccco.edu/Portals/1/SSSP/

Matriculation/Assessment/MultipleMeasuresAndOtherSorrowsMarch1998.pdf

California Community Colleges Chancellor's Office. (2011). California community colleges

matriculation program handbook. Sacramento, CA: Author. Retrieved from

http://extranet.cccco.edu/Portals/1/SSSP/Matriculation/MatriculationHandbookRevSepte

mber2011.pdf

Collins, M. L. (2008). It's not just about the cut score: Redesigning placement assessment

policies to improve student success. Boston, MA: Jobs for the Future.

Conley, D. T. (2007). Toward a more comprehensive conception of college readiness. Eugene,

OR: Educational Policy Improvement Center.

DesJardins, S. L., & Lindsay, N. K. (2007). Adding a statistical wrench to the "toolbox".

Research in Higher Education, 49(2), 172-179.


Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and

passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087–

1101.

Duckworth, A. L., Quinn, P. D., & Tsukayama, E. (2012). What No Child Left Behind leaves

behind: The roles of IQ and self-control in predicting standardized achievement test

scores and report card grades. Journal of Educational Psychology, 104(2), 439-451.

Educational Testing Service (ETS). (2013). ETS Personal Potential Index: Evaluator user’s

guide. Princeton, NJ: Author.

Fain, P. (2013, February 19). Redefining college-ready. Inside HigherEd. Retrieved from:

http://www.insidehighered.com/news/2013/02/19/two-community-colleges-get-serious-

about-working-k12

Florida Senate Bill No. 1720. Retrieved from http://laws.flrules.org/files/Ch_2013-051.pdf

Fong, K., Melguizo, T., Bos, H., & Prather, G. (2013). A different view on how we understand

progression through the developmental math trajectory. Los Angeles, CA: Rossier

School of Education, University of Southern California. Retrieved from

http://www.uscrossier.org/pullias/research/projects/sc-community-college/

Fulton, M. (2012). Using state policies to ensure effective assessment and placement in remedial

education. Denver, CO: Education Commission of the States.

Geiser, S., & Santelices, M. (2007). Validity of high-school grades in predicting student success

beyond the freshman year: High-school record vs. standardized tests as indicators of

four-year college outcomes. Berkeley, CA: Center for Studies in Higher Education.

Geiser, S., & Studley, R. (2003). UC and the SAT: Predictive validity and differential impact of

the SAT I and SAT II at the University of California. Educational Assessment, 8(1), 1-26.


Gerlaugh, K., Thompson, L., Boylan, H., & Davis, H. (2007). National study of developmental

education II: Baseline data for community colleges. Research in Developmental

Education, 20(4), 1-4.

Heckman, J., Stixrud, J., & Urzua, S. (2006). The effects of cognitive and noncognitive abilities

on labor market outcomes and social behavior. Journal of Labor Economics, 24(3), 411–

482.

Hodara, M., Jaggars, S. S., & Karp, M. M. (2012). Improving developmental education

assessment and placement: Lessons from community colleges across the country (CCRC

Working Paper No. 51). New York, NY: Community College Research Center.

Hughes, K. L., & Scott-Clayton, J. (2011). Assessing developmental assessment in community

colleges. Community College Review, 39(4), 327-351.

Jaggars, S. S., & Hodara, M. (2011). The opposing forces that shape developmental education.

Community College Journal of Research and Practice, 37(7), 575-579.

Jaffe, L. (2012, October). Mathematics from high school to community college: Preparation,

articulation, and college un-readiness. Paper presented at the 2012 Annual Conference of

the Research and Planning Group for California Community Colleges.

Jenkins, D., Jaggars, S. S., & Roksa, J. (2009). Promoting gatekeeper course success among

community college students needing remediation: Findings and recommendations from a

Virginia Study. New York, NY: Community College Research Center. Retrieved from:

http://eric.ed.gov/?id=ED507824

Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational Measurement (4th

ed., pp.

17–64). Westport, CT: ACE/Praeger Publishers.


Karp, M. M., & Bork, R. H. (2012). "They never told me what to expect, so I didn't know what to

do": Defining and clarifying the role of a community college student (CCRC Working

Paper No, 47). New York, NY: Community College Research Center.

Lagunoff, R., Michaels, H., Morris, P., & Yeagley, P. (2012). A framework for evaluating the

technical quality of multiple measures used in California community college placement.

San Francisco, CA: WestED.

Lewallen, W. C. (1994). Multiple measures in placement recommendations: An examination of

variables related to course success. Lancaster, CA: Antelope Valley College. (ERIC

Document No. 381 186).

Long Beach College Promise. (2013). 5-Year Progress Report (2008-2013): A breakthrough in

student achievement. Long Beach, CA: Author. Retrieved from: http://www.longbeach

collegepromise.org/wp-content/uploads/2013/03/LBCP-5-Year-ProgressReport.pdf.

Mattern, K. D., & Packman, S. (2009). Predictive validity of ACCUPLACER scores for course

placement: A meta-analysis (Research Report No. 2009-2). New York, NY: College

Board.

Maruyama, G. (2012). Assessing college readiness: Should we be satisfied with ACT or other

threshold scores? Educational Researcher, 41(7), 252-261.

Marwick, J. D. (2004). Charting a path to success: The association between institutional

placement policies and the academic success of Latino students. Community College

Journal of Research and Practice, 28(3), 263-280.

Medhanie, A. G., Dupuis, D. N., LeBeau, B., Harwell, M. R., & Post, T. R. (2012). The role of

the ACCUPLACER mathematics placement test on a student’s first college mathematics

course. Educational and Psychological Measurement, 72(2), 332-351.


Melguizo, T., Bos, H., & Prather, G. (2013). Using a regression discontinuity design to estimate

the impact of placement decisions in developmental math in Los Angeles Community

College District (LACCD). Los Angeles, CA: University of Southern California.

Retrieved from http://www.uscrossier.org/pullias/research/projects/sc-community-

college/

Melguizo, T., Hagedorn, L. S., & Cypers, S. (2008). Remedial/developmental education and the

cost of community college transfer: A Los Angeles County sample. The Review of Higher

Education, 31(4), 401-431.

Melguizo, T., Kosiewicz, H., Prather, G., & Bos, H. (in press). How are community college

students assessed and placed in developmental math? Grounding our understanding in

reality. Journal of Higher Education.

Merisotis, J. P., & Phipps, R. A. (2000). Remedial education in colleges and universities: What's

really going on? The Review of Higher Education, 24(1): 67-85.

National Center for Public Policy and Higher Education & Southern Regional Education Board

(NCPPHE & SREB). (2010). Beyond the rhetoric: Improving college readiness through

coherent state policy. Atlanta, GA: NCPPHE. Retrieved from

http://publications.sreb.org/2010/Beyond%20the%20Rhetoric.pdf

Noble, J. P., & Sawyer, R. L. (2004). Is high school GPA better than admission test scores for

predicting academic success in college? College and University Journal, 79(4), 17–22.

Perin, D. (2006). Can community colleges protect both access and standards? The problem of

remediation. Teachers College Record, 108(3):339-373.

Perry, M., Bahr, P. M., Rosin, M., & Woodward, K. M. (2010). Course-taking patterns, policies,

and practices in developmental education in the California Community Colleges.


Mountain View, CA: EdSource. Retrieved from

http://www.edsource.org/assets/files/ccstudy/FULL-CC-DevelopmentalCoursetaking.pdf

Porchea, S. F., Allen, J., Robbins, S., & Phelps, R. P. (2010). Predictors of long-term enrollment

and degree outcomes for community college students: Integrating academic,

psychosocial, sociodemographic, and situational factors. The Journal of Higher

Education, 81(6), 750-778.

Porter, A., & Polikoff, M. (2012). Measuring academic readiness for college. Educational

Policy, 26(3), 394-417.

Robbins, S. B., Allen, J., Casillas, A., Peterson, C. H., & Le, H. (2006). Unraveling the

differential effects of motivational and skills, social, and self-management measures from

traditional predictors of college outcomes. Journal of Educational Psychology, 98(3),

598-616.

Sawyer, R. (1996). Decision theory models for validating course placement tests. Journal of

Educational Measurement, 33(3), 271-290.

Sawyer, R. (2007). Indicators of usefulness of test scores. Applied Measurement in

Education, 20(3), 255-271.

Sawyer, R. (2013). Beyond Correlations: Usefulness of High School GPA and Test Scores in

Making College Admissions Decisions. Applied Measurement in Education, 26(2), 89-

112.

Scott-Clayton, J. (2012). Do high-stakes placement exams predict college success? (CCRC

Working Paper No. 41). New York, NY: Community College Research Center.


Scott-Clayton, J., Crosta, P., & Belfield, C. (2012). Improving the targeting of treatment:

Evidence from college remediation (NBER Working Paper 18457). Cambridge, MA:

National Bureau of Economic Research.

Sedlacek, W. E. (2004). Beyond the Big Test: Noncognitive Assessment in Higher Education.

San Francisco, CA: Jossey-Bass.

Seymour-Campbell Matriculation Act (1986), California Education Code §§ 78210-78218

Sommerville, J., & Yi, Y. (2002). Aligning K-12 and postsecondary expectations: State policy in

transition. Washington, D.C.: National Association of System Heads.

Stone, D. (2002). Policy Paradox: The Art of Political Decision Making. New York, NY: W.W.

Norton.

Texas Higher Education Coordinating Board. (2012). 2012-2017 Statewide Developmental

Education Plan. Retrieved from http://www.thecb.state.tx.us/index.cfm?objectid=

233A17D9-F3D3-BFAD-D5A76CDD8AADD1E3

van der Linden, W. J. (1998). A decision theory model for course placement. Journal of

Educational and Behavioral Statistics, 23(1), 18-34.

Venezia, A., Bracco, K., & Nodine, T. (2010). One-shot deal? Students' perceptions of

assessment and course placement in California's community colleges. San Francisco, CA:

WestEd.

Wiseley, W. C. (2006). Regulation, interpretation, assessment and open access in California

community colleges. Retrieved from http://home.comcast.net/~chuckwiseley/

Publications/Regulation,%20Interpretation,%20and%20Open%20Access%20(Wiseley,%

202006).pdf


Table 1. Multiple measures used for math placement

College Point Range Academic Background College Plans Motivation

HS Diploma HS GPA Prior Math

A 0 to 4

+

B 0 to 3

+ +

+

C N/A

D 0 to 2

+

E 0 to 3

+

F -2 to 2

+/- +/-

G 0 to 3

+ +

+

H 0 to 4

+

J -2 to 5 + +/- +/-

Note: (+) indicates measures for which points are added, and (-) indicates measures for which points are subtracted. Academic Background

includes whether the student received a diploma or GED, high school GPA, and prior math course-taking (including achievement and highest-level completed). College plans include hours planned to attend class, hours of planned employment, and time out of formal education.

Motivation includes importance of college and importance of mathematics. Multiple measure information was not available for one of the nine

LACCD colleges. Source: LACCD, 2005-2008


Table 2. Students receiving multiple measure boost into higher-level courses (%)

College AR to PA PA to EA EA to IA IA to CLM Total % Observations

A 5.60 7.86 9.15 13.20 6.13 8323

B 0.00 3.95 2.93 3.26 0.87 4470

C - - - - - -

D 3.50 6.29 5.49 4.23 3.80 9316

E - 1.73 5.33 0.00 1.62 5189

F 4.24 0.70 0.14 0.00 0.83 2278

G - - - - - -

H 1.97 1.71 2.43 6.09 2.66 10349

J 27.82 20.10 15.44 20.00 13.76 4303

Total % 5.38 5.26 4.95 5.84 4.23

Observations 9236 13294 9593 3460 44228

Note: Arithmetic (AR); Pre-Algebra (PA); Elementary Algebra (EA); Intermediate Algebra (IA); College-Level Math (CLM). Total percentage and total number of observations includes students who were assigned to AR (N=8645). Multiple measure information was not available for College C. College G

uses a set of diagnostic tests and multiple cutoffs to assign students to developmental math courses, so the use of multiple measures operates differently

from the other colleges. For further details about College G see Author (2014).


Table 3. Placement by math level with and without multiple measures

(a) African-American

College A

College H

w/o MM w/ MM Difference


Arithmetic 0.337 0.303 0.0339*

0.0568 0.0502 0.00655

Pre-algebra 0.403 0.426 -0.0235

0.225 0.225 0

Elem. Algebra 0.151 0.150 0.00188

0.410 0.415 -0.004

Int. Algebra 0.0988 0.107 -0.00847

0.253 0.255 -0.00218

College-Level Math 0.0103 0.0141 -0.00376

0.0546 0.0546 0

Observations 1063 1063 2126 458 458 916

Pearson (p-value) 0.461 0.995

(b) Latino/a

College A

College H



Arithmetic 0.213 0.189 0.0241*

0.0237 0.0198 0.00395

Pre-algebra 0.449 0.451 -0.00192

0.176 0.174 0.00216

Elem. Algebra 0.227 0.231 -0.00466

0.412 0.412 0.000359

Int. Algebra 0.101 0.116 -0.0151**

0.316 0.315 0.000359

College-Level Math 0.0101 0.0126 -0.00247

0.0722 0.0790 -0.00682

Observations 3649 3649 7298 2784 2784 5568


Note: Mean coefficients; Multiple Measures (MM)

Source: LACCD, 2005-2008 * p


Table 4. Placement in pre-algebra with and without multiple measures

College A

College H



Asian 0.114 0.113 0.00166

0.139 0.143 -0.00411

African-American 0.135 0.141 -0.00618

0.0899 0.092 -0.00209

Hispanic 0.516 0.512 0.00391

0.428 0.432 -0.00457

White 0.164 0.163 0.00101

0.235 0.224 0.0106

Other 0.0714 0.0718 -0.0004

0.109 0.109 0.000146

Observations 3179 3217 6396 1146 1120 2266


Note: Mean Coefficients; Multiple Measures (MM) Source: LACCD, 2005-2008

* p


Table 5. Boosted and enrolled students

(a) College A

Non-boosted vs. Boosted

Non-enrolled vs. Enrolled

Non-

boosted Boosted Difference

Non-

enrolled Enrolled Difference

Attempted placed math course 0.636 0.61 0.0261

Received MM boost

0.0654 0.0589 0.00646

Male 0.473 0.478 -0.00576

0.502 0.457 0.0451***

Asian 0.200 0.202 -0.00153

0.237 0.179 0.0578***

African-American 0.128 0.125 0.00237

0.146 0.117 0.0281***

Latino/a 0.436 0.469 -0.0322

0.369 0.478 -0.109***

White 0.164 0.147 0.0172

0.172 0.158 0.0142

Other 0.0710 0.0569 0.0142

0.0759 0.0669 0.00902

Documents

Using Multiple Measures to Make Math Placement Decisions ......underrepresented minority students were being disproportionately placed into remedial courses (Perry, Bahr, Rosin, &