Upload
winfred-shepherd
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
LEARNING FROM EXPERIMENTATION IN DEVELOPMENTAL MATH IN COMMUNITY COLLEGES IN CALIFORNIA: RESULTS FROM A LONG-TERM RESEARCH PARTNERSHIP
The Steinhardt Institute for Higher Education Policy,
New York University, October 19, 2015
Tatiana MelguizoAssociate Professor, University of Southern [email protected]
This research was funded by a grant from the U.S. Department of Education’s Institute of Education Sciences (IES).
Overview
Educational & professional background Key results from a federally funded
long-term research collaboration with a large urban community college district in California
Other research interests Q & A
Inequalities in college access and outcomes by socioeconomic status in Colombia
Only 25% of students from the lowest income strata enter college compared to 60% of students from the highest income strata(Melguizo, Sanchez & Velasco, 2015)
Los Andes University Bogota-Colombia
Banning of Affirmative Action Policies in the U.S. in the mid 1990s
I started my Ph.D. at the time that Affirmative Action policies were banned in California
That major policy shift motivated me to study the impact of attending more selective colleges and universities for students of color
Findings suggested that students of color both in the 1980s and in the 1990s had higher graduation rates at the most selective colleges and universities (Melguizo, 2007)
Findings cited as part of Amicus Brief to the Supreme Court by AERA
Creating a Research Partnership to Study Assessment and Placement in Developmental Math
The purpose of the partnership is to inform developmental education research, policies, and practices Site: The Los Angeles Community College District Research Focus: To examine the impacts and
implementation of test-based, alternative placement policies, and delivery methods for developmental math
Research approaches: Experimental and quasi-experimental, qualitative, & descriptive
Strength of partnership built on commitment & unique expertise of partners A joint commitment to improve achievement in
dev. ed. through policy and practical change
An firm belief that each partner is an equal; brings unique contextual, technical expertise LACCD: Knowledge about local context, access to
administrative records, faculty, and administrators
USC: A strong record of community college research
AIR: Technical expertise in conducting experimental and quasi-experimental research
Problem Statement
• Every year about 80 percent of community college students in California are placed into preparatory mathematics. This percentage is higher than the national average
• Community college students have widely varying initial skills levels
• Colleges have to offer classes to meet these levels and have to keep heterogeneity in the classrooms manageable
• Placing students incorrectly can reduce the likelihood that students succeed
Why LACCD?
Los Angeles Community College District - a natural laboratory
Diverse student population that varies by college.
Nine colleges with 130,000 plus students. “Common data system.” Large number of observations. Presumption of representativeness—likely
to capture the wide variation across community colleges in the United States.
Literature on Impacts of Remedial Education
• Proponents argue that remedial education provides the preparation necessary for students to succeed in college (Boylan, Bliss, & Bonham, 1994; 1997; Lazarik, 1997)
• Critics contend that the benefits that students obtain are not clear (Calcagno & Long, 2008; Martorell & McFarland, 2011; Scott-Clayton & Rodriguez, 2015).
Five Key FindingsFinding 1: Establishing an effective A&P system is complex.
More support and training is needed for faculty and administrators charged with this task. (Melguizo, Kosiewicz, Prather & Bos, 2014).Finding 2: The largest barrier for developmental math students is attempting their initial course (Fong, Melguizo, & Prather, 2015).
Finding 3: Community college faculty and administrators have the opportunity to improve placement and success in developmental math by engaging in a systematic process of calibration of the cut scores of assessment and placement tests (Melguizo, Bos, Ngo, Mills & Prather, 2015).
Finding 4: The diagnostic test places students more accurately than the computer-adaptive test (Ngo & Melguizo, 2015).
Finding 5: The inclusion of multiple measures in the placement process can increase access to higher-level math without decreasing students’ chances of success (Ngo & Kwon, 2014; Fong & Melguizo, 2015).
F3: Systematic Process of Calibration of the Cut Scores
Math faculty set the cut points between the different levels based on who applies and how their course offerings are distributed
If the cut points are too high, too many students languish in remedial courses
If the cut points are too low, too many students fail higher-level courses and present a challenge to the instructors
Getting the cut points just right is important
F3: Different Pathways to Success(Arithmetic vs. Pre Algebra)
Test
Placed in Pre-Algebra
Placed in Arithmetic
Enroll in Pre-Algebra
Enroll in Arithmetic
No enrollment
Success
Failure
Success
Failure
Next course
F3: Ideal Regression Discontinuity Situation
Regression discontinuity analysis is the strongest non-experimental method to estimate causal effects
It depends on a continuous forcing variable and an exogenously established cut point
Those two conditions are present in this situation
F4: Placement Accuracy:Diagnostic versus Computer-AdaptiveTestsDo diagnostics improve placement accuracy?MethodsI.Logistic Regressions
Predicting EA outcomes with skill-specific math information
II.Placement Accuracy Sum of accurate placements
III.Regression Discontinuity Traditional (with single-score) Binding-score (with multiple criteria)
Placement AccuracyTable 3. Percent of accurate placements by level of developmental math.
AR vs PA PA vs EA EA vs IA
ACCUPLACER Colleges
College A 73.2 75.3 72.9
College B - 75.5 48.4
College C 19.8 22.6 24
College D 77.8 47.2 71.4
MDTP Colleges
College G (w/subscores)
51.6 76.3 56.1
College G (w/o subscores)
49.9 68.2 44.9
College H 90.0 65.5 54.1Note: Placement accuracy calculated using method described by Scott-Clayton (2012). Percentage shown is the sum of the proportion of students predicted to pass higher-level course and placed there plus the proportion of students not predicted to pass the higher-level course and placed in the lower level course.
F4: Diagnostic tests are Placing Students more Accurately than Computer-Adaptive Tests
Students placed using results from computer-adaptive tests were more negatively impacted by the placement decision than prior cohorts placed by MDTP.
Students were less likely to enroll and persist onto the next math course after the placement test switch.
Consistent with other studies, we found that the diagnostic test can provide information on student proficiency on a range of subtopics such as fractions, exponents, and reasoning which can improve math placement decisions and/or tailor instruction in math courses.
F5: Findings
Only 6% of the students benefitted from multiple measures at the LACCD
Major benefits for African American and Latino students who could enroll in higher-level math courses
We found no evidence that “boosted” students were less likely to complete the course
Conclusions
Engaging in research partnerships with large districts are a way of increasing research capacity of the district while gaining a much more nuanced understanding of the context for the researcher
Research findings not only contribute to the knowledge but are also policy relevant and actionable
Researcher-practitioner partnerships can be conducive to high-quality and high-impact research
Other Current Projects Mixed-methods evaluation of the Susan
Thompson Learning Communities. Buffett Foundation
Using high school transcript information to refine A&P in math. LAUSD-LACCD-USCNSF: EAGER
Accurately estimating Student Learning Outcomes (SLOs) in higher education Brazil (Melguizo & Wainer, under review) Colombia (Melguizo, Zamarro, Sanchez & Velasco,
2015) Proposed an RCT evaluation to test student
self-placement in Dev Ed Math
THANK YOU!Questions
Tatiana [email protected]
http://www.uscrossier.org/pullias/research/projects/sc-community-college/