Embed Size (px)
Citation preview
Educational Outcomes of I-BEST,Washington State Community and Technical College System’s
Integrated Basic Education and Skills Training Program: Findings from a Multivariate Analysis
Davis JenkinsMatthew Zeidenberg
Gregory Kienzl
May 2009
CCRC Working Paper No. 16
Address correspondence to:
Davis Jenkins Community College Research Center Teachers College, Columbia University 525 West 120th Street, Box 174 New York, NY 10027 [email protected]
Funding for this study was generously provided by the Ford Foundation as part of the Community College Bridges to Opportunity Initiative. We are grateful to our program officers John Colborn and Cyrus Driver for their guidance and support. Thanks also to Tina Bloomer, David Prince, and their colleagues at the Washington State Board for Community and Technical Colleges for sharing the data used in this study and for their insights on the preliminary findings. Thomas Bailey, Shanna Jaggars, and Michelle Van Noy provided helpful comments on earlier drafts. We thank Doug Slater for his excellent work in editing the paper. All errors are our own.
Table of Contents
Executive Summary ............................................................................................................ 2
1. Introduction and Background ......................................................................................... 4
2. Data and Methods ......................................................................................................... 10
3. Findings......................................................................................................................... 12
3.1 Descriptive characteristics ..................................................................................... 12 3.2 Estimates of the probability of earning college credit ........................................... 16 3.3 Estimates of the number of credits earned............................................................. 18 3.4 Estimates of the probability of persisting into 2007-08......................................... 20 3.5 Estimates of the probability of earning an award .................................................. 22 3.6 Estimates of the probability of achieving point gains on basic skills tests............ 24
4. Conclusion .................................................................................................................... 26
Appendix: A Brief Description of Propensity Score Matching ........................................ 28
References......................................................................................................................... 30
Executive Summary
Nationally, relatively few of the more than 2.5 million adults who enroll annually
in basic skills programs advance successfully to college-level coursework. This limits the
ability of such individuals to secure jobs that pay family-supporting wages and that offer
opportunity for career advancement. This paper presents findings from a study conducted
by the Community College Research Center (CCRC) at Teachers College, Columbia
University, on the outcomes of the Integrated Basic Education and Skills Training
program, or I-BEST, an innovative program developed by the community and technical
colleges in Washington State to increase the rate at which adult basic skills students enter
and succeed in postsecondary occupational education and training.
Under the I-BEST model, basic skills instructors and college-level career-
technical faculty jointly design and teach college-level occupational courses for adult
basic skills students. Instruction in basic skills is thereby integrated with instruction in
college-level career-technical skills. The I-BEST model challenges the conventional
notion that basic skills instruction ought to be completed by students prior to starting
college-level courses. The approach thus offers the potential to accelerate the transition of
adult basic skills students to college programs.
The CCRC study reported on here used multivariate analysis to compare the
educational outcomes over a two-year tracking period of I-BEST students with those of
other basic skills students, including students who comprise a particularly apt comparison
group — those non-I-BEST basic skills students who nonetheless enrolled in at least one
workforce course in academic year 2006-07, the period of enrollment in the study. The
researchers examined data on more than 31,000 basic skills students in Washington State,
including nearly 900 I-BEST participants. The analyses controlled for observed
differences in background characteristics of students in the sample.
The study found that students participating in I-BEST achieved better educational
outcomes than did other basic skills students, including those who enrolled in at least one
non-I-BEST workforce course. I-BEST students were more likely than others to:
Continue into credit-bearing coursework;
Earn credits that count toward a college credential;
2
Earn occupational certificates; and
Make point gains on basic skills tests.
On all the outcomes examined, I-BEST students did moderately or substantially better
than non-I-BEST basic skills students in general. The I-BEST group’s comparative
advantage relative to non-I-BEST basic skills students who enrolled in at least one
workforce course was not as large, but was still significant.
The study also compared I-BEST students to a group of non-participants with
similar characteristics who were matched with the I-BEST students using a statistical
technique called propensity score matching (PSM). Using the PSM analysis, the study
estimated that, over the two-year tracking period, the probability that I-BEST students
would earn at least one college credit was 90 percent, while the probability for the
matched students was 67 percent, a 23 percentage point difference. I-BEST students
earned, on average, an estimated 52 quarter-term college credits, compared to an average
of 34 quarter-term credits for the matched comparison group. I-BEST students had a
higher probability of persisting into the second year: 78 percent, compared with 61
percent for the matched group. The chances of earning an occupational certificate was 55
percent for I-BEST students, compared with only 15 percent for the matched group. I-
BEST students also had a higher likelihood of making point gains on the CASAS basic
skills test: 62 percent compared with 45 percent for the matched group.
While the results of this analysis show that participation in I-BEST is correlated
with better educational outcomes over the two-year tracking period, it is important to note
that they do not provide definitive evidence that the I-BEST program caused the superior
outcomes. It could be that, because of the way students are selected into the program,
those who participate have higher motivation or other characteristics not measured in this
study that make them more likely to succeed. Selection bias could also operate in the
other direction if I-BEST students are more disadvantaged in ways we do not measure.
In the future, CCRC researchers plan to conduct fieldwork to better understand
the process by which students are selected into the program. CCRC will also extend this
study by examining degree attainment and labor force outcomes of I-BEST students over
a longer timeframe, by collecting financial data to estimate program cost-effectiveness,
and by examining the practices of I-BEST programs that produce superior outcomes.
3
1. Introduction and Background
Today, most jobs that pay wages sufficient to support a family require at least
some postsecondary education, and, increasingly, a college credential (Barton, 2000;
Baum & Ma, 2007; Osterman, 2008). Yet, according to the Census Bureau (2007a), more
than 75 million Americans 25 years and older have no education beyond high school.
More than 30 million of them lack a high school credential or GED. In addition, over 12
million individuals in this age group lack basic fluency in English (U.S. Census Bureau,
2007b), which also limits the ability to secure a good job.
Community colleges, schools, and community organizations offer programs for
adults with limited skills and education, including adult basic education (ABE) and GED
preparation programs for individuals who do not have a high school credential, and
English-as-a-second-language (ESL) programs for those with limited proficiency in
English. Over 2.5 million students enroll each year in adult basic skills programs funded
by states and the federal government (U.S. Department of Education, 2006). Yet, few
students in these programs advance to college-level education and training, even if they
are enrolled in a program offered at a community college.
For example, a study conducted in Washington State (Prince & Jenkins, 2005),
where ABE and ESL programs are delivered through community and technical colleges,
found that only 31 percent of a cohort of students who started in ABE earned at least one
college credit in five years. The comparable rate for students who started in ESL was
only 12 percent. This same study found that, compared with students who earned fewer
than 10 college credits, those who reached the “tipping point” of earning at least two
semesters of credits and a credential had a substantial average annual earnings advantage:
$7,000 for students who started in ESL, $8,500 for those who started in ABE or GED,
and $2,700 and $1,700 for those who entered with at most a GED or high school
diploma, respectively.
One reason that few students in adult basic skills programs advance successfully
to college-level coursework is that such programs are typically not well aligned with
college-level offerings. Adult basic skills students often do not have access to counselors
and other supports available to students in college programs. Moreover, many students in
4
adult basic skills programs face the challenge of having to balance school and family
while working one or more low-wage jobs.
The study reported on here, conducted by the Community College Research
Center (CCRC) at Teachers College, Columbia University, examined the outcomes of an
innovative approach to increasing the rate at which adult basic skills students enter and
succeed in postsecondary education and training. The model, known as Integrated Basic
Education and Skills Training, or I-BEST, was developed by the Washington State
community and technical colleges, based on the conviction that helping the state’s
growing number of poorly educated immigrants and other low-skill adults succeed in
postsecondary training connected to jobs of importance to local economies had become
an economic imperative (Bloomer, 2008; WSBCTC, 2005). In the I-BEST model, basic
skills instructors and college-level career-technical faculty jointly design and teach
college-level occupational (or what the Washington community and technical colleges
call “workforce”) courses for adult basic skills students.1 Under I-BEST, instruction in
basic skills is combined with instruction in college-level career-technical skills. Students
receive college credit for the workforce portion of the program (though not for the basic
skills instruction).
The program design was motivated by research suggesting that teaching basic
skills in the context of materials that are of interest to the student — sometimes called
“contextual instruction” — can improve learning of basic skills by adults (Resnick, 1987;
Shore, Shore, & Boggs, 2004; Sticht, 1997; Stone, Alfred, Pearson, Lewis, & Jensen,
2005; Weinbaum & Rogers, 1995). In I-BEST, basic skills instruction is typically
customized to the given workforce program. For example, for students enrolled in a
nursing program, there may be increased emphasis on learning medical terms in addition
to mastering everyday vocabulary used in all fields. If a student is having difficulty
understanding technical material because of problems with English, the basic skills
instructor is there to help. The theory is that student motivation and achievement will
increase because students are able to immediately experience the usefulness of their basic
skills education in the learning of technical skills and knowledge.
1 Both instructors of a course are required to be present in class for at least half of the total instructional time.
5
Integrating the teaching of basic skills and college-level career-technical skills, as
is done through the I-BEST model, challenges the conventional notion that basic skills
instruction should be completed prior to starting college-level courses. The approach thus
offers the potential to accelerate the rate at which basic skills students advance to college
programs. Preliminary analyses of I-BEST program outcomes by researchers at the
Washington State Board for Community and Technical Colleges (WSBCTC) found that
participating students were substantially more likely than non-participating adult basic
skills students to advance to college-level workforce programs and to reach the “tipping
point” of having earned at least one year of credits and a credential (WSBCTC, 2005,
2008).
Based on these promising early results, the WSBCTC approved increased funding
of programs using the I-BEST model. I-BEST courses receive 75 percent more funds per
full-time-equivalent student than do regular basic skills courses. With this enhanced
funding, the program model has expanded from pilots at 5 colleges in 2004-05 to
programs at all 34 community and technical colleges in the Washington State system.
Over 115 I-BEST programs are currently offered in such fields as nurse assistant, early
childhood education, and business technology (Bloomer, 2008). Table 1 shows the fields
in which I-BEST students enrolled in 2006-07. The WSBCTC requires that credits earned
in I-BEST programs, which are typically a single quarter term in length,2 should apply to
certificate or degree programs that are part of a “career pathway,” that is, programs that
clearly connect to further education and career-path employment in the given field.
2 The Washington community and technical colleges operate on a quarter system.
6
Table 1. I-BEST Enrollments by Program Description, 2006-073
Program Description Enrollment
Data Entry/Microcomputer Applications (General) 172 Early Childhood Education and Teaching 92 Nurse/Nursing Assistant/Aide and Patient Care Assistant 74Automobile/Automotive Mechanics Technology/Technician 73Welding Technology/Welder 71Medical/Clinical Assistant 71 Criminal Justice/Law Enforcement Administration 54 Home Health Aide/Home Attendant 42Nursing/Registered Nurse (RN, ASN, BSN, MSN) 20Medical Office Management/Administration 18Truck and Bus Driver/Commercial Vehicle Operation 17Medical Reception/Receptionist 17Occupational Safety and Health Technology/Technician 15Office Management and Supervision 15Accounting Technology/Technician and Bookkeeping 13Business/Office Automation/Technology/Data Entry 12Electrical, Electronic and Communications Engineering Technology/Technician 11
Medical Administrative/Executive Assistant and Medical Secretary 5
Graphic and Printing Equipment Operator, General Production 4
Manufacturing Technology/Technician 4Natural Resources Law Enforcement and Protective Services 4Forensic Science and Technology 2Executive Assistant/Executive Secretary 2Business Administration and Management, General 1
3 Due to limitations in the administrative data, 87 of the 896 I-BEST students enrolled in 2006-07 were lacking information on which vocational program they were enrolled in; this table thus reflects the program enrollments of 809 I-BEST participants.
7
The early analyses of I-BEST programs by the WSBCTC were descriptive in
nature, in that they did not control for student characteristics that could bear on outcomes.
In addition, those analyses did not consider that the way students are selected into
programs may influence the results. According to program administrators, I-BEST
students most often find out about I-BEST through word of mouth or by participating in a
non-I-BEST basic skills course (either ABE or ESL). Students are also referred by
persons affiliated with a given college, such as counselors, as well as by outside entities.
Organizations that have referred students to I-BEST include WorkSource, which is
Washington State’s system of public “one-stop” employment centers, WorkFirst,
Washington State’s program for connecting public aid recipients to jobs, and various
retraining programs for dislocated workers. Finally, students may also apply directly to I-
BEST programs, which are publicized through such means as college websites, catalogs,
and flyers.
Students are therefore selected into I-BEST in a non-random manner. WSBCTC
staff members have indicated that the program may be better suited to individuals with
higher basic skills proficiencies. The program may also attract students who are more
motivated than others with similar backgrounds and preparation for success in their
education or career. To accurately estimate the effect of I-BEST on student outcomes,
what is ideally needed is a comparison group of non-I-BEST students who are similar to
the students who enroll in I-BEST in all relevant dimensions other than their enrollment
in the program.
For this study, we chose as the main comparison group those basic skills students
who, on their own accord, also took at least one (non-I-BEST) college-level career-
technical course. We refer to this group as Non-IB Workforce students. This is a small
subset of all basic skills students in the Washington State community and technical
college system. Of the comparison groups that we have ready access to in terms of
available data, this is the one most closely comparable to the I-BEST students because
they, like the I-BEST students, were exposed to both basic skills instruction and
workforce classes during the period under study. We know, however, that student
selection into this comparison group operated differently than in the I-BEST group, most
obviously because the two groups differ on observed characteristics, and, secondly,
8
because they likely differ on characteristics that we do not observe. It is important to keep
in mind these caveats regarding sample selection when considering our results.
This study used multivariate analysis to compare the educational outcomes of I-
BEST students with those of other students with similar characteristics. Our analysis
addressed two main questions:
1) What are the socioeconomic, demographic, and enrollment
characteristics of I-BEST students compared with other basic
skills students?
2) How do the educational outcomes of I-BEST students compare
with those of other basic skills students and, in particular, with
those in the group mentioned above — students who take at least
one workforce course but who are not involved in I-BEST?
9
2. Data and Methods
The data used in this study were drawn from administrative data shared with the
Community College Research Center (CCRC) by the WSBCTC on both I-BEST and
non-I-BEST students who enrolled at any college in Washington State’s community and
technical college system at any time during the academic year 2006-07. We chose to
study students who enrolled in 2006-07 because WSBCTC staff indicated that this was
the first year that the program moved beyond the pilot phase and was in full operation.
We restricted our study to those students who took a non-credit adult basic skills course
(including, of course, the I-BEST students themselves) in that academic year. We did not
include the many students who were enrolled in programs designed to prepare for transfer
to baccalaureate programs because I-BEST programs exist only in occupational fields.
We also restricted our study to students in the 24 colleges that offered I-BEST in 2006-07
(the program was expanded to all 34 colleges the following year). We studied data on
31,078 students, of whom 896 were I-BEST students. Of the remainder, 28,826 were
students who enrolled only in basic skills courses, and 1,356 were basic skills students
who also enrolled in at least one (non-I-BEST) workforce course.
The dataset contains information on the socioeconomic and demographic
characteristics of each student in the sample, as well as transcript data, which we used to
determine the number of credits completed and credentials earned. The transcript data
enabled us to track students through the end of the academic year 2007-08 and back to
the earliest date each student enrolled in the system, making it possible to control for any
credits earned prior to 2006-07.
The study was designed to examine the effects of participation in I-BEST on the
following educational outcomes over two years:
Whether a student earned any college credits;
The total number of college credits earned;
The number of college vocational credits earned;
Whether the student persisted into the following academic year;
Whether the student earned a certificate or associate degree; and
Whether the student achieved gains on basic skills tests.
10
For each of these outcomes, we produced descriptive statistics comparing I-BEST
students with two groups: 1) all basic skills students not in I-BEST (“Non-I-BEST
students”) and 2) those basic skills students not in I-BEST who took at least one
workforce course during 2006-07 (“Non-IB Workforce students”). Note that we also
make reference to basic skills students not in I-BEST who did not enroll in a workforce
course (“Non-IB Non-Workforce students”).
We then performed regressions to compare: a) I-BEST students and Non-IB Non-
Workforce students, b) Non-IB Workforce students and Non-IB Non-Workforce
students, and c) I-BEST students and Non-IB Workforce students. We used linear
regression or logistic regression, depending on the outcome. In each case, we controlled
for student characteristics and enrollment patterns that might bear on the outcomes. For
the logistic outcomes, we measured differences in the probability of the given outcome
between each pair of groups. For the continuous outcomes, we measured the differences
in the outcome itself.
Finally, we compared the outcomes of I-BEST students with those of Non-I-
BEST basic skills students who were matched to the I-BEST students using propensity
score matching (PSM). See the appendix for a brief description of this method. We used
both regression analysis and PSM to see how similar the results from the two methods
would be and thus make as an informal test of the robustness of our findings, although
the two methods cannot be directly compared and draw on different groups of students.
For reasons described in the appendix, we give more credence to the estimates of
treatment effects produced by PSM than to the results of the regressions. Neither
regression analysis nor PSM allows us to correct for selection bias that might be caused
by characteristics we do not observe or measure, however. This remains a limitation of
this study.
We consider the treatment in this study to be enrollment in I-BEST, rather than
completion of an I-BEST program, because we want to view any program attrition effects
as part of the program itself; that is, we want in our estimates of program effects to
account for how successful I-BEST was at retaining students. Nevertheless, we have been
informed by WSBCTC staff that I-BEST programs have high retention and completion
rates.
11
In all of the tables presented here, we report only the main effects and omit the
effects for the controls. We used the same set of controls for all of the regressions and in
our propensity score models. The means for all of the control variables are listed in Table
4, which is discussed in the next section.
3. Findings
We start by giving descriptive statistics on the I-BEST students, the Non-IB Non-
Workforce students, and the Non-IB Workforce students in our sample. We then present
results of the multivariate analyses for each outcome.
3.1 Descriptive characteristics and outcomes
Overall, 896 I-BEST students were enrolled at 24 community or technical
colleges in Washington State in academic year 2006-07 (see Table 2). In this study, all
Non-I-BEST students as well as all those who did enroll in I-BEST programs were, by
definition, enrolled in basic skills coursework. Of the 30,182 Non-I-BEST students in the
sample, 1,356 also took a workforce course. Thus, like the I-BEST students, the latter
enrolled in both basic skills and workforce coursework in 2006-07. However, unlike the
I-BEST students, they did not necessarily take the coursework concurrently, and they did
not take it as part of an integrated program designed to accelerate the transition from
basic skills to college-level workforce programs. These Non-IB Workforce students
comprise the group that we believe is most comparable to the I-BEST group.
Table 2. Distribution of Basic Skills Students, 2006-07
Program Type Enrollment
I-BEST 896
Non-IB Non-Workforce 28,826
Non-IB Workforce 1,356
Total 31,078
12
Table 3 shows that I-BEST students were much more likely than Non-I-BEST
students to advance to college-level coursework and to earn many more college and
vocational credits. This result is descriptive in nature and does not control for differences
among students in these groups. I-BEST students were also more likely than Non-IB
Workforce students to advance and to earn more college and vocational credits. Again,
this result is descriptive.
As Table 3 indicates, over the course of the two-year observation period, I-BEST
students completed slightly more than an academic year’s worth of college coursework,
on average, while Non-I-BEST students earned very few credits. Of these students, the
Non-IB Workforce subset accumulated many more credits, on average, than the rest of
the Non-I-BEST students, but not as many as the I-BEST students. Table 3 also shows
that 54 percent of I-BEST students earned a certificate or degree, as opposed to less than
one percent of all Non-I-BEST students and 18 percent of Non-IB Workforce students.
Virtually all awards earned by anyone in these groups were occupational certificates, not
degrees.
Table 3. Educational Outcomes of Basic Skills Students over Two Academic Years, 2006-07 and 2007-08
Student type
Earned any
college credit
Meannumber
of college credits
Earned any
vocational credit
Meannumber of vocational
creditsEarned a
certificate
Earned an
associatedegree
Earned a certificate
or associate
degreeI-BEST 90.0% 48.7 87.8% 41.5 54.1% 0.2% 54.2% All Non-I-BEST 7.0% 2.3 5.3% 1.4 0.8% 0.0% 0.8% Non-IB Workforce 64.2% 35.7 59.6% 24.8 17.8% 0.2% 18.0%
Table 4 lists the background characteristics that were used as control variables in
the multivariate models. In some ways, I-BEST students appear to be relatively similar to
Non-I-BEST students. There are, however, differences worth noting. All basic skills
students, whether or not they enrolled in I-BEST, were enrolled in either ESL or ABE,
with the latter possibly including a GED component. Table 4 shows that ABE/GED
enrollment was dominant among I-BEST students. More than two thirds of I-BEST
students were enrolled in ABE (or GED) instruction, compared to 36 percent of Non-I-
BEST students. These proportions are reversed when comparing students in the groups
13
who enrolled in ESL (31 percent of I-BEST versus 64 percent of Non-I-BEST students,
respectively, were enrolled in ESL). Other differences of note are the percentage of
students receiving financial aid and the percentage enrolled full time. In both cases, I-
BEST students held an advantage in that they were more likely to receive aid and enroll
full time. In terms of race/ethnicity, I-BEST students were much less likely than Non-I-
BEST students to be Hispanic and much more likely to be Black.
There are also noteworthy similarities and differences between I-BEST students
and the Non-IB Workforce student subset. Both the I-BEST and the Non-IB Workforce
students were mainly ABE/GED students, as opposed to the Non-I-BEST students as a
whole, who were predominantly ESL students. Non-IB Workforce students were also
more likely than others to indicate upon entry that they intended to earn an academic
credential or transfer to a four-year institution. Twenty percent of Non-IB Workforce
students indicated this, compared to 7 percent of I-BEST students and 9 percent of all
Non-I-BEST students.
14
Table 4. Characteristics of Basic Skills Students, 2006-07
I-BEST All
Non-I-BEST Non-IB
Workforce Number of students in program 896 30,182 1,356
Program classification I-BEST student 100% 0.0% 0.0% ABE/GED student 69.0% 36.0% 66.4%ESL student 30.9% 63.8% 33.3% Non-IB Workforce student 0.0% 4.5% 100.0%
Social and economic characteristics Mean age 32.5 32.3 31.9Female 64.8% 60.5% 69.2% Hispanic 18.4% 38.3% 21.3% Black, non-Hispanic 12.1% 6.9% 6.1% Asian/Pacific Islander 12.3% 15.0% 12.4% Single w/ dependent 22.2% 14.0% 22.8% Married w/ dependent 27.8% 26.5% 24.1% Disabled 7.1% 3.8% 11.0% Estimated socioeconomic
4status
aracteristics 5 7 2
mic
1irst enrolled in 4th quarter 10.4% 16.9% 4.2%
aracteristics 1
vocational credits
aduate 1 21
Bachelor’s degree 4.0% 4.6% 5.1%
3.6 3.5 3.5
Current schooling chIntent is vocational 2.4% 2.7% 48.4% Intent is acade 7.4% 9.1% 20.0% Received aid 25.9% 2.1% 14.2%Enrolled full time 67.1% 32.6% 49.0% First enrolled in 1st quarter 30.1% 27.5% 40.0% First enrolled in 2nd quarter 41.0% 33.1% 40.2% First enrolled in 3rd quarter 18.5% 22.5% 5.6% F
Previous schooling chMean college credits 3.9 0.9 8.8 Mean 9.1 0.6 5.8 GED 12.7% 4.0% 10.0% High school gr 27.3% 6.9% 5.7% Some college 0.4% 4.1% 7.5% Certificate 3.7% 1.7% 3.4% Associate degree 2.5% 1.8% 2.2%
4 This is based on the quintile of the average socioeconomic status of the Census block group in which the student’s residence is found. 1 is the highest quintile, and 5 is the lowest. For details, see Crosta, Leinbach, Jenkins, Prince, and Whitakker (2006) and WSBCTC (2006). 5 Vocational and academic intent indicate the type of college program the student means to pursue. If vocational, the student intends to pursue workforce training; if academic, the student intends to pursue a program that leads to a degree and/or transfer to a four-year institution. Students do not always follow their stated intent (see Bailey, Jenkins, & Leinbach, 2006).
15
3.2 Estimates of the probability of earning college credit
f earning 6
any
nts’
e
tarted in ESL.
Both gr
Non-IB
timated
d only 3 percent for Non-IB-
on-W
in the
dit
0 percent; it was 67 percent for
the com
ons with
those of the PSM analysis because each takes a different approach to selecting
Table 5 shows regression estimates of the differences in the probability o
college credit (including college vocational credit ) relative to the Non-IB Non-
Workforce baseline group — those basic skills students who took neither I-BEST nor
other workforce course. As shown in Table 5, even after controlling for demographic
characteristics, enrollment intent and intensity, and previous schooling, I-BEST stude
probability of earning college credit was 81 percentage points higher than that of th
Non-IB Non-Workforce students. There are no significant differences between the
estimates for I-BEST students who started in ABE/GED and those who s
oups appear to have benefited similarly by enrolling in I-BEST.
Table 5 also shows that Non-IB Workforce students (basic skills students who
took at least one workforce course but did not participate in I-BEST) also did better than
Non-IB Non-Workforce students — the former had a probability of earning college credit
that was 47 percentage points higher than the latter. However, the probability that
Workforce students earned college credit was still not as high as that for I-BEST
students. Using the regression results and holding the values of all explanatory variables
other than those corresponding to the three groups of interest at their means, we es
that the probability of earning college credit was 84 percent for I-BEST students,
compared to 50 percent for Non-IB Workforce students an
N orkforce students (results not shown in the table).
Table 6 shows the propensity score matching estimates of the differences
probability of earning college credit between the I-BEST students and both the
unmatched basic skills population and the matched comparison group. By this PSM
method, we estimate that the average difference in probability of earning college cre
between I-BEST students and students in the comparison group was 23 percentage
points. The mean probability for I-BEST students was 9
parison group (results not shown in the table).
As mentioned, we cannot statistically compare the results of the regressi
6 Throughout this paper, the findings we report concerning college credits always include college vocational credits.
16
appropriate comparison groups. However, the fact that these two different methods yield
effect size estimates that are similar in magnitude increases our confidence in the results.
As noted in the appendix, we believe that PSM may give a more accurate estimate of the
program’s effect on a given outcome.
Table 5. Logistic Regression Estimates of Differences in the Probability of Earning College Credit Relative to Non-IB Non-Workforce Students, 2006-08
Overall ABE/GED ESL
0.81*** 0.78*** 0.83*** I-BEST Students (0.02) (0.03) (0.04)
0.47*** 0.54*** 0.40** Non-IB Workforce Students (0.02) (0.02) (0.04)
Pseudo R^2 0.440 0.431 0.427
Observations 27,426 10,058 17,288
Note: * p<0.10, ** p<0.05, *** p<0.01
Effect shown for discrete change of variable from 0 to 1.
Table 6. PSM Estimates of Differences in the Probability of Earning College Credit Relative to Unmatched and Matched Non-I-BEST Students, 2006-08
Unmatched
Average treatment effect on the treated
(ATT)
0.83*** 0.23*** I-BEST Students (0.01) (0.03)
Note: * p<0.10, ** p<0.05, *** p<0.01
17
3.3 Estimates of the number of credits earned
Table 7 reports the results from estimating differences in the number of college
and college vocational credits earned by I-BEST and Non-IB Workforce students
compared to Non-IB Non-Workforce students. Once again using regressions to control
for the factors indicated in Table 4, the left side of Table 7 shows that I-BEST students
earned, on average, an estimated 44 more college credits — equivalent to approximately
one full academic year7 — than Non-IB Non-Workforce students, and 14 more college
credits than Non-IB Workforce students. Using the regression results, we estimated that
I-BEST students earned an average of 45 college credits compared to 31 for the Non-IB
Workforce students and 1 credit for the Non-IB Non-Workforce group (results not shown
in table).
ABE/GED students in I-BEST earned 50 more college credits, and ESL students
in I-BEST earned 35 more college credits than Non-IB Non-Workforce students. These
estimates are 19 and 8 credits more than those earned by Non-IB Workforce students who
were enrolled in ABE/GED and ESL, respectively.
The three columns on the right side of Table 7 show the results for college
vocational credits. Even after controlling for demographic characteristics and other
factors, I-BEST students — overall and by ABE/GED and ESL subgroup — earned more
vocational credits than Non-IB Non-Workforce students and Non-IB Workforce students.
I-BEST students earned, on average, an estimated 40 more college vocational credits than
Non-IB Non-Workforce students and 18 more than Non-IB Workforce students. We
estimated that, on average, I-BEST students earned 40 vocational credits, Non-IB
Workforce students earned 22 and Non-IB Non-Workforce students earned less than one.
ABE/GED I-BEST students earned 45 more college vocational credits than Non-
IB Non-Workforce students and 21 more than the Non-IB Workforce group. ESL I-BEST
students earned 31 more college vocational credits than Non-IB Non-Workforce students
and 14 more than Non-IB Workforce students.
Table 8 shows the PSM estimate of the difference in the number of credits earned
by I-BEST students compared with Non-I-BEST students to whom they were matched
based on similar background characteristics and prior enrollment patterns. The mean
7 The Washington State community and technical colleges operate on a quarter system.
18
number of credits earned by I-BEST students was 52, compared to an average of 34 for
the matched comparison group — a difference of 18 credits. I-BEST students earned an
average of 45 vocational credits, while the matched comparison group earned an average
of 24 vocational credits, a difference of 21 vocational credits. (Only results on the
differences in credits earned are shown in the table.) Though not directly comparable, the
regression and PSM estimates are of similar magnitude, indicating that the results are
robust.
Table 7. OLS Regression Estimates of Differences in the Number of Total College and College Vocational Credits Earned Relative to Non-IB Non-Workforce Students, 2006-08
College credits Vocational credits
Overall ABE/GED ESL Overall ABE/GED ESL
44.35*** 49.74*** 34.81*** 39.64*** 44.58*** 30.97*** I-BEST Students (1.62) (2.23) (2.15) (1.48) (2.03) (1.95)
30.36*** 31.11*** 26.96*** 21.52*** 23.06*** 17.31*** Non-IB Workforce Students (1.28) (1.57) (2.16) (1.04) (1.32) (1.58)
R-squared 0.434 0.455 0.385 0.411 0.427 0.375
Observations 27,426 10,058 17,297 27,426 10,058 17,297
Note: * p<0.10, ** p<0.05, *** p<0.01
Table 8. PSM Estimates of Differences in the Number of Total College and College Vocational Credits Earned Relative to Matched and Unmatched Non-I-BEST Students, 2006-08
College credits Vocational credits
Unmatched
Average treatment effect on the treated
(ATT) Unmatched
Average treatment effect on the treated
(ATT)
49.60*** 18.48*** 43.18*** 21.39*** I-BEST Students (0.56) (2.71) (0.45) (2.26)
Note: * p<0.10, ** p<0.05, *** p<0.01
19
3.4 Estimates of the probability of persisting into 2007-08
We measured persistence into the second academic year, 2007-08, by examining
whether a student had any transcript record in that year. By this definition, in order to
have persisted, students must have completed, though not necessarily passed, a course in
that year. We also considered students as having persisted if they earned an award in
2006-07, even if they did not persist into 2007-08, because these students experienced a
successful outcome. The results of the logistic regressions and PSM models for this
outcome are shown in Tables 9 and 10.
Using a logistic regression model, we estimated that, on average, I-BEST students
had a probability of persisting that was 42 percentage points higher than Non-IB Non-
Workforce students. Among those enrolled in ABE/GED in both these groups, I-BEST
students had a probability that was 47 percentage points higher. The corresponding
difference in chances for ESL students was 41 percentage points. A regression that
compared I-BEST students with Non-IB Workforce students found that the former had a
probability of persisting that was 13 percentage points higher, with an error of 2
percentage points.
Comparing Non-IB Workforce students to Non-IB Non-Workforce students, we
found that the former had a probability of persisting that was 30 percentage points higher
the latter. The corresponding probability differences for the ABE/GED and ESL
subgroups of each of these groups were 35 and 26 percentage points, respectively.
Using the regression results and holding the value of all variables other than the
dummy variables corresponding to the three groups of interest at their means, we
estimated that I-BEST students had an 80 percent probability of persisting into the second
year (or completing a credential), compared to 68 percent for the Non-IB Workforce
students and 38 percent for the Non-IB Non-Workforce group (results not shown in the
table)
As shown in Table 10, our PSM model of persistence found that I-BEST students
had a probability of persisting that was 17 percentage points higher than matched
students. The I-BEST students had a 78 percent probability of persisting, compared to 61
percent for the matched students (not shown in the table). Here again, the results of the
PSM model are similar to those of the regressions.
20
Table 9. Logistic Regression Estimates of Differences in the Probability of Persisting into 2007-08 Relative to Non-IB Non-Workforce Students
Overall ABE/GED ESL
0.42*** 0.47*** 0.41*** I-BEST Students (0.02) (0.02) (0.03)
0.30*** 0.35*** 0.26*** Non-IB Workforce Students (0.02) (0.02) (0.03)
Pseudo R^2 0.058 0.096 0.046
Observations 27,426 10,058 17,297
Note: * p<0.10, ** p<0.05, *** p<0.01
Effect shown for discrete change of variable from 0 to 1.
Table 10. PSM Estimates of Differences in the Probability of Persisting into 2007-08 Relative to Matched and Unmatched Non-I-BEST Students
Unmatched
Average Treatment Effect on the Treated
(ATT)
0.39*** 0.17 I-BEST Students (0.02) (0.05)
Note: * p<0.10, ** p<0.05, *** p<0.01
21
3.5 Estimates of the probability of earning an award
Table 11 shows the results of the regression estimates of the differences in the
probability of earning an award by I-BEST and Non-IB Workforce students relative to
the baseline group, Non-IB Non-Workforce students. Awards may have been earned at
any time within the two academic years of 2006-07 and 2007-08, and include
occupational certificates and associate degrees granted by the system. As was shown in
Table 3, however, virtually all of the awards earned by the students under study here
were certificates.
Controlling for student characteristics and prior enrollment patterns, I-BEST
students had a probability of earning an award that was 51 percentage points higher than
that of Non-IB Non-Workforce students. Non-IB Workforce students had a probability of
doing so that was 16 percentage points higher than Non-IB Non-Workforce students.
ABE/GED I-BEST students had a probability of earning an award that was 42 percentage
points higher than ABE/GED Non-IB Non-Workforce students. For the ABE/GED Non-
IB Workforce group, the corresponding difference is 13 percentage points. For I-BEST
and Non-IB Workforce students enrolled in ESL, the respective differences are 57
percentage points and 10 percentage points. An additional regression model, not shown in
the tables, found that I-BEST students had a probability of earning an award that was 35
percentage points higher than that of Non-IB Workforce students, with an error of 4
percentage points.
Based on the regression results, we estimated that I-BEST students had a 51
percent probability of earning an award, compared to 16 percent for the Non-IB
Workforce students and effectively zero percent for the Non-IB Non-Workforce group
(results not shown in the table).
Table 12 shows the PSM model estimate of the increased probability of earning
an award by I-BEST students compared to matched Non-I-BEST students. Based on this
model, we found that I-BEST students had a 55 percent probability of earning an award,
compared to only 15 percent for the matched group (these results are not shown in the
table) — a 40 percentage point difference. The fact that the PSM estimates are similar to
those from the regression analysis is reassuring. As mentioned, there are reasons to
believe that the PSM estimates are more accurate than those of the regression.
22
Table 11. Logistic Regression Estimates of Differences in the Probability of Earning an Award Relative to Non-IB Non-Workforce Students, 2006-08
Overall ABE/GED ESL
0.51*** 0.42*** 0.57*** I-BEST Students (0.03) (0.04) (0.06)
0.16*** 0.13*** 0.10*** Non-IB Workforce Students (0.02) (0.02) (0.03)
Pseudo R^2 0.672 0.621 0.759
Observations 25,473 9,541 14,535
Note: * p<0.10, ** p<0.05, *** p<0.01
Effect shown for discrete change of variable from 0 to 1.
Table 12. PSM Estimates of Differences in the Probability of Earning an Award Relative to Matched and Unmatched Non-I-BEST Students, 2006-08
Unmatched
Average treatment effect on the treated
(ATT)
0.55*** 0.40*** I-BEST Students (0.00) (0.02)
Note: * p<0.10, ** p<0.05, *** p<0.01
23
3.6 Estimates of the probability of achieving point gains on basic skills tests
Table 13 shows the results of estimates of the increased probability that I-BEST
and Non-IB Workforce students made any point gains in basic skills testing compared to
Non-IB Non-Workforce students. To make point gains, students needed to show a gain
on any of the Comprehensive Adult Student Assessment Systems (CASAS)8 tests,
whether in reading, listening, or math.
The results of logistic regressions indicate that, on average, I-BEST students had a
probability of making CASAS point gains that was 18 percentage points higher than
Non-IB Non-Workforce students. Non-IB Workforce students had a likelihood of making
such gains that was 5 percentage points higher than the Non-IB Non-Workforce group.
ABE/GED I-BEST students had, on average, a probability that was 21 percentage points
higher than ABE/GED Non-IB Non-Workforce students. For the ABE/GED Non-IB
Workforce group, the corresponding difference was 9 percentage points. For I-BEST and
Non-IB Workforce students enrolled in ESL, the respective differences were 20
percentage points and 6 percentage points. Regression analysis (not shown in the table)
indicates that I-BEST students had a probability that was 13 percentage points higher
than Non-IB Workforce students, with an error of 2 percentage points on the estimate.
Based on the regression results, we estimated that the probability of achieving a
CASAS test score gain was 60 percent for I-BEST students, compared with 47 percent
for Non-IB Workforce students and 43 percent for Non-IB Non-Workforce students
(estimates not shown in the tables).
Table 14 shows the PSM model estimates. The I-BEST students had a 62 percent
likelihood of achieving a basic skills point gain, compared to a 45 percent probability for
the matched Non-I-BEST students (these are not shown in the table), a difference of 17
percentage points. Once again, the similarity between the PSM and regression estimates
increases the robustness of the findings.
8 See http://www.casas.org for more information on these tests.
24
Table 13. Logistic Regression Estimates of Differences in the Probability of Achieving Basic Skills Point Gain Relative to Non-IB Non-Workforce Students, 2006-08
Overall ABE/GED ESL
0.18*** 0.21*** 0.20*** I-BEST Students (0.02) (0.03) (0.03)
0.05** 0.09*** 0.06* Non-IB Workforce Students (0.02) (0.02) (0.03)
Pseudo R^2 0.047 0.043 0.052
Observations 27,398 10,050 17,297
Note: * p<0.10, ** p<0.05, *** p<0.01 Effect shown for discrete change of variable from 0 to 1.
Table 14. PSM Estimates of Differences in the Probability of Achieving a Basic Skills Point Gain Relative to Matched and Unmatched Non-I-BEST Students, 2006-08
Unmatched
Average treatment effect on the treated
(ATT)
0.18*** 0.17*** I-BEST Students (0.02) (0.04)
Note: * p<0.10, ** p<0.05, *** p<0.01
25
4. Conclusion
Our findings indicate that students participating in I-BEST achieved better
educational outcomes than did other basic skills students who did not participate in the
program. I-BEST students were more likely to continue into credit-bearing coursework
and to earn credits that count toward a college credential. They were more likely to
persist into the second year, to earn educational awards, and to show point gains in basic
skills testing. On all outcomes, I-BEST students did moderately or substantially better
than basic skills students who did not enroll in any workforce course. Moreover, I-BEST
students had better outcomes than those basic skills students who enrolled in at least one
workforce course in the same academic year. While the I-BEST group’s comparative
advantage relative to this latter group was not as large, it was still significant. The
apparent educational benefits were reaped by I-BEST students who started in either
ABE/GED or ESL.
These results are robust with respect to two methodologies: regression (linear or
logistic) and propensity score matching (PSM). Both methodologies control for observed
differences between the treated (I-BEST) and comparison groups, but neither can control
for selection bias that may be due to unobserved differences between the groups. Some of
these unobserved differences are likely to be related to the selection process, which we
only partially understand. Thus, while the results show that participation in I-BEST is
correlated with better educational outcomes over the two-year tracking period, it is
important to note that they do not provide definitive evidence that the I-BEST program
caused the superior outcomes. It could be that, because of the way students are selected
into the program, those who participate have higher motivation or other characteristics
not measured in this study that make them more likely to succeed. CCRC plans to
conduct further research to better understand the process by which students are selected
into I-BEST. We will explore the feasibility of using quasi-experimental methods to
remedy possible selection bias. The strong positive nature of our findings suggests that an
experimental test of I-BEST, in which students are randomly assigned to a treatment or a
control group, might be warranted.
26
As we are able to follow I-BEST students over time and collect more information
about them, we plan to study their degree attainment and labor force outcomes, such as
employment rates and earnings. CCRC also plans to extend the study to those students
who enrolled in an I-BEST program in academic year 2007-08, when the program was
expanded to include all the institutions in the Washington State community and technical
college system. We will use data on these students to identify I-BEST programs that have
superior educational and labor market outcomes, controlling for student characteristics,
and will conduct field research to identify the practices of effective programs. Finally, we
will also collect data on program finances to estimate the cost-benefit of the I-BEST
approach and thus help to assess the feasibility of offering it on a wide scale.
27
Appendix: A Brief Description of Propensity Score Matching
Propensity score matching (PSM) matches “treated” subjects — in this case,
students served by I-BEST programs — to selected untreated “control” subjects — in this
case, basic skills students who did not enroll in I-BEST — who have similar background
characteristics (Rosenbaum & Rubin, 1983; Winship & Morgan, 1999). PSM conducts
comparisons between similar pairs of students who differ on whether or not they received
the treatment, but have similar other observed characteristics.
PSM first estimates the “propensity score,” which is an assessment of the
propensity to be treated. It does this by performing a logit or probit regression of the
treatment dummy variable on all available covariates that, in the researcher’s judgment,
have the potential to influence the chances of being treated. Treated and untreated
observations are then matched based on having similar propensity scores, and then the
average treatment effect on the treated (ATT) can be estimated, which is the average
difference on an outcome of interest between the matched treated and untreated
observations.9 The ATT is the average effect of the treatment on the sort of person who
participates in the program. The effectiveness of PSM is, in part, a function of having
enough relevant information about the cases to accurately estimate the propensity score,
and thus accurately estimate the ATT using the matching process that uses this score.
The matching process selects from those observations for which there is
“common support,” that is, whose distribution of propensity scores are deemed by the
algorithm to be sufficiently close to the propensity scores of the treated observations. The
fact that PSM draws its comparison group from the observations that give common
support, rather than all observations as is typically done when regression is employed, is
one reason why PSM estimates may be more accurate.
In addition, unlike regression, PSM does not assume a particular functional
relationship between an outcome of interest and the available relevant covariates,
including treatment status. In contrast, if we estimated a linear regression model of an
outcome, such as college credits earned, on a treatment status dummy variable (here I-
9 There are many variants of PSM, many of which match each treated observation to a weighted set of matched untreated observations, rather than a single observation. Herein, we have used probit to estimate the propensity score and a local linear regression estimator, which is one method of conducting such a match (Todd, 1999).
28
BEST participation) and a number of controls, such as demographics, etc., we would
obtain an estimate of a fixed effect of treatment across all of the cases (assuming that we
did not interact the treatment status dummy variable with any other covariates). PSM
does not do this; the treatment effect varies with each matched pair of treated and
untreated cases, and is the difference in the outcome between the two cases.
29
References
Bailey, T. R., Jenkins, D., Leinbach, D. T. (2006, September). Is student success labeled institutional failure? Student goals and graduation rates in the accountability debate at community colleges. CCRC Working Paper No.1. New York: Columbia University, Teachers College, Community College Research Center. Available at http://ccrc.tc.columbia.edu/Publication.asp?uid=342
Barton, P. (2000). What jobs require: Literacy, education, and training, 1940–2006.Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/PICJOBS.pdf
Baum, S., & Ma, J. (2007). Education pays 2007: The benefits of higher education for individuals and society. New York: College Entrance Examination Board. Retrieved from http://www.collegeboard.com/prod_downloads/about/news_info/trends/ed_pays_2007.pdf
Bloomer, T. (2008, October). Education + work skills = jobs. Literacy Update. New York: Literacy Assistance Center. Retrieved from http://www.lacnyc.org/resources/publications/update/Update2008-9/Oct08Rev.pdf
Crosta, P., Leinbach, T., Jenkins, D., Prince, D., & Whitakker, D. (2006, July). Usingcensus data to classify community college students by socioeconomic status and community characteristics. CCRC Research Tools No. 1. New York: Columbia University, Teachers College, Community College Research Center. Available at http://ccrc.tc.columbia.edu/Publication.asp?uid=430
Osterman, P. (2008, August). College for all? The labor market for college-educated workers. Washington, DC: Center for American Progress.
Prince, D., & Jenkins, D. (2005). Building pathways to success for low-skill adult students: Lessons for community college policy and practice from a statewide longitudinal tracking study. CCRC Research Brief No. 25. New York: Columbia University, Teachers College, Community College Research Center. Available at http://ccrc.tc.columbia.edu/Publication.asp?UID=288
Resnick, L. B. (1987). The 1987 presidential address: Learning in school and out. Educational Researcher, 16(9), 13-20, 54.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55.
Shore, M., Shore, J., & Boggs, S. (2004). Allied health applications integrated into developmental mathematics using problem based learning. Mathematics and Computer Education, 38(2), 183-189.
30
31
Sticht, T. G. (1997). The theory behind content-based instruction. Focus on Basics, 1(D),6-10. Retrieved from http://www.iu13.org/ipq/content_based.pdf
Stone, J. R., Alfred, C., Pearson, D., Lewis, M. V., & Jensen, S. (2005). Buildingacademic skills in context: Testing the value of enhanced math learning in CTE, pilot study. Columbus, OH: National Research Center for Career and Technical Education. Retrieved from ERIC database.
Todd, P. E. (1999). A practical guide to implementing matching estimators. Philadelphia: University of Pennsylvania Department of Economics. Retrieved from http://athena.sas.upenn.edu/~petra/papers/prac.pdf
U.S. Census Bureau. (2007a). Educational attainment (American Fact Finder, Table S1501). Retrieved from U.S. Census Bureau: http://tinyurl.com/9bao99
U.S. Census Bureau. (2007b). Language spoken at home (American Fact Finder, Table S1601). Retrieved from U.S. Census Bureau: http://tinyurl.com/cas8nh
U.S. Department of Education. (2006). State-administered adult education program: Program year 2004-2005 enrollment. Washington, DC: Office of Vocational and Adult Education, Division of Adult Education and Literacy. Retrieved from http://www.ed.gov/about/offices/list/ovae/pi/AdultEd/aedatatables.html
WSBCTC [Washington State Board for Community and Technical Colleges]. (2005). I-BEST: A program integrating adult basic education and workforce training.Olympia, WA: Author. Retrieved from http://www.sbctc.ctc.edu/college/d_basicskills.aspx
WSBCTC. (2006). The socioeconomic well-being of Washington State: Who attends community and technical college. Olympia, WA: Author. Retrieved from http://www.sbctc.ctc.edu/docs/data/research_reports/resh_06-4_socioeconstudy.pdf
WSBCTC. (2008). Increasing student achievement for basic skills students (Research Report No. 08-1). Olympia, WA: Author. Retrieved from http://www.sbctc.ctc.edu/college/education/resh_rpt_08_1_student_achieve_basic_skills.pdf
Weinbaum, A., & Rogers, A. M. (1995). Contextual learning: A critical aspect of school-to-work transition programs. Washington, DC: National Institute for Work and Learning. Retrieved from http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/13/d5/a1.pdf
Winship, C., & Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25, 659-707.
