Using Community and Technical College Student Achievement Data for State-Level Accountability Purposes in Washington StatePNAIRP 2010 Conference

Deb StephensWashington State Board for Community and Technical Colleges

• Government Management Accountability and Performance

• Governor implemented in 2005 – first in nation to apply model to state government performance

• Data-based management model to improve the results of statewide programs and services

• Requires state agencies to develop and use performance data to set targets and inform decisions

What is GMAP?

• Governor can hold state government accountable

• Public forums are held quarterly and are organized around priorities, including education

• GMAP supports:– Transparency to the Public (Annual Report to

Citizens)– Evaluations of Program Effectiveness– Decisions about Operational Priorities and

Resource Allocation

What is GMAP? (cont’d)

• Performance funding system for WA’s community and technical colleges– Improves public accountability– Rewards colleges for increasing student

achievement

• Rewards are based on key academic benchmarks that improve students’ chances of college completion

• Shift from funding only inputs (enrollments) to also funding outcomes (student progress)

Washington’s Student Achievement Initiatives

• 4 Categories of Achievement measures– Building toward college level skills (basic skills, pre-

college)– First year retention (first 15 and 30 college-level

credits)– Completing college math– Completions (degrees, certificates, apprenticeship

training)

• Measures are meaningful for all students– Across demographics (race, age, income, employment)– Across missions (academic, basic skills, workforce,

remedial)– Regardless of enrollment patterns (full or part time)– Across types of institutions (rural, urban, community or

technical)

Student Achievement Initiative (cont’d)

• 2006-07: first Baseline • 2007-08: Learning year• 2008-09: first Performance year with financial

rewards to colleges (Rewards distributed Oct 2009)

• Evaluation is underway by the Community College Research Center (CCRC) at Columbia University and the Institute for Higher Education, Leadership and Policy at California State University

Student Achievement Initiative (cont’d)

• Measures are simple, understandable, and relevant to citizens, colleges, students, and policy makers

• Measures focus on shorter-term, intermediate outcomes that provide meaningful momentum towards college completion

• Measures can be influenced by colleges on a timely basis

• Student achievement successes at individual colleges provide evidence to support statewide application of effective strategies

Why Student Achievement in GMAP?

Student Achievement in a Logic Model Context

• All GMAP Student Achievement slides (as the Governor sees them) provided at the end of the presentation

• The slides are very data-rich….Pre-College slide shown as an example

• Information in slides allows Governor and Staff to understand trends, issues and what the college system is doing to improve performance

GMAP Student Achievement Presentation Slides

1. Graph shows total points for all students in each point category over 4 years shown on a graph (shows scale)

2. Chart shows change in relevant achievement points from baseline to first performance year for a significant subset of students (shows efficiency)

3. Summary Analysis describes trends and provides insight into strengths and challenges

4. Action Plan maps system activities and research surrounding key issues

GMAP Student Achievement Slide Elements

Are underprepared college students succeeding in pre-college and college-level work?

Pre-College Math Students’ Math Achievement Growth Over Baseline

Measure: Number of pre-college math and English courses students pass with a qualifying grade to advance toward college-level workSource: SBCTC Data WarehouseLink to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf

Summary Analysis• Nearly half (48%) of high school graduates who come to

community and technical colleges directly from high school need math remediation.

• Colleges are working hard to get students through pre-college math quicker and ensure they successfully complete college-level math. There is much work to be done to increase math success.

• The graph to the left shows the number of pre-college gains completed by all students, and the table shows math achievement growth by those students who attempted pre-college math.

• Gains in pre-college math are keeping pace with the increase in the number of students (14% growth in each over the baseline); however, students are still behind in moving to college-level math.

• Through the Transition Math Project, Achieving the Dream and Gates funded work, colleges are evaluating and piloting ways to move more students into college math faster.

Action Plan• Provide financial rewards to colleges each year for increasing

the number of pre-college courses completed with a qualifying grade.

• The new Re-Thinking Pre-College Math Initiative, funded by Gates Foundation grant, will support 7 colleges to improve curriculum, instructional practices and assessment approaches in pre-college math programs.

• Through Gates Foundation funding, expand I-BEST to 10 pilot colleges to integrate below college-level courses with college-level courses.

GMAP Slide Elements: The Main Graph

• Main graph shows trend in all college students who completed pre-college (both English and Math) courses over 4 years

• Chart shows achievement gains of students who attempted pre-college math from baseline to first performance year

• Governor can see that pre-college math is keeping pace with increases in students but only half are making it through college-level math

GMAP Slide Elements: The Chart

Pre-College Math Students’ Math Achievement Growth Over Baseline

GMAP Slide Elements: Summary Analysis

• Nearly half (48%) of high school graduates who come to community and technical colleges directly from high school need math remediation.

• Colleges are working hard to get students through pre-college math quicker and ensure they successfully complete college-level math. There is much work to be done to increase math success.

• The graph to the left shows the number of pre-college gains completed by all students, and the table shows math achievement growth by those students who attempted pre-college math.

• Gains in pre-college math are keeping pace with the increase in the number of students (14% growth in each over the baseline); however, students are still behind in moving to college-level math.

• Through the Transition Math Project, Achieving the Dream and Gates funded work, colleges are evaluating and piloting ways to move more students into college math faster.

• Provide financial rewards to colleges each year for increasing the number of pre-college courses completed with a qualifying grade.

• The new Re-Thinking Pre-College Math Initiative, funded by Gates Foundation grant, will support 7 colleges to improve curriculum, instructional practices and assessment approaches in pre-college math programs.

• Through Gates Foundation funding, expand I-BEST to 10 pilot colleges to integrate below college-level courses with college-level courses.

GMAP Slide Elements: Action Plan

• Total Student Achievement Points increased by 19% in 2008-09 over the baseline year.

• Basic skills points increased 36% over the baseline, improving achievement in basic skills and beyond.

• The number of basic skills students reaching the tipping point nearly quadrupled. Three quarters of students’ success in reaching the tipping point can be attributed to I-BEST programs.

Highlights of Other Student Achievement Slides

• Students starting the year with some (but less than 30) college credits grew primarily in the tipping point milestone. Workforce students generated most of this growth through short certificate growth in students with a year of college.

• While the percent of transfer students completing college math in 2008-09 over the baseline just kept pace with the increase in the number of students, pre-college math points increased at a faster rate.

Highlights (cont’d)

• Measures are real-time, process measures for public sector entity, something not historically available in education

• Highlights system challenges• Governor can ask for and expect action

items and quarterly updates• Allows Governor and staff to see what the

college system is doing NOW• Generates timely information to support

operational decisions and strategic planning

Value of Information to Governor and Staff

• Held up as a model performance funding mechanism

• Governor has encouraged other entities to consider a similar performance system, in education and elsewhere

• Governor immediately clued in on math• Governor responded favorably to I-BEST

Responses by Governor and Key Staff

• Promotes credibility, particularly important in the current budget climate

• Many government agencies do not possess the same quality of data, so the community and technical college system stands out in a very positive way

Other Considerations

Links and All GMAP Slides

GMAP Websitehttp://www.accountability.wa.gov

Student Achievement Websitehttp://www.sbctc.edu/college/e_studentachievement.aspx

Are two-year college students making academic progress?

2008-09 Change in Achievement Points Over Baseline

Measure: Number of key academic benchmarks (achievement points) that students meet on the path to successfully completing degrees and certificates.Source: SBCTC Data WarehouseLink to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf

Summary Analysis• The Student Achievement Initiative is a new performance

funding system for community and technical colleges aimed at improving public accountability and providing financial rewards to colleges for increasing the levels of achievement attained by their students.

• Students come to colleges with varying levels of college readiness. The initiative rewards colleges when students pass through critical academic milestones: getting ready for college, succeeding in the first year of college, completing college math, and reaching the tipping point, a certificate with at least a year of college or a degree.

• SBCTC and national research demonstrates that college students experience a substantial boost in earnings gain when they reach the tipping point or beyond.

• The first performance awards in October totaled $1.8 million--$1 million from the Legislature and $0.8 million from the Gates and Ford Foundations.

• Compared to the 2006-07 baseline, Student Achievement Points increased by 19 percent overall in 2008-09, the first year colleges received a portion of their funding based on performance. ALL colleges increased performance and gained points as students are going further in college success.

Action Plan• Provide financial rewards to colleges each year for

increasing the number of significant gains in critical academic milestones.

• Colleges will identify and share strategies that are working well to increase student achievement in an effort to foster continuous improvement.

Are adult literacy students increasing their achievement in basic skills and beyond?

Measure: Number of times adult literacy students make nationally recognized standardized test gains in math or English as measured by pre- and post-testing or by earning a GEDSource: SBCTC Data WarehouseLink to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf

Basic Skills Students’ Achievement Growth Over Baseline

Summary Analysis• The graph and table show gains made by those students

who took basic skills courses, regardless of their purpose in college.

• Basic skills students come to college the furthest below college level readiness.

• Colleges have made substantial progress increasing basic skills students’ achievement. In 2008-09, 56,218 students earned a total of 100,000 points, 36% over the baseline.

• Basic skills students increased achievement in basic skills as well as beyond basic skills—the number of basic skills students reaching the tipping point nearly quadrupled.

• Three quarters of students’ success in reaching the tipping point can be attributed to I-BEST programs.

Action Plan• Provide financial rewards to colleges each year for

increasing the number of significant gains made by ABE students on their way to college readiness.

• Expand programs that integrate basic skills education with workforce education (I-BEST)

• Provide training to colleges on using data to better understand factors that contribute to adult literacy students’ success.

• Provide training for new faculty on administration of the standardized test (CASAS) used to evaluate adult literacy student progress.

Are underprepared college students succeeding in pre-college and college-level work?

Pre-College Math Students’ Math Achievement Growth Over Baseline

Measure: Number of pre-college math and English courses students pass with a qualifying grade to advance toward college-level workSource: SBCTC Data WarehouseLink to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf

Summary Analysis• Nearly half (48%) of high school graduates who come to

community and technical colleges directly from high school need math remediation.

• Colleges are working hard to get students through pre-college math quicker and ensure they successfully complete college-level math. There is much work to be done to increase math success.

• The graph to the left shows the number of pre-college gains completed by all students, and the table shows math achievement growth by those students who attempted pre-college math.

• Gains in pre-college math are keeping pace with the increase in the number of students (14% growth in each over the baseline); however, students are still behind in moving to college-level math.

• Through the Transition Math Project, Achieving the Dream and Gates funded work, colleges are evaluating and piloting ways to move more students into college math faster.

Action Plan• Provide financial rewards to colleges each year for increasing

the number of pre-college courses completed with a qualifying grade.

• The new Re-Thinking Pre-College Math Initiative, funded by Gates Foundation grant, will support 7 colleges to improve curriculum, instructional practices and assessment approaches in pre-college math programs.

• Through Gates Foundation funding, expand I-BEST to 10 pilot colleges to integrate below college-level courses with college-level courses.

Are college students increasing their achievement in the critical first year of college?

Achievement Growth Over Baseline of Students Who Startedthe Year with at Least 15 But Less Than 30 College-Level Credits

Measure: Number of students who complete the first 15 and 30 college-level creditsSource: SBCTC Data WarehouseLink to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf

Summary Analysis• The number of students reaching the first 15 and 30

college-level credits increased by 16% and 15% over baseline, respectively, as shown in the graph to the left.

• The achievement of a subset of those students, those who start the year with at least 15 but less than 30 credits, is shown in the table to the left.

• Those students are getting further in achievement in the performance year (2008-09) over the baseline, as total points increased 13% compared to 6% growth in the number of students.

• The tipping point was the fastest growing milestone for those students. Workforce students generated most of this growth as colleges added short certificates that students with 45 credits combined to earn a point in the tipping point category. These students will need to complete college-level math if they are to advance further.

Action Plan• Provide financial rewards to colleges each year for

increasing the number of college students who reach the first 15 and first 30 college-level credits.

• Expand promising practices such as integrating achievement points into tracking and advising.

• Engage in global consortia to provide tutoring and virtual library services to all students around the clock to help them succeed in college.

Are college students increasing their achievement in college-level math?

Math Achievement Growth Over Baseline of Transfer Students Starting the Year With No College Math

Measure: Number of students who complete the first 5 credits in college-level mathSource: SBCTC Data WarehouseLink to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf

Summary Analysis• The graph to the left illustrates the 6% increase in the

number of all CTC students who completed college-level math in 2008-09 over the baseline.

• College-level math is required for all students who are planning to transfer to a 4-year institution. The table to the left illustrates math achievement of transfer students.

• In 2008-09, nearly 100,000 students who were planning to transfer entered the year with no college math.

• While the percent of transfer students completing college math in 2008-09 over the baseline just kept pace with the increase in the number of students, pre-college math points increased at a faster rate.

• Colleges are focusing on making improvements in math remediation, advising students to take math sooner and are working with high schools to improve college prep.

Action Plan• Provide financial rewards to colleges each year for

increasing the number of college students who reach the first 15 and first 30 college-level credits.

• Look for promising practices that prepare students for college level math and advise and guide those students into appropriate courses for their degrees, through the Gates-funded initiatives and college efforts.

• Colleges will work to provide students appropriate math options for specific fields of study.

Are more community and technical college students reaching the “tipping point”?

Tipping Point Growth Over Baseline of All Students

Measure: Number of students who complete a degree, a certificate backed by at least one year of college or an apprenticeship (the “tipping point”)Source: SBCTC Data WarehouseLink to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf

Summary Analysis• The minimum threshold for the “tipping point” is a

short-term certificate backed by at least a year of college. Most students go beyond that to complete a long-term certificate, a degree or an apprenticeship.

• SBCTC and national research demonstrates that college students experience a substantial boost in earnings gain when they reach the tipping point or beyond.

• In 2008-09, the number of students reaching the tipping point or higher increased by 11% over the baseline, as compared to 4% growth in the number of students.

Action Plan• Provide financial rewards to colleges each year for

increasing the number of college students who reach the tipping point.

• Provide Opportunity Grants to help low-income adults reach the tipping point in an educational program leading to careers in high-demand fields.