One day the educational opportunities available to children will be determined solely by their

talent, motivation, and effort.

Is Our Program Working? How to Partner with Evaluators and Get Results

Dr. Chris Avery, Roy E. Larsen Professor of Public Policy and Management, Malcolm Wiener Center for Social Policy, Harvard Kennedy School of Government

Traci Kirtley, Director of Programming and Evaluation, College Possible

Presentation and Workshop Agenda

► I: Presentation: Overview of evaluation partnership and key learnings

► II: Small group exercise: Overcoming potential evaluation challenges

► III: Question and Answer Period and Discussion

20002001

20022003

20042005

20062007

20082009

20102011

20122013

20142015

20162017

20182019

20200

5000

10000

15000

20000

Introduction to College Possible

2012-134 markets

12,000 students

2000-011 market35 students

College Possible is a national college access and success organization currently serving 12,000 students in Minnesota’s Twin Cities; Milwaukee, Wisconsin; Omaha, Nebraska and Portland, Oregon, with a growth plan to expand to serving 20,000 students in 10 metro areas by 2020.

2019-202010 markets

20,000 students

Program Model

College Success Program

Ongoing support through college persistence & graduation

AmeriCorps coach ‘near-peer mentor’

College planning workshops for 9th and 10th grade students are led by coaches and high school seniors to develop a college-going school culture early on.

College Success Program

Weekly touch point with college coach

Academic and campus resource connections

Social network support

Financial aid consulting

Leadership development

Structured after-school program during junior & senior years

College Access Program

320+ hours time on task in sessions & one-on-one

College application assistance

Standardized test prep

Financial aid consulting

College transition guidance

College Access Program College Success Program

"Tom Mortensen, Postsecondary Education Opportunity, 2010"

All Low Income

CollegePossible

All Income

Series1

74

94

59

Percentage of students who enroll in college

Percentage of students who graduate from college

Series1

5558

11

College Possible students are 10 times more likely to graduate from college than their low-income peers.

Results

Introduction to Dr. Christopher Avery

► Roy E. Larson Professor of Public Policy at Harvard Kennedy School

► Extensive research in college admissions

► Current focus on college application and enrollment patterns, especially among low-income students

► Recent studies have highlighted the issue of “undermatching” of high-achieving, low-income students in applying to and enrolling in more selective institutions

Why Evaluate?

3 main reasons to devote resources to program evaluation:

- Shows if program is working, and for whom and in what conditions

- Learn how to improve programmatic practices

- Increases organization’s credibility with funders and other

constituents

As College Possible grows to serve more students, rigorous and systematic evaluation of program implementation and outcomes becomes increasingly important to ensure the organization’s credibility and sustainability.

Experience with Evaluations

2005Wilder

Research

Showed College Possible to be “very effective in helping the students who complete the program gain admission and financial assistance to college.”

2010 ICFInternational

ProgramAudit

External evaluation verified College Possible’s reported student ACT score increases and college acceptance outcomes.

2011Comparative

HistoricalAnalysis

Study by Dr. Chris Avery demonstrated that College Possible students were more than twice as likely to enroll in 4-year instituation as similarly situated peers.

Case Study: Avery study of College Possible

- During spring 2010 student recruitment, College Possible identified 8 high schools with significantly more eligible sophomore applicants than available spots in cohort.

- School teams selected 80% of cohorts (32/40) as they usually would,leaving a “last-in, first out” group of about 240 students.

-Among “last-in, first-out” group, the evaluator randomly assigned students to enroll in the program or be on waitlist. This created treatment and control groups.

-Treatment and control groups were solidified in fall 2010, when some students were admitted off waitlist to replace students who had moved or transferred. Ultimately, around 130 students were in treatment group.

Case Study: Avery study of College Possible

- From their junior year onward, evaluator compared following outcomes of treatment and control groups using data from independent sources.

- ACT attendance and score increases- Number of college applications sent- Types of colleges applied to (2 year or 4 year, selectivity, etc.) - Admissions decisions- College enrollment rates in fall and spring

- Plans to continue following students to study rates of college persistence and graduation.

Lessons Learned: Evaluation Challenges

► Conducting an evaluation takes resources and attention away from running the program

► Evaluations can challenge core beliefs when you “just know it works”

► Self-selection issues may exist– How are participants recruited?– How are participants selected?

► Evaluation needs must be balanced with program design needs or preferences

► Find an evaluator you trust!

Lessons Learned: Evaluation prerequisites

► Measurable outcomes

► Logical explanation for how results are achieved

► Historical evidence– Observational study/regression discontinuity– As a program, are you ready to be evaluated?

► Data collection systems and process

► Staff buy-in

Special Issues: Randomized Controlled Trials

Benefits: Clean comparison of program effects

“Gold standard” establishes strong evidence base

Evidence of effectiveness helps build support

Challenges: Consent process – organizations and participants

Paperwork vs. program activities

Institutional Review Boards

Staff understanding – design issues, interpreting results

Uncertainty about Results

Small Group Discussion

Question and Answer/

Discussion