New Delhi

July 2014

How to move to scale?

(1) Describe an Intervention

(2) Statistical Considerations for

Measuring Effectiveness

(3) Recommendations for Moving to

Scale

An Intervention Needs to Solve

a Problem

What is my problem?

Large numbers of low income and minority

students are not attending college even

though qualified to enroll.

What Are The Solutions? How Can We Fix This?

Ambitious Generation

What was learned?

• Students unaligned – education and work goals are inconsistent

Aligned students:

• Have coherent life plans

• Are realistic about their futures

Taking a Wrong Turn

303.9 1,155.8 661.1

50.9

64.7

80.7

0

10

20

30

40

50

60

70

80

90

0

200

400

600

800

1,000

1,200

1,400

Low income (<20%) Middle income (20-80%)

High income (>80%)

Perc

enta

ge

Num

ber

(Thousa

nds)

Number Enrolled in College (Thousands)

Percent Enrolled in College

SOURCE: American College Testing Program, unpublished tabulations, derived from Census data.

College Attendance by Family Income (2012)

Conceptual Plan

• consult with near-age mentors

• visit colleges

visualize oneself as a college

student

transform interest into

realistic actions

• choose realistic

colleges

create strategic plans

• complete FAFSA

• find scholarships

• support fall

enrollment

Related Work on Postsecondary Enrollment

Tutoring/Mentoring/

Coaching

Course Counseling Financial Aid College Visits

Avery & Christopher (2009)

Bergin, Cooks, & Bergin (2007)

Byndloss (2013)

Carroll & Sacerdote (2012)

Castleman, Arnold, & Wartman

(2012)

Castleman & Page (2013)

Castleman & Page (2014)

Domina (2009)

Hill & Winston (2010)

Hoxby & Avery (2012)

Hoxby & Turner (2013)

Jackson (2012)

Long & Boatman (2013)

Roderick, Nagoka, Coca, & Moeller

(2008)

Sabates, Harris & Staff (2011)

Sherwin (2012)

Smith (2013)

Smith & Howell (2011)

Smith, Pender, & Howell (2012)

Adelman (2006)

Allensworth, Nomi, Montgomery

& Lee (2008)

Avery (2010)

Bettinger & Baker (2011)

Conley (2005)

Constantine, Seftor, Sama

Martin, Silva, & Myers (2006)

Farmer-Hinton & McCullough

(2008)

Hargrove, Godin, & Dodd (2008)

Hurwitz & Howell (2013)

Perna, Rowan-Kenyon, Thomas,

& Bell (2008)

Riegle-Crumb, Farkas, & Muller

(2006)

Riegle-Crumb, King, Grodsky,

Muller (2012)

Seftor, Mamun, & Schrim (2009)

Stephan & Rosenbaum (2013)

Avery & Hoxby (2003, 2004)

Avery, Hoxby, Jackson, Burek,

Pope, Raman (2006)

Bettinger, Long, Oreopoulos, &

Sanbonmatsu (2012)

Cohodes & Goodman (2013)

De La Rosa (2006)

Dynarski & Scott-Clayton (2013)

Dynarski & Wiederspan (2012)

Fryer, Jr. (2010)

Grodsky & Jones (2004)

Goldrick-Rab, Harris, Kelchen, &

Benson (2012)

Harris (2013)

Heller (2006)

Horn, Chen, & Chapman (2003)

Jackson (2010)

Kane (2003)

McDonough & Calderone (2006)

Pallais & Turner (2006)

Scott-Clayton (2012)

Engle, Bermeo,& O’Brien (2006)

Gandara (2004)

Hill (2008)

Holland (2011)

Klein & Washburn (2012)

Myers, Olsen, Seftor, Young, &

Tuttle (2004)

Rocca (2013)

Roderick, Coca, & Nagaoka (2011)

How CAP is Different?

• Whole school design

• Public high schools

• Includes rural schools

• Built on relationships

• Highlights STEM college programs

• Scale-up potential

The College Ambition Program

• CAP is an intervention with a staggered rollout

• Began with two treatment and two control high schools in 2010

• Thirteen in 2013-2014

CAP Description

• CAP Center open to all students

• CAP coordinates with school counseling staff and teachers

• Staffed by a site coordinator

• Offers: (1) tutoring and mentoring; (2) course-counseling and advising; (3)

financial aid planning; and (4) college visits

CAP Site Coordinators

Measures and Materials

Sample

• Treatment criteria - lower than

state average college enrollment

rates

• Identified with state

administrative data, census

data, and data from the

Common Core of Data

• Each treatment school matched

with five control schools using

propensity score matching

techniques

School Matching Criteria

Postsecondary enrollment rates, school size, census

designation, poverty rate, and racial/ethnic diversity

Participants

• Urban schools serve a racially diverse population of students; while rural schools serve a predominately white student population

• Sizes vary from around 700 to 1,200 students for the urban sites and between 300 and 600 students at the rural schools

• Schools serve a majority of economically disadvantaged students, with 70 percent of their students eligible for free and reduced lunch

• All treatment schools have a substantial number of students who would be the first in their family to go to college

2013-2014 Treatment and Comparison School Descriptive Characteristics

4-Year College Going Rates

2-Year College Going Rates

Graduation Rate

Dropout Rate

% Free and Reduced % Minority School Size

Urban Treatment (n=9) 28% 42% 53% 16% 71% 88% 905

Comparison (n=45) 35% 39% 63% 11% 57% 59% 1031

Rural Treatment (n=4) 37% 37% 80% 6% 29% 4% 488

Comparison (n=20) 37% 44% 84% 6% 38% 9% 625 SOURCE: Michigan Department of Education, 2009

Research Design

2010-2011 2011-2012 2012-2013 2013-2014

Treatment

Schools

2 4 7 13

Matched

Comparisons

10 20 35 65

Total Schools 12 24 42 78

Power analysis using Optimal Design Plus (Spybrook et al. 2011) shows that for 2012-2013, a total of 42 schools put us close to 90% power and that in 2013-2014, we will exceed 90%.

Preliminary Analysis

12th Grade CAP Participants vs. Non-participants in Treatment Schools 2011-2012

CAP Participants Non-Participants Difference

12th Graders in 2012 (N=519) 29% 71%

Attend a 4-year College 56% 41% 15%**

Attend a Competitive College 51% 35% 16%**

Attend a 2-year Institution 44% 58% -14%**

Interest in a STEM Major 29% 18% 11%*

SOURCE: NSC data, 2012 CAP Exit Survey, Mentor Contact Logs

*p<.10, **p<.05

Four Year College Enrollment Treatment vs. Control Schools

Two Year College Enrollment Treatment vs. Control Schools

Any College Enrollment Treatment vs. Control Schools

Research Take Aways

• Schools are dynamic places that are changing

• Partnerships are key for research in schools

• Changing student behaviors/and plans depends on personal interaction with a trusted knowledgeable individual

• College interventions should be low-cost and designed to be scaleable

*Some Recommendations

*Training and full disclosure of limitations

*Full documentation including measures of

fidelity of implementation

*A comparison group with appropriate measures

of measuring statistical differences

*Finding an objective third party evaluator

Pictures of Staff

This material is based in part upon work supported by the National Science Foundation under Grant #s 0929635, DRL-1255807, DRL-1316702, NIH Grant

# 1R01GM102637-01, the Michigan Department of Education, and a Pre-Doctoral Training Grant from the Institute of Education Sciences, U.S.

Department of Education Award # R305B090011. Any opinions, findings, and conclusions or recommendations expressed in this material are those of

the author(s) and do not necessarily reflect the views of the funding agencies.

Pictures of CAP Participants

Sophia Bozzo Michigan State University

Majoring in Food Science

Zahna Woodson Alma College

Majoring in Communications,

Minor in Cultural Studies

Michael Rogers Michigan State University

Majoring in Food Science

Nate Hall University of Michigan

Double majoring in Economics

and Informatics

Kenny Heindel University of Michigan

Science of Information