The Effects of Student Coaching:

An Evaluation of a Randomized Experiment in Student Mentoring

October 3, 2012

Eric Bettinger, Stanford UniversityRachel Baker, Stanford University

Defining the problem: Trends in college attendance and completion

More students are taking classes online

SOURCE: Going the Distance: Online Education in the United States, 2011 Survey by Babson Survey Research Group

College attendance in the United States has consistently increased over the last four decades

SOURCE: The College Board, Trends in College Pricing 2010, Figure 17A and Figure 17B.

College completion has not

SOURCE: Turner 2004.

Voicing concerns about completion

• President Obama (2011): “This country needs and values the talents of every American. That is why we will provide the support necessary for you to complete college and meet a new goal by 2020: America will once again have the highest proportion of college graduates in the world.”

• Vice President Biden (2011): “We have to make the same commitment to getting folks across the graduation state that we did to getting them to the registrar’s office.”

• Financial barriers/liquidity constraints (e.g. Dynarski & Deming 2010)

• No access to appropriate channels of information

(e.g. Bettinger, Long, Oreopolous and Sanbonmatsu 2010)

• Weak academic preparation and performance (e.g. Adelman & Gonzalez 2006)

• Lack of social and academic integration (e.g. Bloom & Sommo 2005, Tinto 1975)

Why do students not complete college?

Today’s Focus is on Student Coaching

• What is coaching?– Individualized instruction/guidance aimed at

helping students overcome barriers• Why coaching?

– Help students to build study skills– “Nudge” students to complete complex tasks– Provide information related to college success

• Previous studies have looked at similar interventions

InsideTrack

• Student coaching service• Business model focuses on being an external,

third-party advising service– Claim to build an economy of scale for counseling

services

• Partners with a number of types of institutions– Most students are studying in vocational tracks.

InsideTrack’s Coaching

• Emphasis on training and hiring coaches• Coaching takes place via phone, email, and

text. • Coaching is “Active” not “Passive”

Our key goal is to identify the effects of this coaching on student retention.

Our Experiment

• InsideTrack wanted to “prove” itself to college partners. They used randomized trials to show colleges their impact.

Selection into Randomization

1. Colleges selected the number of students to be treated and submitted lists of students to InsideTrack.

2. InsideTrack randomly divided college lists into two groups.

Selection into Randomization, con’t

3. InsideTrack presented the list to the schools.

4. Colleges chose which group would receive treatment.

Basic Descriptive Statistics and Balance

Characteristic Control Group Mean

Difference for Treatment(std error)

Sample Size Number of Lotteries

Female.488 .009

(.009)12,525 15

Missing Gender.675 -.001

(.001)13,555 17

Age30.5 .123

(.209)9,569 8

Missing Age.294 .0001

(.0010)13,555 17

SAT Score886.3 -11.01

(16.19)1,857 4

Missing SAT.827 .001

(.002)13,555 17

Living On Campus

.581 -.005(.017)

1,955 4

0.0

1.0

2.0

3.0

4.0

5

0 20 40 60 80Age

Treatment Age Control Age

0.0

005

.001

.001

5.0

02

0 500 1000 1500SAT

Treatment Control0

.2.4

.6.8

0 1 2 3 4HS GPA

Treatment Control

Age Distributions SAT Score Distributions

HS GPA Distributions

Distributions of Treatment and Control Groups

Significant Differences by LotteryLottery # Charac-

teristics# Significant Diff (90%)

1 (n=1583) 2 0

2 (n=1629) 2 0

3 (n=1546) 2 0

4 (n=1552) 2 0

5 (n=1588) 2 0

6 (n=552) 3 0

7 (n=586) 3 0

8 (n=593) 3 0

9 (n=974) 9 0

Lottery # Charact-eristics

# Significant Diff (90%)

10 (n=326) 6 0

11 (n=479) 6 0

12 (n=400) 2 0

13 (n=300) 1 0

14 (n=600) 1 0

15 (n=221) 3 1

16 (n=176) 14 0

17 (n=450) 12 0

Methodology

• Basic Regression Analysis

Y = α + βTreatment + γ1Lottery1 +. . . + γ17Lottery17 +Xδ + ε

Y is an outcome of interest focusing on retention after 6, 12, 18 or 24 months

Treatment is a binary variable for being coached.Lottery# is a binary variable indicating student

participation in a specific lottery.X is a vector of student characteristics

Baseline Results with Covariates

Model 6-month retention

12-month retention

18-month retention

24-month retention

Control Mean .580 .435 .286 .242

1. Baseline

Treatment Effect(std error)

.052***(.008)

.053***(.008)

.043***(.009)

.034**(.008)

Lottery Controls Yes Yes Yes Yes

N 13,552 13,553 11,149 11,153

2. Baseline w/ Covariates

Treatment Effect(std error)

.051***(.008)

.052***(.008)

.042***(.009)

.033**(.008)

Lottery Controls Yes Yes Yes Yes

N 13,552 13,553 11,149 11,153

Robustness: Effects in Each Lottery

Robustness:Effects Across Years

Model 6-month Retention

12-month Retention

18-month Retention

24-month Retention

Control Mean .617 .479 .381 .356

2004 Cohorts

Treatment Effect(std error)

.088***(.020)

.070***(.020)

.068***(.021)

.030(.020)

Covariates Yes Yes Yes Yes

N 1,774 1,745 1,520 1,524

2007 Cohorts

Control Mean .573 .426 .265 .217

Treatment Effect(std error)

.044***(.008)

.049***(.009)

.037***(.010)

.034***(.009)

Lottery Controls Yes Yes Yes Yes

N 11,808 11,808 9,629 9,629

Robustness:50/50 Splits

Model6-month

Retention12-month Retention

18-month Retention

24-month Retention

Completed Degree

Control Mean

.769 .614 .366 .350 .312

1. Baseline Model

Treatment Effect

.037***(.012)

.050***(.014)

.070***(.021)

.027(.020)

.040*(.024)

Lottery Controls

Yes Yes Yes Yes Yes

N 3,527 3,527 1,344 1,348 1,346

2. Baseline w/ Covariates

Treatment Effect

.037***(.012)

.050***(.014)

.070***(.021)

.027(.020)

.040*(.024)

Lottery Controls

Yes Yes Yes Yes Yes

N 3,527 3,527 1,344 1,348 1,346

Effects on Subgroups

Model 6-month Retention

12-month Retention

18-month Retention

24-month Retention

Females

Control Mean .661 .497 .346 .299

Treatment Effect(std error)

.025**(.012)

.045***(.013)

.033**(.014)

.022*(.013)

N 6,045 6,045 4,740 4,744

Males

Control Mean .536 .403 .260 .215

Treatment Effect(std error)

.061***(.012)

.054***(.012)

.047***(.012)

.047***(.011)

N 6,479 6,480 5,457 5,457

Effects by Gender

Effects by Age GroupModel 6-month

Retention12-month Retention

18-month Retention

24-month Retention

Students Under 30

Control Mean .600 .438 .234 .184

Treatment Effect(std error)

.037***(.010)

.052***(.011)

.040***(.012)

.041***(.011)

N 7,850 7,850 5,671 5,671

Students Over 30

Control Mean .513 .400 .311 .266

Treatment Effect(std error)

.062***(.017)

.044***(.017)

.034**(.016)

.024(.015)

N 3,958 3,958 3,958 3,958

Effects on Subgroups

Cost-Benefit Analysis• Most studied intervention focused on retention is

financial aid– Effect sizes are usually around 3 percentage points

per $1000 in aid.– Effect is contemporaneous and doesn’t extend into

future years.• InsideTrack cost about $1000 per year per

student– Contemporaneous effect was about 5 percentage

points– Effects persisted into subsequent year (3 percentage

points)

Conclusion/Discussion• College advisement is a widespread intervention• “Adult” learners are becoming an increasingly important

group of students in higher education; effects were symmetric across age.

• Online education is also rising; multiple campuses in our study were online campuses.

• InsideTrack offers 3rd party advising/coaching– Attempts to exploit economy of scale.– Loosely affiliated with college.– Active rather than passive coaching.

• Effects were large and cost effective

Degree Completion

• Degree completion information come from 3 lotteries

• Definition of degree is generally four-year degree. It could include some two-year degrees.

• Control Group Graduation Rate = 31.2%• Treatment Effect = 4.0% with standard error

of (2.4%)