Virtual Classrooms: How Online College Courses Affect Student …€¦ · Virtual Classrooms: How Online College Courses Affect Student Success Eric Bettinger Stanford. Lindsay Fox

Virtual Classrooms: How Online College Courses

Affect Student Success

Eric BettingerStanford

Lindsay FoxMathematica

Susanna LoebBrown

Eric TaylorHarvard

CUNY Office of Policy Research, Higher Education Policy Seminar Series October 25, 2019

American Economic Review, 2017

2

0.1

.2.3

.4

Prop

ortio

n of

enr

ollm

ent

degr

ee-s

eeki

ng, u

nder

grad

and

gra

d

2002 2005 2008 2011 2014 2017Year

any online (Allen and Seamen 2013)

any distance

entirely distance

for-profit for-profit, any distance

for-profit, entirely distance

(Source: Authors calculations of NCES data, except “any online” series.)more detail on institution type

3

Motivation

Growth of online courses in college among degree-seeking students roughly ⁄1 3 take at least

one course online, and ⁄1 6 take at all courses online

generally described as an effort to reduce costs and increase access

Growth (and decline) of for-profit colleges first estimates of learning outcomes (now a few more)

our work complements the literature on labor-market outcomes for for-profit students

4

This paper: defining “online course”

Wide diversity in what counts as “online education”…MOOCs, micro-credentials, badges, flipped-classroom, (a)synchronous, blended, hybrid,…

Focus in our paper

online courses which

i. replicate the features of a traditional in-person class (e.g., syllabus, text, lectures, assignments, tests)

except that

ii. student-teacher and student-student interactions are online and asynchronous

5

This paper: quasi-experiment

Quasi-experiment, instrumental variables

treatment = 1 if the student takes a given course online = 0 if in-person

we pool 750+ courses, 100+ campuses, and 27 terms

2.3M student-course observations

instrument = interaction of two reasons students take a course online or in-person

a. the distance (miles) between the student’s home and the nearest campus

b. whether the course is offered in-person at that campus in the current term

6

This paper: headline results

Students learn less when they take a course online average student earned C online v. B− in-person larger losses for lower-achieving students

Students are worse-off the next term less-likely to enroll lower grades, especially when the course was the same

subject as the treated course fewer credits

One year later, students are making slower academic progress

7

① Introduction

❷ Setting and treatment

③ Econometrics

④ Estimates

8

Setting

institution large for-profit university 100+ campuses across the U.S. (map) 750 courses, 168K course sections courses ⁄2 3 online, ⁄1 3 in-person majors: business mgmt, technology, health

time 2009-2013 27 terms (6 terms per year)

students undergraduate, degree-seeking ⁄4 5 seeking bachelors 230K students 2.3M student-course observations

9

0.1

.2.3

.4

Prop

ortio

n of

enr

ollm

ent

degr

ee-s

eeki

ng, u

nder

grad

and

gra

d

2002 2005 2008 2011 2014 2017Year

any online (Allen and Seamen 2013)

any distance

entirely distance

for-profit for-profit, any distance

for-profit, entirely distance

(Source: Authors calculations of NCES data, except “any online” series.)

10

Setting

11

Setting

12

Treatment contrast

in-person online

same syllabus, textbook, lecturesassignments, group projects

quizzes, tests, grading rubricsclass size (caps)

term lengthinstructors

different conventional in-person class meetings

lecture videos

class discussion is asynchronous and online

13

① Introduction

② Setting and treatment

❸ Econometrics

④ Estimates

14

Instrumental variables specification

𝑦𝑦𝑖𝑖𝑖𝑖𝑖𝑖 = 𝛿𝛿𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑖𝑖𝑖𝑖𝑖𝑖 + 𝛼𝛼𝐷𝐷𝑂𝑂𝐷𝐷𝐷𝐷𝐷𝐷𝑂𝑂𝐷𝐷𝑂𝑂𝑖𝑖𝑖𝑖 + 𝛾𝛾𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑏𝑏𝑖𝑖𝑖𝑖+ 𝑿𝑿𝛽𝛽 + 𝜋𝜋𝑖𝑖 + 𝜙𝜙𝑖𝑖 + 𝜓𝜓𝑏𝑏 + 𝜀𝜀𝑖𝑖𝑖𝑖𝑖𝑖

instrument for 𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑖𝑖𝑖𝑖𝑖𝑖 with 𝐷𝐷𝑂𝑂𝐷𝐷𝐷𝐷𝐷𝐷𝑂𝑂𝐷𝐷𝑂𝑂𝑖𝑖𝑖𝑖 × 𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑏𝑏𝑖𝑖𝑖𝑖

student 𝑖𝑖, course 𝑐𝑐, term 𝑡𝑡, nearest campus 𝑏𝑏 = 𝑏𝑏 𝑖𝑖𝑡𝑡

𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑖𝑖𝑖𝑖𝑖𝑖 = 1 if online, = 0 if in-person 𝐷𝐷𝑂𝑂𝐷𝐷𝐷𝐷𝐷𝐷𝑂𝑂𝐷𝐷𝑂𝑂𝑖𝑖𝑖𝑖 miles between student 𝑖𝑖’s home and campus 𝑏𝑏 𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑏𝑏𝑖𝑖𝑖𝑖 = 1 if course 𝑐𝑐 offered at campus 𝑏𝑏 in term 𝑡𝑡

𝜋𝜋𝑖𝑖 , 𝜙𝜙𝑖𝑖, 𝜓𝜓𝑏𝑏 fixed effects for course, term, and campus

𝑿𝑿 includes prior GPA, gender, age, and major fixed effects

(first stage, covariate balance)

15

Exclusion restriction assumption

With one exception, (a) and (b) hold:a. any mechanism by which student distance from campus

affects outcomes, affects students the same way in terms with and without an in-person class option

b. any mechanism causing outcomes to differ between terms with and without an in-person class option, does not depend on a student’s distance from campus

The exception: In terms when students have a choice between online and in-person, distance may affect that choice, and taking the course online could harm (improve) outcomes

16

Exclusion restriction potential violations

students change their residence from term to term in response to in-person class offerings

the university chooses when and where to offer in-person classes based on students’ potential outcomes

… but to violate the exclusion restriction the decision rule would need to give systematically differential weight based on student distance from campus

students take a course in a later (earlier) term, predicting the likelihood of in-person offerings over time

17

In-person course offerings

Variation typical course is offered 1:4 terms at a given campus

half of variation is within course-by-campus

Explanations prior demand essentially nothing (0.1-0.6%)

current demand (test based on “fewer than 5” rule)

seasonality largest predictor (25% explained by one pattern alone: “offer the course every other term”)

professor availability, anecdotally

18

Distance between residences and campuses

Variation (cdf)

typical student lives 10 miles from campus (IQR 5,28)

10% live more than 100 miles from nearest campus

Student residence choices 1 in 8 moved; half moved closer to campus, half further

no correlation between moves and student observables

19

Distance between residences and campuses

University campus location choices 10 of 102 campuses opened during our data, none closed

University’s stated criteria: financial performance, local market competition

demand seems likely (Cellini 2009)

… but empirically demand does not predict openings in our data (event study)

20

① Introduction

② Setting and treatment

③ Econometrics

❹ Estimates

Course grade

Enrolled next term

GPA next term

Enrolled one year later

-.6 -.4 -.2 0Estimated effect of taking a

course online, instead of in-person

21

sample mean 2.82

sample mean 2.78

sample mean 0.88

sample mean 0.69

Headline results

Course grade

A or higher

B or higher

C or higher

Passed (D or higher)

Withdrew

-.6 -.4 -.2 0 .2Estimated effect of taking a


22

sample mean 2.82

sample mean 0.41

sample mean 0.88

sample mean 0.09

sample mean 0.69

sample mean 0.83

Grades (learning) in course taken online

23

Larger losses for lower-achieving students

GPA next term

GPA next term, in same subject

GPA next term, was prerequisite

GPA next term A or higher

GPA next term B or higher

GPA next term C or higher

Credits next term

-1 -.8 -.6 -.4 -.2 0Estimated effect of taking a


24

The term following an online course

Enrolled next term

Credits next term


Credits one year later

-1.8 -1.6 -1.4 -1.2 -1 -.8 -.6 -.4 -.2 0Estimated effect of taking a


25

Lower persistence and academic progress

26

Main estimate

Required for major

Intro/intermediate course

Advanced course

New student

Took online course before

Business

Health

Technology

-1.2 -1 -.8 -.6 -.4 -.2 0 .2Estimated effect of taking a


Course grade

Heterogeneity (very little)

(other outcomes)

27

Robustness

non-linear function of distance (figure, table)

intertemporal course choices (new students test)

students who move (table)

(non-normal) distribution of grades

28

Discussion, Conclusion

Taking a course online, instead of in-person, makes students worse off learn less

less likely to persist and progress

Increased inequality, with respect to student human capital

These conclusions (may only) apply to students who can choose between online and in-person

students at similar institutions

Virtual Classrooms: How Online College Courses

Affect Student Success

Eric BettingerStanford

Lindsay FoxMathematica

Susanna LoebBrown

Eric TaylorHarvard

CUNY Office of Policy Research, Higher Education Policy Seminar Series October 25, 2019

American Economic Review, 2017

30

(McPherson and Bacow 2015)

31

32

33

34

35

36

Main estimate

Required for major


Advanced course

New student


Business

Health

Technology



GPA next term

37

Main estimate

Required for major


Advanced course

New student


Business

Health

Technology



Enrolled next term

38

Main estimate

Required for major


Advanced course

New student


Business

Health

Technology




39

40

41

Documents

Virtual Classrooms: How Online College Courses Affect Student …€¦ · Virtual Classrooms: How Online College Courses Affect Student Success Eric Bettinger Stanford. Lindsay Fox