Interactive Learning Online At Public Universities

7/31/2019 Interactive Learning Online At Public Universities

1/53

May 22, 2012

Interactive Learning Online

at Public Universities:Evidence from

Randomized TrialsWilliam G. Bowen

Matthew M. Chingos

Kelly A. Lack

Thomas I. Nygren


2/53

Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 1

Ihaka S+R is a sraegic consuling and research service provided by IHAK,a no-or-pro organizaion dedicaed o helping he academic communiy usedigial echnologies o preserve he scholarly record and o advance research andeaching in susainable ways. Ihaka S+R ocuses on he ransormaion o schol-arship and eaching in an online environmen, wih he goal o ideniying hecriical issues acing our communiy and acing as a caalys or change. JSOR,a research and learning plaorm, and Porico, a digial preservaion service, arealso par o IHAK.

Copyrigh 2012 IHA K. Tis work is licensed under he Creaive CommonsAtribuion No Derivaive Works 3.0 Unied Saes License. o view a copy ohe license, please see htp://creaivecommons.org/licenses/by-nd/3.0/us
http://creativecommons.org/licenses/by-nd/3.0/ushttp://creativecommons.org/licenses/by-nd/3.0/us


3/53


Preface

Higher educaion is acing serious challenges in he Unied Saes. Tere isincreasing concern abou rising coss, he qualiy o educaion, and ha henaion is losing is compeiive edge. Online learningspecically highlyineracive, closed-loop, online learning sysems ha we call ILO or IneraciveLearning Onlineholds he promise o broadening access o higher educaiono more individuals, while also lowering coss or sudens. Bu is he qualiyhere?

In our rs repor in his area, Barriers o Adopion o Online Learning Sysemsin U. S. Higher Educaion, we highlighed a broad, widely held concern abouhe qualiy o learning oucomes achieved hrough online learning. Bu do weacually know how ineracive online learning sysems really compare o hein-classroom experience? Tis second repor was designed o help nd answers.

We used a sricly quaniaive mehodology o compare he wo learning approachesin a rigorous way. In six dieren public insiuions, we arranged or he sameinroducory saisics course o be augh. In each insance, a conrol group wasenrolled in a radiional classroom-based course; hen, a reamen group ook ahybrid course using a prooype machine-guided mode o insrucion developed aCarnegie Mellon Universiy in concer wih one ace-o-ace meeing each week.Sudens were assigned o hese wo groups by means o a careully designedrandomizaion mehodology. Te research we conduced was designed o answerhese quesions:

Can sophisicaed, ineracive online courses be used o mainain or improvebasic learning oucomes (masery o course conen, compleion raes, andime-o-degree) in inroducory courses in basic subjecs such as saisics?

Are hese courses as eecive, or possibly more eecive, or minoriy andlow-socioeconomic-saus sudens and or oher groups subjec o sereoype

hrea? Or, are hese groups less well suied o an online approach?

Are such courses equally eecive wih no-so-well-prepared sudens andwell-prepared sudens?

Te resuls o his sudy are remarkable; hey show comparable learning oucomes orhis basic course, wih a promise o cos savings and produciviy gains over ime.

More research is needed. Even hough he analysis was rigorous, i was a singlecourse. We need o learn more abou he adapabiliy o exising plaorms oroering oher courses in dieren environmens. Ihaka S+R is commited oconinuing his research and shar ing our ndings broadly.

We look orward o coninuing o engage wih al l hose who care abou highereducaion o help deliver on he poenial ha new echnologies provide.

DEANNA MARCUM

[email protected]

Managing Director, Ithaka S+R
mailto:Deanna.Marcum%40ithaka.org?subject=mailto:Deanna.Marcum%40ithaka.org?subject=


4/53


Interactive Learning Online at Public Universities:

Evidence from Randomized Trials

2 Preface

4 Introduction

9 Educational Outcomes in Public Universities

23 Costs and Potential Savings

26 Summary Observations

29 Acknowledgements

30 Appendices


5/53


Interactive Learning Online at

Public Universities: Evidence fromRandomized TrialsWilliam G. Bowen, Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren1

May 22, 2012

Introduction

Te opic o online learning in higher educaion is o obvious imporance. Teserious economic and social problems acing he U.S.high unemploymen,slow growh, and severe inequaliiesare relaed, many believe, o ailures o he

1 The authors are all associated with Ithaka S+R (the Strateg y and Research arm of ITHAKA), which sponsored

this study. Bowen is a senior advisor to Ithaka S+R, Chingos is a senior research consultant at Ithaka S+R

and a fellow at the Brookings Institut ion's Brown Center on Education Policy, Lack is a research analyst,

and Nygren is a project director and senior business analyst for Ithaka S+R. The authors wish to thank the

foundat ions that suppor ted t his work: t he Carnegie Corporation of New York, t he William and Flora Hewlett

Foundation, the Spencer Foundation, and a fourth foundation that has asked to remain anonymous. We also

thank our colleagues at ITHAK Aand Larry Bacow, Johanna Brownell, Jackie Ewenstein, and Kevin Guthrie inparticularfor their generous help all along the way. But most of all, we wish to thank our faithful friends on

the participating campuses for their hard work and patience with us; their names are appended to this report.

A number of these individuals (as well as others) have commented on a draft of the report, but the authors

are, of course, fully responsible for the views expressed here and for any errors that remain.

Ithaka S+R has sponsored three s tudies of online learning, of which this is the longest lasting. The t wo other

studies are now available on the Ithaka S+R website. See Barriers to Adoption of Online Learning Systems

in U.S. Higher Education by Lawrence S. Bacow, William G. Bowen, Kevin M. Guthrie, Kelly A. Lack, and

Matthew P. Long, and Current Status of Research on Online Learning in Postsecondary Education by William

G. Bowen and Kelly A. Lack (both available online at htt p://www.sr.ithaka.org/).


6/53


U.S. educaion sysem, including higher educaion.2 Levels o educaionalatainmen in his counry have been sagnan or almos hree decades, whilemany oher counries have been making grea progress in educaing largernumbers o heir ciizens. Tere is growing concern ha he U.S. is losing iscompeiive edge in an increasingly knowledge-driven world. Also, subsanialachievemen gaps relaed o race and socioeconomic saus persis and have agrea deal o do wih worr ying inequiies. Moreover, here are good reasons o

believe ha hese wo problems are closely relaed.3

The Cost Squeeze in Higher Education

A he same ime, higher educaion, especially in he public secor, is increasinglyshor o resources. Saes coninue o cu back appropriaions in he ace o scalconsrains and pressures o spend more on oher hings, such as healh careand reiremen expenses.4 Caliornia is a dramaic case in poin. Lack o undinghas caused Caliornia colleges and universiies o reduce he size o heir ener-ing classes a he very ime when increasing numbers o sudens are seekingo enroll.5 Higher uiion revenues migh be an escape valve, bu here is greaconcern abou uiion levels and increasing resenmen among sudens andheir amilies ha is having poliical reverberaions. Presiden Obama, in his

2 The authors agree that there is an important connection between educational outcomes and the economic

performance of a country. But we would warn against exaggerating the power of the connection. In the case

of the U.S., for example, the recent recession and the slow rate of growth seen in the last few years surely

owe more to the 2008 financial excesses than they do to deficiencies in the countrys higher education

system. As Jacob Weisberg pointed out in Newsweek in 2010 with respect to the recent recession, there are

no strong candidates for a single factor that would have caused the crisis in the absence of any others

(Weisbergs piece can be f ound online at http://www.thedailybeast.com/newsweek/2010/01/08/what-

caused-the-great-recession.html).

3 SeeEquity and Excellence in American Higher Education by William G. Bowen, Martin A. Kurzweil, and

Eugene M. Tobin (2005) for an ext ended discussion of the historical record and of t he likely connections,

going forward, between achievement gaps and overall levels of educational attainment. See also David

Leonhardts October 8, 2011 column in the New York Times, The Depression: If Only Things Were That Good,

in which he argues that the U.S. is worse off today t han it was in the 1930s because innovation is lagging

which he attributes in no small part to deficiencies in education (http://www.nytimes.com/2011/10/09/sun-

day-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=all.) Of course, lagging

rates of educational at tainment have their origins in low high school graduation rates. See Henr y M. Levin

and Cecilia E. Rouse, The True Cost of High School Dropout, New York Times, January 25, 2012. (http://

www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.html). But these problems

are then compounded by low completion rates among those who both graduate from high school and enter

college; see Crossing the Finish Line: Completing College at Americas Public Universities (2009) by William

G. Bowen, Matthew M. Chingos, and Michael S. McPherson.

4 A report released in spring 2012 by the State Higher Education Executive Officers, entitled State Higher

Education Finance FY 2011 (http://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdf), docu-

ments the dire economic circumstances of many public institutions.

5 In November 2008, California State University became the first public university to limit enrollment when,

despite a 20% increase in applications from prospective first-year students, it decided to reduce its student

body by 10,000 students, following a $200 million decrease in tax revenue that academic year coupled with

an additional $66 million cut (see Under Financial Stress, More Colleges Cap Enrollments (November

26, 2008) in TIME, http://www.time.com/time/nation/article/0,8599,1861861,00.html). The University of

California and California Community College systems have since followed suit in the face of limited funding

available from the state (see the August 5, 2009 article Budget cuts devasta te California higher educa-

tion in The Washington E xaminer, http://washingtonexaminer.com/science-and-technology/2009/08/

budget-cuts-devastate-california-higher-education).

Levels of educational attainment in

this country have been stagnant for

almost three decades, while many

other countries have been making

great progress in educating larger

numbers of their citizens.
http://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdfhttp://www.time.com/time/nation/article/0,8599,1861861,00.htmlhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://www.time.com/time/nation/article/0,8599,1861861,00.htmlhttp://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdfhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.html


7/53


2012 Sae o he Union address and in subsequen speeches, has decried risinguiions, called upon colleges and universiies o conrol coss, and proposed o

wihhold access o some Federal programs or colleges and universiies ha didno address aordabiliy issues or mee compleion ess.6

oday, a variey o higher educaion insiuions mus conron he challengeo how o manage coss in he ace o igher unding. Whi le he proporion oeducaion spending drawn rom uiion revenues rose across all insiuions,

increases in uiion oen oupaced increases in educaion and relaed spending(i.e. spending on insrucion, suden services, and some suppor and maine-nance coss relaed o hese uncions), calling ino quesion he susainabiliyo he curren unding model.7 Moreover, he rs sur vey o provoss and chieacademic ocers byInside Higher Ed ound ha on he quesion o ins iuionaleeciveness in conrolling coss, over 15 percen o all provoss gave heirinsiuions marks o 1 or 2 on eeciveness [on a scale rom 1 o 7, wih 7 being

very eecive].8 I is equally noeworhy ha very ew chie academic ocers(and especially hose a boh public and privae docoral universiies) gave heirinsiuions high marks on his meric. Recogniion o he problem is widespread;soluions have been hard o come by.

A undamenal source o he problem is he cos disease, based on he handicranaure o educaion wih is atendan lack o opporuniies or gains in produc-iviy, which one o he auhors o his repor [Bowen] promulgaed in he 1960s,in collaboraion wih William J. Baumol. Bu he ime may (nally!) be a hand

when advances in inormaion echnology will permi, under he righ circum-sances, increases in produciviy ha can be ranslaed ino reducions in he

6 See Remarks by the President in State of the Union Address, January 24, 2012 (transcript available at (http://

www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-address). Three days later,

Obama spoke about college af fordability at the University of Michigan (transcript available at http://www.

whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michigan).

This speech does not, however, contain more details concerning how affordability is to be measured or

what penalties are to be imposed on those who fail to pass the requisite tests. As Molly Broad, president of

the American Council on Education, said after the speech: The devil is in the [unspecified] details (Mixed

Reviews of Obama Plan to Keep Down College Costs, January 28, 2012, New York Times, http://www.nytimes.

com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.html).7 According to the College Boards 2011 Trends in College Pricing Report (http://trends.collegeboard.org/

downloads/College_Pricing_2011.pdf), tuition at public two-year universities increased, on average, by

8.7% relative to the 2010-2011 academic year, and tuition at public four-year institutions for the 2011-2012

academic year increased, on average, by 8.3% for instate students and by 5.7% for out of state student s.

In keeping with the trend over the previous four years, students attending private institutions experienced

smaller percentage increases (4.5% for private not-for-profit four-year institutions and 3.2% for private for-

profit institutions).

8 See Scott Jaschik, Mixed Grades: A Survey of Provosts, Inside Higher Education, January 25, 2012, http://

www.insidehighered.com/news/survey/mixed-grades-survey-provosts.

Higher education, especially in the

public sector, is increasingly short

of resources. States continue to

cut back appropriations in the

face of scal constraints and

pressures to spend more on

other things, such as healthcare and retirement expenses.
http://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-address


8/53


cos o insrucion.9 Greaerand smareruse o echnology in eaching iswidely seen as a promising way o conroll ing coss while also reducing achieve-men gaps and improving access. Te exploding growh in online learning isoen cied as ev idence ha, a las, echnology may oer pahways o progress.10Online learning is seen by a growing number o people as a way o breakingree o cenury-old rigidiies in educaional sysems ha we have inheried. Temuch-discussed book on disrupive echnologies and universiies by ClayonChrisensen and Henry Eyring is perhaps he bes example o he atenion beinggiven o online echnologies as a way o changing prooundly he way we educaesudens.11

Tere are, however, also concerns ha a leas some kinds o online learning arelow qualiy and ha onl ine learning in general de-personalizes educaion.In thisregard, it is crit ically important to recognize issues o nomenclature: online learningis hardly one thing. I comes in a dizzying variey o favors, ranging rom simply

videoaping lecures and posing hem or any-ime access, o uploading maeri-als such as s yllabi, homework assignmens, and ess o he Inerne, all he wayo highly sophisicaed ineracive learning sysems ha use cogniive uors andake advanage o muliple eedback loops. Te varieies o online learning can be

used o each many kinds o subjecs o dieren populaions in diverse insiuional

9 Bowens co-author in the promulgation of the cost disease, William J. Baumol, has continued to discuss

its relevance not only for education but also for sectors such as the performing arts and heath care. For

the initial statement of this proposition, see William J. Baumol and William G. Bowen,Performing Ar ts: The

Economic Dilemma, Twentieth Century Fund(1968). In essence, the argument is that in fields such as t he

performing arts and education, there is less oppor tunity than in other fields to improve productivity (by, for

example, substituting capital for labor), that unit labor costs will there fore rise inexorably as these sec tors

have to compete for labor with other sectors in which productivity gains are easier to come by, and that the

relative costs of labor-intensive activities such as chamber music and teaching will therefore continue to

rise. As Bowen argued in his Romanes lecture, for a number of years advances in information technology

have in fact increased productivity, but these increases have been enjoyed primarily in the form of more

output (especially in research) and have generally led to higher, not lower, total costs. (For the text of the

Romanes lecture, see William G. Bowen, At a Slight Angle to the Universe: The Universit y in a Digitized,

Commercialized Age, Princeton University Press, 2001; the tex t is also available on the Andrew W. Mellon

Foundation website: http://www.mellon.org/internet/news_publications/publications/romanes.pdf.)

10 A November 2011 report by the Sloan Consort ium and the Babson Survey Research Group shows that be tween

fall 2002 and fall 2010, en rollment s in online courses increased much more quickl y than total enrollment s

in higher education. During this time period, the number of online course enrollments grew from 1.6 million

to 6.1 million, amounting to a compound annual rate of 18.3% (compared with a rate of 2% for course

enrollments in general)although between fall 2009 and fall 2010 online enrollments grew more slowly,

at 10.1%. More than three of every 10 students in higher education now take at least one course online. Inaddition to the growth in what we call online or hybrid courseshowever nebulous that terminology may

bewe also feel the pervasiveness of the Internet in higher education by the increasing use of it in the

form of course management systems or vir tual reading materials /elect ronic textbooks incorporated into the

curriculum. Even courses that are c alled traditional almost always involve some use of digital resources.

11 See Clayton M. Christensen, and Henry J. Eyring, The Innovative University: Changing the DNA of Higher

Education from the Inside Out, San Francisco: Jossey-Bass, 2011. An October 2, 2011 New York Times op-ed

piece by Bill Keller, aptly titled The University of Wherever, is another illustration of the high visibility and

high stakes of the debate over online education (http://www.nytimes.com/2011/10/03/opinion/the-universi-

ty-of-wherever.html?pagewanted=all).

There are also concerns that at

least some kinds of online learning

are low quality and that online

learning in general de-personalizes

education. In this regard, it is

critically important to recognize

issues of nomenclature: onlinelearning is hardly one thing. It

comes in a dizzying variety of

avors.
http://www.mellon.org/internet/news_publications/publications/romanes.pdfhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.mellon.org/internet/news_publications/publications/romanes.pdf


9/53


setings. A key poin, i an obvious one, is ha here is no one approach ha isrigh or every suden or every seting. In imporan respecs, he online learn-ing markeplace refecs he diversiy o A merican higher educaion isel.12

As resisan as some may si ll be even o hink abou seeking produciv iy gainsin order o reduce eaching coss, here is simply no denying he need o lookmore closely han ever beore a he relaion beween cerain oupus (approxi-maed, or example, by degrees conerred) and inpus (he mix o labor and

capial ha denes educaional producion uncions).13

I is essenial ha helimied resources available o higher educaion be used as eecively as possible.For hese reasons, he research repored here is concerned wih boh educaionaloucomes and coss, seen as wo blades o he scissors.

Organization of This Report

Te nex secion o his repor describes a wo-year eor we have made o esrigorously he learning oucomes achieved by a prooype ineracive learningonline course delivered in a hybrid mode (wih some ace-o-ace insr ucion) onpublic universiy campuses in he Norheas and Mid-Alanic. Beore preseningour ndings, we devoe space o explaining our randomizaion mehodology

boh because he ndings can only be undersood agains he backdrop o he

mehodology and because he research design may be o independen ineres osome readers.14 Tis secionwhich conains he resuls o he main par o ourresearchis ollowed by a brieer discussion o he poenial cos savings hacan conceivably be achieved by he adopion o hybrid-orma online learningsysems. We explain why we avor using a cos simulaion approach o esimaepoenial savings, bu we relegae o Appendix B he highly provisional resuls

we obained by employing one se o assumpions in a cos simulaion model. Weend he main body o he repor wih a shor conclusion ha considers barrierso he adopion o online learning sysems ha are ruly ineracive, seps hamigh be aken o overcome such barriers, and he need o ake a sysem-wideperspecive in addressing hese exremely imporan issues.

12 As Henry Bienen (president emeritus of Northwestern and chairman of the board of Rasmussen College, a

for-profi t university, as well as chairman of ITHAKA) point s out, for many institutions seeking to address the

needs of adult learners and others who are not candidates for places in traditional colleges and universi-

ties, there is no choice: online education, in some form, is the only way that many people can acquire more

skills and earn a college degree, the returns on which have skyrocketed in the past three decades. But online

education is also increasingly common in colleges and universities that educate traditional students. It is

seen as a revenue-generating force in many institutions, both four-year and two-year and both public and

private. See Barr iers to Adoption of Online Learning Systems in U.S. Higher Educat ion by Bacow et al.13 Some argueand we heartily agreethat the output of higher education has broader dimensions and

includes both research results and also the contribution that the entire system of higher education makes

to the effective functioning of a democratic society. But it will not do to allow emphasis on these larger (and

hard-to-measure) contributions to obscure the need to look carefully, and with a somewhat skeptical eye, at how

effectively institutions utilize resources to achieve straightforward aims such as improving graduation rates.

14 Readers interested in methodology may be especially interested in Appendix C to this report, which contains

a detailed discussion of lessons learned from our experience in carrying out this complicated research

project. We wish only that we had had access to this recitation of what to do and what not to do before we

started on this adventure! We learned many of these lessons the hard way.


10/53


Educational Outcomes in Public Universities

Te rs and mos ambiious par o our research was direced a assessing heeducaional oucomes associaed wih wha we erm ineracive learning onlineor ILO. By ILO we reer o highly sophisicaed, ineracive online coursesin which machine-guided insrucion can subsiue or some (hough no usu-ally all) radiional, ace-o-ace insrucion. Course sysems o his y pe akeadvanage o daa colleced rom large numbers o sudens in order o oer eachsuden cusomized insrucion, as well as allow insrucors o rack sudensprogress in deail so ha hey can provide heir sudens wih more argeed andeecive guidance. As several leaders o higher educaion made clear o us inpreliminary conversaions, absen real ev idence abou learning oucomes here isno possibiliy o persuading mos radiional colleges and universiies, and espe-cially hose regarded as hough leaders, o push hard or he inroducion o ILOechnologies ha begin o subsiue machine-guided insrucion or radiionalorms o eaching in appropriae setings.

We se ou o provide a leas enaive answers o hese quesions:

Can sophisicaed, ineracive online courses be used o mainain or improvebasic learning oucomes (masery o course conen, compleion raes, andime-o-degree)?

Are hese courses as eecive, or possibly more eecive, or minoriy andlow-socioeconomic-saus sudens and or oher groups subjec o sereoypehrea?

Are hey equal ly eecive wih no-so-well-prepared sudens and well-pre-pared sudens?

Are hey equal ly eecive in a var iey o campus setingscommuniy col-leges versus our-year colleges, commuer colleges versus colleges wih more

sudens in residence?

Research Design

In hinking abou research design, we began by looking closely a exisingresearch. Tere have been lierally housands o sudies o online learning, buunorunaely he grea majoriy are decien in one way or a noheroen orreasons beyond he conrol o he principal invesigaors.15 Very ew look direclya he eaching o large inroducory courses in basic elds a major public univer-siies, where he grea majoriy o undergraduae sudens pursue eiher associaeor baccalaureae degrees, presumably because very ew ILO courses have been

15 A detailed summary of existing research has been compiled by our staff (especially Lack); but it is too lengthy

to include here. See Current Sta tus of Research on Online Learning in Postsecondary Education by Bowen

and Lack.

The most ambitious part of our

research was directed at assessing

the educational outcomesassociated with what we term

interactive learning online or

ILO. By ILO we refer to highly

sophisticated, interactive online

courses in which machine-guided

instruction can substitute for some

(though not usually all) traditional,

face-to-face instruction.


11/53


oered in hese setings.16 Very ew o he sudies use randomized assignmenechniques o creae reamen and conrol groups ha can be used o reduceoherwise ubiquious selecion eecs ha make i hard o inerpre ndings.

o overcome hese limiaions, we decided o work wih seven insances o aprooype ILO saisics course a six public universiy campuses (including woseparae courses in wo deparmens on one campus) ha agreed o cooperae ina careully designed research projec uilizing random assignmen echniques.

wo o hese campuses are par o he Sae Universiy o New York (SUNY);wo are par o he Universiy o Maryland; and wo are par o he Ciy Univer-siy o New York (CUNY). Te individual campuses involved in his sudy were,rom SUNY, he Universiy a A lbany and SUNY Insiue o echnology; romhe Universiy o Maryland, he Universiy o Maryland, Balimore Couny andowson Universiy; and, rom CUNY, Baruch College and Ciy College. Teseven courses, wih heir all 2011 enrollmens, are shown in able 1.

We also atemped o include hree communiy colleges in New York andMaryland. We were ulimaely unable o include daa rom hese campuses inour sudy or several reasons. A one o he hree communiy colleges, muliplechanges in leadership compromised he implemenaion o he randomizedresearch proocol. A he second communiy college, a large number o sudyparicipans never ook he course, and among hose who did, almos a quar-er swiched ino a orma dieren rom he one o which hey were randomlyassigned. Addiionally, daa on nal exam and sandardized es scores wereunavailable or a subsanial proporion o his campus sudy paricipans. A hehird communiy college, much o he daa were provided oo lae o incorporaeino our primary analysis. We srongly cauion readers agains assuming ha hendings repored here or our-year colleges necessarily apply o communiy col-leges. Vigorous eors nowihsanding, we were unable o obain hard evidenceon his key quesion.

16 Our focus on students attending public institutions is not meant to denigrate the importance of either the

private non-profit sector or t he for-profit sector. Nor is it meant to denigrate professional programs aimed

at working adults. But it is t he public colleges and universities, which educate more than three- quarters of

undergraduates at degree-g ranting institutions (according to the College Boards 2011 report, cited above),

that face the most consequential challenges in raising attainment rates and closing achievement gaps while

simultaneously reducing costs and rest raining tuition increases.


12/53


TABLE 1. PARTICIPATING COURSES/INSTITUTIONS, FALL 2011

Course Enrollment Study Participants

Institution A 850 97

Institution B 877 229

Institution C 235 92

Institution D 86 16

Institution E, Department 1 337 31

Institution E, Department 2 473 50

Institution F 188 90

Total 3,046 605

Notes: Study participants are students who consented to be in our study and were randomly assigned to a traditional or

hybrid format of the introductory statistics class.

We do no claim ha hese six campuses are a sais ical ly valid sample o evenpublic higher educaion, never mind all o higher educaion. Bu his se o si xdoes include: (a) major urban universiies wih large commuing populaionso sudens, as well as universiies wih more residenial sudens; and (b) largenumbers o minoriy sudens and sudens rom low-socioeconomic-saus

amilies (as shown in ables 2 and 3). Tus, he populaion o insiuions andsudens in he sudy is boh large enough and diverse enough o allow us oexplore mos o he quesions lised above in he conex o our-year publicaioninsiuions.

More specically, his research was designed o es as rigorously as possiblehe learning eeciveness o a paricular ineracive saisics course developeda Carnegie Mellon Universiy (CMU)viewed as a prooype o oher ILO

The population of institutions

and students in the study is both

large enough and diverse enoughto allow us to explore most of

the questions listed above in the

context of four-year publication

institutions.


13/53


courses.17 While he CMU course can be delivered in a ully online environmen,in his sudy i was used in a hybrid mode in which mos o he insrucion wasdelivered hrough he ineracive online maerials, bu he online insrucion

was supplemened by a one-hour-per-week ace-o-ace session in which sudenscould ask quesions or be given a rgeed assisance.

Te exac research proocol varied by campus in accordance wih local policies,pracices, and preerences, and we describe hese proocols in deail in Appendix

able A1, and on Ihaka S+Rs websie where here is a narraive descripion;Appendix able A1 also presens summar y daa on enrollmens and secion sizesin each orma (oen he hybrid-orma secions were somewha smaller han heradiional-orma secions). Te general procedure ollowed was: 1) a or beorehe beginning o he semeser, sudens regisered or he inroducory sais-ics course were asked o paricipae in our sudy, and modes incenives wereoered;18 2) sudens who consened o paricipae lled ou a baseline survey;3) sudy paricipans were randomly assigned o ake he class in a radiional orhybrid orma; 4) sudy paricipans were asked o ake he CAOS es o saisi-cal lieracy19 a he beginning o he semeser; and 5) a he end o he semeser,

17 We prefer the ILO acronym to others, including the OLI acronym used by CMU to stand for Open LearningInitiative. The term ILOfor interactive learning onlineis not specific to CMUs suite of courses, and

ILO emphasizes the interactive fea tures of this kind of online learning. This is in contrast with more com-

mon types of online learning which largely mimic classroom teaching without taking advantage of the unique

online environment to provide added value, that is, anything beyond that which can be achieved in a physi-

cal classroom.

The CMU statistics course (which can be accessed at http://oli.web.cmu.edu/openlearning/) includes textual

explanations of concepts and an inventory o f worked examples and practice problems, some of which require

the students to manipulate data for themselves using a statistical sof tware package. Both the st atistics

course and other courses in the OLI suite were originally intended to be comprehensive enough to allow stu-

dents to learn the material independently without the guidance of an instructor; since it was developed, how-

ever, the statistics course has been used at a variet y of higher education institutions, sometimes in a hybrid

mode. (Taylor Walsh describes the history of the development of this course, which was financed largely by

the Hewlett Foundation over a number of years, in her 2010 book Unlocking the Gates: How and Why Leading

Universities Are Opening Up Access to Their Courses, Princeton University Press, 2010.) Among the main

strengths of the CMU sta tistics course is its ability to embed interac tive assessments into each instructional

activity, and its three key feedback loops: system to student, as the student answers questions; system to

teacher, to inform student-instructor interactions; and system to course developer, to identif y aspects of the

course that can be improved. In addition to offering assessments to measure how well students understand a

particular concept, the CMU course also asks students to complete self-assessments, to give the instruct or

and/or learning scientists a sense of how well students think they understand the concept. However, while

instructors can delete and re-order modules, CMU does not offer much opportunity for customization, nor is

the course adaptive in terms of redirecting students to ex tra practice sessions or additional reading if their

incorrect answers indicate that they do not understand a concept and need more help. Thus, although the

CMU statistics course is cert ainly impressive, we refer to it as a prototype because we believe it is an early

representative of what will likely be a wave of even more sophisticated systems in the not-too-distant future.18 See Appendix A for a description of the research protocol and incentives used on each campus.

19 The CAOS test, or Comprehensive Assessment of Outcomes in Statistics, is a 40-item multiple-choice

assessment designed to measure students statist ical literacy and reasoning skills. One characteristic of

the CAOS test is that (for a variet y of reasons) scores do not increase by a large amount over the course

of the semester. Among students in our study who took the CAOS test at both the beginning and end of the

semester, the average score increase was 5 percentage points. For more information about the CAOS tes t,

see https://app.gen.umn.edu/artist/caos.html, or delMas, Robert, Joan Garfield, Ann Ooms, and Beth Chance,

Assessing Students Conceptual Understanding After a First Course in Statistics, 6.2 (2007): 28-58,

accessed July 28, 2010, http://www.stat.auckland.ac.nz/~iase/serj/SERJ6(2)_delMas.pdf.
http://oli.web.cmu.edu/openlearning/https://app.gen.umn.edu/artist/caos.htmlhttp://www.stat.auckland.ac.nz/~iase/serj/SERJ6%282%29_delMas.pdfhttp://www.stat.auckland.ac.nz/~iase/serj/SERJ6%282%29_delMas.pdfhttps://app.gen.umn.edu/artist/caos.htmlhttp://oli.web.cmu.edu/openlearning/


14/53


sudy paricipans were asked o ake he CAOS es o saisical lieracy again,as well as complee anoher quesionnaire. Appendix able A2 provides henumbers o sudens on each campus who were randomized ino each ormaand who compleed each daa collecion insrumen.

Adminisraive daa on paricipaing and non-paricipaing sudens were gah-ered rom he paricipaing ins iuions daabases. Te baseline survey admin-isered o sudens included quesions on sudens background characerisics,

such as socioeconomic saus, as well as heir prior ex posure o saisics andhe reason or heir ineres in possibly aking he saisics course in a hybridorma. Te end-o-semeser survey asked quesions abou heir experiences inhe saisics course. Sudens in sudy-aliaed secions o he saisics courseook a nal exam ha included a se o iems ha were idenical across al l heparicipaing secions a ha campus (or, in he case o he campus ha had wodeparmens paricipaing in he sudy, all par icipaing secions in ha depar-men). Te scores o sudy paricipans on his common porion o he exam

were provided o he research eam, along wih background adminisraive daaand nal course grades o all sudens (boh paricipans and, or comparisonpurposes, nonparicipans) enrolled in he saisics course in he all 2011

semeser. All o hese daa are described in deail on he Ihaka S+R websie,which also includes copies o he survey insrumens.

Our inenion was o provide a rigorous side-by-side comparison o speciclearning oucomes or sudens in his hybrid version o he saisics course andcomparable sudens in a radiionally-augh version o he same course. Werecognize, however, ha while we were reasonably successul in randomizingsudens beween reamen and conrol groups (see documenaion in he nexsecion o his repor), we could no randomize insrucors in eiher group andhus could no conrol or dierences in eacher qualiy.20 Tis is one reason,among ohers, ha we do no regard he research design o his projec as

20 Instructor surveys reveal that, on average, the instructors in traditional format sections were much more

experienced than their counterparts teaching hybrid-format sections (median years of teaching experience

was 20 and 5, respectively). Moreover, almost all of the instructors in the hybrid-format sect ions were using

the CMU online course for either the first or second time, whereas many of the instructors in the traditional-

format sect ions had t aught in this mode for years. The experience-advantage, therefore, is clearly in favor

of the teachers of the traditional-format sections. The questionnaires also revealed that a number of the

instructors in hybrid-format sections began with negative perceptions of online learning. In part for these

reasons, a leader of one of the sets of institutions in this study believes that results for the hybrid-format

sections would be improved vis--vis results in the traditional-format sections if the exper iment were

repeated and instructors in the hybrid-format sections were bet ter motivated and bet ter trained. But this is,

of course, a conjecture.

Our intention was to provide a

rigorous side-by-side comparison

of specic learning outcomesfor students in this hybrid

version of the statistics course

and comparable students in a

traditionally-taught version of the

same course. However, while we

were reasonably successful in

randomizing students between

treatment and control groups, we

could not randomize instructorsin either group and thus could not

control for differences in teacher

quality.


15/53


anyhing close o perec.21 Sill, his is he rs eor o which we are awareo carr y ou he kind o randomized sudy o oucomes in large inroducorycourses on public universiy campuses ha we hink has been needed.

One wise decision we made was o conduc spring-erm pilos on as many cam-puses as possible in advance o he al l-erm 2011 research phase o he sudy

when we reaed oucomes as suiable or measuremen. Te spring-erm pilosidenied a number o pracical aspecs in which he sudy could be improved,

and a memo on lessons learned rom he spring-erm pilos is included in hisrepor as Appendix C.22

I remains only o add ha, as Appendix C illusraes, his is very diculresearch o do, in large par because so many deailshow bes o presenhe course, o recrui suden and aculy paricipans, o randomize sudens

beween reamen and conrol groups, o collec good daa including back-ground inormaion abou he suden paricipans, and o saisy InsiuionalReview Board requiremens in a imely wayneed o be worked ou wih heday-o-day involvemen o campus sa no direcly responsible o us. We havegrea respec or oher invesigaors who have coped wih hese problems, oenin setings ouside higher educaion.

Findings

Te grea advanage oindeed, he main moivaion orconducing a ran-domized experimen is ha sudens in he reamen and conrol groups areexpeced o have he same average characerisics, boh observed and unob-served. Te resuls in able 2 indicae ha he randomizaion worked properly inha radiional and hybrid-orma sudens in ac have similar characerisics.Tere are a handul o small dierences ha are saisically signican bu, ingeneral, he dierences beween sudens augh in he radiional orma andsudens augh in he hybrid orma are no meaningul.23

21 Randomization procedures were limited by the fact that Institutional Review Board (IRB) requirements

precluded randomization of students enrolled in the course without their consent. Instead, we had first to

use incentives to encourage students to participate in the study, with the understanding that they would then

be randomized between trea tment and control groups. We were able, however, to compare the character-

istics of participants and non-participants, and the two groups turned out to be very similar; see Table 3.

The study is, o f course, limited in that it involves only a single course, but having a common hybrid course

across the six campuses (i.e. the CMU statistics course) controls for one source of variance in outcomes.

We deliberately chose the CMU statistics course because we think that the greatest near-term opportunity

to take advantage of interact ive online technologies is in introductory-level courses that serve large student

populations in fields in which there is more or less one right answer to most questions. Somewhat dif ferent

pedagogies would be needed, we suspect, in courses that are more value-laden and dependent on discussion

of various perspectives.22 We are indebted to James Kemple, now Executive Director of the Research Alliance for New York City Public

Schools, and formerly the Director of the K-12 Education Policy division at MDRC, for much useful advice. Dr.

Kemple has long experience with randomized t rials. Lessons learned from the pilots included how to present

the project, the ef fective use of modest incentives for par ticipants, and techniques that could improve ran-

domization. We hope that others will benefit from our experience (see Appendix C) in mounting this research

project.

23 A regression of format assignment on all of the variables listed in Table 2 (and institution dummies) fails to

reject the null hypothesis of zero coefficients for all variables (except the institution dummies) with =0.12.

A Hotelling test fails to rejec t the null of no difference in means with =0.27.


16/53


In addiion o esing he success o our eors o randomize sudens, able 2also serves o describe he populaion o sudens who paricipaed in our sudy.Tey are a very diverse group. Hal o he sudens come rom amilies wihincomes less han $50,000 and hal are rs-generaion college sudens. Fewerhan hal are whie, and he group is abou evenly divided beween sudens wihcollege GPAs above and below 3.0. Mos sudens are o radiional college-goingage (younger han 24), are enrolled ull-ime, and are in heir sophomore or

junior year.

Tese sudens are a diverse group, bu do hey resemble he enire populaiono sudens enrolled in he inroducory sais ics courses included in our sudy?Sudy paricipans were randomly assigned o a secion orma, bu he sudyparicipans hemselves are a sel-seleced populaionbecause o InsiuionalReview Board requiremens only sudens who agreed o be in he sudy wererandomly assigned, and scheduling complicaions also l imied he populaiono paricipans. Overall, 605 o he 3,046 sudens enrolled in hese saisicscourses paricipaed in he sudy. An even larger sa mple size would have beendesirable, bu he logisical challenges o scheduling a leas wo secions (onehybrid secion and one radiional secion) a he same ime, so as o enable

sudens in he sudy o atend he saisics course regardless o heir (random-ized) orma assignmen, resriced our prospecive paricipan pool o helimied number o paired ime slos available. Also, as already noed, Insi-uional Review Boards required suden consen in order or researchers orandomly assign hem o he radiional or hybrid orma. No sur prisingly, somesudens who were able o make he paired ime slos eleced no o paricipaein he sudy. All o hese complicaions nowihsanding, our nal sample o 605sudens is by no means smalli is in ac quie large in he conex o his ypeo research.24

24 Of the 46 studies examined in the Means et al. (2009) meta-analysis, only 5 had sample sizes of over 400,

and of the 51 independent effect sizes the authors abstract ed, 32 came from studies with fewer than 100

study participants.

The students who participated

in our study are a very diverse

group. Half of the students come

from families with incomes less

than $50,000, and half are rst-

generation college students. Fewer

than half are white.


17/53


TABLE 2. RANDOMIZATION OF STUDY PARTICIPANTS

Traditional Hybrid Adj. Diff. Sig?

Male 46% 39% -7% +

Asian 24% 23% -1%

Black 14% 14% 0%

Hispanic 20% 14% -5% +

White 41% 46% 4%

Other/Missing 1% 3% 2%

Average Age 21.9 22.0 0.0

Age


18/53


TABLE 3. STUDENT CHARACTERISTICS BY STUDY PARTICIPATION

Participant Non-Part. Adj. Diff. Sig?

Male 42% 44% -1%

Asian 23% 17% 1%

Black 14% 13% 0%

Hispanic 17% 10% 3% *

White 44% 47% 6% *

Other/Missing 2% 13% -10% **

Average Age 21.9 21.6 -0.3

Age


19/53


Te resuls in able 3 indicae ha he 605 sudy par icipans, while no ul lyrepresenaive o all saisics sudens in any ormal sense, have broadly simi-lar characerisics. Tere are sais ically signican dierences beween sudyparicipans and non-paricipans on several characerisics, bu mos o hedierences are small in magniude. For example, paricipans are more likelyo be enrolled ull-ime, bu only by a margin o 90 versus 86 percen. Courseoucomes are also broadly similar, wih par icipans earning similar grades and

being only sl ighly less likely o complee and pass he course as compared onon-paricipans.

Our analysis o he daa is sraighorward; we compare he oucomes o su-dens randomly assigned o he radiional orma o he oucomes o sudensrandomly assigned o he hybrid orma. In a small number o cases4 perceno he 605 sudens in he sudyparicipans atended a dieren orma sec-ion han he one o which hey were randomly assigned. In order o preservehe randomizaion procedure, we associaed sudens wih he secion y pe o

which hey were randomly assigned. Tis is someimes called an inen o reaanalysis. Under cerain assumpions, he eec o acually ak ing he course inhe hybrid orma (as opposed o jus being randomly assigned o do so) can be

calculaed by increasing our esimaes by 4 percen.25

Tis is someimes calledhe reamen on he reaed esimae, which in our sudy is very similar o heinen o rea esimae because mos sudens ook he course in he orma o

which hey were randomly assigned.

How did learning oucomes compare across he reamen and conrol groups?We rs examine he impac o assignmen o he hybrid orma, rela ive o heradiional orma, in erms o he rae a which sudens compleed and passedhe course, heir perormance on a sandardized es o saisics (he CAOSes), and heir score on a se o nal exam quesions ha were he same in hewo ormas.26 Our main resuls are summarized in Figure 1 (page 19), and heregression resuls are repored in Appendix able A3.27 We nd no saisicallysignican dierences in learning oucomes beween sudens in he radiional and hybrid-orma secions. Hybrid-orma sudens did perorm sl ighly beterhan radiional-orma sudens on hree oucomes, achieving pass raes ha

25 The key assumption is that being randomly assigned to hybrid or traditional did not have an effect on student

outcomes independent of its effect on the format in which students were enrolled. This assumption would

be violated if, for example, students hoped for a cer tain outcome of the random assignment and were disap-pointed when they did not get their preferred assignment, which in turn caused them to do worse in

the course.

26 All of our results control for course-specific dummy variables, since students were randomized within cours-

es; these variables also control for unobserved student characteristics that are constant within institutions.

However, we obtain similar results when we do not control for institution dummies, as would be expected

given that the probability of being assigned to the hybrid sec tion was constant across courses (50%).

27 Note that the pass rate in Figure 1 and Appendix Table A3 cannot be used to calculate the percentage of stu-

dents who failed the course because the non-passing group includes students who never enrolled or withdrew

from the course without receiving a grade.


20/53


wereabou hree percenage poins higher, CAOS scores abou one percenagepoin higher, and nal exam scores wo percenage poins higherbu none ohese dierences passes radiional ess o saisical signicance.28

Figure 1. Effect of Hybrid Format on Student Learning Outcomes

0 10 20 30 40 50 60 70 80%

Traditional

Hybrid

PassRate

CAOSPost-Test

FinalExam

54.7%

56.7%

76.4%

79.7%

46.9%47.5%

Notes: None of the traditional-hybrid differences above were statistically signicant at the 10% level. See Appendix Table

A3 for more information about the results depicted here.

I is imporan o repor ha hese dierences (or raher, he lack o saisicallysignican dierences) are ai rly precisely esimaedsee boh he acual coe-ciens and he small sandard errors o he eec es imaes repored in Appen-dix able A3.29 In oher words, we can be quie conden ha he averageeecs were in ac close o zero. As we explain shorly, we also nd ha he same

basic resuls hold or subgroups, and ha disribuions o key oucomes are verysimilar or boh he reamen and conrol group sudens. One commenaor,Michael S. McPherson, presiden o he Spencer Foundaion, observed ha wha

we have here are quie precisely esimaed zeros.

Ta is, i here had in ac been pronounced dierences in oucomes beweenradiional-orma and hybrid-orma groups, i is highly likely ha we would

have ound hem.30

Our ndings are srik ingly dieren in his consequenialrespec rom a hypoheical nding o no signican dierence which resuledrom a coecien o some magniude accompanied by a very large sandard erroror by big dierences in he disr ibuions o oucomes. A hypoheical nding ohis k ind would mean, in eec, ha we don know much: ha he rue resulscould be almos anywhere.

28 The effect on CAOS test scores in standard deviation units (using the distribution of the control group) was

0.05. We also examined performance using item-level CAOS post-test data. Specifically, we grouped the 40

items into the 20 items on which delMas et al.s (2007) national sample of students exhibited significant

growth (over the course of a semester) and the remaining 20 items. We found similar hybrid-format effec tsfor the two g roups of items.

29 We cluster standard errors by section in order to capture section-specific shocks to student outcomes (such

as the quality of the instructor). Students who were randomly assigned but never enrolled in the course are

grouped as a section within each course for t he purpose of computing clustered st andard errors.

30 Some degree of caution is warranted in interpreting the results for the CAOS post-test because the average

students CAOS score only increased by five percentage points over the course of the semester (among

students who took both the pre-test and the post-test ). This may have resulted in part from some students

not taking the CAOS test seriously because, in most cases, it was not part of their grade. However, it is

reassuring that the results for the CAOS test are consistent with results for pass rates and final exam scores.

In other words, we can be quite

condent that the average effects

were in fact close to zero. We alsond that the same basic results

hold for subgroups, and that

distributions of key outcomes are

very similar for both the treatment

and control group students.


21/53


Tese ndings conrol or suden characerisics, including race/ehniciy,gender, age, ull-ime versus par-ime enrollmen saus, class year in college,parenal educaion, language spoken a home, and amily income. Tese conrolsare no sricly necessary since sudens were randomly assigned o secion or-ma, bu we include hem in order o increase he precision o our resuls and oconrol or any remaining imbalance in observable characerisics. However, weobain nearly idenical resuls when we do no include hese conrol variables

jus as we would expec given he apparen success o our random assignmenprocedure.

Our resuls are also robus o a variey o alernaive mehodologies used oanalyze he experimenal daa. Tese resuls are repored in Appendix able A4,and one is worh highlighing. A limiaion o our main resuls or CAOS pos-es and nal exa m scores is ha we only observe hese oucomes or sudens

who compleed he course and ook hese exams. Tis is unlikely o be a signi-can limiaion given ha we do no nd any signican eecs o secion ormaon course compleion and pass raes. Bu as an addiional check, we assignedsudens or whom we did no observe a CAOS pos-es score heir score on heCAOS pre-esin oher words, we assumed ha heir score did no change over

he course o he semeser. Sudens who did no ake eiher he pre-es or hepos-es were assigned he average pre-es score a heir insiuion. Te resul-ing se o real and impued pos-es scores yielded very similar resuls o hoseobained using only he real daa.

Te lack o dierences in mean oucomes beween ormas could mask dier-ences in he disribuion o oucomes.31 Figure 2 (page 21) shows ha his is nohe case or CAOS pos-es scores. Te disribuions o scores or radi ionaland hybrid orma sudens are largely similar, alhough scores are sl ighly morespread-ou or hybrid-orma sudens. We obain a similar nding or nal examscores (no shown).32 (Tis kind o comparison o disribuions is no possible orpass raes, which only ake on a value o 0 or 1 or an individual suden.)

Resuls broken down by individual insiuion (Appendix able A5) do noreveal any noeworhy paterns. Tese resuls are much noisier because heyare based on smaller numbers o sudens, bu hey do no indicae ha hehybrid orma was paricularly eecive or ineecive a any individual insiu-ionwih he possible excepion o Insiuion F, where coeciens are posiiveacross all our oucomes, alhough only saisically signican in he case o oneoucome.

31 We are indebted to Stephen Stigler, a professor at the University of Chicago and a member of the ITHAKA

board, for emphasizing to us the importance of considering this question.

32 In general, results that use final exam scores should be interpreted cautiously given limitations in these

exams and their implementation. Some institut ions included only a handful of questions that were common

across the sections of t he course (and we only use data from the common questions). At one institution,

common questions were administered to some students after the end of the semester because the actual

final exam only included common questions in two ou t of six sect ions. At another institution, f inal exam data

were not available for the students of two instructors (covering three out of six traditional-format sections).

Excluding either or both of these institutions produces similar results, but it should be noted that the ef fect

on final exams in standard deviation units is substant ial in size (0.19; see Appendix Table A4) and imprecisely

estimated.

Results broken down by

individual institution do not

reveal any noteworthy patterns.


22/53


Figure 2. Distributions of CAOS Post-Test Scores

Traditional

Hybrid

Traditional

4

Density

CAOS Post-Test Score

Kernel =epanechinov, bandwidth=0.03335

2

0

0 .2 .4 .6 .8

Hybrid

We also calculaed resuls separaely or subgroups o sudens dened in ermso var ious characeris ics, including race/ehniciy, gender, parenal educa-ion, primary language spoken, CAOS pre-es score, hours worked or pay, andcollege GPA. We did no nd any consisen evidence ha he hybrid-orma

eec varied by any o hese characerisics (Appendix able A6).33

Tere wereno groups o sudens ha beneed rom or were harmed by he hybrid ormaconsisenly across muliple learning oucomes.

Our main resuls provide compelling evidence ha, on average, sudens learnedjus as much in he hybrid orma as hey would have had hey insead aken hecourse in he radiional ormawih learning measured in radiional ways,in erms o course compleion, course grades, and perormance on a naional eso saisical lieracy. Tis seemingly bland resul is in ac very imporan, in l igho perhaps he mos common reason given by aculy and deans or resising heuse o ILO-ype insr ucion: We worry ha basic suden learning oucomes(pass raes and masery o conen) will be hur, and we won expose our su-

dens o his risk . Te research repored here suggess srongly ha such worriesare no well ounded.

We also examined how much sudens liked he hybrid orma o he course rela-ive o radiional orma sudens (Figure 3 [page 22] and Appendix able A7).

We ound ha sudens gave he hybrid orma a modesly lower overal l rainghan he one given by sudens aking he course in radiional orma (he raing

was abou 11 percen lower). By similar margins, hybrid sudens repored eel ing

33 The one exception is our finding that completion and pass rates were significantly higher in the hybrid course

for students with family incomes of at least $50,000 per year, but not for students with family incomes of

less than $50,000 per year. However, we hesitate to attach much significance to this result given that we do

not find such a patt ern for our other measure of socioeconomic status (parental education) or for measures

of academic preparation. This finding could be the result of random noise in the coef ficients (especially given

the large number of coefficients reported in Appendix Table A6).


23/53


ha hey learned less and ha hey ound he course more dicul.34 Tese hreedierences, hough modes in size, were saisically signican a he 10 percenlevel. Bu here were no saisically signican dierences in sudens repors ohow much he course raised heir ineres in he subjec mater.

Figure 3. Effect of Hybrid Format on Student Evaluations of Course

0.0

0.5

1.0

1.5

2.0

2.5

DifficultyAmountLearned

RaisedInterest

OverallRating

Traditional

Hybrid

Raiting(0

4)

2.3

2.0+

2.0+

2.5+

1.7 1.7

2.22.3

Notes: + indicates traditional-hybrid difference is statistically signicant at the 10% level. See Appendix Table A7 for moreinformation about the results depicted here.

A leader o one o he universi ies ha ac ively paricipaed in our sudyopined ha a deec o he CMU prooype course is ha i has no addiciveor Disney-like appeal; i was, as his person pu i, designed by cogniivescieniss (no oense inended!). In conras, some sudens in he radiionalorma may have been reaed o an occasional colorul sory, personal recol-lecions o he insrucor, or oher reamens someimes used by aculy oimprove sudens opinions o heir course.35

We also asked sudens how many hours per week hey spen outside o class

working on he sais ics class. Hybrid-orma sudens repored spending 0.3hour more each week, on average, han radiional-orma sudens. Tis dier-

34 Students responses to the open-ended questions on the end-of-semester surveys indicate that many

students in the hybrid format would have liked more face-to-face time with the instructor than one hour

each week; others felt that the instructor could have better used t he face-to-face time to make the weekly

sessions more structured and/or helpful in explaining the material and going over concepts students did not

understand. A number of students in the hybrid course also indicated they would have benefited from more

practice problems or examples, and many were frustrated by the dif ficulty of checkpoint assessments in the

course and by problems they encountered using the statistical software packages to complete assignments.

35 The question of what is really going on herewith no differences in learning outcomes as measured con-

ventionally combined with a (to be sure, small) difference in qualitative assessmentsrelates t o a larger

literature on measured learning outcomes versus more subjective measures of student satisfaction with

online or hybrid courses relative to their satis faction with face-to-face courses. Studies (some more rigorous

than others) pertaining to the lat ter topic abound; a few examples include: Hannay, Maureen, and Tracy

Newvine. Perceptions of Distance Learning: A Comparison of Online and Traditional Learning. 2.1 (2006):

1-11. Accessed April 24, 2012. http://jolt.merlot.org/documents/MS05011.pdf; Horspool, Agi, and Carsten

Lange. Applying the Scholarship of Teaching and Learning: Student Perceptions, Behaviours and Success

Online and Face-to-Face. 37.1 (2011): 73-88. Accessed April 24, 2012. http://www.tandfonline.com/doi/abs

/10.1080/02602938.2010.496532; and Meyer, Katrina A. Student Perceptions of Face-to-Face and Online

Discussions: The Advantage Goes To . .4: 53-69. Accessed April 24, 2012. http://sloanconsortium.org/

jaln/v11n4/student-percept ions-face-face-and-online-discuss ions-advantage-goes.

Our results indicate that hybrid-

format students took about

one-quarter less time to achieve

essentially the same learning

outcomes as traditional-format

students.
http://jolt.merlot.org/documents/MS05011.pdfhttp://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://jolt.merlot.org/documents/MS05011.pdf


24/53


ence, which is no saisically signican, implies ha, in a course where a radi-ional secion mees or 3 hours each week and a hybrid secion mees or 1 hour,he average hybrid-orma suden would spend 1.7 less hours each week in totaltime devoed o he course, a dierence o abou 25 percen. Tis resul is con-sisen wih oher evidence ha ILO-ype ormas do succeed in achieving hesame learning oucomes as radiional-orma insrucion in less imewhichpoenially has imporan implicaions or scheduling and he rae o coursecompleion.36

In sum, our resuls indicae ha hybrid-orma sudens ook abou one-quarerless ime o achieve essenially he same learning oucomes as radiional-ormasudens. Te hree main limiaions o his analysis are: (1) we were no able orandomly assign insrucors o secion ormaswhich would have been di-cul, i no impossible, o do, especially in a small scale sudy, or o conrol ordierences in how radiional-orma secions were augh;37 (2) he limiaionso he CMU prooype o an ILO courseno cusomizaion and no addiciveeaures; and (3) our inabiliy o repor resuls or communiy colleges. Despiehese limiaions, hese resuls refec a ser ious, rigorous, assessmen o he rela-ive ecacy o echnology-enhanced learning (ILO-syle hybrid insrucion)

compared o he radiional mode o insr ucion. Tey are, we believe, he besevidence available o dae on an imporan se o quesions. Tere is, wihoudoub, much more research ha can and should be carried ou bu, a he mini-mum, his sudy suppors a no-harm-done conclusion regarding a leas onecurren prooype o an ILO sysem.

36 The authors of this paper, interested to see whether the hybrid course might enable students to learn the

material in the statistics course in a shorter period of time, conducted a separate, quasi-experimental study

in the summer of 2011, involving one of the campuses used in our larger fall 2011 study. This summer probe

occurred over two shortened (five or six-week) summer sessions; while we collected a large amount of

data and used numerous controls in our analysis, we did not randomly assign st udents to the hybrid or theface-t o-face format so that we could obtain larger sample si zes (f rom what was already a much smalle r pool

of students taking the course during the summer). The results of the summer study revealed no statistically

significant differences in the percentage of students who passed the course, in final exam grades, or in the

end-of-semester CAOS test results, between the students who took the course in a hybrid format and stu-

dents who took the course in a face-to-face format. A separate st udy by Marsha Lovett, associate director of

the Eberly Cente r for Teaching Excellence, and her colleagues at CMU produced similar findings. In that study,

which was conducted at CMU, the performance of students who were randomly assigned to the face-to-face

format of a s tatis tics course, which met four times a week for 50 minutes each time, was compared with

the performance of students who were randomly assigned to a hybrid format, which met twice a week for 50

minutes each time. The researchers found little difference in the amount of time students reported spending

on the course outside of class each week (about 2.8 hours for the hybrid-format student s, compared with

about 2.7 hours per week for face-to-face s tudents). In addition, students also were able t o learn in eight

weeks the same amount of material that students would ordinarily take 15 weeks in a face-to-face formatto learn. In this respect, the use of the CMU course could be said to increase learning ef ficiency. (For more

information about this study, see Lovett, Meyer, and Thilles 2008 art icle in the , entitled The Open Learning

Initiative: Measuring the effectiveness of the OLI s tatistics course in accelerating student learning, available

online at http://jime.open.ac.uk/2008/14.)

37 One commentator, Michael McPherson, noted that right now, quite apart from any use of online technologies,

very different instructional methods are used to teach introductory statist ics courses even within the same

universitynever mind across universities. But little effor t appears to be made to compare learning out-

comes and cost effec tiveness across different approachesa point Derek B ok, former president of Harvard

who continues to write on this subject, keeps emphasizing.
http://jime.open.ac.uk/2008/14http://jime.open.ac.uk/2008/14


25/53


26/53


amorized over ime. Tere are also ransiion coss enailed in moving rom heradiional, mosly ace-o-ace model (ha may, however, employ some elemenso simple online models, such as video-aped lecures or homework assign-mens online) o a hybrid model ha akes advanage o more sophisicaed ILOsysems employing machine-guided insrucion, cogniive uors, embeddedeedback loops, and some orms o auomaed grading. Insrucors need o berained o ake ull advanage o such sysems. Tere may also be conracual lim-is on secion size ha were designed wih he radiional model in mind bu hado no make sense or a hybrid model. I is possible ha hese consrains can bechanged in a nex round o conrac negoiaions, bu ha oo wi ll ake ime.

o overcome (or avoid!) hese problems, we hink here is much o be said orcarrying ou simulaed cos probes, and we made a very rough atemp o do

jus his on hree o he campuses included in he lea rning oucomes par ohe sudy. We concepualize he research quesion here no as how much willinsiuions save righ now by shiing o hybrid learning? bu raher as under

wha assumpions wi ll cos savings be realized, over ime, by shiing o a hybridorma, and how large are hose savings likely o be? Our basic approach waso sar by looking in as much deail as possible a he acual coss o eaching a

basic course in radiional orma (usually, bu no always, he saisics course)in a base year. Ten, we worked wih leaders on hese campuses o simulae heprospecive, seady-sae coss o a hybrid-online version o he same course,looking hree o ve years ino he uure. Tese ex ploraory simulaions were

based on explici assumpions, especially abou sang, ha were incorporaedino spreadsheeswhich in urn allowed us o see how sensiive our resuls

were o variaions in key assumpions. We ocused heavily on personnel coss,because o boh heir imporance and our abiliy o examine hem wih someprecision. Oher coss, including space coss, were also considered. We hopedha he simulaions would, a he minimum, give us a leas a rough sense o hepoenial impac on coss o inroducing hybrid learning and, more specically,

show us how much room here would be or insiuions o share cos savingswih acul y and sudens on a coninuing basis.

We ocus on insrucor compensaion because hese coss comprise a subsan-ial porion o he recurr ing cos o eaching and are he mos sraighorwardo measure. Space coss are also an imporan caegory o coss ha are likely o

be reduced by shiing o a hybrid lear ning model (he mos imporan caegoryin some siuaions), bu such coss are more dicul o measure accuraely ahe level o an individual course. A hybrid model also aords boh aculy andsudens signicanly greaer scheduling fexibiliy, a poenially very imporan

bene ha will no be capured by our simulaions. On he oher hand, here arealso oher y pes o coss ha we do no consider here, such as increases in inor-maion echnology (I) suppor coss associaed wih moving a signi can shareo learning aciviies online. Such added coss can be ar rom riv ial.

We did exploraory simulaions or wo ypes o radi ional eaching models: (1)a model in which sudens are augh in secions o roughly 40 sudens per sec-ion; and (2) a model in which all sudens atend a common lecure and are henassigned o small discussion secions led by eaching assisans. We compare hecurren coss o each o hese radi ional eaching models o simulaed coss oa hybrid model in which more insrucion is delivered online, sudens atend

Our simulations illustrate that

hybrid learning offers opportunities

for signicant savings incompensation costs, but that the

degree of cost reduction depends

on the exact model of hybrid

learning used.


27/53


weekly ace-o-ace sessions wih par-ime insrucors, and he course is over-seen by a enure-rack proessor (wih adminisraive responsibiliies delegaedo a par-ime insrucor).

We have decided, however, no o presen he acual calculaions and resuls ohese simulaions in he body o his repor. Tey are oo speculaive and subjeco considerable variaion depending on how a par icular campus waned oorganize is eaching. Te danger o specious precision is grea, and i would be

wrong o atach much imporance o paricular numbers. Suce i o say ha hecrude models we employed sugges savings in compensaion coss ranging rom36 percen o 57 percen in he all-secion model, and 19 percen in he lecure-secion model. Appendix B presens hese resuls and many more calculaionsand gures showing how sensiive poenial savings are as we var y assumpionsabou secion sizes and compensaion.

Tese simulaions illusrae ha hybrid learning oers opporuniies or sig-nican savings in compensaion coss, bu ha he degree o cos reduciondepends (o course) on he exac model o hybrid learning usedespecially herae a which insr ucors are compensaed and secion size. A large share o cossavings is derived rom shiing away rom ime spen by expensive proessors

oward boh machine-guided insrucion ha saves on sang coss overall andoward ime spen by less expensive sa in Q&A setings. O course, enuredproessors canno be laid o in order o realize hese savings and, in any case,orce reducions are no required o save signican amouns o money. Insiu-ions ha ace pressures o expand enrollmen are in an especially good posiiono realize savings by shiing he mix o eaching ime. When more sudens areo be augh, hybrid models make his possible wihou increasing he demandsmade on enured aculy. Recruimen coss may hereby be reduced along wihcompensaion coss per suden, and debaes over mainaining commimenso exising aculy are avoided. Over ime, cerainly, sa size can be aleredhrough atriion. A lso, he ime o proessors can be reallocaed oward smaller,more advanced classeswhich many preer o each (such reallocaions mayno save he insiuion money, hough hey may improve he overall educaionalexperience o many sudens).

In hese simulaions, we have assumed ha he number o sudens in he coursewill remain consan. However, as already suggesed, many insiuions aceincreasing demand or places in heir classes. Te hybrid learning model is veryatracive in such circumsances or wo primary reasons: (a) less space is neededin general; and (b) hybrid courses provide boh sudens and eachers wihgreaer scheduling fexibiliy. Increased enrollmen can a lso lead o increasedcompensaion cos savings (per suden) because he xed coss o he proessor

in charge o he course, and an adminisraive coordinaor, would be spread overa larger number o sudens. For he same reason, he larges savings will be real-ized in courses wih he larges enrollmen, all else equal.

Our simulaion approach underesimaes subsanially he savings rom mov-ing oward a hybrid model in many setings because we do no accoun or spacecoss. We are relucan o pu a dollar gure on space coss because capial cossare dicul o apporion accuraely down o he course level. However, i is moresraighorward o ca lculae he percenage change in he need or classroom


28/53


space ha would resul rom shiing oward a hybrid model. Te hybrid coursemees or one hour each week, whereas he radiional course ypically meesor hree or our hours each week. Consequenly, he hybrid course requires 67percen o 75 percen less classroom use han he radiional course, assumingha he course is augh in secions, ha secion size is held consan, and hahe hybrid course does no have addiional space requiremens o is own, such asaddiional compuer labs.

In he shor run, insiuions canno sell or demolish heir buildings. However,in he long run, using hybrid models or some large inroducory courses wouldallow insiuions o expand enrollmen wihou a commensurae increase inspace cossa major cos savings (cos avoidance) relaive o wha insiuions

would have had o spend had hey sayed wih a radiional model o insrucion.An imporan poin here is ha he hybrid model need no jus save moneyican also suppor an increase in access o higher educaion. I serves he accessgoal boh by making i more aordable or he insiuion o enroll more sudensand by accommodaing more sudens because o greaer scheduling fexibiliy,

which is especia lly imporan or sudens wih complicaed lives who have obalance amily responsibiliies and work wih course compleion, as well as or

sudens who may live a disance rom campus.41

o repea, we regard his highly speculaive cos simulaion eor as primarilyillusraive o an approach ha we believe has meri. I is no surprise ha under aplausible se o assumpions ILO sysems have he poenial o save sang cossand classroom space, and o increase scheduling fexibiliy. o go beyond haobvious saemen requires, a he minimum, deailed knowledge o local campussiuaions and realisic assessmens o wha is easible.

Summary Observations

In he case o a opic as acive as online learning, where new aricles appearevery day and millions o dollars are being invesed by a wide variey o eni-ies, we should perhaps expec ha here wil l be infaed claims o specacularsuccesses. Te ndings in his sudy warn agains oo much hype. o he beso our knowledge, here is no compelling evidence ha online learning sysemsavailable odayno even highly ineracive sysems, o which here are veryewcan in ac deliver improved educaional oucomes across he board, ascale, on campuses oher han he one where he sysem was born, and on a sus-ainable basis. Tis is no o deny, however, ha hese sysems have grea poen-ial. We believe ha hey do, and ha vigorous eors should be made o aggres-sively explore uses o boh he relaively simple sysems ha are prolieraing al l

around us, oen o good eec, and more sophisicaed sysems ha are si ll inheir inancy. Tere is every reason o expec hese sysems o improve over ime,perhaps dramaically, and hus i is no oolish o believe ha learning oucomes

will also improve.

41 The MOOCs (Massive Open Online Courses), like edX and Udacity, are another example of a technologically-

driven effort t o give many more students, of all kinds, access to low-cost instruction of high quality. It

remains to be seen, however, whether the certificates and badges that MOOCs propose to confer will be

accepted as credits toward degrees by mainline universitiesand how much this will matter to students

themselves, as well as to employers and others who want to assess learning outcomes.


29/53


Te barriers o adopion o even he simpler sysems, le alone hose o moresophisicaed sysems ha are ruly ineracive, are deailed in he companionIhaka S+R repor by Lawrence S. Bacow e al. ha has been cied previously.42Tere is no need o summarize hose ndings here. I is sucien o re-empha-size he need o:

a. work closely wih aculy, who undersandably wil l wan o pu a leas some-hing o heir own samp on all such courses;

b. conron direc ly and imaginaively worries abou loss o jobs;

c. encourage serious research by rused hird paries on evidence o learningoucomes;43 and

d. recognize, and even embrace, he need o use such echnologies o increaseproduciviy and lower insrucional coss wihou sacricing learning ou-comes.

Te research repored here demonsraes he poenial o ruly ineracivelearning sysems ha use machine-guided proocols (wha we have been call-ing ILO) o provide some orms o insrucion, in properly chosen courses,

in appropriae setings. Our ndings demonsrae ha such an approach needno negaively impac learning oucomesand conceivably could, in heuure, improve hem as hese sysems become ever more sophisicaed anduser-riendly. I is also enirely possible ha by (poenially) saving signicanamouns o resources, such sysems can lead o more, no less, opporuniyor sudens o bene rom exposure o modes o insrucion such as direcedsudyi scarce aculy ime can be benecially redeployed. Bu none o his will

be easy.

In he spir i o hinking ou loud, here are some houghs as o wha is requiredi we are o ake ull advanage o he poenial o ILO oerings. Tese observa-ions, we should noe, are based less on he specic ndings in his sudy han

on conversaions wih a wide

Documents

Interactive Learning Online At Public Universities