Deciphering the MOOC ecosystem: what makes successful MOOCs

DECIPHERING THE MASSIVE OPEN ONLINE COURSES (MOOCS) ECOSYSTEM: WHAT

MAKES SUCCESSFUL MOOCS?

Ido Roll, Will Engle, Simon Bates!@hummus_monster @infology @simonpbates @CTLT_UBC!

Centre for Teaching, Learning, and Technology!University of British Columbia

1

@hummus_monster @simonpbates @UBC_CTLT #STLHE2014

Please raise your hand if you have pre-teen child(ren) at home.!

Please keep it up if you know where the following sentence is taken from:!

“The snow glows white on the mountain tonight”

2


LEARNING FROM VIDEO !IS POSSIBLE

3

If someone wants to learn, you cannot stop them.!@eric_mazur #STLHE2014


WE KNOW A LOT ABOUT UNIVERSITY TEACHING

What works?!

Small classrooms!

Interactivity!With material!With peers!With instructors!

Feedback!

…

4

MOOCs


THREE KEY QUESTIONS

5

Who are !our learners?

What defines !success?

What course design supports that?


CONTEXT

Four UBC courses (Spring/Summer 2013)!

Climate Literacy; Useful Genetics; Program Design; Game Theory!

330,000 learners registered. !

Five Edinburgh courses (Winter/Spring 2013)!

Introduction to Philosophy; Artificial Intelligence; Astrobiology; Nutrition; Critical Thinking.!

309,000 learners registered.!

https://www.era.lib.ed.ac.uk/handle/1842/6683!

6


VIDEO PRODUCTION

7

Aver

age

leng

th p

er v

ideo

(min

)

0

6

12

18

24

Average # of videos per week0 4 8 12 16

UBC 1UBC 2UBC 3UBC 4


THREE KEY QUESTIONS

8

Who are our learners?


What course design supports that?


WHO ARE OUR LEARNERS?

General demographics consistent across courses and with Coursera.

9

Education level

% re

spon

ses

0

25

50

75

100

UBC 1 UBC 2 UBC 3 Coursera

High School DiplomaProfessional SchoolSome CollegeBachelor’s DegreeMaster’s DegreeDoctorate Degree


WHO ARE OUR LEARNERS?

General demographics consistent across courses and with Coursera.

10

Ethnicity

% re

spon

ses

0

25

50

75

100

UBC 1 UBC 2 UBC 3 Coursera

White or CaucasianSouth AsianEast AsianOther AsianBlack or African AmericanDecline to State


ROLE OF GEOGRAPHY

Does location matter? Are MOOCs “local”?

11

Country of Origin

% re

spon

ses

0

15

30

45

60

UBC overall Coursera Edinburgh

113.63.9

United StatesIndia CanadaUnited KingdomSpainBrazilGermany


ROLE OF GEOGRAPHY

Does location matter? Are MOOCs “local”?

12

Country of Origin

% re

spon

ses

0

15

30

45

60

UBC overall Coursera Edinburgh

43.95.2

United StatesIndia CanadaUnited KingdomSpainBrazilGermany


ROLE OF GEOGRAPHY

Break down by course. weighting by overall registration, population of country:!

How many more students are local, per 10,000 registrants, per capita?!

In Canada, !~150 additional local students!~500 Canadians overall

13

Extr

a lo

cal l

earn

ers

per 1

0,00

0 re

gist

rant

s, 1

,000

,000

resi

dent

s

0

2.5

5

7.5

10

UBC Edinburgh

2.0

1.0

55.7

3.3

5.54.8


THREE KEY QUESTIONS

14


What defines success?

What course design support that?


WHAT DEFINES SUCCESS?

Definitions:!Visitors: log in at least once. Our denominator.

15

Activities

Videos

Website

Week0 1 2 3 4 5 6 7 8


METRICS OF SUCCESS

Definitions:!Visitors: log in at least once. Our denominator.!Tasters: watch a video on Week 2.

16

Activities

Videos

Website

Week0 1 2 3 4 5 6 7 8


METRICS OF SUCCESS

Definitions:!Visitors: log in at least once. Our denominator.!Tasters: watch a video on Week 2.!Learners: watch videos all the way through.

17

Activities

Videos

Website

Week0 1 2 3 4 5 6 7 8


METRICS OF SUCCESS

Definitions:!Visitors: log in at least once. Our denominator.!Tasters: watch a video on Week 2.!Learners: watch videos all the way through. !Active learners: problem sets, discussion forums.

18

Activities

Videos

Website

Week0 1 2 3 4 5 6 7 8


METRICS OF SUCCESS

Definitions:!Visitors: log in at least once. Our denominator.!Tasters: watch a video on Week 2.!Learners: watch videos all the way through. !Active learners: problem sets, discussion forums.

19

Activities

Videos

Website

Week0 1 2 3 4 5 6 7 8


METRICS OF SUCCESS

Metrics:!

Attractiveness: convert Visitors to Tasters.!

Retention: convert Tasters to Learners!

Interactivity: convert Learners to Active Learners.

20

Activities

Videos

Website

Week0 1 2 3 4 5 6 7 8

Visitors

Tasters Learners

Active Learners


METRICS OF SUCCESS

21

0

0.15

0.3

0.45

0.6

Attractiveness Retention Interactivity

48%54%

35%50%

31%29%

52%58%

36% 35%

57%

24%

Activities

Videos

Website

Week0 1 2 3 4 5 6 7 8

Visitors

Tasters Learners

Active Learners

17k 13k 49k 85k 2k 3k 4k 16k


THREE KEY QUESTIONS

22



What course design support that?


WHAT AFFECTS CONVERSION RATES?

Correlate with:!

Average length per video!

Average length per week

23

Activities

Videos

Website

Week0 1 2 3 4 5 6 7 8

Visitors

Tasters Learners

Active Learners


ATTRACTIVENESS

Visitors -> Tasters: How many of those who visited the website watched Week 2 videos?!

By length per week: r = -0.72*, r² = 0.52!

24

% o

f Visi

tors

who

ar

e al

so Ta

ster

s

0%

20%

40%

60%

Overall legth of videos per week (min)0 45 90 135 180

UBCEdinburgh


ATTRACTIVENESS

Visitors -> Tasters: How many of those who visited the website watched Week 2 videos?!

By length per video: r = -0.33!

25

% o

f Visi

tors

who

ar

e al

so Ta

ster

s

0%

20%

40%

60%

Average length per video5 8 11 14 17

UBCEdinburgh


RETENTION

Tasters -> Learners: How many of those who watched @ Week 2 also watched @ Week 5?!

By length per week: r = -0.29!

26

% o

f Tas

ters

who

ar

e al

so L

earn

ers

0%

26.667%

53.333%

80%

Overall legth of videos per week (min)0 45 90 135 180

UBCEdinburgh


RETENTION

Tasters -> Learners: How many of those who watched @ Week 2 also watched @ Week 5?!

By length per video: r = 0.55, r² = 0.30

27

% o

f Tas

ters

who

ar

e al

so L

earn

ers

0%

20%

40%

60%

80%

Average length per video5 8 11 14 17

UBCEdinburgh


INTERACTIVITY

How many of the students who watch videos also submit problem sets?

28

% o

f vid

eo w

atch

ers

who

subm

itted

pr

oble

m s

ets

20%

30%

40%

50%

60%

70%

Week0 2 4 6 8

UBC 1UBC 2UBC 3UBC 4


SUMMARY

Local matters!

Courses have unique “signatures”!Structure, e.g., # and length of videos!

Engagement, e.g., % active learning.!

Overall length of videos is negatively correlated with retention!

But length of individual videos does not show such relationship. !

True for both universities

29


DESIGN RECOMMENDATIONS

Long videos may be okay!

Especially later in the course!

Keep overall load reasonable.!

Active learning is not affected by week-by-week variations!

Two-stage courses: attractiveness and retention.

30