Monitoring MOOCs : Which Information Sources Do Instructors Value?

1

Monitoring MOOCs:Which Information Sources Do Instructors Value?

Kristin Stephens-Martinez,Marti A. Hearst, Armando Fox

UC Berkeley

2

Main Takeaways• Goal: Information display for MOOC instructors

• Survey: What information MOOC instructors value?

• Results:• Quantitative information valued same as smaller online courses

• Assignment grades, demographics, activity patterns

• Qualitative information is important• Forums, student surveys

• Visualization mockups useful while course running and for review

3

Prior Work: Data to Visualize• Performance

• Grades, cumulative concept performance

• Access and Activity Patterns• What, how much, how long, and when content opened• Path through content• Assignment status

• Forum Discussions• Student forum activity• Post meta data• Post structure

• Student Demographics• Location, age, learning style• Course goals

4

Prior Work: How Visualize Data• Standard Graphs

• Scatter plots, bar graphs, line graphs, pie charts, heat maps

• Innovative Visualizations• Node graphs, icons, 3D, timeline spiral graphs

• Interactive Techniques• Sorting• Filtering• Drill down• Clustering

5

Why need a survey?

• Prior work on• Long distance learning• Course management systems

• What about MOOCs?• Massive student body• Open enrollment

“Small” Courses

6

Survey Participants

• Recruit• Coursera• edX• Udacity

• 92 MOOC Instructors• 67 complete survey (73%)

7

Survey Participants

1 2 3 4+ N/A0%

20%

40%

60%

80%

100%

Number of MOOCs Created

Perc

ent o

f Par

ticip

ants

1 2 3 4+ N/A0%

20%

40%

60%

80%

100%

Number of MOOCs Run

94% Run 1 or more MOOCs95% Create 1 or more MOOCs

8

Survey Participants

Coursera edX

Blackb

oard

Moodle

Udacity

Piazza

WebCT

Canva

sOther

N/A0

10

20

30

40

50

60

70

Platforms used by Participants

Num

ber o

f Par

ticip

ants

9

Survey Participants

Students at Course End100s 1,000s 10,000s Total

Engineering 3 10 7 20Science 2 25 10 37Humanities 1 8 10 19Other 1 3 7 11Total 7 46 34 87

10

Course Monitoring Goals• During class• Assignment issues• Find trouble students and what struggle with• Gauge difficulty of exam problem• Course is appropriate difficulty level

• After class• Find most engaging content• Struggling parts of class• TA effectiveness• Improve topic presentation• Least interesting content.

11

Information Sources• Quantitative• Assignment grades• Student answers to problems• Student’s view pattern of online content• Number of times students attempt a problem• Grades for self-check questions

• Qualitative• Discussion forum• Class survey• Discussion with students• Ask the TA• Chat room logs

12

Assignment issues

Find trouble students and what struggle with

Cross: Goals X Info SourcesAssignment issues

Find trouble students and what struggle with

Gauge difficulty of exam problem

Course is appropriate difficulty level

Find most engaging content

Struggling parts of class

TA effectiveness

Improve topic presentation

Least interesting content

Assignment issues

Find trouble students and what struggle with

13

Find trouble students and what struggle with

Assignment grades

Student answers to problems

Student’s view pattern of online content

Number of times students attempt a problem

Grades for self-check questions

Discussion forum

Class survey

Discussion with students

Ask the TA

Chat room logs

Assignment grades

Student answers to problems

Student’s view pattern of online content

Number of times students attempt a problem

Grades for self-check questions

Discussion forum

Class survey

Discussion with students

Ask the TA

Chat room logs

Cross: Goals X Info SourcesAssignment grades

Student answers to problems

Student’s view pattern of online content

Number of times students attempt a problem

Grades for self-check questions

Discussion forum

Class survey

Discussion with students

Ask the TA

Chat room logs

Assignment issuesAssignment grades

Student answers to problems

Student’s view pattern of online content

Number of times students attempt a problem

Grades for self-check questions

Discussion forum

Class survey

Discussion with students

Ask the TA

Chat room logs

Assignment grades

Student answers to problems

Student’s view pattern of online content

Number of times students attempt a problem

Grades for self-check questions

Discussion forum

Class survey

14

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Use Would use if available Do not/Would not use

Find trouble students and what struggle with

Cross: Goals X Info SourcesAssignment grades

Student answers to problems

Student’s view pattern of online content

Number of times students attempt a problem

Grades for self-check questions

Discussion forum

Class survey

Discussion with students

Ask the TA

Chat room logs

Assignment issues

Assignment grades

Student answers to problems

Student’s view pattern of online content

Number of times students attempt a problem

Grades for self-check questions

Discussion forum

Class survey

Use Would use if available Do not/Would not use

15

Value of Information Sources: Across GoalsFo

rum

Clas

s Sur

vey

Stud

ent D

is.

Teac

hing

Ass

ist.

Assg

n. G

rade

Prob

. Ans

wer

View

Patt

erns

Prob

. Atte

mpt

s

Self

Chec

k Q

Chat

Logs

0%

25%

50%

75%

100%Not UseSkipUse/Would Use

16

Value of Information Sources: Across GoalsFo

rum

Clas

s Sur

vey

Stud

ent D

is.

Teac

hing

Ass

ist.

Assg

n. G

rade

Prob

. Ans

wer

View

Patt

erns

Prob

. Atte

mpt

s

Self

Chec

k Q

Chat

Logs

0%

25%

50%

75%

100%Not UseSkipUse/Would Use

17

Value of Information Sources: Across GoalsFo

rum

Clas

s Sur

vey

Stud

ent D

is.

Teac

hing

Ass

ist.

Assg

n. G

rade

Prob

. Ans

wer

View

Patt

erns

Prob

. Atte

mpt

s

Self

Chec

k Q

Chat

Logs

0%

25%

50%

75%

100%Not UseSkipUse/Would Use

18

Value of Information Sources: Across GoalsFo

rum

Clas

s Sur

vey

Stud

ent D

is.

Teac

hing

Ass

ist.

Assg

n. G

rade

Prob

. Ans

wer

View

Patt

erns

Prob

. Atte

mpt

s

Self

Chec

k Q

Chat

Logs

0%

25%

50%

75%

100%Not UseSkipUse/Would Use

19

Value of Information Sources: Across GoalsFo

rum

Clas

s Sur

vey

Stud

ent D

is.

Teac

hing

Ass

ist.

Assg

n. G

rade

Prob

. Ans

wer

View

Patt

erns

Prob

. Atte

mpt

s

Self

Chec

k Q

Chat

Logs

0%

25%

50%

75%

100%Not UseSkipUse/Would Use

Qualitative Quantitative Outlier

20

Value of Information Sources: By Goal

Use/Would Use

Skip

Not Use

F SD AG VPSC

Q0%

25%

50%

75%

100%

TA Effectiveness

F SD AG VPSC

Q0%

25%

50%

75%

100%

Improve Topic Presentation

F SD AG VPSC

Q0%

25%

50%

75%

100%

Least Interesting Content

F SD AG VPSC

Q0%

25%

50%

75%

100%

Struggling Parts of Class

F SD AG VPSC

Q0%

25%

50%

75%

100%

Find most Engaging Content

F SD AG VPSC

Q0%

25%

50%

75%

100%

Course is Appropriate Difficulty Level

F SD AG VPSC

Q0%

25%

50%

75%

100%

Gauge Difficulty of Exam Problem

F SD AG VPSC

Q0%

25%

50%

75%

100%

Find Trouble Students and what Struggle with

F SD AG VPSC

Q0%

25%

50%

75%

100%

Assignment Issues

21

Value of Information Sources: By Goal

Use/Would Use

Skip

Not Use

F SD AG VPSC

Q0%

25%

50%

75%

100%

TA Effectiveness

F SD AG VPSC

Q0%

25%

50%

75%

100%

Improve Topic Presentation

F SD AG VPSC

Q0%

25%

50%

75%

100%

Least Interesting Content

F SD AG VPSC

Q0%

25%

50%

75%

100%

Struggling Parts of Class

F SD AG VPSC

Q0%

25%

50%

75%

100%

Find most Engaging Content

F SD AG VPSC

Q0%

25%

50%

75%

100%

Course is Appropriate Difficulty Level

F SD AG VPSC

Q0%

25%

50%

75%

100%

Gauge Difficulty of Exam Problem

F SD AG VPSC

Q0%

25%

50%

75%

100%

Find Trouble Students and what Struggle with

F CS SD TA AG PAn VP PAtSCQ

Assignment Issues

22

Value of Information Sources: By Goal

Use/Would Use

Skip

Not Use

TA Effectiveness

Improve Topic PresentationLeast Interesting ContentStruggling Parts of Class

Find most Engaging ContentCourse is Appropriate Difficulty

LevelGauge Difficulty of Exam ProblemFind Trouble Students and

what Struggle withAssignment Issues

Qualitative

Quantitative

23

Value of Information Sources: By Goal

Use/Would Use

Skip

Not Use

Qualitative

Quantitative

24

Value of Information Sources: By Goal

Use/Would Use

Skip

Not Use

Qualitative

Quantitative

Qualitative + Quantitative

Qualitative + some View Patterns No Agreement

25

What does qualitative mean?

• Qualitative Data:• Forums• Class Surveys

• Student Discussions• Teaching Assistant

Student Voice

People

26

Visualization Mockups

1. Usefula. (Likert response)

2. Easy to understanda. (Likert response)

3. When use it?a. While the course is runningb. When preparing new materialc. When preparing by reviewing past courses

27

Visualization Mockups

• Majority participants find useful and understandable

• Majority participants use• During class All⇨• Review All, except (quantitative) forum⇨

28

Limitations

• Survey• Some questions ask predict future behavior • (people not necessarily good at)

• Instructors not experienced with all information sources

• Forum• Small sample of student body

29

Other Work: Metrics Tab

Stanford edX Branch

30

Summary• Goal: Information display for MOOC instructors

• Survey: What information MOOC instructors value?

• Results:• Quantitative information valued same as smaller online courses• Qualitative information is important• Visualization mockups useful while course running and for

review

• Future work:• More visualizations• Improve communication of student voice to instructor

31

Spare Slides

Mockup: Boxplot Grades Distribution

Mockup: Stacked Bar Grades Distribution

Mockup: Stacked Bar Attempts Distribution

Mockup: Stacked Bar Views Distribution

Mockup: Line Graph Forum

New Posts

Views