1
Monitoring MOOCs:Which Information Sources Do Instructors Value?
Kristin Stephens-Martinez,Marti A. Hearst, Armando Fox
UC Berkeley
2
Main Takeaways• Goal: Information display for MOOC instructors
• Survey: What information MOOC instructors value?
• Results:• Quantitative information valued same as smaller online courses
• Assignment grades, demographics, activity patterns
• Qualitative information is important• Forums, student surveys
• Visualization mockups useful while course running and for review
3
Prior Work: Data to Visualize• Performance
• Grades, cumulative concept performance
• Access and Activity Patterns• What, how much, how long, and when content opened• Path through content• Assignment status
• Forum Discussions• Student forum activity• Post meta data• Post structure
• Student Demographics• Location, age, learning style• Course goals
4
Prior Work: How Visualize Data• Standard Graphs
• Scatter plots, bar graphs, line graphs, pie charts, heat maps
• Innovative Visualizations• Node graphs, icons, 3D, timeline spiral graphs
• Interactive Techniques• Sorting• Filtering• Drill down• Clustering
5
Why need a survey?
• Prior work on• Long distance learning• Course management systems
• What about MOOCs?• Massive student body• Open enrollment
“Small” Courses
6
Survey Participants
• Recruit• Coursera• edX• Udacity
• 92 MOOC Instructors• 67 complete survey (73%)
7
Survey Participants
1 2 3 4+ N/A0%
20%
40%
60%
80%
100%
Number of MOOCs Created
Perc
ent o
f Par
ticip
ants
1 2 3 4+ N/A0%
20%
40%
60%
80%
100%
Number of MOOCs Run
94% Run 1 or more MOOCs95% Create 1 or more MOOCs
8
Survey Participants
Coursera edX
Blackb
oard
Moodle
Udacity
Piazza
WebCT
Canva
sOther
N/A0
10
20
30
40
50
60
70
Platforms used by Participants
Num
ber o
f Par
ticip
ants
9
Survey Participants
Students at Course End100s 1,000s 10,000s Total
Engineering 3 10 7 20Science 2 25 10 37Humanities 1 8 10 19Other 1 3 7 11Total 7 46 34 87
10
Course Monitoring Goals• During class• Assignment issues• Find trouble students and what struggle with• Gauge difficulty of exam problem• Course is appropriate difficulty level
• After class• Find most engaging content• Struggling parts of class• TA effectiveness• Improve topic presentation• Least interesting content.
11
Information Sources• Quantitative• Assignment grades• Student answers to problems• Student’s view pattern of online content• Number of times students attempt a problem• Grades for self-check questions
• Qualitative• Discussion forum• Class survey• Discussion with students• Ask the TA• Chat room logs
12
Assignment issues
Find trouble students and what struggle with
Cross: Goals X Info SourcesAssignment issues
Find trouble students and what struggle with
Gauge difficulty of exam problem
Course is appropriate difficulty level
Find most engaging content
Struggling parts of class
TA effectiveness
Improve topic presentation
Least interesting content
Assignment issues
Find trouble students and what struggle with
13
Find trouble students and what struggle with
Assignment grades
Student answers to problems
Student’s view pattern of online content
Number of times students attempt a problem
Grades for self-check questions
Discussion forum
Class survey
Discussion with students
Ask the TA
Chat room logs
Assignment grades
Student answers to problems
Student’s view pattern of online content
Number of times students attempt a problem
Grades for self-check questions
Discussion forum
Class survey
Discussion with students
Ask the TA
Chat room logs
Cross: Goals X Info SourcesAssignment grades
Student answers to problems
Student’s view pattern of online content
Number of times students attempt a problem
Grades for self-check questions
Discussion forum
Class survey
Discussion with students
Ask the TA
Chat room logs
Assignment issuesAssignment grades
Student answers to problems
Student’s view pattern of online content
Number of times students attempt a problem
Grades for self-check questions
Discussion forum
Class survey
Discussion with students
Ask the TA
Chat room logs
Assignment grades
Student answers to problems
Student’s view pattern of online content
Number of times students attempt a problem
Grades for self-check questions
Discussion forum
Class survey
14
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Use Would use if available Do not/Would not use
Find trouble students and what struggle with
Cross: Goals X Info SourcesAssignment grades
Student answers to problems
Student’s view pattern of online content
Number of times students attempt a problem
Grades for self-check questions
Discussion forum
Class survey
Discussion with students
Ask the TA
Chat room logs
Assignment issues
Assignment grades
Student answers to problems
Student’s view pattern of online content
Number of times students attempt a problem
Grades for self-check questions
Discussion forum
Class survey
Use Would use if available Do not/Would not use
15
Value of Information Sources: Across GoalsFo
rum
Clas
s Sur
vey
Stud
ent D
is.
Teac
hing
Ass
ist.
Assg
n. G
rade
Prob
. Ans
wer
View
Patt
erns
Prob
. Atte
mpt
s
Self
Chec
k Q
Chat
Logs
0%
25%
50%
75%
100%Not UseSkipUse/Would Use
16
Value of Information Sources: Across GoalsFo
rum
Clas
s Sur
vey
Stud
ent D
is.
Teac
hing
Ass
ist.
Assg
n. G
rade
Prob
. Ans
wer
View
Patt
erns
Prob
. Atte
mpt
s
Self
Chec
k Q
Chat
Logs
0%
25%
50%
75%
100%Not UseSkipUse/Would Use
17
Value of Information Sources: Across GoalsFo
rum
Clas
s Sur
vey
Stud
ent D
is.
Teac
hing
Ass
ist.
Assg
n. G
rade
Prob
. Ans
wer
View
Patt
erns
Prob
. Atte
mpt
s
Self
Chec
k Q
Chat
Logs
0%
25%
50%
75%
100%Not UseSkipUse/Would Use
18
Value of Information Sources: Across GoalsFo
rum
Clas
s Sur
vey
Stud
ent D
is.
Teac
hing
Ass
ist.
Assg
n. G
rade
Prob
. Ans
wer
View
Patt
erns
Prob
. Atte
mpt
s
Self
Chec
k Q
Chat
Logs
0%
25%
50%
75%
100%Not UseSkipUse/Would Use
19
Value of Information Sources: Across GoalsFo
rum
Clas
s Sur
vey
Stud
ent D
is.
Teac
hing
Ass
ist.
Assg
n. G
rade
Prob
. Ans
wer
View
Patt
erns
Prob
. Atte
mpt
s
Self
Chec
k Q
Chat
Logs
0%
25%
50%
75%
100%Not UseSkipUse/Would Use
Qualitative Quantitative Outlier
20
Value of Information Sources: By Goal
Use/Would Use
Skip
Not Use
F SD AG VPSC
Q0%
25%
50%
75%
100%
TA Effectiveness
F SD AG VPSC
Q0%
25%
50%
75%
100%
Improve Topic Presentation
F SD AG VPSC
Q0%
25%
50%
75%
100%
Least Interesting Content
F SD AG VPSC
Q0%
25%
50%
75%
100%
Struggling Parts of Class
F SD AG VPSC
Q0%
25%
50%
75%
100%
Find most Engaging Content
F SD AG VPSC
Q0%
25%
50%
75%
100%
Course is Appropriate Difficulty Level
F SD AG VPSC
Q0%
25%
50%
75%
100%
Gauge Difficulty of Exam Problem
F SD AG VPSC
Q0%
25%
50%
75%
100%
Find Trouble Students and what Struggle with
F SD AG VPSC
Q0%
25%
50%
75%
100%
Assignment Issues
21
Value of Information Sources: By Goal
Use/Would Use
Skip
Not Use
F SD AG VPSC
Q0%
25%
50%
75%
100%
TA Effectiveness
F SD AG VPSC
Q0%
25%
50%
75%
100%
Improve Topic Presentation
F SD AG VPSC
Q0%
25%
50%
75%
100%
Least Interesting Content
F SD AG VPSC
Q0%
25%
50%
75%
100%
Struggling Parts of Class
F SD AG VPSC
Q0%
25%
50%
75%
100%
Find most Engaging Content
F SD AG VPSC
Q0%
25%
50%
75%
100%
Course is Appropriate Difficulty Level
F SD AG VPSC
Q0%
25%
50%
75%
100%
Gauge Difficulty of Exam Problem
F SD AG VPSC
Q0%
25%
50%
75%
100%
Find Trouble Students and what Struggle with
F CS SD TA AG PAn VP PAtSCQ
Assignment Issues
22
Value of Information Sources: By Goal
Use/Would Use
Skip
Not Use
TA Effectiveness
Improve Topic PresentationLeast Interesting ContentStruggling Parts of Class
Find most Engaging ContentCourse is Appropriate Difficulty
LevelGauge Difficulty of Exam ProblemFind Trouble Students and
what Struggle withAssignment Issues
Qualitative
Quantitative
23
Value of Information Sources: By Goal
Use/Would Use
Skip
Not Use
Qualitative
Quantitative
24
Value of Information Sources: By Goal
Use/Would Use
Skip
Not Use
Qualitative
Quantitative
Qualitative + Quantitative
Qualitative + some View Patterns No Agreement
25
What does qualitative mean?
• Qualitative Data:• Forums• Class Surveys
• Student Discussions• Teaching Assistant
Student Voice
People
26
Visualization Mockups
1. Usefula. (Likert response)
2. Easy to understanda. (Likert response)
3. When use it?a. While the course is runningb. When preparing new materialc. When preparing by reviewing past courses
27
Visualization Mockups
• Majority participants find useful and understandable
• Majority participants use• During class All⇨• Review All, except (quantitative) forum⇨
28
Limitations
• Survey• Some questions ask predict future behavior • (people not necessarily good at)
• Instructors not experienced with all information sources
• Forum• Small sample of student body
29
Other Work: Metrics Tab
Stanford edX Branch
30
Summary• Goal: Information display for MOOC instructors
• Survey: What information MOOC instructors value?
• Results:• Quantitative information valued same as smaller online courses• Qualitative information is important• Visualization mockups useful while course running and for
review
• Future work:• More visualizations• Improve communication of student voice to instructor
31
Spare Slides
Mockup: Boxplot Grades Distribution
Mockup: Stacked Bar Grades Distribution
Mockup: Stacked Bar Attempts Distribution
Mockup: Stacked Bar Views Distribution
Mockup: Line Graph Forum
New Posts
Views