Upload
rebecca-ferguson
View
3.087
Download
2
Embed Size (px)
Citation preview
Moving through MOOCs: Pedagogy, Learning and Patterns of EngagementRebecca Ferguson, Doug Clow (OU)Russell Beale, Alison J Cooper (Birmingham)Neil Morris (Leeds)Siân Bayne, Amy Woodgate (Edinburgh)
Current context
Students seek not merely access, but access to success
“John Daniel, 2012
% complete from: www.katyjordan.com/MOOCproject
”
Patterns of engagement: Coursera
●Samplinglearners explored some course materials
●Auditinglearners watched most videos, but completed assessments rarely, if at all
●Disengaginglearners completed assessments at the start of the course and then reduced their engagement
●Completinglearners completed most assessments
MOOC designers can apply this simple and scalable categorization to target interventions and develop adaptive course features
“”
Coursera study Kizilcec, R., Piech, C., and Schneider, E., 2013. Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. LAK13
4
ReplicationOpen University FutureLearn data
Replication
MOOC1 MOOC2 MOOC3 MOOC4
Subject area Physical sciences
Life sciences
Arts Business
M 51% 39% 32% 35%
F 48% 61% 67% 65%
Participants 5,069 3,238 16,118 9,778
Fully participating 1,548 684 3,616 1,416
Participation rate 31% 21% 22% 14%
5
Calculating an activity profileReplicating the method
●T = on track (3) undertook the assessment on time
●B = behind (2) submitted the assessment late
●A = auditing (1)engaged with content but not assessment
●O = out (0)did not participate
Replication
6
ReplicationIdentifying dissimilarity between engagement patterns
Assigned numerical value to each label●On track = 3●Behind = 2●Auditing = 1●Out = 0Calculated L1 norm for each engagement patternUsed that as the basis for one-dimensional k-means clusteringRepeated clustering 100 times and selected solution with highest likelihoodFocused on extracting four clusters
Replication
7
ReplicationCoursera and FutureLearn results were different
●Samplinglearners explored some course materials
●Auditinglearners watched most videos, but completed assessments rarely, if at all
●Disengaginglearners completed assessments at the start of the course and then reduced their engagement
●Completinglearners completed most assessments
√
√
xx
They also differed when we tried●Different values for k●A one-dimensional approach●Running k means directly on engagement profiles
Replication
8
FutureLearn is differentConversation is a central feature
Sharples, M., & Ferguson, R. (2014). Innovative Pedagogy at Massive Scale: Teaching and Learning in MOOCs. ECTEL 2014.
9
Revising the numeric values
OU FL study
1 only visited content (for example, video, audio, text)
2 commented but visited no new content
3 visited content and commented
4 did the assessment late and did nothing else that week
5 visited content and did the assessment late
6 did the assessment late, commented, but visited no new content
7 visited content, commented, late assessment
8 assessment early or on time, but nothing else that week
9 visited content and completed assessment early / on time
10 assessment early or on time, commented, but visited no new content
11 visited, posted, completed assessment early / on time
10
Typical engagement profilesThese profiles apply to an eight-week course
●Samplers visit only briefly[1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content
●Strong Starters do first assessment[9, 1, 0, 0, 0, 0, 0, 0] – 9 means they visited content and did assessment on time
●Returners come back in Week 2 [9, 9, 0, 0, 0, 0, 0, 0]
●Mid-way Dropouts[9, 9, 9, 4, 1, 1, 0, 0] – 4 means they submitted assessment late
●Nearly There drop out near the end[11, 11, 9, 11, 9, 9, 0, 0] – 11 means full engagement, 8 means submission on time
●Late Completers finish[5, 5, 5, 5, 5, 9, 9, 9] – 5 means they viewed content and submitted late
●Keen Completers do almost everything [11, 11, 9, 9, 11, 11, 9, 9]
OU FL study
11
Samplers & Strong starters
Samplers (1, 0, 0, 0, 0, 0, 0, 0)
●The largest group in all MOOCs
●Typically accounted for 37% – 39% of learners
●Visited the materials, but only briefly
●Active in a small number of weeks
●25% – 40% joined after Week 1
●Very few Samplers posted comments (6% – 15%)
●Almost no Samplers submitted any assessment
Strong starters (9, 1, 0, 0, 0, 0, 0, 0)
●All Strong Starters submitted the first assignment
●Engagement dropped off sharply after that
●A little over a third of them posted comments
●Typically posted fewer than four comments
OU FL study
12
Returners & Mid-way dropouts
Returners (9, 9, 0, 0, 0, 0, 0, 0)
●Completed the assessment in the first week
●Completed the assessment in the second week
●Then dropped out
●Over 97% completed those two assessments, although some submittted late
●No Returner explored all course steps
●Average amount of steps visited varied (23% – 47%)
Mid-way dropouts (9, 9, 9, 4, 1, 1, 0, 0)
●A much smaller cluster (6% of learners on MOOC1, 7% on MOOC4)
●These learners completed three or four assessments
●They dropped out around halfway through the course
●Mid-way dropouts visited about half the steps on the course
●Just under half posted comments
●Posted just over six comments on average
OU FL study
13
Nearly There
Nearly there (11, 11, 9, 11, 9, 9, 0, 0)
●Another small cluster (5% – 6% of learners)
●Consistently completed assessments
●Dropped out just before the end of the course
●Visited around 80% of the course
●Submitted assignments consistently (>90%) and typically on time until Week 5
●Activity then declined steeply
●Few completed the final assessment
●None completed the final assessment on time
OU FL study
14
Late completers & keen completers
Late completers (5, 5, 5, 5, 5, 9, 9, 9)
●Submitted the final assessment
●Submitted most other assessments
●However, either submitted late or missed some assessments
●Each week, more than 94% of this cluster submitted their assessments
●More than three-quarters submitted the final assessment on time (78% – 90%)
●Around 40% of them postedcomments (76% did so on MOOC3)
Keen completers (11, 11, 9, 9, 11, 11, 9, 9)●Accounted for 7% – 23% of learners●All Keen Completers submitted all assessments
●More than 80% of these were submitted on time
●Typically, Keen Completers visited more than 90% of course content
●Over two-thirds contributed comments(68% – 73%)
●Mean number of comments varied from 21 to 54
OU FL study
15
Cross-university datasetFutureLearn data from four universities
Cross-university
Name Duration University Discipline Active learners
LongMOOC1 8 OU Hard science
5,069
LongMOOC2 7 Edinburgh Hard science
10,136
TalkMOOC3 6 Edinburgh Politics 6,141
ShortMOOC4 3 Birmingham Medical science
6,839
ShortMOOC5 3 Leeds Medical science
4,756
16
Values for k used in this studyDifferent values were used for the three study phases
Cross-university
Name Phase 1 Phase 2 Phase 3
LongMOOC1 7 – –
LongMOOC2 7 – –
TalkMOOC3 – 7 3
ShortMOOC4 – 7 4
ShortMOOC5 – 7 5
Phase 1: Best-fit value for k aligned with OU studyPhase 2: Testing k=7 where this was not the best fitPhase 3: Most suitable value for k in each set of data
17
Phase two: k ≠ 7Why k≠7 in Talk MOOC3
Phase two
The absence of assessment in TalkMOOC3 limited its coding profile
1 only visited content (for example, video, audio, text)2 commented but visited no new content3 visited content and commented4 did the assessment late and did nothing else that week5 visited content and did the assessment late6 did the assessment late, commented, but visited no new content7 visited content, commented, late assessment8 assessment early or on time, but nothing else that week9 visited content and completed assessment early / on time10 assessment early or on time, commented, but visited no new content11 visited, posted, completed assessment early / on time
18
Phase two: k ≠ 7Why k≠7 in ShortMOOC4 and ShortMOOC5
Phase two
Three clusters are indistinguishable in a three-week MOOCReturners who come back in Week 2
Mid-way Dropouts who drop out mid-course
Nearly There who drop out near the end
With only three opportunities for late submission, there are no
Late Completers (who typically submit assessment late five times)
The three-week course design means other clusters emerge, such as:
Surgers concentrate their effort after the first week of a three-week course
Improvers fall behind in Week 1, begin to catch up in Week 2 and complete on time
19
Phase three: suitable values for kTalkMOOC3: k=3
Phase three
Quiet (1, 0, 0, 0, 0, 0)
●The largest cluster
●Visit a quarter of course steps
●Do not comment in first week
●Only 7% comment at all
●Only 9% engage with second half of course
Contributors (3, 1, 1, 0, 0, 0)
●19% of cohort
●Visit 38% of course steps
●Every cluster member posts in first week of course
●Half do not comment again
Consistent engagers(3, 3, 3, 3, 1, 1)
●11% of cohort
●Visit 82% of course steps
●Engage throughout course
●Every cluster member posts a comment
●95% contribute more than three comments
●7% contribute more than 100 comments
20
Phase three: suitable values for kShortMOOC4: k=4
Phase three
Very weak starters (2, 1, 0)
●The largest cluster
●Visit 20% of steps
●20% do not engage in first week
Strong starters (truncated) (10, 1, 0)
●17% of cohort
●Submit week 1 assessment
●Do not submit another assessment
●Almost half post comment
Returners (truncated)(3, 3, 3, 3, 1, 1)
●Most submit week 1 assessment
●All submit week 2 assessment
●Half submit at least one comment
Keen completers (truncated) (9, 9, 9)
●Visit more than 90% of steps
●Submit work on time
●Engage throughout
21
Phase three: suitable values for kShortMOOC5: k=5
Phase three
Samplers (truncated) (1, 0, 0)
●Visit few steps
●Includes many latecomers (>25%)
●Very few submit assessment Strong starters (truncated)
(9, 1, 0)
●Submit week 1 assessment
●Do not submit another assessment
Returners (truncated)(8, 8, 2)
●Most submit week 1 assessment
●All submit week 2 assessment
Keen completers (truncated) (9, 9, 9)
●Visit more than 90% of steps
●Submit work on time
●Engage throughout
Improvers (5, 6, 9)
●Activity increases each week
●Final assessment submitte on time
22
Learning design and pedagogy
●The Coursera Study suggested that MOOC designers would be able to apply the four engagement patterns they had identified ‘to target interventions and develop adaptive course features’
●These subsequent studies show that this is not necessarily the case –engagement patterns are not consistent across MOOCs
●Changes to the basic pedagogic elements of a course are associated with shifts in patterns of engagement.
●Shifts in pedagogic approach can change the elements of a course that can be regarded as key
●Changes to some elements of learning design can change learners’ patterns of engagement with a MOOC
23
Shorter courses
●Reducing course length does not necessarily increase engagement
●Many learners do not approach a three-week course in the same way as an eight-week course
●Many focus their attention on later weeks and may miss out the content and activities in the first week
●A three-week course offers limited opportunities to get ahead of, or behind, the cohort
●It is possible to dip in at different points without losing the sense of being a cohort member
24
Improving learning and learning environments
Closing the loop
●Previews of course material would allow Samplers to make a more informed decision about whether to join the course
●Sign-up pages could draw attention to the problems experienced by those who are out of step with the cohort
●Discussion steps for latecomers could support those who fall behind at the start
●Prompts might encourage flagging learners to return and register for a subsequent presentation
●Bridges between course weeks could indicate links and point learners forward
25
View these slides at www.slideshare.net/R3beccaF
Rebecca Ferguson @R3beccaFDoug Clow @dougclow