Claudia Hauff
Web Information Systems, TU Delft
Large-scale Learning Analytics
It’s data that is on the Web … Web data … lets find the Web Information Systems people!
✤ 40+ MOOCs ✤ 1+ Million enrollments ✤ From primary school to PhD level ✤ Lots of user data (click logs)
Our goals
Data
Knowledge
Application to learning
Gain actionable insights into learner behaviours at scale. a. Data Science b. Big data processing
Increase our knowledge about learners by looking beyond the learning platform a. Web data analytics
Design technology interventions that enable adaptive learning at scale. a. Web data analytics b. Human-centered design c. Learning technologies
Learner profiling beyond the MOOC platform
ACM WebScience 2016
Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, and Geert-Jan Houben. Beyond the MOOC platform: Gaining Insights about Learners from the Social Web, ACM WebScience, pp. 15-24, 2016.
Whythis research?
Learner
Before the MOOC
NOTHING
Engagement, retention, …
During the MOOC
NOTHING
After the MOOC
Howto solve the problem?
We propose:
a deeper understanding about learnerscan be gained by exploring their traces on the Social Web.
Whatresearch questions?
1 On what Social Web platforms can a significant fraction of MOOC learners be identified?
Are learners who demonstrate specific traitson the Social Web drawn to certain types of MOOCs? 2
To what extent do Social Web platforms enable us to observe (specific) user attributes
that are relevant to the online learning experience? 3
Learner identificationacross Social Web platforms
edX learners
Email Login name Full name+ +
1. Explicit Matching
Profile images & links
Identification via emails
2. Direct Matching
Identification via profile links from Step 1
3. Fuzzy Matching
Search learners by their login & full names
Compare: 1. profile link2. profile image3. login & full names
Social Web platformsinvolved in our work
Matching resultsfor 18 DelftX MOOCs
Lowest Highest Overall
Gravatar 4.37% 23.49% 7.81%
Twitter 4.99% 17.58% 7.78%
Linkedin 3.90% 11.05% 5.89%
StackExchange 1.23% 21.91% 4.58%
GitHub 3.43% 41.93% 10.92%
Matching resultsfor 18 DelftX MOOCs
Lowest Highest Overall
Gravatar 4.37% 23.49% 7.81%
Twitter 4.99% 17.58% 7.78%
Linkedin 3.90% 11.05% 5.89%
StackExchange 1.23% 21.91% 4.58%
GitHub 3.43% 41.93% 10.92%
On average, 5% of learners can be identified on globally popular Social Web platforms.
Learners on
Linkedin- Using job titles & skills to characterise learners
Spreadsheet MOOC- Software Engineer- Business Analyst- …
Design Approach MOOC- Co founder- UX designer- …
Learners onStackExchange
- Functional Programming learners in StackOverflow
- To what extent do learners change their question/answering behaviour during and after a MOOC?
Take-homeMessages
On average, 5% of learners from 18 DelftX MOOCscan be identified on 5 globally popular Social Web platforms. 1
Learners with specific traits prefer different types of MOOCs.2
Learners’ post-course behaviour can be investigated by using their external Social Web traces.3
Learning Transfer: does it take place?
Best Paper Nominee at ACM Learning At Scale 2016
An Investigation into the Uptake of Functional Programming in Practice
Guanliang Chen, Dan Davis, Claudia Hauff and Geert-Jan Houben, Learning Transfer: does it take place in MOOCs?, ACM Learning At Scale, pp. 409-418, 2016.
Whatis learning transfer?
Learning transfer is the application of knowledge or skills gained in a learning environment to another context.
Whydo we care?
Learning transfer is a more important measure of learning in MOOCs than retention, success or engagement.
FP101x
@flickr:christiaan_008
Course programming language: Haskell
Run as a typical video-lecture based MOOC
Assessment: 288 Multiple Choice questions
Introduction to Functional Programming
37,485 learners registered.41% engaged with the course. 5% completed the course.33% were active on GitHub (1.1M events).
Whatdid we do?
FP101xlogs surveys coding
activities
3 months 2.5 years + 0.5 years
+ +
email address
Are changes made in a functional language?
GitHub
10+ million registered users
hosting, collaboration and organisation
the most popular social coding platform
founded in 2007long-term
large-scale
detailed
detailed logs
code changes
project meta-data
A sanity check
Are “GitHub learners” different? GitHub
learnersNon-GitHub
learners
#Learners 12,415 25,070
Completion rate 7.71% 4.03%
Avg. time watching videos 49.1 min 27.7 min
Avg. #questions attempted 31.3 17.5
Avg. accuracy of learners’ answers 23.4% 12.9%
GitHub learners are more engaged than non-GitHub learners and exhibit higher levels of knowledge.
Are “Expert learners” different? Expert GitHub
learnersNovice GitHub
learners
#Learners 1,721 10,694
Completion rate 15% 6.5%
Avg. time watching videos 78.6 min 44.4 min
Avg. #questions attempted 57.9 27.0
Avg. accuracy of learners’ answers 38.0% 21.1%
Expert learners are more engaged than Novice learners and exhibit higher levels of knowledge.
To what extent do engaged learners exhibit learning transfer?
5-10% >30%10-30%<5%
To what extent do engaged learners exhibit learning transfer?
5-10%
Which type of learner is more likely to display learning transfer?
flickr@ConalGallagher
Intrinsically motivated Extrinsically motivated
Which type of learner is more likely to display learning transfer?
flickr@ConalGallagher
Intrinsically motivated
Which type of learner is more likely to display learning transfer?
Experienced Inexperienced
Which type of learner is more likely to display learning transfer?
Experienced
Which type of learner is more likely to display learning transfer?
High-spacinglearning routine
Low-spacinglearning routine
Which type of learner is more likely to display learning transfer?
High-spacinglearning routine
Learners who transfer quickly move to Scala
FP101x
Conclusions
Most transfer learning findings from the classroom hold.
The observed transfer rate is low: 8.5%.
Learners quickly moved on after the course to industrially-relevant functional languages.
@flickr:torsten-reuschling
From Learners to Earners: Enabling MOOC Learners to Apply their Skills and Earn Money in an
Online Market Place
IEEE Transactions on Learning Technologies
Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff and Geert-Jan Houben. Can Learners be Earners? Investigating a Design to Enable MOOC Learners to Apply their Skills and Earn Money in
an Online Market Place, IEEE Transactions on Learning Technologies.
WhatMOOCs aim to educate the world. Most successful learners are already highly educated. Learners from developing countries are underrepresented.
is the problem?
Whatis the problem?
EX101x: Data Analysis to the MAX()
HowPay learner at scale: recommend tasks from online market places to learners that are relevant to the course material.
can we tackle it?
Howcan we tackle it?
What1) To what extent do online market
places contain relevant tasks? 2) Are learners able to solve
real-world tasks with high quality?
do we need to look at?
Setup1) Weekly spreadsheet “bonus
exercises” drawn from Upwork (manually checked) in EX101x
2) Accuracy check 3) Quality check (code smells)
Howare learners doing?
Good accuracy & quality.
Built a workingrecommender.
Deployed in a MOOC by the end of October.
Our goals one more time…
Data
Knowledge
Application to learning
Gain actionable insights into learner behaviours at scale.
Increase our knowledge about learners by looking beyond the learning platform
Design technology interventions that enable adaptive learning at scale.
MOOCs are vital to bring higher education to the world. Lots of unexplored potential. Plenty of data. Many users.http://bit.ly/lambda-lab
Overall …
Recommended