Analysing at-risk students at The Open University
Date: 18th March, 2015
Author: Jakub Kuzilek, Martin Hlosta, Drahomira Herrmannova, Zdenek Zdrahal, Annika Wolff
The Open University
• Largest distance learning university in the UK (200k students, over 500 courses)
• Main campus in Milton Keynes
• Many efforts of the OU aims at improving retention rate of students
• Several university units exist only for providing help to at-risk students
Problem specification
• Given:– Demographic data at the Start (may include information about student’s previous
modules studied at the OU and his/her objectives)
– Assessments (TMAs) as they are available during the module
– Virtual Learning Environment activities between TMAs
– Conditions student must satisfy to pass the module
• Goal: – Identify students at risk of failing the module as early as possible so that OU intervention
is efficient and meaningful.
Genesis of OU Analyse
time
Darkness OU Analyse prehistory
2011 20132012Project with
MSR Cambridge
Analysis ofVLE data 3 courses
Experimentaldashboard
Genesis of OU Analyse
time
Darkness OU Analyse prehistory
2011 2013
OU Analyse success stories
2014Feb
2 courses
2012Project with
MSR Cambridge
Weekly support for OU courses
4 predictivemodels
Analysis ofVLE data 3 courses
Experimentaldashboard
Genesis of OU Analyse
time
Darkness OU Analyse prehistory
2011 2013
OU Analyse success stories
2014Feb
2 courses
2012Project with
MSR Cambridge
Weekly support for OU courses
2014Oct
12 courses
4 predictivemodels
DashboardAnalysis ofVLE data 3 courses
Experimentaldashboard
Genesis of OU Analyse
time
Darkness OU Analyse prehistory
2011 2013
OU Analyse success stories
2014Feb
2 courses
2012Project with
MSR Cambridge
Weekly support for OU courses
2015Feb
2014Oct
12 courses 18 courses
4 predictivemodels
Dashboard Dashboard& recommender
Analysis ofVLE data 3 courses
Experimentaldashboard
Genesis of OU Analyse
time
Darkness OU Analyse prehistory
2011 2013
OU Analyse success stories
2014Feb
2 courses
OU Analyse future
2012Project with
MSR Cambridge
Weekly support for OU courses
2015Feb
2014Oct
12 courses 18 courses
4 predictivemodels
Dashboard Dashboard& recommender
Analysis ofVLE data 3 courses
Experimentaldashboard
What is the best time for at-risk student identification?
• Students, who fail first Tutor Marked Assignment
(TMA) in fourth week has high probability of course
failure (>95%)
We need to start predicting before first TMA
Data
• Demographic data
– Static data during the course
– Gender, Age, Highest education, New/Continuing student, Index of multiple deprivation, Number of previous course attempts, Student workload during the course
• Virtual Learning Envinronment (VLE) data
– Data from student interaction with VLE
– One day summary data
Importance of VLE data
• Demographic data
– New student
– Male
– No formal qualification
Sex
Education
N/C
TMA1
Without VLE:Probability of failing at TMA1 = 18.5%
Sex
Education
N/C
VLE
TMA1Clicks Probability Nr of students
0 64% 4
1-20 44% 3
21-100 26% 5
101-800 6.3% 14
With VLE:
Identifying module fingerprints
• Identification of the
most informative VLE
“resources”
• Course specific
• Using historical data
Important VLE activities
• Identified VLE activities: Forum (F), Subpage (S),
Resource (R), OU_content (O), No activity (N)
• Possible activities each week are: F, FS, N, O, OF, OFS,
OR, ORF, ORFS, ORS, OS, R, RF, RFS, RS, S
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
Start
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
Pass Fail No submit TMA-1time
VLE opens
Start
Activity space
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
Start
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
Pass Fail No submit TMA-1time
VLE opens
Start
VLE trail: successful
student
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
Start
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
FSF RFSOFS ORFN O SRFROF OR ORSORFS OS RS
Pass Fail No submit TMA-1time
VLE opens
Start
VLE trail: student who
did not submit
Results
• Four predictive modules
• Important activities identification -> recommendations
• Support of 18 modules
• Weekly predictions
• Dashboard (from scratch to working application in 1 year)
Future work
• Scaling up
• 2nd round of evaluations
• Addressing new challenges: modules without historical
data, model voting, new models, module finger prints,
alignment of assessments
• Evaluation of interventions