Upload
phungtuyen
View
214
Download
1
Embed Size (px)
Citation preview
A lens on descriptive and learning analytics at UNISA
Mr Glen Barnes & Mr Dion van Zyl
Department of Institutional Statistics & Analysis Unisa
SAHELA Conference - June 2013
Objectives
• A recent report for the VP: Academic teaching &
Learning
– Progress on all initiatives relating to teaching & learning
– Placement within the context of the approved conceptual
model
• Selected results pertaining to the areas of:
– Descriptive analytics
– Predictive analytics
– Learning analytics Numerous contributors acknowledged …
Learning
Analytics
A conceptual-hypothetical model
• For enhancing student success
– Student walk at the core
– Both student and institution as situated agents
– Identifies shaping attributes and conditions for ‘fit’ towards
success
– Implementation framework
• Conceptual modelling
• Metrics & measures
• Data, information & analytics
• Interventions
• Monitoring & evaluation
FRAMEWORK FOR THE
MANAGEMENT OF STUDENT EXPERIENCE, SUCCESS, THROUGHPUT &
GRADUATENESS
Conceptual
Modeling
Identifying
what is
relevant,
measurable,
available &
actionable
Statistical &
Analytic
Modeling
producing
Actionable
Intelligence
Data
Gathering:
“Broad”
Tracking/
Intelligence
System
Proactive
Learner
Support
Interventions
Monitoring &
Evaluation
Systematic gathering and disseminating of information
• Identifying modules ‘at risk’
• Further cohort retention and throughput studies – Re-defining throughput timelines
• Exam success – Sitting view
– Module/Course success view
– Degree credit view
• Graduate success – Academic graduation view
– HEMIS graduation view
– Ceremony graduation view
Identifying modules ‘at risk’
• To refine the old definition of ‘high risk’ and apply a nuanced metric for different
requirements
• Objectives include:
– The prioritising of funding for additional interventions (face-to-face tutoring)
– Revision of e-Tutor provisioning (student : tutor ratios)
– College specific interventions
• Estimation of a ‘Risk Appetite’
– Curriculum based (compulsory, pre-requisite modules for qualifications)
– Student based (curriculum and electives)
Driver Description Metric
ENR Enrolment ranking on tota l enrolments Enrol led
REG Registration ranking on regis tered modules Registered
ATT Attri tion ranking Cancel led/Enrol led
PTN Participation in the exam Wrote/Registered
NPR Exam Pass Rate Passed/Wrote
CSR Module or Course Success Rate Passed/Registered
GSR Gross Success Rate Passed/Enrol led
Tota l ENR
ENR Rank
(No)
1 CSET EUP15 0 1 (0 ) ETHICAL INFORM ATION AND COM M UNICATION TECHNOLOGIES FOR … 5 26 631 0
2 CEMS ECS15 0 1 (1) ECONOM ICS IA 5 19 502 22,93
3 CEMS FAC15 0 2 (1) FINANCIAL ACCOUNTING PRINCIPLES, CONCEPTS AND PROCEDURES 5 18 952 25,1
4 CEMS ECS15 0 1 (2 ) ECONOM ICS IA 5 17 478 30,93
5 CEMS FAC15 0 2 (2 ) FINANCIAL ACCOUNTING PRINCIPLES, CONCEPTS AND PROCEDURES 5 17 132 32,29
6 CEMS ECS16 0 1 (1) ECONOM ICS IB 6 12 934 48,88
7 CLAW CLA15 0 1 (1) COM M ERCIAL LAW IA 5 12 664 49,95
8 CEMS MNB15 0 1 (1) BUSINESS M ANAGEM ENT IA 5 12 011 52,53
9 CLAW CLA15 0 1 (2 ) COM M ERCIAL LAW IA 5 11 053 56,32
10 CEMS INM10 13 (1) INTRODUCTION TO THE ECONOM IC AND M ANAGEM ENT ENVIRONM ENT 1A 5 10 785 57,38
NQFNo Colle ge Code (Se me ste r) Module De sc ription
One or more drivers … Module ranking index
Module
Risk Index
Module
Risk
Category
Student
‘Risk Appetite’
Qualification
‘Risk Appetite’
Risk index
Risk index
Risk index
Risk index
Compulsory
Modules
Risk index
Risk index
Risk index
Risk index
Elective
Modules
Towards a ‘Risk Appetite’
Curriculum Risk
Student Risk
Barriers to Graduation
Cohort retention & throughput studies
• Increased awareness and need for these
• Identified the need to move away from aggregated
analyses
– Regular updates
– Analyses per qualification
– Move to specialisations and majors within qualification
• Tracking system and analysts geared to update all
cohorts in a short period
– Move to ‘provisional’ more recent data
– Use of the ‘academic’ graduate view Less reliance on HEMIS
Constrained by IT capacities / hardware
Statistical analysis and predictive modelling
• Profiling cohorts
• Researching student access to ICTs
– ICT sophistication index
• Understanding the habits and behaviours of students
• Understanding dropouts
• Predicting module enrolments
• Predicting student registrations
• Predicting students ‘at risk’
Profiling Cohorts
• To develop and refine a survey questionnaire that can be used to profile students
according to academic and non-academic risk indicators defined in the conceptual
model
• To evaluate most appropriate method(s) of data collection (single vs. multi-phase
approach) that can facilitate the roll out to the broader student body
• To evaluate aspects of validity and reliability relating to the measuring of items and
constructs
• To conduct exploratory analyses to identify academic and non-academic factors
explaining student success
Educational Background: Results
50% 40% 60%
Exam Pass Rate
Did receive career guidance at school
EPR 53%/59%
Career guidance
received
54%/50%
Mothers education:
Post Matric
30%/25%
n-275
20
11
n-301
20
12
Fathers education:
Post Matric
33%/31%
First in family to
enrol in HE
40%/52%
Did not receive career guidance at school
EPR 43%/50%
Highest level of education: Mother Post
Matric
EPR 61%/50%
Highest level of education: Mother
Matric or below
EPR 44%/57%
Highest level of education: Father Post
Matric
EPR 59%/59%
Highest level of education: Father Matric or below
EPR 42%/43%
Not the first person in family to enrol in HE
EPR 53%/55%
First person in family to enrol in
HE
EPR 45%56%
2011/2012
Exam Pass Rate: 2011/2012
p=0,020*
p=0,020*
Life Circumstances: Results
n-275
20
11
n-278
20
12
Exam Pass Rate
Family
38%/43%
Single
63%/57%
50% 40% 60%
Life stage grouping: Family
EPR 36%/40%
Life stage grouping: Single
EPR 56%/50%
Life stage grouping
Number of
financial
dependents (1+)
45%/48%
None financial dependants
EPR 57% /56%
(1+) Financial dependants
EPR 38%/53%
Sufficient funds
for studies
51%/55%
Disagree: sufficient funds for
studies
EPR 46%/49%
Agree: sufficient funds for studies
EPR 51%/61%
2011/2012
Exam Pass Rate: 2011/2012
p=0,003*
p=0,004*
Researching student access to ICTs
• The key research questions were:
– what is the extent of ICT access among Unisa students?
– what are the technological capabilities of these students?
The 2011 research culminated in a proceeding at the ODL
Conference in September 2012 entitled ‘Student Access to and
Skills in Using Technology in an ODL Context’
Moving toward identifying an ‘ICT sophistication index’
Conceptual: ICT Sophistication How Unisa students adopt & engage
Ab
ilit
y t
o e
ng
ag
e w
ith
IC
T (
hard
ware
& s
oft
ware
)
Success
64%
Typically connect via own computer,
mobile phone and tablet devices (at
home)
Highest levels of ICT engagement for
academic purposes
Communicating with fellow students
and lecturers via ICT lower than
expected!
Highest LSM
Parents also studied
ICT adoption (access/ownership to ICTs in general)
Higher adoption
Higher ability
Higher adoption
Lower ability
Lower adoption
Lower ability
Lower adoption
Higher ability
Success
50%
Success
39%
Success
37%
High prevalence of connectivity via own
computer, mobile phone and tablet devices
(at home)
Mid to high levels of ICT engagement for
academic purposes
Communicating with fellow students and
lecturers via ICT 1 in 2
Mid to high LSM
Large proportion of parents also studied
Connectivity via own computer, mobile
phone and tablet devices lowest
Traditional - Pen & paper?
Lowest levels of ICT engagement for
academic purposes
Communicating with fellow students and
lecturers via ICT higher than expected
Lower LSM
Low proportion of parents also studied
Connectivity via own computer, mobile
phone and tablet devices (at home) low
More work and Internet cafe
High levels of ICT engagement for
academic purposes
Communicating with fellow students
and lecturers via ICT highest 6 in 10)
Lower LSM
Low proportion of parents also studied
Conceptual: ICT Sophistication Interventions
Ab
ilit
y t
o e
ng
ag
e w
ith
IC
T (
hard
ware
& s
oft
ware
)
Success
64%
ICT adoption (access/ownership to ICTs in general)
Higher adoption
Higher ability
Higher adoption
Lower ability
Lower adoption
Lower ability
Lower adoption
Higher ability
Success
50%
Success
39%
Success
37%
Predicting Students ‘At Risk’
• e-Tutor pilot project module ACN203S in 2012
• Attempting to deal with both academic and non-academic
factors
• Retain ‘early alert’ factors and as few as possible
• Move from CHAID analyses to logistic regression
• Model built on 2011 cohort
– Results for 2011 and 2012
• Aim to have sequenced parameters
– Follow the student walk
Students ‘At Risk’ - Results Structural Demographics
Gender
Race
Home language
English (1,14)
Other (0,81)
Nationality
Age
Correspondence language
College/School CEMS - Accounting Sciences (0,3)
CSET – Sciences (0,3)
CEDU - Educational Studies (2,3)
CEDU - Teacher Education (2,1)
Academic period (semester, year)
CESM
Entering status Transfer (T) (1,2)
Entering (E) (1,0)
Non-entering/Returning (N) (1,4)
Exam sitting May/June (15,6)
Qualification type – UG (0,96)
Tutorial attended – Yes (1,2)
Module repeat – No (7,5)
Accuracy
2011 = 73,0%
2012 = 72,1%
2012 success using the 2011 model
Variables to be considered…
Structural Demographics
ICT sophistication
Living Standard Measure (LSM)
Employment status
Life stage grouping
Parents in HE
Credit load
Matric certificate
English matric mark
Entrance testing
Assignments
‘Risk Appetite’
Registration Tuition & learning Exam
Learning analytics??
Student Support and Interventions
• Academic Literacies
– Reading and writing, numeracy - are provided at the various
regional offices, face-to-face 2013, online 2014
• The Integrated Tutor model (e-Tutors)
– Pilot in 2012, all NQF 5 in 2013
• Measurement of under-preparedness
– More use of NBT and similar testing, test all 2014 students in
two colleges
• Interactive study materials
– Podcasts, vodcasts & simulations
Links to profiling, ICT sophistication and early alerts
Ongoing Monitoring & Evaluation
• Enrolment management
– Setting enrolment targets per qualification
• Setting module success targets
• Setting cohort success targets
• The impact of the re-admissions policy
• Monitoring concessions to improve throughput
• Monitoring student satisfaction
• Undertaking module evaluations
Underpinning Requirements
• Re-think of the metrics to be measured
• Re-assess those being used
• Highlight the need for improved ‘systems’
– Operations
– ICT
– Interventions/support
• Results to inform
– Procurement and Development of IT
Structures Processes Systems
Conclusions
• A lens …
– Change to reflect on the various initiatives
– Opportunity to share some of the results
– Systematically move ahead under the ‘cover’ of the
conceptual model
– Indication of our involvement in areas of analytics and the
move to ‘learning’ analytics
Acknowledgements:
Mr G Barnes, Director: Information and Analysis, DISA
Mr D van Zyl, Manager: Information Services, DISA
Ms Y Chetty, Director: Institutional Research, DISA
Ms H Liebenberg, Senior Specialist: Institutional Research, DISA
Prof EO Mashile, Executive Director: Tuition and Facilitation of Learning
Prof G Subotzky, Past Executive Director: Institutional Statistics Analysis
Dr P Prinsloo, Directorate for Curriculum and Learning Development (DCLD)
Registration – Entrance testing/evaluation
Bridging / Alternative pathways (Attitudes, habits, ability, resource use)
ICT
sophistication
Interventions
Training & interventions
Structures Processes Systems
Observe the frequency patterns …
Determine appropriate categories …
Risk Index
Risk
Category