The power of learning analytics to measure learning gains: an OU, Surrey and Oxford Brookes story

The power of learning analytics to measure learning gains: an OU,

Surrey and Oxford Brookes story

https://twitter.com/LearningGains

https://abclearninggains.com/

The results of the ABC project were made possible due to Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Jekaterina Rogaten, Bart Rienties, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise Whitelock. Please contract [email protected] if you want to know more about ABC learning gains



https://abclearninggains.com/2015/12/17/aims-of-the-abc-model-of-learning/


mailto:[email protected]

Why measure Learning Gains

Talk to your neighbour:

1. What do you think Learning Gains are?

2. Why should universities worry about Learning Gains?

3. Why should I/we need to think about Learning Gains?

Why measure Learning Gains

• Green paper UK (November 2015) states – learning gain (and student outcomes) is a key aspect to consider when recognizing teaching excellence and quality.

• Learning Gains are key elements of TEF, but how to measure?

• 13 Pilot projects have been awarded a total of 4m to advance our understand of learning gains in HE and develop valid and reliable assessment of learning gains.

• Our project is ABC mixed-method learning gains research and for quantitative part of the study we are looking at what affective, behavioural and cognitive learning gains students show.

Agenda for today

1. What are learning gains?

2. How are we working within ABC project to measure learning gains?

3. How are learning gains measured: a meta-analysis

4. Do students make learning gains, and what is the power of LA to predict learning gains?

5. Policy implications

1 What are learning gains?

In line with Cronbach & Furby, 1970 and Lord, 1956, we define Learning Gains as:

Growth or change in knowledge, skills, and abilities over time that can be linked to the desired learning outcomes or learning goals of the course

Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective. Educational Research Review. Impact factor: 3.860

2. How are we working within ABC project to measure learning gains?Affect

Behaviour

Cognition

Academic performance

VLE

Satisfaction

Quantitative

QualitativeAcademic

performance

VLE

Satisfaction

Interviews

Diary

Module design

Socio-Demographic

Structure of the ABC project

Open University Team• Strength• Learning analytics• Quantitative data analysis• Assessment• Online learning and learning design• Cross-sector thinking and working

• Responsibilities• Overall leading the project• Leading the Phase 1 of the project

(secondary data analysis)• Phase 1 ethics applications• Developing models of learning gains

Dr Bart RientiesDr Jekaterina Rogaten

Prof Denise Whitelock

Dr Simon CrossProf Allison Littlejohn

Open University1. Learning AnalyticsLargest VLE datasets of affect, behaviour and cognition of 200K students, with 2 large scale implementations of predictive modelling

2. Learning DesignExtensive modelling of learning designs (OULDI)

3. Student Experience on a ModuleExtensive datasets of students’ experiences with blended and online learning

Oxford Brooks University Team

Prof Rhona SharpeDr Ian Scott

StrengthExpertise in learners’ experienceMixed-methods approachMeasures of learning and engagementLearning analytics

ResponsibilitiesCo-leading Phase 2 of the projectIn depth-interviews and diary data collectionTesting validity of self report measures of learning gainsTriangulation

Oxford Brookes University1. Academic Performance Tracking Tool Presents staff with data from student information system, NSS and module evaluations in form of dashboards used for module and programme reviews.

2. Grade Point AverageProvides students with information about their progress. Provides staff with a more nuanced indicator of attainment.

3. Survey of Student EngagementDeveloped own scales for monitoring student engagement with Assessment Compact, Academic Advising and Graduate Attributes.

University of Surrey Team

ResponsibilitiesCo-leading Phase 2 of the projectTesting the validity of the self-reported measures of learning gains (+OB)Testing the validity of learning analytics modelling (+OU)Dissemination: website and Twitter

Prof Ian Kinchin Prof Steven Warburton Dr Simon Lygo-Baker

StrengthUniversity pedagogy and concept mapping Educational technologiesExpertise in innovative teaching approaches development

Ceri Hitchings

University of Surrey

• Business Intelligence - Progression Analysis project has built / is building:• Live data on students who are at risk, to act as an early warning indicator so that active support

measures can be put in place;• Visualisation of data via dashboards for academics to put in place appropriate actions;• Extension into WP & Outreach domain with specific data feeds, modelling, and a WP progression

dashboard.

• Institutional success: • Ability to makes links between what gains our students make in correlation with an institution which

has high levels of satisfaction and has enhanced learning and teaching across all Faculty.

3. How are learning gains measured: a meta-analysis

• The concept of learning gain is primarily used to examine the effect of any particular educational ‘intervention’• There is a gradual increase in

studies examining learning gains all across the world• All learning gains can be

classified into ABC53%

16% 21%

10% Behaviour-Cognitive Learning Gains

Affective-Cognitive Learning Gains

Affective-Behaviour-Cognitive Learning Gains

Cognitive Learning Gains


A wide divergence in methods and reported effects


Results will be shared when published

4. Do students make learning gains, and what is the power of LA to predict learning gains?

• Learning gains are usually measured using pre-post standardised testing • It is resource intensive and becomes even more so if one wants to estimate

learning gains across various disciplines and number of universities • Using assessment results for estimating learning gains has number of

advantages:• Assessment data readily available• Widely recognized as appropriate measure of learning • Relatively free from self-reported biases• Allows a direct comparison of research finding with the results of other studies

Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia

Three-level Growth Curve Model

Level 1

Level 2

Level 3

TMA1

Student1

TMA3 TMA1 TMA2 TMA3 TMA1 TMA2 TMA3TMA2

Student2 Student3

Module1 Module2

TMA1 TMA2 TMA3

Student4

TMA1 TMA2 TMA3

Student5

Module3

Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia

VLE VLEVLE

Study 1• Participants

• 11,909 Social Science students of whom 72% were females and 28% were males with average age of M = 30.6, SD = 9.9

• 5,791 Science students of whom 58.2% were females and 41.8% were males with average age of M = 29.8, SD = 9.6.

• Measures• Tutor Marked Assessments (TMA)• Socio demographics (gender, ethnicity,

prior educational qualification)• Across 111 modules

Study 1: Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, EstoniaStudy 2: Rogaten, J., Rienties, B., Whitelock, D. (Submitted: 17-10-2016). Multi-level growth modelling of learning gains: the impact of VLE engagement on learning Arts and Business. Paper submitted to LAK2017.

Study 2• Participants

• 2,875 Arts students of whom 64% were females and 36% were males with average age of M = 34, SD = 13.3

• 5,082 students of whom 54% were females and 46% were males with average age of M = 29.6, SD = 9.3

• Measures• Tutor Marked Assessments (TMA)• Socio demographics (gender, ethnicity, prior

educational qualification, work part-time or full-time, and disability)

• VLE engagement• Across 36 modules

Three level growth model

Study 1: Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, EstoniaStudy 2: Rogaten, J., Rienties, B., Whitelock, D. (Submitted: 17-10-2016). Multi-level growth modelling of learning gains: the impact of VLE engagement on learning Arts and Business. Paper submitted to LAK2017.

Results will be shared when published

The power of learning analytics to measure learning gains: an OU,

Surrey and Oxford Brookes story


https://abclearninggains.com/

The results of the ABC project were made possible due to Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Jekaterina Rogaten, Bart Rienties, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise Whitelock. Please contract [email protected] if you want to know more about ABC learning gains





mailto:[email protected]