MOOCs and Quality Issues - QAQE Steering Group

Quality Assurance and Quality Enhancement in E-learning Special Interest Group (QE-SIG)

Helen Barefoot

University of Hertfordshire

Jon Rosewell

The Open University

Session Overview

• Definition of MOOCs

• Online learning

• The Quality Code

MOOCs and Quality Issues

3

So what is a MOOC?

• Massive

• Open

• Online

• Course

MOOC background information

• Term coined in 2008

• Connectivism and Connective Knowledge ▫ 25 tuition paying students

▫ 2300 other students from the general public taking course online and free of charge

Providers

Context

What are the key challenges associated with assuring the quality of MOOCs?

quality for UG - levelness?

feedback could blossom or get lost - managing amount of feedback

is quality relevant?

reputational benefit/risk

if offering something for free should there be an obligation for it to be good?

good for widenening participation

employers must know what the course means

authentication

MOOCs and Quality Issues

8

Why bother with quality?

• Students – know what they are committing to

• Employers – recognition of content and skills

• Authors – personal reputation, 'glow' of success

• Institutions – brand reputation

• Funders – philanthropic, venture caps, governments

• Quality agencies – on behalf of above

MOOCs and Quality Issues

9

Tensions

Delivery: face-to-face . distance

Accreditation: credit . none

Price: cost . free

Entry: selective . open

Scale: personal . massive

Support: intensive . not supported

Pedagogy: constructivism . transmission

Teacher: star . anonymous

Massive

• Issue is not large size, but scale independence

• Aspects: ▫ Financial

▫ Technical

▫ Pedagogical

Learning design at scale

• Individual learning

• Small group collaboration

• Large communities

Open

• Different meanings: ▫ Open = free = gratis

▫ Open = free = libre

▫ Open entry

▫ Open = transparency

Course

• What distinguishes MOOC from OER or learning object? ▫ Size?

▫ Goal, learning outcomes?

▫ Measure of completion / achievement / attainment?

▫ Structure As narrative (xMOOC)?

Negotiated experience (cMOOC)?

▫ Time?

Quality and learners

“What are MOOCs actually aiming at? “Can the quality of MOOCs be assessed in the same way as any defined university course with traditional degree awarding processes? “Or do we have to take into account a different type of objective with MOOC learners? Are the learners mostly interested in only small sequences of learning, tailored to their own individual purpose, and then sign off and move to other MOOCs because their own learning objective was fulfilled?” Ulf-Daniel Ehlers, Ebba Ossiannilsson, Alastair Creelman http://mooc.efquel.org/

Quality Assurance Agency

“Our job is to safeguard quality and standards in UK universities and colleges, so that students have the best possible learning experience” Quality Code for Higher Education sets out the expectations all providers of UK higher education are required to meet. It gives all higher education providers a shared starting point for setting, describing and assuring the academic standards of their higher education awards and programmes and the quality of the learning opportunities they provide.

QAA event

• Quality Code

“Factors which apply to all learning opportunities regardless of location, mode of delivery, academic subject; MOOCs are no exception to that”

Anthony McClaran, QAA event (July, 2013)

The Quality Code

The Quality Code Part A: Setting and maintaining threshold academic standards Part B: Assuring and enhancing academic quality Part C: Information about higher education provision

The Quality Code Part A: Setting and maintaining threshold academic standards

Part B: Assuring and enhancing academic quality

• A1: The national level

• A2: The subject level and qualification level

• A3: The programme level

• A4: Approval and review

• A5: Externality

• A6: Assessment of achievement of learning outcomes

• B1: Programme design and approval

• B2: Admissions

• B3: Learning and Teaching

• B4: Enabling student development and achievement

• B5: Student engagement

• B6: Assessment of students and the recognition of prior learning

• B7: External examining

• B8: Programme monitoring and review

• B9: Academic appeals and student complaints

• B10: Managing HE provision with others

• B11: Research degrees

Part C: Information about higher education provision

Considering MOOCs through the lens of the Quality Code

B1: Programme design and approval

• Procedures for design and approval for MOOC courses?

• Are external partners involved in the design and approval?

• Assessing the effectiveness of the course and processes for monitoring and reviewing the course?

Course approval

B3; Learning and Teaching

• Constructive alignment

• Levelness

• Teaching methods ▫ Videos

▫ Embedded questions

▫ Tutorials

▫ Discussion forums

▫ Wikis

▫ Assessments

Considering MOOCs through the lens of the Quality Code

• B5; Student engagement

• “improving the motivation of students to engage in learning and to learn independently ▫ data

• the participation of students in quality enhancement and quality assurance processes, resulting in the improvement of their educational experience”.

Considering MOOCs through the lens of the Quality Code

Source: edX – Presented by Lori Breslow – Enhancement Themes conference, June 2013

“the participation of students in quality enhancement and quality assurance processes, resulting in the improvement of their educational experience” Traditional courses MOOCs

• B6; Assessment of students and the recognition of prior learning

Considering MOOCs through the lens of the Quality Code

Mozill

a b

adge infr

astr

uctu

re

http://o

penbadges.o

rg

http://www.dmlcompetition.net/

A badge is a validated indicator of

accomplishment, skill, quality or

interest that can be earned in any

environment

http://scouts.org.uk/

Back to basics…

• Are students learning? ▫ Self assessment

▫ Quizzes

▫ Peer assessment

▫ Analytics

▫ Survey evaluation

Monitoring and evaluation

Completion rates

• Reported completion may be very low (1-10%)

• Does that matter? ▫ With very large starting numbers, there are still many

learners completing

▫ Maybe learners achieve personal goals even if they don’t complete

▫ Can MOOCs encourage access to HE if >90% have an experience which is a ‘failure’?

Provenance Reputation

Brand

creation

use

user recommendation

peer review

MOOC

Quality points

checking

Quality Dimensions

Content

Accuracy

Currency

Relevance

Pedagogic Effectiveness

Learning objectives

Prerequisites

Learning design

Learning styles

Assessment

Reusability & openness

Format & interoperability

Localisation

Discoverability: metadata

Digital preservation

Accessibility

Ease of use

Clarity

Visual attractiveness,

engaging

Clear navigation

Functional!

Benchmarks / indicators (NB: an unashamed plug!) • Conventional HE benchmarks not be appropriate

• E-learning benchmarks a better starting point ▫ E-xcellence NEXT

http://e-xcellencelabel.eadtu.eu/

• MOOC benchmarks better! ▫ OpenupEd / E-xcellence MOOC (to come…)

http://openuped.eu/

OpenupEd www.openuped.eu

• ‘Reflecting European values such as equity, equality and diversity’

• Courses should show features: ▫ Openness to learners ▫ Digital openness ▫ Learner-centred approach ▫ Media supported interaction ▫ Recognition options ▫ Quality focus ▫ Spectrum of diversity

UK’s first credit rated MOOC • Level 4 • 20 credit points • Taught as MOOC for

undergraduate students and as open access

• 1000 students • 200 students going for credit • Assessed by a

• 10-minute podcast presentation (40%)

• 1500-word critical blog (60%)

And the latest… • University of California, Irvine,

• “Tackle serious scientific issues, related to events in the show”

▫ Physics- "science behind decay“

▫ Public health department- "What can we learn from past epidemics?“

▫ Maths - "post-calculus maths can be used to model population and epidemic dynamics"

• TV series - audience of 10 million

• Student numbers for online course expected to be in hundreds of thousands