Monitoring and Evaluation

Harvey HurreeDavid Tye

@LondonMetUni

/londonmetuni

londonmet.ac.uk

Agenda

1 Overview – Monitoring & Evaluation - Harvey Hurree 

2 Performance Enhancement Meetings - David Tye 

3 Course Evaluation - Harvey Hurree 

londonmet.ac.uk

The Quality Code (Chapter B8) sets out the following Expectation about programme monitoring and review, which higher education institutions are required to meet.

Higher education providers, in discharging their responsibilities for setting and maintaining academic standards and assuring and enhancing the quality of learning opportunities, operate effective, regular and systematic processes for monitoring and for review of programmes.

The following Expectation (Chapter A: Expectation A3.3) is also relevant to this topic:

Degree-awarding bodies ensure that processes for the monitoring and review ofprogrammes are implemented which explicitly address whether the UK thresholdacademic standards are achieved and whether the academic standards required by the individual degree-awarding body are being maintained.

londonmet.ac.uk

The Indicators

Indicator 1Higher education providers maintain strategic oversight of the processes for,and outcomes of, programme monitoring and programme review, to ensure processes are applied systematically and operated consistently.

Indicator 2Higher education providers take deliberate steps to use the outcomes of programme monitoring and review processes for enhancement purposes.

Indicator 3Higher education providers operate a process to protect the academic interests ofstudents when a programme is closed.

Indicator 4Higher education providers define processes, roles and responsibilities for programme monitoring and programme review and communicate them to those involved.

londonmet.ac.uk

Indicator 5Higher education providers evaluate their processes for programme monitoring andreview and take action to improve them where necessary.

Indicator 6Higher education providers make use of reference points and draw on expertise from those outside the programme in their processes for programme monitoring and review.

Indicator 7Higher education providers involve students in programme monitoringand review processes.

Indicator 8Higher education providers enable staff and other participants to contribute effectively to programme monitoring and programme review by putting in place appropriate arrangements for their support and development.

londonmet.ac.uk

londonmet.ac.uk

Course design in line with agreed

benchmarks, frameworks and

regulations

Course Validation

Student Feedback

Course Performance

Data / External

Examiner

Monitoring

Module and Course

Modification

Course Review

Quality Cycle at London Met

Monitoring & Evaluation Working Group (MEWG)

Set up April 2013

Chaired by Dominic Palmer BrownAll 4 faculties represented

Office of Institutional EffectivenessAcademic Registry

QEU

londonmet.ac.uk

Monitoring & Evaluation Working Group (MEWG)

Tasks• To consider the effectiveness and value of current monitoring and evaluation

procedures• To consider the UK Quality Code Expectations in respect of monitoring and

evaluation• To consider sector good practice in respect of monitoring and evaluation

Objectives• To recommend a cohesive, timely and effective set of quality monitoring and

evaluation practices to the University’s Undergraduate and Postgraduate Committee

• To identify actions (quick wins) that might be achieved quickly and easily• To identify resources that might yield early results or that could deliver impact

londonmet.ac.uk

Monitoring & Evaluation Working Group (MEWG)

What did it look at?

londonmet.ac.uk

Procedures• Module Logs• Module evaluation questionnaires• External Examiner reports• PEMS• Course Committees• Course Logs• PEMs• FAMs• Student Engagement• Employer feedback

Data• RPA data• DLHE data• NSS data

Recommendations

New Module and Course Logs with prepopulated data (course logs only).Move from consideration of module performance at an early July PEM to consideration of course performance and modules of concern, at a late July PEM.PEM Chair Report (PCR) of Course and Module logs when final data are available in November.Development of a new Faculty Annual Monitoring Statement (FAMS) that will provide the University with:

a) an evaluation of Faculty performance at undergraduate and postgraduate level, b) confirmation of the implementation of the University’s quality assurance procedures, c) information on validation/review/modification/deletion activity and d) an update on enhancement within the Faculty (QEU0022-FAMS).New Course and Module Modification Deadlines – 2nd February 2015 for modifications that impact on programme planning and two weeks post PEM for modifications that arise from PEM discussionsOnline course evaluation – pilot in May.

londonmet.ac.uk

Monitoring and evaluation process

PerformanceEnhancement

Meeting (PEM) in July, then

PEM Chair Report in November

Faculty UGPG Committee

UniversityUGPG

Committee

Module monitoring data - available from OIE Annual Monitoring Livelink site

Supplementary module performance data- Published by Academic Registry

Marksheets- Component marks produced by Module Leader

External Examiner comments - Module Confirmation Proforma (MCP) - Published by Academic Registry SSB Secretary

Results from module questionnaires - from Faculty records

External Examiner Annual Report ORExternal Examiner written commentary in advance of PEM (where the External Examiner cannot attend the PEM)-available from the EE and QEU Livelink sites-Published by Academic Registry/Assessment on EE Livelink siteCourse (& SSB) monitoring data- Available from OIE Annual Monitoring Livelink site

Course Committee minutes - from Faculty records

Results from course questionnaires - from Faculty records

NSS (previous year data only for July)Supplied by OIE via Faculty senior managers

DLHE data- Supplied by OIE in course monitoring data

PTES data- Available from Research Office

Module Log Completed by Module Leader

Course LogData provided by OIE, Commentary

sections completed by Course Leader

PEM Chair ReportData provided by OIE, Commentary sections

completed by PEM Chair

Faculty Annual Monitoring

Statement (FAMS)Completed by Associate Dean

Scrutiny Forum

Boxes on the left denote information type and the documentation into which it feeds. Information is either supplied by the Office of Institutional Effectiveness (OIE) or must be actively sourced by the person responsible for the production of the core document. All Course Logs, PEM Chairs’ Reports,

FAMS, and PEM minutes need to be provided to QEU who will hold them on Livelink.

Core documentationInformation and source

Annual Monitoring for Collaborative Partners

Module Level Student Feedback

Module Level External Examiner Comment

Course Level External Examiner Comment (aggregated from modules)

Course Level Performance Data (available from the Office of Institutional Effectiveness (OIE) the Academic Registry, and/or the collaborative partner)

Overall summary of good practice/issues to be addressed and annual update on partner activity

University evaluation of collaborative partner performance

QEU Collaborative Performance Report

 

 Module Logs are shared with the Academic Liaison Tutor

 

Faculty UGPG Committee

Collaborative Course Log produced by

collaborative partner

Faculty update on partner activity

University UGPG Committee

Collaborative Module Log produced by collaborative

partner

Collaborative Annual Monitoring Statement (CAMS) produced by collaborative partner

Partnership Annual Monitoring Statement

(PAMS) produced by Faculty

Collaborative Partner PEMs

Course Level Student Feedback (aggregated from modules)

Other relevant course information e.g. employment statistics (available locally)

Module Level Performance Data

The boxes on the left denote the action and the documentation into which it feeds. Advice is supplied (variously) by the Quality Enhancement Unit (QEU), the Registry and the Office of Institutional Effectiveness (OIE). The boxes on the right denote the forum at which

the annual monitoring documents are considered

Scrutiny DocumentationInformation

londonmet.ac.uk

Core Documentation for Module, Course, Subject and Faculty Monitoring & Evaluation

PEM Chair Report (PCR)

Aggregated and individual data

provided by OIE, Commentary

sections completed by

PEM Chair

Course LogData provided

by OIE, Commentary

sections completed by Course Leader

Module Log Completed by Module Leader

Faculty Annual

Monitoring Statement

(FAMS)Completed

by Associate Dean

londonmet.ac.uk

Review Forums for Module, Course, Subject and Faculty Monitoring & Evaluation

PerformanceEnhancement Meeting (PEM)

Faculty UGPG Committee

UniversityUGPG

Committee

PEM Chair’s Report (PCR)

July PEM November PEM (Chair only)

November Faculty UGPG

December/January University UGPG

londonmet.ac.uk

Monitoring & evaluation activity July 2015    

Action Responsibility Date Original date

Academic Registry produce & distribute module data to PEM Chairs and Secretaries, for distribution to Module Leaders. Based on module results from standard publication date 25 June with subsequent updates to 10 July. Module Log pro-formas included.

Academic Registry Fri 10 July Original date proposed was 1 July; extended to allow for late marks

OIE issue data for Course Logs, together with data consolidated at PEM level, plus DLHE and NSS data as available, to PEM Chairs / Secretaries for onward distribution to Course Leaders. OIE also make data available to Academic Registry and QEU. Data based on awards and progression data from standard publication date of 7 July with subsequent updates to 10 July. Course Log proformas included.

OIE Tue 14 July

Original date proposed for Course data was 10 July; extended to

allow for late awards and progression arising from late

marks. Original date for PEM data was 17 July

Module Leaders complete Module Logs & distribute to Course Leaders, PEM Chair/Secretary, and Academic Registry

Module Leaders Wed 15 JulyOriginal date proposed was Wed 8 July

Academic Registry distribute to External Examiners the relevant course data (not Course Logs) and completed Module Logs as available

Registry Fri 17 JulyOriginal date proposed was 15 July

Course Leaders complete Course Logs & distribute to PEM Chair/Secretary and QEU

Course Leaders Fri 17 JulyOriginal date proposed was 17 July

External Examiners unable to attend PEMs feedback any comments to PEM Chair & Secretary, cc Academic Registry

External Examiners with Academic Registry support for queries

Thur 23 JulyOriginal date proposed was 24 July

PEMs meetPEM Chairs, Secretaries, and other members

Mon 20 to Thur 23 JulyOriginal dates proposed 20 - 23 July

PEM minutes produced, approved and distributed to all PEM Members. It is up to PEM Chairs whether they produce also a draft PEM Chairs Report at this stage; the formal requirement for a PCR is in November.

PEM Chairs & PEM Secretaries Friday 31 JulyOriginal date proposed was 30 July

External Examiner Annual Reports (UG modules/courses only)

External Examiners to submit to Academic Registry, who publish via Livelink and circulate to Chairs and designated faculty contacts

Fri 4 SeptOriginal date proposed was 4 Sept

londonmet.ac.uk

Course Evaluation

New process

Consistent practice

Central accessibility

Online evaluation

Reaching all students of London Met

Collaborative students

Asking the question

Why core questions

What questions • National Student Survey (NSS)?

What about local level questions

londonmet.ac.uk

Course Evaluation Pilot 2015

Programme EvaSys

Options Hard Copy Online

What courses

25 of the largest from across the

Faculties

Data Where does it go Who looks at it