ANNUAL SUBJECT REVIEW 2018-19UNDERGRADUATE PROVISION

CONTENTS

Section Page

1. Introduction 1

1.1 The Process 1

1.2 ASR and the Student Voice 1

2. The University’s Framework for Quality and Standards 1

3. The ASR Process 2

3.1 Who is responsible for producing ASR reports? 2

3.2 What should be covered in ASR reports? 2

3.3 The role of Faculty and University AQSC in ASR 3

3.3.1 Faculty Academic Quality and Standards Committee 3

3.3.2 Academic Quality and Standards Committee 4

3.4 The timetable of activity for ASR 4

4. Information for ASR 5

4.1 Boards of Examiners and Module Review Board minutes / External Examiners’ reports

5

4.2 National surveys, module/programme evaluations 5

4.3 Alumni feedback 5

4.4 Board of Studies and Faculty AQSC minutes 5

4.5 Degree results data 5

4.6 Employment data 5

4.7 Minutes from Student/Staff Liaison Committees 5

4.8

4.9

Student appeals and complaints

Student Experience Key Performance Indicators (KPIs)

6

6

Appendices

Appendix 1 Template for ASR Reports 2017-18 7

Appendix 2 Aide-Memoire 10

Appendix 3 Template for Faculty Reports on the ASR Exercise 12

Appendix 4 Student Experience Key Performance Indicators 15

1

ANNUAL SUBJECT REVIEW 2018-19UNDERGRADUATE PROVISION

(for provision in 2017-18)

1. INTRODUCTION

1.1 The Process

Annual Subject Review (ASR) is one element of the University’s monitoring and review processes for taught programmes of study. It is a process which supports and facilitates constructive reflection and self-appraisal, but most importantly it looks to celebrate and promote good work. The process comprises:

(i) An ASR report produced by the head of an agreed reporting unit* for all undergraduate taught programmes of study for which the unit is responsible.

(ii) A Faculty ASR report which summarises all the ASR reports it has received as part of this process.

(iii) Feedback from the Faculty AQSC to the reporting units* and from the University AQSC to the Faculties and reporting units*.

* Please see s.3.1.1 below for more information about reporting units.

1.2 ASR and the Student Voice

It is important that students are able to contribute to the ASR process and to do this a separate process involving Student-Staff Liaison Committees has been established to complement ASR. The process is as follows:

(i) The ASR report produced by each reporting unit should be submitted to the first meeting in the academic year of the SSLC relevant to the reporting unit for consideration.

(ii) With support from the Liverpool Guild of Students, each SSLC should produce a commentary on the report it received for submission to the first Faculty AQSC meeting in the 2019 calendar year.

(iii) The Faculty AQSC should consider the commentary reports from the SSLCs and identify any actions arising from them and monitor progress against these actions as appropriate.

(iv) The SSLC commentary and minutes from their meetings during 2017-18 should inform the focus of ASR reporting in 2018.

2. THE UNIVERSITY’S FRAMEWORK FOR QUALITY AND STANDARDS

The ASR process is intended to reflect the University’s approach to quality and enhancement as detailed in the University’s Framework for Quality and Standards. The following principles are embodied in the Framework:

The primary responsibility for the quality of the student experience lies at the point of delivery, with staff engaged in teaching and supporting learning across the University’s Faculties, its Professional Services and, where appropriate, its Partner Institutions;

1

Quality Assurance Processes exist to support the University’s aim of delivering excellence in learning, teaching and assessment. They will lead to enhancement of learning and teaching and will be subject to ongoing review of their effectiveness;

Quality Assurance Processes will be evidence-based, making full use of available management information and contributing to the further development of that information.

Please see Appendix 2: Aide Memoire for suggestions about what might be reflected on in your report.

3. THE ASR PROCESS

The ASR process consists of subject level reports which should be submitted to the Faculty Academic Quality and Standards Committees (FAQSC). The FAQSC will provide feedback on the reports and identify any actions arising from them. Each FAQSC will provide a summary Faculty ASR report for submission to the University AQSC, which will provide feedback on each of the Faculty reports.

3.1 Who is responsible for producing ASR reports?

3.1.1 It is the responsibility of each FAQSC to agree the reporting units applicable within their Faculty. ASR is a reflective process for evaluating and reviewing programmes and, therefore, it is expected that in most areas reports will be produced at department level (level 1), though exceptions can be agreed by FAQSC. Throughout this document, the level which FAQSC agrees as being responsible for producing ASR reports will be referred to as the ‘reporting unit’.

3.1.2 The head of the reporting unit will be expected to involve the programme team in the process of producing the report. S/he is free to involve Students, Boards of Studies, Directors of Studies, Year Tutors, and any other bodies or individuals relevant to the programmes under consideration; it is suggested that it is good practice to draw upon the perspectives of as wide a group of staff and students, and other stakeholders, as possible. The head of the reporting unit is responsible for the co-ordination of the information used in ASR reports.

3.1.3 Faculty AQSCs are responsible for producing a Faculty ASR report that goes to the University AQSC for consideration and approval. The Faculty report is a summary of the ASR reports from the reporting units.

3.2 What should be covered in ASR reports?

3.2.1 A template for ASR reports is at Appendix 1 and requires reporting units to cover the following in their reports:

(i) A list of all undergraduate taught programmes – this includes joint and interdisciplinary programmes and accredited CE/CPD provision which the reporting unit is responsible for, and any provision which forms a substantial or essential part of any other programmes not ‘owned’ by the reporting units.

(ii) Significant changes or events during 2017-18, e.g. changes to staff, curriculum, events or reviews which have affected the reporting unit.

(iii) Achievements and good practice in 2017-18, specifically any features of good practice which might be flagged for broader dissemination, e.g. using information cited by External examiners, PSRB activity, students and other projects.

(iv) Consideration of the maintenance of academic standards against the Framework for Higher Education Qualifications (FHEQ), specifically using the data available, analysing trends and referencing external examiner reports. This will include commenting on programme data for 2017-18, making reference to trends/changes from previous years and differentiating as relevant for cohort groups.

2

(v) A commentary on actions taken in response to any key issues identified by the External Examiners and from any other external review of the programme (i.e. periodic review, reports of PSRBs etc).

(vi) A commentary on the issues and actions arising from student evaluations, SSLC feedback and from surveys (e.g. NSS, Module evaluations) and how they have been responded to.

(vii) A commentary on the reporting units response to Strategy 2026 and TEF metrics, this should include details of any actions taken and to be taken in response to Strategy 2026 (Education Plans), its KPIs and plans to address TEF metrics. There should be a focus on addressing any major weaknesses highlighted in student feedback and especially actions being taken to improve student employability and academic support.

(viii) Identify any issues to be addressed and discussed at Faculty and/or University level.

3.2.2 In consideration of the points above, reporting units should also reflect on the following:

(i) Activity to enhance student employability.

(ii) Where applicable, provide a brief commentary on any international exchange partners/Erasmus exchange opportunities undertaken in connection with the reporting unit. Reflect on any issues or problems encountered, good practice and plans for future development.

(iii) Where applicable, provide a commentary on credit-bearing work/industrial placements and responses to issues regarding placements raised in feedback from students.

3.2.3 In reporting on progress against actions from the previous year, the Reporting Unit should list the actions and explain progress against each action. The FAQSC will be able to use this list to oversee progress and development. This action plan will be updated (to include Subject Action Plans) and monitored by FAQSC at mid-point in the academic year

3.2.4 The report should be critically reflective in appraising the reporting unit’s provision during the year. Reporting units should avoid producing reports that are merely descriptive of what they have done and the processes they have used or developed. In conducting a genuinely reflective and critical appraisal, the reporting unit should report on the effectiveness of actions it has taken and should list the actions it will be undertaking over the forthcoming year.

3.3 The role of Faculty and University AQSC in ASR

3.3.1 Faculty Academic Quality and Standards Committee

(i) Reviewing ASR Reports:

ASR reports should be submitted to FAQSC in time for their November 2017 meeting. At this meeting FAQSC should review and evaluate the ASR reports; seeking to identify any emerging issues, common aspects to the reports, points of particular interest or note, and/or matters that need to be brought to the attention of the University.

(ii) Providing feedback to the reporting units on the ASR Reports:

The FAQSC will provide feedback to each reporting unit on its ASR report. If it is considered necessary, FAQSC may request further information from a reporting unit, or ask a unit to re-submit its report if FAQSC considers that it was not sufficiently reflective or self-appraising. This should not cause any delay in the Faculty ASR report being submitted to AQSC, but it is expected that the Faculty ASR report will include details about actions being taken by the Faculty to get further information and/or an improved report from any individual reporting units.

3

(iii) Producing the Faculty ASR Report:

The FAQSC is responsible for producing a report for University AQSC summarising the ASR reports. The report should cover the points noted in (i) and (ii) above and follow the template at Appendix 3. The Faculty report will need to be produced in time for submission to the University AQSC meeting in November 2018.

(v) FAQSC actions arising from ASR Reports:

ASR reports may highlight actions that will need to be addressed by the Faculty. The FAQSC should ensure that these actions are followed up and closed down, with feedback provided to relevant parties as appropriate.

(vi) Ongoing monitoring of ASR:

The FAQSC is responsible for monitoring actions identified for reporting units through the ASR reports and the student commentary reports (including those received after the ASR reports). Reporting units will be expected to update their action plan during the academic year and this will be received by FAQSC.

3.3.2 University Academic Quality and Standards Committee

The University AQSC will receive the Faculty ASR reports from the FAQSCs and review and evaluate them. It will provide feedback on the reports and it will progress or oversee any institutional level actions that have been identified as a result of the ASR process.

3.4 The timetable of activity for ASR

When Who Activity

Summer Reporting Unit

The reporting unit should start compiling its ASR report.

October/November

Reporting units will be informed of FAQSC dates

FAQSC The FAQSC will receive and consider the ASR report from the reporting units in the Faculty and provide feedback accordingly. The FAQSC will produce a Faculty ASR report for the University AQSC.

November AQSC AQSC will receive the Faculty ASR reports for consideration at AQSC at its December meeting and provide feedback accordingly.

2018-2019 AQSC / FAQSC

The Committees will progress, close down and monitor actions arising from the ASR process as appropriate.

4

4. INFORMATION FOR ASR

4.1 Boards of Examiners and Module Review Boards minutes / External Examiners reports

Comments from external examiners, as recorded in the minutes of the Board of Examiners and Module Review Board meetings can inform the ASR report, as can their written reports if these are received in time.

4.2 National Surveys and Module/Programme evaluations

Reporting units are expected to make use of these in reflecting on their provision.

4.3 Alumni feedback

Reporting units should consider feedback from alumni where this is available and to the extent that it informs the themes being reported on and/or it raises a point of particular interest of note that the unit feels should be reported.

4.4 Board of Studies and Faculty AQSC minutes

Minutes from these may be helpful in reflecting on developments to provision, matters of academic practice, and policies and/or procedures that have been developed or have impacted on provision.

4.5 Degree results data

Indicative data on this should be available as a result of the Final Board of Examiners and should be reported on as appropriate.

4.6 Data on student employment and further study outcomes, engagement with careers and employability activities, and career intentions

Data on graduates’ employment and further study outcomes (via the HESA DLHE Survey) is available via the link below and should be reported on as appropriate. The data is updated in mid-May to show the employment statistics for the cohort graduating in the previous academic year and will include the data for earlier cohorts as well.

https://www.liv.ac.uk/intranet/careers/graduatedestinations/

Data on student levels of career awareness, levels of work experience, and career intentions for each student cohort is available through the Careers Registration process. Contact the Careers and Employability Service for more information.

Additionally, data on student engagement with careers and employability activities, such as attendance at employer presentations and fairs, and advice and guidance appointments is available from the Careers and Employability Service. careers@liverpool.ac.uk.

4.7 Minutes from Student/Staff Liaison Committees and SSLC Trend Analysis

Reporting units should use the SSLC minutes to inform the ASR report. It may be appropriate to liaise with the SSLC Chair and/or student representatives from the SSLC to identify the key issues raised and discussed at SSLCs during the 2017-18 session. The Guild of Students produces an Annual Student Voice Coordinators’ Report for the SSLCs, and this should be considered and reported on, as appropriate, along with other matters discussed at SSLC meetings.

5

4.8 Student Appeals and Complaints

Reporting units should report on action taken, or planned, or changes in processes or practices as a result of student appeals and complaints.

4.9 Student Experience Key Performance Indicators (KPIs)

Data will be available on the following:

(i) Degrees awarded: proportion of 1st and 2.1 degrees;

(ii) Continuation status, entry tariff, proportion of staff achieving formally recognised standards of teaching (as defined by HESA);

(iii) Progression;

(iv) Contact time (as defined by KIS); accessed at https://unistats.direct.gov.uk/

(v) Student population, student numbers by UG, HEI, OSI, proportion of OSI students;

(vi) TEF metrics.

See Appendix 4 for further information regarding available data. Further support on accessing the data can be obtained from Strategic Planning.

6

APPENDIX ONE

TEMPLATE FOR UNDERGRADUATE ASR REPORTS 2018-19For provision in 2017-18

A word limit of 500 words is advised for the report sections below; tables may be used to summarise data, as appropriate.

1. Programmes covered by this report

Please list all taught programmes for which the reporting unit is responsible, this includes accredited CE and CPD provision and any provision which forms a substantial part of any other programmes which are not ‘owned’ by the reporting unit.

Type here ...

Date of report and author

2. Significant changes or events during 2017-18

Briefly comment here on any significant changes (e.g. changes to staff, curriculum, events or reviews) which have affected the reporting unit during 2017-18).

Type here ...

3. Achievements and good practice in 2017-18

Outline any features of good practice which might be flagged for broader dissemination. E.g. using information cited by External Examiners, PSRB activity, students and other projects.

Type here....

4. Issues relating to academic standards and outcomes

7

Provide a reflective and evaluative summary of programme data for 2017-18, making reference to trends/changes from previous years and differentiate as relevant for cohort groups.

Confirm standards are in line with national expectations, e.g. the FHEQ

i) Intake: numbers, entry qualifications, relevant characteristics of the cohort

ii) Progression between years (failure rates, retention, module, programme level averages)

iii) Final award outcomes of students

iv) Graduate destinations

5. External evaluation

Comment on actions taken in response to any key issues identified by the External Examiners and from any other external review of the programme (such as periodic review, reports of PSRBs etc.)

Type here ...

6. Actions arising from student feedback

In this section you should indicate issues arising from student evaluations, SSLC feedback and from surveys (e.g. NSS) and how they have been responded to.

Type here …

7. Actions in response to Strategy 2026 and TEF metrics

In this section you should include details of any actions taken and to be taken in response to Strategy 2026 (Education Plans), its KPIs and plans to address TEF metrics. You should comment specifically on how you are addressing any major weaknesses highlighted in student feedback and especially actions being taken to improve student employability and academic support.

Type here….

8

8. Issues to be raised at Faculty and/or University level

You may use this section to highlight issues for attention beyond the subject area

Type here….

9. Action Plan for 2017-18

This is the Action Plan updates with progress on all actions from last year’s ASR

Action Success measure

Responsibility Timeframe for completion

Progress

10. Action Plan for 2018-19

This Action Plan should show the actions identified from this year’s ASR report.

The three actions identified in the Subject Action Plan should be added at a later date and monitored through the academic year via FAQSC.

Action Success measure

Responsibility Timeframe for completion

Progress mid-year

Progress year-end

9

APPENDIX TWOAide Memoire

The following can be used as a prompt to support the reporting unit in reflecting and evaluating the delivery of provision during the previous academic year. This is for guidance only and is not intended to be prescriptive about what to include in your report.

Significant changes Have you reviewed your curricula, as a result have you identified areas that you wish to

develop or enhance? Have there been any developments to introduce or increase international study opportunities

or work placement opportunities? Have there been any significant staff changes?

Achievements and good practice Are there any particular good examples of: e-learning success; enhancing employability skills;

developments in research-led teaching; international opportunities? Have staff been awarded any grants or have any staff received awards for learning and

teaching development of excellence? Have any initiatives been undertaken to provide or promote teaching development or

enhancement? Are there any particular examples of excellence in teaching which you have disseminated or

which you wish to report for wider dissemination? Any particular examples from external feedback?

Issues relating to academic standards and outcomes Do the data indicate any trends, either positive or negative, that are of particular interest or

need to be addressed? Have there been any activities or developments to address problems that have been

highlighted through performance indicators? Have these activities/developments been effective? To what extent have you involved students in addressing the problems?

Are there any factors outside your control which are affecting performance indicators, with positively or negatively? Is there any support which could be provided at School, Faculty or University level which could help with this? What support would this be?

Do the data show that academic standards are being met, as set out in the FHEQ and the University’s Code of Practice on Assessment?

External Evaluation Have there been any PSRB visits/review of programmes? What were the outcomes? Are

there any actions as a result of the process? Comment on the feedback from External Examiners and any action that has been identified

from their report(s). Have there been any developments or initiatives to engage with other external bodies such as

employers, industry representatives? If so, in what way do they contribute/enhance the student experience?

Actions arising from student feedback What have been the main points (positive and negative) that have been raised and discussed

at the SSLCs? What have you done or what do you intend to do in response? Are the students satisfied with the response? Was identified action discussed with them before taking forward?

How effective is your process for getting students’ evaluations of modules/programmes?

10

What issues have been raised in student evaluations? How are these being addressed? Are students satisfied with the response? Was identified action discussed with them before taking forward?

Have you developed or are you developing any other initiatives to engage with students?

Actions in response to Strategy 2026 and TEF Metrics How is the reporting unit addressing the KPIs in Strategy 2026? What action has been

identified? You should comment specifically on how you are addressing any major weaknesses

highlighted in student feedback and especially actions being taken to improve student employability and academic support.

What are the plans to address any areas in the TEF metrics that are below the benchmarks?

11

APPENDIX THREE

TEMPLATE FOR FACULTY REPORTS ON THE 2018-19 UG ASR EXERCISE FOR 2017-18 PROVISION

Please note: The Faculty ASR report should highlight the most significant points or issues and whether there appear to be any trends or common areas of success or areas of concern across the Faculty.

1. Receipt of Annual Subject Review summary reports

Faculties should confirm that all reporting units have submitted their ASR reports. If reports have not been submitted, please identify which reports are missing and outline the action being taken by the Faculty to follow this up. If the Faculty has sought further information from a reporting unit or has asked a reporting unit to re-submit their report, it should explain the reasons for this and the outcomes so far.

Faculties should also confirm that feedback on ASR reports has been provided to the reporting units.

Type here ...

2. Significant changes or events during 2017-18

Faculties should highlight any significant changes that need to be noted at institutional level

Type here ...

3. Achievements and good practice in 2017-18

The Faculties should highlight any points of particular interest and highlight any areas of good practice for the attention of the University.

12

Type here ...

4. Issues relating to academic standards and outcomes

The Faculties should highlight any points of particular concern AND report whether or not they are satisfied with the action identified. They should report any actions for the Faculty and highlight any points for the attention of the University.

Type here....

5. External Evaluation

The Faculties should highlight any points of particular concern and report whether or not they are satisfied with the action identified. They should report any actions for the Faculty and highlight any points for the attention of the University.

Type here....

6. Student feedback

Faculties should report on any actions arising from students’ feedback and how these will be progressed and monitored.

Type here ...

Faculties should list any actions identified for the Faculty and any points they feel should be addressed by the University.

7. Actions in response to Strategy 2026 and TEF metrics

13

Faculties should confirm that relevant action is taking place across the Faculty in response to Strategy 2026, its KPIs and TEF metrics, specifically in relation to employability and student support.

Type here ...

Faculties should list any actions identified for the Faculty

8. Issues to be raised at University level

Faculties should identify any issues that need addressing at University level.

Type here ...

9. Action Plan for 2017-18

This is the Action Plan updated with progress on all actions from last year’s ASR Faculty level report

Action Success measure

Responsibility Timeframe for completion

Progress

10. Action Plan for 2018-19

This Action Plan should show the actions identified from this year’s ASR Faculty level report.

Action Success measure

Responsibility Timeframe for completion

Progress (mid-year)

Progress (year-end)

14

APPENDIX FOUR

Student Experience Key Performance Indicators (KPIs)

Data is not currently held in a single location for ASRs. The following begins to explore the ways in which some of the required data can be obtained.Updates will be provided as and when they become available

Accessing Business ObjectsIntranet home page > ‘More tools and online services’ (left hand menu) > Business Intelligence > ExplorerPlease note, Business Objects does not work on Google Chrome – it is best to use Internet Explorer

Degrees awarded: proportion of 1st and 2.1 degreesASR UG Degree awards and tariff: http://bi4live/explorer/index.jsp?application=portal&documentId=5636a2e1-9843-4367-9534-cbd577190b7c

Use the following to add filters:

When you click on the + above, an alphabetical field list will appearSelect field 1 as listed belowSelect from the list that appears then click the top arrow to the right of the list – your selected item will then appear in the right hand side. You can do this for multiple items (e.g. year)Click ‘Done’Repeat this for the field 2 below

1. Level 1 (Department)EXAMPLE:

15

2. Effective term code (select multiple years for comparison)

A graph will be returned at the bottom. Above the graph, change the 2nd and 3rd drop downs so they reflect the following:

Your graph will show the number of each award split by year. Please bear in mind that the years are not chronological, nor are the degree awards in a standard order.

You can then export data using the menu at the top:

Select Excel from the left hand side, then the second option on the main panel to export the visualisation data, ‘OK’ and save to a location you can then find!

From here, you can use the data to calculate percentages etc.

16

Entry tariffUse exactly the same query as for degrees awarded – The ‘Displaying By’ drop down above the graph should be changed to show ‘Tariff Grouped’.

Continuation/ProgressionTEF metrics and internal indicators look at non continuation – i.e. the proportion of students who do not progress to the next year of study at UoL or as part of a transfer to another institution.

ASR UG Progression data: http://bi4live/explorer/index.jsp?application=portal&documentId=daf8dd82-e12d-47dd-ae0b-02353860aea5

Repeat the process as above adding level 1 (department) and effective term codes filters.The ‘Displaying By’ drop down above the graph should be changed to show ‘Continuation Status (HESA Defined).

Proportion of staff achieving formally recognised standards of teaching (as defined by HESA);These will be circulated shortly once an agreed reporting mechanism has been established

Contact timeThis measure is defined by the Key Information Set (KIS) and can be accessed here: https://unistats.direct.gov.uk/Institutions/Details/10006842

Find the relevant course using the search under the map and then use the ‘Study information’ tab along the top to review contact time:

17

Student population, student numbers by UG, HEI, OSI, proportion of OSI students;

Use exactly the same query as for continuation/Progression – The ‘Displaying By’ drop down above the graph should be changed to show ‘Residence Description’

Please note, data can be ‘sliced’ to show any of the fields listed across the top of the screen, including:GenderDisability flagEthnicityPolar groupsSchool types

TEF metrics

At present, TEF is only available officially at institution level.

Until it is replicated at subject level by HEFCE, Strategic Planning have developed an interim solution.Please note, these are not reflective of what may published in the future by TEF and they must be utilised alongside the following caveats:

Departmental recreations Published TEF

Based on 1 year of data 3 years of data averaged

No benchmark, just an average of all institutions Calculated benchmark

Flags based solely on difference between actual and average

Flags based on difference between actual and benchmark AND standard deviation

Quartiles based on all institutions data No quartiles given

No splits Splits on WP markers

Non-continuation measures how many students are returned to HESA at the end of the year as leaving the course, without gaining an award or transferring to another HEI

Tracks students into the next year to see if they return

The data can be accessed here: https://www.liverpool.ac.uk/intranet/strategic-planning/teaching-excellence-framework/

18