Higher Education Data Workshop 22 July 2010 · & ASC • Credits assigned to column 1 or 2 • Credits assigned to column 3 where module outcome is known and credits are not completed

Higher Education Data Workshop

22 July 2010

Agenda

– Data and coding issues

– Audit outcomes

– Audit timetable

– Using HESA data for end of year monitoring

– The inclusion of FECs in Unistats and the NSS

– Timetable for 2010/11

– Update from HESA

– For our Future indicators

– Changes to 2011/12 funding and the funding

review

Data and Coding Issues – HESA

Data

• LANGPCNT

• Provision funded through initiatives

• Students no longer active

• Module data

• Data requirements circular

LLWR Coding Issues

Structure showing relationships between different

datasets on Lifelong Learning Wales Record

LEARNER

LEARNING

PROGRAMME

LEARNING

ACTIVITY

HIGHER

EDUCATION

LLWR Coding Issues Example structure of a HEFCW fundable programme of

study

LEARNER

LEARNING

PROGRAMME

e.g. BSc Computing

LEARNING ACTIVITY

e.g. modules:

Basic Computing

Introduction to the internet

etc.

HIGHER

EDUCATION

LLWR Coding Issues

Learning activities should be coded at module level

e.g. ‘Introduction to the Internet’ module which is part of ‘BSc

Computing’ learning programme:

o LP17 – Type of learning programme = 56 - HE First Degree

o LA06 – Learning aim reference = LDKKHE07

o LA08 – Title of learning activity = Introduction to the Internet

o LA23 – Credit value of learning activity = 20

o LA31 – Completion status = 2 - Learner has completed the

learning activity

o HE12 – Credit value of the course = 360

LLWR Coding Issues

Credit value of course should be coded in HE12

Completion of the HE dataset

Completion of LA08 - Title of learning activity with

actual title of the module

LLWR Coding Issues HE specific changes for 2010/11 data collection

(list not exclusive)

Amendments to existing fields:

HE06 – Nationality

HE07 – New entrant to HE

HE09 – Fee band

HE10 – Major source of HE tuition fees

HE11 – UCAS number



New fields:

HE13 – Mode of attendance

01 (Full-time), 02 (Sandwich year out), 03(Part-time)

HE14 – General qualification aim of programme

To be coded with codes provided in Annex 19

HE15 – Type of year of programme

1 - Course academic year contained within the reporting year 1

August – 31 July

2 - Course academic year not contained within the reporting

year 1 August – 31 July



New fields continued:

HE16, HE18, HE20 - Subject of learning programme

HE17, HE19, HE21 – Subject of learning programme

percentage

HE22, HE24 – Learning programme qualification

awarded

HE23, HE25 – Learning programme qualification

awarded – grade

LLWR Coding Issues HE validations for 2010/11 data collection


Validations:

If learner provision funding is recurrent funding for

teaching from HEFCW then the credit level of

learning activity is at HE level.

i.e. if LA11 = 2 then LA22 is one of 04, 05, 06, 07 or 08

If learner provision funding is recurrent funding for

teaching from HEFCW or the learning aim reference

indicates a HE module then the credit value of the

learning activity must be present

i.e. if LA11=2 or LA06 one of LDKKHE07 or LDLLHE07 then

LA23 not 000.0, 888.8, 999.9

LLWR Coding Issues Useful addresses

Version 3 of the 2010/11 LLWR manual :

www.learnerdata-wales.org.uk/LLWR-

Inform/info/Documents.aspx

Joint Academic Coding System (JACS) subject classification :

www.hesa.ac.uk/index.php/content/view/1776/296/

Annex E of HEFCW Higher Education Students Early Statistics -

W09/33HE for definitions of mode of study used in HE13

www.hefcw.ac.uk/publications/circulars/circulars_2009_all.aspx

HE dataset fields additional guidance from HESA student record

www.hesa.ac.uk/index.php?option=com_studrec&Itemid=232&

mnl=10051

Audit Outcomes – External Audits

Findings of the audits:

• Errors in recording of completion status

• Withdrawals not being recorded correctly

• Inconsistencies between the EYM return and

supporting schedules

• Lack of supporting evidence for data used in

premium and per capita funding

• Lack of documentation for Third Mission funding

data

Audit Outcomes – Internal Audits

Findings and recommendations included:

– Lack of internal procedural documentation

– Risks not included in risk register

– Incorrect allocation of modules to ASCs

– Over-reliance on key members of staff

– Issues with accuracy of forecasts on EYM

– Quality checking not performed effectively

– The importance of accurate data returns should be impressed upon

senior staff

– Copies of data submitted to HESA should be kept

– Welsh medium activity should be properly recorded

– JACS codes should be assigned at the validation stage of a new

course

– Data submissions should be reconciled

Audit Timetable

• August 2010

– Feedback from internal and external audits

– Official notification of external audits for 2010

• Autumn 2010

– External audits take place

• Future

– Have extended the contract with KPMG for another

two years – this covers the five year cycle

– May in future choose to audit specific initiatives or

provision depending on HEFCW priorities

Using HESA Data for End of Year

Monitoring

– Proposal last year to set up a project to

look at the feasibility of using HESA data in

place of EYM

– Technical group established as a result

• Two meetings so far


Monitoring

– Algorithm developed to extract end of year data from

HESA

– Data extracted for 2007/08 and 2008/09

– First meeting looked at proposed algorithm and

methods for estimating completion status for modules

without outcomes

– Second meeting looked at issues with the 2008/09

extraction, a comparison of thresholds for estimating

completion status, the amount of estimation and timing

of the consultation


Monitoring

The algorithm:

– Uses the HESA/HESES/EYM mapping shown

in Annex K of the EYM circular

– Countable credits categorised by mode, level

& ASC

• Credits assigned to column 1 or 2

• Credits assigned to column 3 where module

outcome is known and credits are not completed

• Completion status estimated for modules with

unknown outcomes and a proportion included in

column 3


Monitoring – 2008/09 extraction

• Issues with 2008/09 extraction:

– Large proportions of modules with outcome not

known contained within the reporting year

– Module status incorrectly coded for modules which

span more than one year

– Categorisation into ASC

– Coding of sandwich year out

– Suitability of algorithm for institutions with large

numbers of modules not contained within the

reporting year


Monitoring

• Method of estimation:

– For modules with known outcomes, the proportion of

credits in each outcome category is calculated and this

is applied to the credits with unknown completion

status

– Proportions are calculated at level, mode, module

subject, ASC and module status level of aggregation

• Threshold:

– Minimum number of values used to calculate

proportions in each module outcome category

– Higher level of aggregation used if threshold not met

– Minimum of 10 or 20 values tried for 2008/09

Estimation of Module Outcomes

– 13% of student on module outcomes for modules started in 2008/09 are not known (14% in 2007/08)

– Of these, 64% (43,814 outcomes) were contained within the reporting year (65% in 2007/08)

Estimation of Completion Status

– Undergraduate: • 10% of full-time credits

• 47% of sandwich year out credits

• 31% of part-time credits

– Postgraduate: • 32% of full-time credits

• 41% of part-time credits

Threshold for Calculation of Proportions

• Thresholds of 10 and 20 tried

• Proportions calculated at following levels: – Mode, level, ASC, module subject, year of

programme, module status

– Mode, level, ASC, year of programme, module status

– Mode, level, ASC, year of programme, module status

– Mode, level

(For estimation purposes, module status has two categories – is contained within reporting year or spans reporting years)

Comparison with EYM 2008/09

Percentage difference between HESA and EYM:

• Undergraduate: – 2% full-time

– -27% sandwich year out

– 4% part-time (excluding franchised out)

– 1% part-time franchised out

• Postgraduate: – -2% full-time

– 1% part-time

Revised Timetable 15 July 2010 Second meeting of technical group

22 July 2010 Report back to 2010 data workshop

April 2011 Extraction of end of year data from the 2009/10 record

May 2011 Third meeting of technical group

June 2011 Report back to 2011 data workshop

August 2011 Return of EYM 2010/11

September 2011 Start consultation with sector

Autumn 2011 Extracted data from the 2010/11 record made available through

the IRIS system

November 2011 Consultation outcomes published and decisions made about the

future of the EYM return

Early 2012 Extraction of end of year data from the 2010/11 record (clawback

outcomes compared to 2010/11 EYM clawback outcomes)

August 2012 No EYM data collected directly from institutions (subject to the

outcome of the consultation)

Early 2013 Extraction of end of year data from the 2011/12 record used for

calculating clawback

HESA IRIS system

– Aiming to implement for Wales for the 2010/11 record

– Will include: • EYM extraction

• Premium and per capita summaries and data

• Other data extractions as necessary

– Data will automatically be sent to HEFCW, analysis will be carried out at HEFCW and data sent back to HESA

– Summaries and data available at TEST COMMIT and COMMIT stages of the submission system

Unistats and the National

Students’ Survey

Unistats website:

• Presents information about provision at the institution and

student demographics

• Shows the results of the NSS

• Covers HEIs in the UK and directly funded FECs in

England

National Students’ Survey:

• Undergraduates in their final year surveyed about

satisfaction with their experience at their institution

• Covers HEIs in England, Wales, Northern Ireland and some

in Scotland and directly funded FECs in England

The inclusion of FECs in Unistats

and the NSS

Currently on the Unistats website:

• Students franchised out to an FEC from an HEI included

with the franchisor HEI

• Directly funded HE at FECs in Wales not included in the

website and not surveyed in the NSS

Changes for 2010:

• Information about franchised out provision, including

NSS results, will be shown against the college the

provision is franchised to

• Franchised out provision will not be included in the

summary data and NSS results shown for the HEI

The inclusion of FECs in Unistats

and the NSS

Future changes for Wales:

• The NSS will be carried out for students at directly

funded FECs from 2012

• Directly funded provision at FECs will be presented

on the Unistats website

• Changes to coding and improvements in quality in

the LLWR data will enable this to happen

• Information about directly funded provision will be

shown against the FEC along with any franchised

provision at the FEC

Timetable for 2010/11

22 July Workshop for institutions

4 August EYM return date

19 August EYM verifications sent out

1 September Return date for redistribution exercise

9 September EYM verifications return date

Sept/October Audit of data (selected institutions)

Mid September Data submitted to HESA

Mid Sept/Oct HESA data quality checking

Mid October HESES survey issued

Mid November HESES returned

For our Future Indicators

1. Students from Communities First areas

2. Module completion rates

3. Student’s overall satisfaction with the quality of

their course

4. Students studying through the medium of Welsh

5. Overseas students

6. Leavers employed or studying six months after

leaving


7. Participation rates

8. Part-time students

9. Graduate spin-out companies, active and surviving

at least three years

10. Income from Research Councils

11.Annual income of HEIs

12.Risk status of HEIs


Data sources:

Student record

Destination of leavers from HE survey

HE business and community interaction survey

Finance record

National Student Survey

HEFCW participation rates analysis

HEFCW institutional risk review process


Targets cover the period 2008/09 to 2012/13

Some are similar to Reaching Higher targets

Some are new indicators

Based on HESA registration population where

relevant

Open University in Wales and Further Education

Colleges included where possible

Algorithms will be provided for indicators where

relevant


Students from Communities First areas

Data from HESA student record and LLWR

Similar to reaching higher target, except all

students, not new entrants, updated community first

areas

Includes OU and FECs

Key fields used: POSTCODE


Module completion rates

Data from HESA student record

Includes OU

Key fields used: MODSTAT – module status,

MODOUT – module outcome

Completed modules

MODOUT one of

1 - Completion - gained full credit or

2 - Completion - did not gain credit


Module completion rates continued

Denominator = modules expected to get results

MODSTAT is one of

1 - Continuing from previous reporting year or

2 - Contained within reporting year

and

MODOUT is one of

1 - Completion - gained full credit

2 - Completion - did not gain credit

3 - Partial completion (HEFCW HESES Rules)

4 - Student did not complete module

6 - Module outcome not yet known


Students studying through the medium of Welsh

Data from HESA student record; from verifications

for FECs

Similar to reaching higher target


Key fields used: POSTCODE, MODSTAT – module

status, MODLANG, - Module taught in a Celtic

language, LANGPCNT - Percentage of module

taught in Celtic language

POSTCODE used to determine Welsh domiciled


Students studying through the medium of Welsh

continued

Welsh medium where MODLANG is 1 – Welsh or

LANGPCNT > 0

Results only modules excluded, MODSTAT is 4

Target changed from percentage to specific number


Students overall satisfaction with the quality of

their course

Data from overall satisfaction question from

National Student Survey

Three year rolling average used

Overseas students

Data from HESA student record

Similar to reaching higher target

DOMICILE field used to determine students with

Non-European Union domicile


Leavers employed or studying six months after

leaving

Taken directly from the UK PI E1a - Leavers

obtaining first degrees from full-time courses

Participation rates

Based on data calculated in HEFCW’s participation

rates method

POSTCODE field used to determine Welsh domicile

Compares participation rate in the five UAs targeted

by UHOVI with the national participation rate


Part-time students

Data from HESA student record and LLWR


Key field used: MODE

Graduate spin-out companies, active and

surviving at least three years

Data from Higher Education Business and

Community Interaction Survey (HEBCIS)


Income from Research Councils

Data from HESA Finance Statistics Return Table 5b

Similar to Reaching Higher target

Target constructed differently

Annual income of HEIs

Data from HESA Finance Statistics Return Table 1

Risk status of HEIs

Risk classification from HEFCW institutional risk

review process


HEFCW circular W10/21HE requests forecasts for

following indicators by 10 September:

Communities First students

Module completion rates

National Student Survey ‘overall satisfaction’ scores

Welsh medium activity

Employability performance

Spin-out company performance

Forecasts for other indicators from financial and

student forecasts and other data sources