DLHE advanced 2012. Objectives Methodology Recap and reflect on the methodology Questionnaire and data Review the questionnaire and data whilst learning

DLHE advanced

2012

Objectives

MethodologyRecap and reflect on the methodology

Questionnaire and dataReview the questionnaire and data whilst learning how to overcome common issues and challenges

Data submission and quality checkingUnderstand the validation kit and changes to data collection

Preparing for auditFind out about the audit process and findings from HEFCE

DLHE and KISLearn how to use the downloadable files

Where to begin?

The review

• The review of DLHE took place ahead of 2011/12• The sector was consulted through DLHE contacts

who were invited to respond to the proposals• Changes were subsequently introduced for 2011/12

and communicated through seminars, circulars and the coding manual

Today we recap and reflect on the changes and aim to draw on institutional experience to improve on

procedures, practices and dealing with challenges…

CoverageStudent

completed during HESA

reporting period

Obtained applicable

qualification

Reported as active (or as

dormant for PGR students only)

Studied for 8 weeks+ in UK

Not a visiting/exchange

student

Reason for ending not a

result of death

Coverage

The survey population has been extended to include a wider range of qualifications

QUAL

The survey population now includes all qualifying postgraduate research students

including those from a dormant statusMODE

The survey population no longer includes incoming exchange or visiting students

EXCHANGE

The survey population now includes all non-EU international leavers

DOMICILE

DLHE Coverage Flowchart

So much to do in so little time…

• HESA recognise that some institutions will now be surveying a higher volume of leavers in comparison to earlier years

– The contact period has not been increased – Resource and budget will be impacted

It is important to note

Surveying a wider range of students generates a more diverse set of results and will potentially provide more data…

DLHE 2011/12

Target response rates

0%

20%

40%

60%

80%

80%50%

70% 80%

How did we do last time?

60.0%

65.0%

70.0%

75.0%

80.0%

85.0%

90.0%

95.0%

100.0%

Response rates 2010/11 UK-domiciled leavers who studied full-time (First degree)

Response rateTarget

Institutions

How did we do last time?

50.0%

55.0%

60.0%

65.0%

70.0%

75.0%

80.0%

85.0%

90.0%

95.0%

100.0%

Response rates 2010/11 UK-domiciled leavers who studied part-time (First degree)

Response rateTarget

Institutions

The KIS agenda

• Key Information Set – Course level information for prospective students– Uses DLHE data

KISCourse information

Financial information

Accreditation

Accommodation

Employment

Assessment

At course level, there must be at least 23 or more responses and for salary information, at least 50% of the leavers (if 23+) who are

employed full-time must have provided salary information

Promoting DLHE

Evidence shows there is a link between those institutions who promote DLHE and those who exceed target response rates

Why not consider graduation ceremonies, graduate contact, internal publicity or offering prizes?

Surveying the correct students

What if it goes wrong?

• Missed entirely (cohort)– Contact institutional liaison for further information

• Missed entirely (individual)– Resurvey using correct questionnaire for intended census

These coverage errors can be corrected at a later date but are best avoided

What if it goes wrong• Dormant

– Students sometimes reported as dormant if in the final stages of postgraduate study at the start of reporting period 2. This is a student reporting discrepancy and will result in the institutions own record systems not identifying students for DLHE purposes

• Wrong census – Some institutions own record systems only allow for one end

date to be recorded. HESA defines the end date as at the point the structured part of the course finishes whereas some institutions consider this to be the date they are awarded. If the end date is recorded as the award date then the student could be potentially surveyed for the wrong census

These coverage errors usually go unmissed and cannot be easily corrected at a later date and therefore must be avoided

Methodology

Methodology

• 2011/12 saw the introduction of a new more relaxed approach to DLHE methodology

• Institution’s now decide when and how to contact leavers during the contact period (initial and follow-up no longer exist)

• Using predetermined methods to contact leavers was viewed as restrictive…

It’s now up to you • The new methodology now allows for the use of as many methods as possible during the contact period

• With the view that by increasing the lines of communication, the number of responses will increase

• Institutions are advised to review previous collection response rates for an indication of methods which attract the highest response rates

Highest response rate is key

Methods

Number of

methods

Sequence of

methods

Methods of contact

• Postal mailing

• Telephone questionnaire

• PDF version (e-mail)

• Centrally-hosted online questionnaire

Paper questionnaire

• HESA provide paper questionnaires for sending out by post

• Questionnaires are also available in Welsh

• Institutions are contacted by HESA with a set of questions about the size of the DLHE target population and the planned surveying arrangements in January (for April) and June (for January)

Common issues

Pros• One version of the

questionnaire simplifies the DLHE process, is more cost effective and easier to manage

Cons• Third party questions are

not identified• Coding boxes have been

removed

• Laminated copies showing third party questions

• Create coding reference material

Suggestions

PDF questionnaire

• A PDF version of the questionnaire is available in both English and Welsh

• Available to download from the HESA website

• PDF questionnaires are emailed to leavers who are then required to print, complete, sign, date and return the form back to the institution

• Institutions should consider using postage-paid returns to increase likelihood of responses. Leavers will not respond well to having to pay postage

What are the chances

Smartphone anyone?Do you open your emails from your smartphone?Will you go back to emails read from your smartphone?Is your smartphone remotely connected to a printer?

• Whilst PDF remains a fairly problem free approach to surveying leavers, it is worth keeping in mind that response rates to emailed PDFs may be lower than other methods

• Therefore this method should only be considered in combination with one or more other methods

DLHE online system

• Online version of the questionnaire (Welsh version also available)

• Institutions provide link www.dlhe.ac.uk to leavers

• HESA provide free of charge to institutions who register for system

• Also know as ‘Centrally-hosted online DLHE survey’

• Has in-built validation to ensure questions and sections are answered

• Leavers can complete it at their own accord and at their own pace

More than just…

An online DLHE survey tool

Why not use the (free, maintained and updated by HESA) DLHE online survey to input responses?

Inbuilt validation will ensure all valid questions are both answered and internally consistent

Remember to adjust the STATUS field to reflect how the information was captured!

Telephone

• Often the most effective method – accounts for majority of responses but requires the most significant amount of resource

• Callers require perseverance, an excellent telephone manner and a sound grasp of the requirements. Some institutions recruit students as callers and multilingual speakers are sometimes used

Telephone best practice

When to call• Experience shows calling

outside of standard working hours yields better results

• Specifically between 5.30pm and 8.30pm is often considered the best time to call

• Also consider employing staff to contact leavers at the weekend

• A cut off time should be agreed and adhered to

• Log the times of calls to establish peak or optimum times for future surveys

Preparation• Time is of the essence - know

your leaver before calling (create a profile)

• Note the qualification awarded to the leaver vs. what they had originally aimed for

• Do not block your telephone number

• Ensure permissions are set to allow calls to mobiles

• Have the correct questionnaire for the survey period

Third parties

• If a leaver is unobtainable, a third party can be used as an alternative source in providing the required information on behalf of the leaver

• Most commonly this scenario will take place during a telephone call (some third parties have been known to also return paper questionnaires)

• Third party information is considered legitimate on the basis they are fully confident the answers are correct (and relate to the applicable census date)

• Third parties should only be asked a specific subset of the questions

Other methods

• Direct contact– During the field work period e.g. they visit the careers

office or during the graduation ceremony

• Academic departments Last resort – Can also be used as a source of information as they may

have on-going interactions with the leaver(s)

• Employers Last resort – If employer is known to the institution they too may be

contacted

Covering letter and email

It is mandatory for all leavers to receive specific information in connection with DLHE. The covering letter, email or verbal confirmation provided during a telephone call;

It is possible to include the covering letter or email for all methods with the exception of using telephone. If the institution opts to use the telephone as their first method it is important to remember to confirm this information verbally as part of the introduction

Tells the leaver that they will be contacted again if they do not reply

Informs the leaver that

data may be collected from a third party if they can’t be

contacted themselves

Gives the leaver an

opportunity to refuse to give

data or be contacted

again

Tells them that they may be

contacted again for the longitudinal

survey

Covering letter and email

• Printed on headed paper (covering letter)

• Do not amend HESA text

• Include institutions own data protection notice

• Include literature about the careers service, forthcoming alumni events

• Branding helps

Interactive task

Discussion questions

• Map the DLHE process within your institution -consider what additional tasks you might do as an institution

• What barriers exist when undertaking DLHE?

• How is the DLHE data used internally within your institution?

The questionnaire

Understanding the questionnaire

• Only one version of the questionnaire is available and this is to be used for all methods

• Specific questions must be answered for a valid response! Institution’s commonly follow up incomplete or illegible responses

• Remember, third parties should only be asked set questions. There are no markings on the form to indicate these questions

• The need to ‘pad out’ (‘X’ or default codes) the data/answers for questions which are not required or returned is no longer necessary due to the format (XML)

Official use only

STATUS

• The top boxes are used to record the dates of attempted contact

• These are not used to indicate how many times an institution should attempt contact

• Many institutions attach their own labels to include more dates

• The boxes on the bottom row are used to record the STATUS of the response:

• (2) Telephone survey: HE leaver• (3) Telephone survey: third

party• (5) Own institution’s student

record• (6) Institutional third party• (7) Deceased• (8) Explicit refusal NEW

Section A

Activities

Section A: Q1

ALLACT and MIMPACT

Required for a valid response

To be asked third parties

Ensures remainder of information returned is internally consistent

If the leaver does not complete any activities or completes more than one activity but does not identify which is most important, in the first instance try to contact the leaver. If this is not possible use professional judgement to code ALLACT and MIMPACT based on completed section(s)

A new concept…

• The leaver (or third party) must decide which one activity is ‘most important’

• This doesn’t necessarily mean it has to be the job which takes up the most time or pays the most salary, or any further study, training or research which takes up the most time. Some leavers may opt to select the activity which will support their future circumstances/aspirations

Section B

Employment

Section B: Q2

JOBSNO

Not required for a valid response


Will be used to understand the extent of portfolio careers

No requirement to return less than 2 jobs and only able to return up to 9 jobs

A new concept…

• If JOBSNO is more than one, leavers are asked to provide details about what they consider to their main job

• This doesn’t necessarily mean it has to be the job which takes up the most time or pays the most salary! Some leavers may opt to choose ‘on an internship’ or ‘Developing a professional portfolio/creative practice’ above full-time, paid work

Section B: Q3 and Q4

JOBTITLE and JOBDUTIES

Required for a valid response (at least one of the two)


Determines occupations that recent leavers from HE have entered and identifies leavers who are employed on postdoctoral research contracts

Must code up to highest level of detail. SOCDLHE2010 allows for up to 5 digits of detail with 423 codes available in total!

SOCDLHE2010 and KIS

• SOCDLHE2000 replaced with SOCDLHE2010, a more developed coding frame reflecting the changing nature and scope of employment activities (423 codes)

• A new version of CASCOT has been released to reflect the move to SOCDLHE2010 and is downloaded from Aardvark

• KIS will include a proportion of those leavers who go into occupations classified within SOC groups 1-3 ‘Professional/Managerial’

SOCDLHE2010 and KIS

• Some previous SOC codes (SOC2000) have been moved from groups <4 into >3 (e.g. paramedics)

• The table below shows examples of the relationships between SOC2010 codes, SOC groups and KIS

Section B: Q5

EMPBASIS



Allows analysis of trends in the labour market and the demand for HE leavers

If more than one box ticked, code the one nearest the top of the list as ordered in the valid entries (coding manual not questionnaire)

Section B: Q6, Q7 and Q8

SALARY, PTHOURS and ESTEARN


Not to be asked third parties

SALARY enables analysis and comparison of pay in different employment sectorsPTHOURS will be used to calculate the full-time equivalent of part-time salaries

If leaver has indicated unpaid work as their EMPBASIS then no value should exist for SALARYJOBNO must exist in order to ESTEARN to be returnedException warnings exist for SALARY <£10,000 and >£75,000

SALARY and KIS

• The information returned in SALARY will feed into the KIS (providing threshold criteria met)

• Only salaries for full-time employment (ALLACT) will be included in KIS

• Any salaries below £10,000 will not be included in KIS




Determines the type of industry that leavers enter

HESA now derive SIC2007 codes on behalf of institutions based on information provided in Q9 and Q10Informative answers (to both questions) allows for more accurate coding (up to 60 characters can be returned for EMPTITLE and 200 for EMPDUTIES)

EMPNAME and MAKEDO

Section B: Q11

Required for a valid response (at least one has to be provided)


Enables analysis of graduate mobility and supports the SIC2007 coding

Use online tools to support investigative work – Google maps, Royal Mail, search engines etc. Postcodes are the most accurate form of recording location(s) and are easily found online

EMPPLACE, EMPPCODE and EMPCOUNTRY

Standard Industrial Classification (SIC) code

• Centralised SIC coding will be completed by Oblong on behalf of HESA (on behalf of institutions)

• Data items – EMPCODE– EMPCOUNTRY– EMPPLACE– MAKEDO– EMPNAME

• 4 days for completed coding following submission




Allows for better understanding of the qualification requirements and variances for obtaining different jobs


QUALREQ and EMPIMP

Section B: Q14



Captures the different reasons why leavers decide to take different jobs

If the leaver does not complete any reasons or completes more than one reason but does not identify which is the main reason, please try to contact the leaver as it is not possible to use professional judgement to code this particular question

JOBRSNALL

Section B: Q15



Provides information to current students on the different sources used to find jobs


JOBFOUND




Used to find out how employment histories affect outcomes and to provide current students with information

If more than one box ticked (PREVWORK), code the one nearest the top of the list as ordered in the valid entries (coding manual not questionnaire)

PREVEMP and PREVWORK

Section C

Newly qualified teacher status

Section C: Q18 and Q19

EMPLDTEACH required for a valid responseGTCSTIS not required for a valid response (and only applicable to Scottish institutions)

EMPLDTEACH to be asked third partiesGTCSTIS not to be asked third parties

Used to identify those in teaching jobs and those in teaching jobs through the GTC Scotland Induction Scheme

EMPLDTEACH and GTCSTIS

Section C: Q20 and Q21



Identifies the split between the sectors and primary and secondary levels


TEACHFUND and TEACHPHS

Section C: Q22



To identify leavers who are seeking a teaching post

SEEKTEACH

Section D

Further study, training or research

Section D: Q23



Used to capture the total number of courses leavers are undertaking

No requirement to return less than 2 courses and only able to return up to 9 courses

COURSENO

A new concept…

• If COURSENO is more than one, leavers are asked to provide details about what they consider to their main study, training or research

• This doesn’t necessarily mean it has to be the study, training or research which takes up the most time or awards the highest qualification. Some leavers may base their decision on their future plans

Section D: Q24



Used to monitor the types of further qualifications leavers aim for next


TYPEQUAL

Section D: Q25 and Q26

Required for a valid response (at least one of the two)


Allows analysis of subjects of further study, training and research

Third parties should now be asked both questions (change to coding manual)

COURSENAME and COURSESUBJECT

Joint Academic Coding system (JACS 3.0) code

• Owned, maintained and updated (Version 3.0) by both HESA and UCAS and used to code HE provision subjects

• Should be coded to the highest level possible (1552 codes available), however no coding tool available

• Field only allows one JACS code to be returned. If combined studies, leaver must specify one and if returned by post, institution must use professional judgement in returning one code (minor/major will provide an indication)

Section D: Q27



Used to derive UCPROV which monitors the movements of leavers

If the name of the institution cannot be derived to a UK university or college, Non-UK university or college or UK FE college, institutions should use their own judgement when deciding on coding as Other public body in the UK or Other private body in the UK

UCNAME

Section D: Q28



Provides information to current students about out how leavers fund their study, training or research


STUDYFUND

Section E

Overall Higher Education experience

Section E: Q29, Q30 and Q31



Will be used to understand how well the leaver feels that their HE experience provided them with skills, competencies and knowledge for the applicable activities

Leavers should complete (and be asked) all questions regardless of current activities / activities on census date

HEWORKEXP, HESTUDYEXP and HEBUSNEXP

Section F

Thank you

Section F Thank you for completing this questionnaire

The end is near but the final part of the questionnaire will ultimately determine whether the responses can be returned to HESA…

The form must be signed and dated!

This can be by the leaver, caller or even a third party

Remember only leavers should be asked if they do not want to be included in the follow-up survey. If the questionnaire is completed by a third party the leaver must be excluded from longitudinal DLHE

Data collection

The data model

XML

• XML standards for eXtensible Mark-up Language

• It is the international standard for data transfer

• Hierarchical data structures in a single file

• Survey responses will be inputted into a student record system or database which will be capable of producing an XML DLHE file

• The correct format structure must be followed to avoid errors

XML template in Excel can be used exceptionally

Validation kit

• Available to download from HESA website now• Allows for local validation (download it to PC as

opposed to a network)…• …ensuring you maximise submission time• User friendly interface• One kit framework for all HESA XML data collections• Framework connects to web to download latest rule

version• Demonstration of kit….

Submitting data

• The process of submitting data remains unchanged• NEW TEST_COMMIT transaction• NEW! Data supply files available at COMMIT….• …downloadable, flattened data files which also

contain derived fields used for analysis• Match reports B and C should be looked at, however

it is okay to send data for students not in POPDLHE – we simply will disregard it

• Use the TQI downloadable files!

Check documentation….

Sign-off

• Data should be signed-off by the Head or Acting Head of institution

• Some evidence to suggest this is not happening

• Escalate responsibility!

HEFCE

Preparing for audit

Welcome slide

DLHE Advanced SeminarPreparing for audit

Leeds25 September 2012

Ewa Wawrzynska

In spring 2012 HEFCE conducted a pilot exercise of audits of the DLHE 2009/10 survey:

• New area of HEFCE audit activity;

• 12 institutions selected;

• Reasons for audit activity:

• DLHE feeds into KIS;

• concerns over data reliability within the sector;

• need for assurance over the DLHE process in institutions;

• identifying good practice at institutions;

• identifying areas of difficulty, providing advice on these and improving training and guidance on these where required.

Pilot exercise of audits of the DLHE survey

• HESA involvement;

• Summary report on the findings to be published on the HEFCE website.

Pilot exercise of audits of the DLHE survey (continued)

• Audit of the DLHE process and how institutions implement it, rather than the primary source of the data;

• No evidence of data manipulation found;

• Our audit work in this area will continue.

Key messages

Overview of the audit process:

• Pre-visit data analysis;

• One day on-site visit;

• Discussion of the DLHE process;

• Discussion of the data;

• Detailed testing – sample;

• Feedback meeting at the end of the visit;

• Reporting arrangements.

Audit programme

• Insufficient Data Protection Notice;

• Lack of evidence of attempts made to contact the leavers, for leavers for whom no response was obtained;

• Sign-off by individuals other than the Head of institution;

• Implications of using the centrally hosted online DLHE survey as a data entry tool;

• Institutions returning data for unanswered multiple choice questions;

• PROFSOCT field coding;

• Salary question;

Issues encountered

• Audit trail to the supplementary data;

• Lack of supporting evidence for data returned to HESA (individual cases);

• Responses obtained from leavers not returned to HESA (individual cases at two institutions, sample size was 50 in both cases);

• Sporadic data entry errors.

Issues encountered (continued)

• Timing of the visits;

• Variable practices around advertising the DLHE survey;

• Good practice points;

• Complaints about the timing of the survey;

• Match B and Match C reports;

• Validation of responses;

• Interpretation of responses;

• Date on the questionnaire (front and back).

Discussions with HESA.

Other observations

• Summary report to be published on the HEFCE website;

• Audit programme to be published on the HEFCE website;

• This activity will continue for the foreseeable future.

Next steps

Thank you for [email protected]

DLHE and KIS

TQI tables

Key Information Set (KIS) and DLHE

• DLHE data is used extensively in the KIS• Remember the publication thresholds:- 23 respondents must have responded- For salary data to be published at least 23 students who are

employed full-time must have given a salary and these must represent at least 50% of the students employed full-time

• Where thresholds are not met the data will be aggregated from a course level to a subject level

• Salaries under £10,000 will not be included• Respondents who are working full-time and studying full-

time will not be included

Using the TQI tables to benchmark for KIS…

[email protected]

HESA information and discussion forum for HESA DLHE

These lists are used to circulate email news alerts from HESA regarding data requirements, coding manuals and validation kit releases

Want to join? email [email protected] or visit www.jiscmail.ac.uk

mailto:[email protected]

http://www.jiscmail.ac.uk/

Data collection and submission

The Institutional Liaison team are there to help with all aspects of the return from local collection of data items through to final submission and sign-off

Contact the [email protected] 211144

mailto:[email protected]

Training support

If you require additional training help, including bespoke visits to your institution, get in touch with the training department…

w: www.hesa.ac.uk/traininge: [email protected]: 01242 211472

Follow us on Twitter: @HESATraining

Documents

DLHE advanced 2012. Objectives Methodology Recap and reflect on the methodology Questionnaire and data Review the questionnaire and data whilst learning