Quality Indicators for Learning and Teaching

2014 Survey Managers’ Information Forum18 July 2014

Dr Andrew TaylorBranch Manager, Higher Education Data and Analysis

2014-15 Budget Measure

• 2014-15 Budget measure Upholding Quality – Quality Indicators for Learning and Teaching (QILT)

• QILT – suite of surveys over the student life cycle, from commencement to employment

• New, purpose built website to publish results and inform student choice

• Indicator development for performance funding

• AQHE Reference Group, chaired by Professor Ian O’Connor

• Consultation process:

– Discussion papers

– Submissions

– Roundtables

Development of Indicators

• Survey burden on students and institutions

• Centralised administration

• Employer surveys ad hoc and not well done

• Need for robust measures of teaching performance

Key issues

AQHE Report

Key Reference Group recommendations:• Centralised administration of the suite of performance

measurement instruments under a single third-party provider• The suite of instruments to initially comprise the University

Experience Survey (UES) and a new Graduate Outcomes Survey (GOS)

• Scoping study on a new instrument to assess employer needs and satisfaction with graduates

Not recommended:• Composite Teaching Quality Indicator• Collegiate Learning Assessment

MyUniversity website

• Live in April 2012• Clear, meaningful and transparent information• Confidence intervals• Subject by institution level data

• Amendments made to the Higher Education Support Act 2003 (HESA) in November 2013

• Personal information can now be provided to a range of bodies, including– TEQSA– higher education providers and some peak bodies– state governments

• Safeguards remain– Limited uses– No on-sharing– Provider consent

• Universities Australia data sharing agreement allows access for member institutions

Data sharing

QILT Components• Three surveys:

University Experience Survey (UES)Graduate Outcomes Survey (GOS)Employer Satisfaction Survey (ESS)

• Plus new purpose built QILT website to publish results from the surveys

• QILT contractor selected through tender to administer all elements

Timing

2014 2015 2016 2017

University Experience Survey

August August August August

Graduate Outcomes Survey

n/a October round April and October rounds

April and October rounds

Employer Satisfaction Survey

October June June June

Website August – initial release

February and August updates

February and August updates

February and August updates

University Experience Survey

• 2014 UES administered by GCA/SRC consortium • 2015 UES and 2016 UES comprise part of QILT• Methodology in line with previous administrations

National results, key items, 2011-2013

2011 2012 201350

55

60

65

70

75

80

85

90

Percentage satisfied, quality of teaching item

Commencing Later Year Total

2011 2012 201350

55

60

65

70

75

80

85

90

Percentage satisfied, quality of en-tire education experience item

Commencing Later Year Total

National results, scales, 2012-2013

Teaching Quality Learner Engagement Learning Resources Student Support Skills Development0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

81

58

82

53

8279

57

83

53

79

Percentage satisfied, UES focus areas

2012 2013

International benchmarking

2008 2009 2010 2011 2012 201370

72

74

76

78

80

82

84

86

88

90

Entire educational experience rated positively, UES and NSSE, 2008-2013

NSSE 1st year NSSE senior year UES 1st year UES later year

Perc

enta

ge s

atisfi

ed

Graduate Outcomes Survey

• GOS will provide timely labour market information• Review existing AGS instrument to develop new GOS

instrument and methodology • Reporting methods need review• Survey administered in two rounds October and April

(following year), though alternative proposals considered• 2015 AGS administered by GCA• GOS to replace AGS from the second half of 2015 • 2016 GOS and 2017 GOS comprise part of QILT

Employer Satisfaction Survey• ESS measures employer satisfaction with generic skills, technical

skills and work readiness of graduates• Recent graduates provide contact details of direct supervisor• Pilot ESS in 2013-14 across 4 universities and 5 fields of education• Pilot ESS report available from: www.education.gov.au/employer-satisfaction-survey• Further development of ESS in October-December 2014

instrument and survey methodology to be reviewed and tested on larger scale

universities participate opt-in basis• 2014 ESS trial, 2015 ESS and 2016 ESS comprise part of QILT

ESS – key resultsOverall 92 per cent of employers stated they would be confident in recommending someone with the same qualification from the same university for a similar position to the graduate’s role

Overall satisfaction by key factors (skill clusters) Foundation skills — 95 per cent Adaptive skills — 89 per cent Teamwork and interpersonal skills — 96 per cent Disciplinary skills — 93 per cent Employability skills — 93 per cent Enterprise skills — 79 per cent

QILT Website• QILT website aims to inform student choice through provision of survey

results by institution and course • Two phases:1. Initial release

August 2014 release Existing data from the AGS and UES, to be provided by department Limited scope – undergraduates at Table A and B universities

2. Further development Developed progressively to August 2015 Expanded scope – all institutions and levels of study Enhanced functionality, e.g. search and comparison functions

enhanced Development and release of mobile app

• Website updated every six months from August 2014 with latest available QILT survey data

Governance

• QILT Working Group will provide advice to department on implementation of QILT

• QILT Working Group largely comprises former members of AQHE Reference Group

• QILT Working Group will include representation from non-university higher education institutions (NUHEIs)

General methodology

• Independent administration• Random stratified sample• Stratification variables are specified for each instrument• HEIMS data to be made available for constructing sample

frames• Additional information for the construction of sample frames,

along with student and graduate contact details, to be collected from institutions

Quality assurance• Biases and unrepresentativeness minimised• Quality assurance is based on achieving required confidence

interval, on specific items, for a set proportion of strata• UES (entire educational experience) and GOS (further full-

time study)– Interval of + or – 7.5 pp, at 90% confidence level– 75% of strata at the institution by subject area level,

excluding strata with a population of fewer than six• ESS (overall supervisor rating)– Interval of + or – 7.5 pp, at 90% confidence level– 8 out of 10 broad fields of education– 25 responses for each field at each institution

• Confidence intervals calculated as per 2013 UES

NUHEIs

• Currently around 85 Non University Higher Education Institutions (NUHEIs) accredited under HESA

• Limited participation in AGS to date, not in scope for UES• NUHEIs in scope for QILT:

2015 ESS 2015 UES (trial of NUHEIs in 2014 UES)2016 GOS (from October 2015 round)

• Survey results for NUHEIs to be included on QILT website subject to data being robust

Tender• Tender opened 13 June• Tender closed 11 July• Professor Shirley Alexander, UTS, to assist

Department in assessing tenders• Expect to announce outcome by the end of July• Market testing through tender for 2014-16 triennium

23

CONFIDENCE INTERVAL PAPER - ANU

http://unistats.anu.edu.au/surveys/toolkit/Confidence_Intervals_Sheet.pdf