Just before we get started… Who am I? How questions will be handled Resources available after the webinar Key QILT Dates & the AGS Item Review 2

Just before we get started…

• Who am I?

• How questions will be handled

• Resources available after the webinar

Key QILT Dates & the AGS Item Review 2

Key QILT dates &the AGS Item Review

Quality Indicators for Learning and Teaching

About the Social Research Centre

• We are a private, for-profit company owned by ANU Enterprise

• Our services include: survey design and execution, qualitative research, survey data management, statistical consulting and analytical reporting.

Key QILT Dates & the AGS Item Review4

Overview

1. Student Experience Survey (SES) timeline

2. Graduate Outcomes Survey (GOS) timeline

3. Employers Experience Survey (ESS) timeline

4. Australian Graduate Survey (AGS) Item Review


SES timeline

• Six main components in the SES timeline

• Sample draw & verification

• Additional items / populations specification

• Pre-fieldwork promotion

• Fieldwork / data collection

• Data preparation, coding & output production

• Dissemination & distribution

• The data collection period is fixed so we have very little flexibility with these dates or the ‘upstream’ dates


SES: sample draw & verification

• The sample draw and verification procedure is identical to the procedure used for the 2014 University Experience Survey (UES).

• The Social Research Centre:

• Identifies in-scope students from HEIMS

• Requests institutions append students’ contact information

• Requests institutions flag students that have withdrawn, deferred or have passed away.


SES: fixed dates

• 8th June: Close off for inclusion of additional populations.

• 22nd – 26th June: Sample files sent to institutions.

• 3rd July: Close off for inclusion of additional questions.

• 10th July: Final day to return sample files.

• 17th July: Final sign-off on scripted additional questions.

• 20th July: National social media campaign commences.

• 3rd August: Data collection commences.

• 31st August: Data collection concludes.


SES: data cleaning & production

• Strict data input controls = very clean quantitative data

• Cleaning of course names is required where the student has updated the course name in the survey

• Please respond as quickly as possible to coding queries (this is the main delay in producing final files)

• Draft data files are sent FYI but you are welcome to do a ‘sense check’ particularly if this is your first SES


SES: (slightly less) fixed dates

• 7th – 25th September: Coding queries sent to institutions.

• 14th – 16th October: Draft data files sent for checking.

• 23rd October: Final day to return data files (if necessary).

• Before 30th November: Final Institutional Data Files.

• 31st December: Final Institutional Reports.

• 31st December: Final National Report.

• 31st December: Final National Data File.

• Report and national data release date?


SES timeline

The SES data collection period is 'locked in'. Start planning now!

If you have non-standard teaching periods or holidays in August, contact us ASAP

If this all looks confusing / overwhelming, contact us ASAP


GOS timeline

• The SES and the GOS follow the same process:

• Sample draw & verification

• Additional items / populations specification

• Pre-fieldwork promotion

• Fieldwork / data collection

• Data preparation, coding & output production

• Dissemination & distribution

• The data collection period is also fixed.


GOS: sample draw & verification

• The sample draw and verification procedure is identical to the SES.

• The Social Research Centre:

• Identifies in-scope students from HEIMS

• Requests institutions append students’ contact information

• Requests institutions flag students that did not graduate or have passed away.

• (We are still working on the postgrad research issue)


GOS: courses vs majors

• Some GOS questions ask students to think about their qualification

• Institutions can reference GOS responses to:

• Courses or programs

• Majors from administrative records, or

• Majors supplied by the student.

• We do not recommend the use of student supplied major (reduced data quality)

• Student supplied majors will attract an additional coding fee


GOS: fixed dates

• 7th September: Close off for inclusion of additional populations.

• 7th September: Final day to indicate that majors will be used.

• 21st – 25th September: Sample files sent to institutions.

• 2nd October: Close off for inclusion of additional questions.

• 9th October: Final day to return sample files.

• 16th October: Final sign-off on scripted additional questions.

• 19th October: National social media campaign commences.

• 2nd November: Data collection commences.

• 30th November: Data collection concludes.


GOS: data cleaning & production

• Strict data input controls = very clean quantitative data

• Cleaning of course names & majors is required where the student has updated or provided these in the survey

• Please respond as quickly as possible to coding queries

• Draft data files are sent FYI – we would suggest a ‘sense check’ given this is the first GOS


GOS: (slightly less) fixed dates

• 1st – 11th December: Coding queries sent to institutions.

• December: Draft data files sent for checking.

• December: Final day to return data files (if necessary).

• January: Final mid-round Institutional Data Files.

• Institutional Reports and the National Report are not produced until the April Round has been finalised (tentatively, August 2016).


GOS timeline

How will you communicate the change from the AGS to the GOS within your institution

and to your potential graduates?

Do you need to supplement the GOS with questions or graduate populations?


ESS timeline

• The ESS is ‘live’ at the same time as the GOS.

• All employed graduates are asked to provide contact information for their supervisors

• Supervisors can receive an email about the survey as soon as the graduate has provided contact information.

• There is nothing for you to do with the ESS – just be informed about the survey so you can answer general questions if they arise.


ESS: dates

• 19th October: National communications campaign commences.

• 2nd November: Data collection commences.

• 22nd December: Data collection concludes.

• ESS has a CATI non-response follow-up phase

• February: Mid-round data files sent to institutions.


ESS timeline

The ESS is ‘set-and-forget’ – you don’t need to do anything!

Fact sheets and FAQs will be provided to institutions to help with briefings and any

questions


AGS Item Review

• In December 2014, 35 universities provided feedback about the GDS and the CEQ.

• Each AGS item was categorised as essential, important, interesting and irrelevant to institutional KPIs and performance management frameworks

• Many institutions provided useful commentary on their ‘ratings’ and additional feedback about the AGS


GDS feedback

• Of the 70 GDS items and 38 were identified as essential or important. Of these 38:

• 8 demographic items are available from HEIMS

• 23 are labour force items

• 5 relate to further study, and

• 2 are demographic items that may not be available from HEIMS.


CEQ feedback

• The Overall Satisfaction Item, the Generic Skills Scale and the Good Teaching Scale were regarded as essential.

• The Graduate Qualities Scale and the open-ended items (Best Aspects & Needs Improvement) were identified as important.

• Additional comments suggested that these CEQ concepts are still very relevant to institutions but that the implementation is no longer contemporary.


Next steps for the GOS

• GDS items have been aligned with Australian Bureau of Statistics standards (mostly from the Labour Force Survey).

• The CEQ scales are still under consideration.

• An initial version of the Graduate Outcomes Questionnaire (GOQ) has been drafted. We will require feedback regarding the ‘non-ABS’ items to ensure that they are still fit for purpose.

• Feedback guidelines for the draft GOQ are being prepared.


GOS trial

• For almost all institutions, the next round of the AGS will be ‘business as usual’ – no institution will be required to participate

• As soon as a draft questionnaire has been approved by the department we will approach potential trial institutions

• If you would like to be a trial institution, you are welcome to contact us to discuss options


ESS trial

• The GOS and the ESS are linked so institutions participating in the GOS trial will also be part of the second ESS trial.

• The next ESS trial will:

• Refine the supervisor recruitment methodology

• Test a modified and improved questionnaire


AGS Item Review

Essential and important GDS items are being redeveloped.

Key CEQ scales are 'moving to the next round'.

A draft GOQ will be circulated as soon as possible.


Working with the QILT team

• We are always available to

answer questions (no matter how vague)

provide advice about research methodology or data collection

explain technical aspects of the QILT surveys or the data files

assist with the inclusion of additional questions or populations

• If you have been liaising with a specific person, please contact them directly, otherwise, [email protected] will always work.


To summarise

• The SES data collection period is 'locked in'. Start planning now!

• Do you need to communicate internally about the GOS? Do you need to supplement the GOS?

• Be aware of the ESS so you can answer questions.• Essential and important GDS items are being

redeveloped. Key CEQ scales are 'moving to the next round'.


And finally…

• Block out the SES survey period so participation for your students can be maximised.

• Please start discussions now about any additional populations or questions for the QILT surveys (noting the close-off dates).

• If you have any concerns or questions, please contact the QILT Team.


Documents

Just before we get started… Who am I? How questions will be handled Resources available after the webinar Key QILT Dates & the AGS Item Review 2