Calibrating and benchmarking disciplinary standards: A practical session

HEQN Conference-Assessment, Integrity, Review: A sector in transformation

4th June, 2019

Dr Sara Booth, Academic Director, Online Peer Solutions

Presentation outcomes

UNDERSTANDING HIGHER EDUCATION STANDARDS AND DEFINITIONS

1

UNDERSTANDING A PROCESS TO ASSURE ACADEMIC STANDARDS

2

UNDERSTANDING AN ASSESSMENT QUALITY CYCLE

3

CALIBRATING AND BENCHMARKING DISCIPLINARY STANDARDS

4

USING THE PEER REVIEW PORTAL

5

Higher Education Standards

Higher Education Standards Framework (2015):

Monitoring, review and improvement Standards 5.3.1, 5.3.4 and 5.3.7; the specification of learning outcomes at Standards 1.4.1, 1.4.3, 1.4.4

TEQSA Guidance Note: External Referencing, including Benchmarking (v2.5 16 April 2019)

A number of approaches: benchmarking, peer review and moderation.

ELICOS Standards 2018, Standard P4.1c (ii) (Beta version) (23 April 2019)

in the case of ELICOS courses which are provided under a direct entry arrangement to a tertiary education course, formal measures must be in place to ensure that assessment outcomes are comparable to other criteria used for admission to the tertiary education course of study, or for admission to other similar courses of study.

TEQSA Guidance Note: ELICOS Direct Entry Beta Version 1.0 (23 January 2019)

The ELICOS Provide must be able to provide evidence that a valid and reliable mechanism (or combination of mechanisms) is in place to independently demonstrate comparability. Mechanisms include:▪ External referencing/benchmarking ▪ Benchmarking to validated language proficiency frameworks ▪ Tracer studies of student cohorts ▪ External testing

Key definitions

Academic calibration is an efficient and effective process that verifies academic judgements around the assessment standards across the sector.

Calibration is about a shared knowledge of standards, often achieved through social moderation processes in order to create ‘calibrated’ academics.

Benchmarking as a continuous and systematic process of comparing products, services, processes and outcomeswith other organisations or exemplars, for the purpose of improving outcomes by identifying, adopting and implementing best practice approaches.

External Referencing:External referencing should include national and international comparators.

Peer review of assessmentis a valuable means of validating that grades awarded reflect the level of student attainment, including through calibration of different markers’ grading (refer to Standard 5.3.4).

Flipping the Curriculum: A process to assure standards

KEY 1: Design PLOs

Assure the quality of programme level outcomes and

graduate capabilities in course design and review

KEY 2: Map PLOs

Ensure effective mapping of

programme level outcomes to

unit level ones

KEY 4: Grading

Confirm you have an agreed

picture of what indicators are

used to allocate different grade

levels

KEY 3: Assessment

Ensure assessment is fit-for-

purpose and valid

Constructive

alignment +

curriculum design for

assuring learning +

policies & processes

in quality & review

KEY 5: Calibration and

Benchmarking

Confirm that everyone who

will be grading assignments

applies the key indicators to

allocate grades in the same

way

KEY 6: Learning Methods & Resources

flipcurric.edu.au/Emeritus Professor Geoff Scott

Assessment quality cycle flowchart [Bloxham & Boyd, 2007]

Design Phase

Delivery Phase

Declaration

Phase

Review Phase

● Subject Coordinator/Teaching team designs assessment items and develops marking criteria, assessment rubrics to aid shared understanding.

● Review of assessment design by another member of staff with appropriate discipline expertise. ● Students completes assessment item

● Pre-marking calibration of marking team. Marking team marks completed assessments. ● Consensus marking: markers gather together and mark a sample of the same papers whilst addressing consistency issues as they

arise. ● Rotational marking-the assessment is divided into sections; each section is marked by the same marker● Post-marking moderation

● Post-marking Review ● Spot checks, check outliers, check borderlines, administrative error checking● Analysis of the pattern of marks and grades to identify trends and anomalies ● Reporting through academic unit results meetings of any issues that have been identified during the assessment quality cycle

process● Marks/grades released to students

● Review ● Blind/sample marking to check marking consistency ● Consideration of, and action in response to, feedback and outcomes from assessment quality assurance activities in order to

inform and improve subsequent delivery of the subject. This includes during subsequent course, subject and assessment design;teaching team calibration; marking and grade finalisation.

Why should academics participate in calibration sessions?

Achievement Matters Professor Kim Watty

https://www.youtube.com/watch?time_continue=272&v=8JrQYEySQD4

These sessions are critical for discussing their understanding of what the Threshold Learning Standards are and how they can be applied

Importance of developing a shared understanding of standards

It is important to unpack what has been met and what hasn’t been met…

Calibrating and benchmarking English language standards

University English Language Centres (UECA), 20 universities, 60 assessors, 3

months (2018-2019)

Common European Framework for Reference (CEFR), Calibration workshop

Broad consensus was that it was a

valuable tool [CEFR], though there was

discussion of other methods of

benchmarking and scales that could

be employed (e.g. Global Scale of

English from Pearson) or types of

analysis (e.g. Rasch Modelling of test

scores). The tool could be improved

through work samples at the levels, and

genre specific (e.g.

essay/report/annotated bibliography)

scales

One step further, unpack some issues

around vocabulary and grammar…

pointed out an iceberg, it needs to be

unpacked and addressed. If we do

have consensus broadly speaking, we

need to revisit it in a more granular level.

We would need to invest some

resources, or broader working group

to interpret different CEFR levels and

samples. We acknowledge the issues.

We have to make sure we have this

discussion.

This [the project] is brilliant and it has

given us an opportunity to collaborate.

Will it be an annual event? Will it be

every two years?

It was a professional development

exercise. We got our teaching staff to do

it.

TEQSA want evidence for external referencing, how do we formally communicate

that? We are benchmarking and we are also doing this… Real desire to move

away from IELTS and position ourselves in regards to academic literacy. How

do we form standards around academic literacy? How can we differentiate

ourselves?

Calibrating and benchmarking disciplinary standards

From 2018-2019, 200 subjects were externally reviewed with 33 HE institutions in 3 months (IHEA/ITECA)

Benefits Challenges

Explicit guidelines and process

• Process is straight forward matching

units of study

• Previous experience has proved to

be useful

• Clear expectations

• Guidelines for matching a reviewer

could be useful

Experts

• Broad range of experts

• The importance of content experts

• The importance of assessment

experts on constructive alignment

• Sense of competitiveness reduced

when operating at an individual

discipline level rather than

institutional level

Online Support

• Portal is easy to use, very stable and

secure

• Anonymous reports

• Potential to use A1 for matchmaking

Professional Development

• Professional development for staff

• Calibration of assessment and

standards

Operationalising external referencing

• Time required to collect all the self-

material requirements

• Matchmaking reviewers on Excel

spreadsheet

• Zoom meeting size with some

disciplines

• Friday afternoon a challenge to make

meetings

Reviewers

• Finding reviewers for engineering,

hospitality and tourism

• Concern about the other side’s

standards, qualifications and

experience

• Do reviewers go through a certification

process?

• Finding international reviewers

Online Support

• Can reviewers be incentivised?

• Search engine for discipline experts

• Problems with Portal with uploads

What is the Peer Review Portal?

The Peer Review Portal, which has been endorsed by TEQSA, is an optional online support mechanism for higher education institutions for external peer review and external referencing activity.*

This cloud-based system has the added advantage of no infrastructure costs or ongoing IT development costs. This allows your institution to establish an institutional reporting system at a minimal cost which makes more effective use of your resources.

https://www.peerreviewportal.com

*See TEQSA Guidance Note on external referencing [including benchmarking]TEQSA Guidance Note on ELICOS Direct Entry

What can the Peer Review Portal be used for?

Assessment:

Inputs/Outputs

Course and Unit Review Benchmarking Accreditation Support

Assessing learning outcomes

and assessment through peer

review of units of study

Assessing program-level outcomes

to support course and unit review

Assessing/calibrating data and

processes to compare and

improve standards

Supporting course and professional

accreditation requirements

● Intuitive dashboard

● A range of templates for

external review of

assessment

● Add de-identified student

work samples under

each assessment task

● Intuitive dashboard

● Unit Reports and Course

Reports

● A range of templates for

annual course reports

● A range of templates for

semester unit reports

● Calibration

● Intuitive dashboard

● Benchmarking Reports

● Benchmark themes

across the sector or

discipline group

● A range of benchmarking

templates

● Calibration

● Intuitive dashboard

● Accreditation reports to

support professional and

course accreditation

● Add Accreditation Panel

● A range of templates for

accreditation

Practical Session Calibration Activity

Click on the URL that was sent to you by email and

complete the review. Anyone with the link can join this

review project as a Reviewer.

https://www.peerreviewportal.com/self-join-

reviewer/CYS6rlQUuc122PeE1pvJvMZczsb8kezNa

gfeRXUOiJhdJvbSRLjor1euoUYJ18Yu

Practical session: Calibrating standards

Student Work Sample A

Soil Poster Task

Create a colourful poster which communicates a specific soil issue to the general public and farming community.

Task length A1 or A2 size poster • Creative thinking, creative expression, succinct communication skills.

Assessment criteria / guidelines • Engaging and creative poster • Clarity of message i.e., not too wordy • Tidiness of layout, quality and finish of poster • Use of clear and relevant references • No plagiarism

Review and discuss the assessment task, assessment criteria and student work sample. Please provide a grade for each work sample and a rationale for your decision.

Practical session: Calibrating standardsSoil Poster Task

Create a colourful poster which communicates a specific soil issue to the general public and farming community.

Task length A1 or A2 size poster • Creative thinking, creative expression, succinct communication skills.

Assessment criteria / guidelines • Engaging and creative poster • Clarity of message i.e., not too wordy • Tidiness of layout, quality and finish of poster • Use of clear and relevant references • No plagiarism

Review and discuss the assessment task, assessment criteria and student work sample. Please provide a grade for each work sample and a rationale for your decision.

Student Work Sample B

Practical session: Calibrating standards

Student Work Sample C

Soil Poster Task

Create a colourful poster which communicates a specific soil issue to the general public and farming community.

Task length A1 or A2 size poster • Creative thinking, creative expression, succinct communication skills.

Assessment criteria / guidelines • Engaging and creative poster • Clarity of message i.e., not too wordy • Tidiness of layout, quality and finish of poster • Use of clear and relevant references • No plagiarism

Review and discuss the assessment task, assessment criteria and student work sample. Please provide a grade for each work sample and a rationale for your decision.

Practical session: Calibrating standards▪ One assessment task and 3-4 student work samples in Agriculture

▪ Face-to-face workshop with 90 assessors across a range of disciplines (30 independent providers)

▪ Face-to-face workshop with 30 assessors across a range of disciplines (Chinese University of Hong Kong)

▪ Presentation at HES Conference

Key Themes

▪ Assessment criteria does not contain enough information regarding the quality of assessment and what level of work is expected. No weighting on different criteria.

▪ No assessment rubric to provide a framework for grading

▪ Learning outcomes are subjective rather than objective

▪ No indication of AQF level

▪ Poor assessment design

▪ Each sample was strong in some criteria and poor in others

▪ Student work samples ranged from fail to DN+

ReferencesBloxham, S. & Boyd, P. (2007). Developing effective assessment in higher education: A practical guide. Maidenhead: Open University Press.

ELICOS Standards (2018) (Beta version) (23 April 2019)

Peer Review Portal. Go to https://peerreviewportal.com

Scott, G. (2016). FLIPCurric. Go to https://flipcurric.edu.au

Higher Education Standards Framework (2015)

TEQSA Guidance Note: External Referencing (including benchmarking) (v2.5) (16 April 2019)

TEQSA Guidance Note: ELICOS Direct Entry Standards (Beta version 1.0) (23 January 2019)

Contact details for further information

Dr Sara Booth

Academic Director, Online Peer Solutions Pty Ltd

Mobile: 0474301991

Email: sara.booth@peerreviewportal.com