ASSURANCE OF ASSESSMENT QUALITY IN HIGHER EDUCATION UQ Vietnam Professional Development Seminar Series 4: April 2012 Ho Chi Minh City: Hanoi: Hue: Danang

ASSURANCE OF ASSESSMENT QUALITY IN HIGHER EDUCATION

UQ Vietnam Professional Development Seminar Series 4: April

2012Ho Chi Minh City: Hanoi: Hue: Danang

Dr Clair Hughes [email protected]

Senior Lecturer in Higher Education

The Teaching and Educational Development Institute (TEDI)

UQ-VN PDSS

Session overview

This session will address:

1. The influence of the international standards agenda on quality assurance practices

2. The importance of assessment in providing credible assurance of the quality of student learning

3. Approaches to the assurance of the quality of assessment tasks

4. Characteristics of assessment quality5. Approaches to the assurance of assessment judgements6. Implications for assessment quality assurance

responsibilities

Definitions

• Program - a period of study of several years duration leading to the award of a degree

• Course - a period of study usually of a semester’s duration – a component of a program

• Assessment - the process of forming a judgment about the quality and extent of student achievement or performance (Sadler 2005, p. 177).

• Evaluation - the process of forming a judgement about the quality of (in this context) an educational experience or one or more of its components

• Criterion – a distinguishing property or characteristic of anything, by which its quality can be judged or estimated. (Sadler 2005, p. 178)

• Standard - a definite level of excellence or attainment.(Sadler 2005, p. 189)

The assessment task

• Your colleague Ms Hoa is new to your department and has asked for your advice on an assessment task she has developed for a new course in English language. The main learning objectives are for students to:– improve their English vocabulary and pronunciation– develop awareness of the type of information discussed in a job

interview– present their skills and knowledge well to a future employer.

CRICOS Provider No 00025B

Activity 1:

• Ms Hoa has not taught this type of course before and so she asks your advice on this task:

• Her ideas include asking students to:A. think of 3 likely questions they will be asked in an interview and

prepare written answers to these questions

B. role-play a job interview with a partner using questions provided by the teacher

C. write an essay on how to behave in a job interview

D. prepare a CV and a letter of application

E. watch a video of a job interview and write about what went well and what could have been improved

• What feedback would you give her on these suggestions? Why?


Task quality

Focus of assessment quality assurance (1)

Assessment as

TEXTproduction

PURPOSE

TEXT TYPE

PurposeDoes the task allow students to demonstrate intended learning outcomes – it is aligned?

Type of text Does the task require the students to produce the type of work common to your discipline?

ModeDoes the task require students to use forms of communication used in your discipline – spoken, written, visual?

MediumDoes the task require students to use the media used in your discipline – face-to-face, webpages, posters, journals etc?

Subject matter or topic Does the task address significant subject matter?

Roles and relationshipsDoes the task give students a realistic role or audience – is it authentic?

Hughes 2009

Subject Matter

Roles and Relationships

Mode & Medium

Task quality- Example from Engineering (Project report)

Focus of assessment quality assurance

Assessment as

TEXTproduction

PURPOSE

TEXT TYPE

PurposeTask requires students to address learning outcomes related to teamwork, engineering construction and project management

Type of text The report is to be of the same quality as those produced by professional engineers

ModeThe report contains written and visual material and is also presented orally

MediumAn electronically produced report is common in the engineering industry

Subject matter or topic Project requires students to use wide range of engineering skills and reflect on teamwork skills

Roles and relationshipsReport requires students to take the role of a professional engineer working with a client

Hughes 2009

Subject Matter


Mode & Medium

Think of an assessment task with which you are familiar and analyse it as for the example on the previous slide

Activity 3: Task analysis

Assessment as

TEXTproduction

PURPOSE

TEXT TYPE

Purpose Type of text

Mode

MediumSubject matter or topic

Roles and relationships

Hughes 2009

Subject Matter


Mode & Medium

Focus of assessment quality assurance (2) • Also consider task in relation to:

• compliance with assessment policy and rules• clarity of assessment description• other tasks in this course (how they fit together)• assessment tasks in other courses studied at the same time

(spread, timing, workload)• assessment tasks across the program (how they allow for

progression in difficulty and skill development)• accreditation requirements of professional bodies• provision of feedback• its contribution to overall grade


Activity 2

• After seeking advice from a few people Ms Hoa decides on a 2-part assessment task:

1. Students develop a CV and letter of application that addresses a particular job they have found in a newspaper

2. Students role play a job interview with one student in the role of the interviewer and one in the role of the applicant

• She is anxious not to make any mistakes and asks you for further advice:

• Select two of the following as being important things for her to do:

A. Read the university assessment policy and guidelines

B. Talk to the program coordinator

C. Not to do anything as the task is already quite good

D. Look at the assessment for other courses in this department

E. Look at student survey ratings for other courses in this department

F. Read some articles on assessment in educational journals

G. OtherCRICOS Provider No 00025B

Approval

• The university has an approval process in place for new courses. Ms Hoa gains approval for the task from:– the program coordinator – The university Course and Program Approval Committees


Activity 3

• Ms Hoa wants to collect some evidence of her assessment quality so you advise her to (select two of the following):A. Make notes while students are doing the role-play

B. Nothing as it is more important to spend time on her research

C. Look at the assessment items on the student survey ratings

D. Ask some students for their opinion in a group interview

E. Ask the program coordinator to sit in on some of the role plays

F. Ask the program coordinator to mark some of the CVs and letters of application

G. Analyse what students did well or poorly on their written CVs and letters of application

H. Other?


Activity 4

• As there were a lot of students in the class, Ms Hoa had the help of 2 tutors in marking the CVs, letters of application and role plays. Some students complained that one of the tutors gave much lower marks than Ms Hoa and the other tutor. What advice would you offer her so that this does not happen next time the course is offered? Select two of the following.A. Keep some of the CVs and letters of application (with student permission) from

the first course and use them to train the two tutors before they mark student work

B. Develop a detailed marking guide and give it to the tutors

C. Collect a random sample of marked work to check the marks of both tutors

D. Video all student role-plays and look at a random sample of them to check the marks

E. Sit in on the first couple of role-plays and discuss tutor marks before they go ahead with the rest of the marking

F. Mark a couple of CVs and letters together before doing the rest of the marking

G. Meet with the tutors to discuss examples of high and low marks and any that were difficult to mark

H. Other


Activity 5

Global activity in Quality Assurance reflects a shift of emphasis from teaching to learning • Tuning in Europe, South America• Tuning USA – Lumina Foundation• Learning and Teaching Academic Standards project of the

Australian Learning and Teaching Council (now OLT)

• As a result, the Ministry of Education is asking universities to collect evidence that they are encouraging teachers to engage in the assurance of assessment quality. Your University conducts a survey and Ms Hoa is one of a group of randomly selected teachers who is asked to write a short report on what she has done to assure assessment quality in her course. List all the activities she could include in her report.


SUMMARY


The international standards agenda Assuring the quality of student learning outcomes through:

• articulation of minimum or threshold standards• standardised testing (AHELO, CLA)• external examining and moderation - comparison of

standards within and between institutions• development or revision of student satisfaction surveys

Funded projects to identify and support effective practice and explore emerging issues

http://www.itl.usyd.edu.au/projects/aaglo/

Assessment – a key curriculum component

Learning Objectives

What do we want students

to learn?

AssessmentHow will students demonstrate their

learning?

Learning activitiesWhat will students

do to develop their learning?

Teaching activities

How will teachers facilitate student

learning?Credible

evidence of student learning

outcomes

Assessment quality is the responsibility of.....

Senior management Faculty and school leaders

Program and course staff

• VC• DVC (Academic)• President of academic

Board• Chair of Teaching and

Learning Committee• Chair of Teaching and

Learning Sub-Committee• Chair of Program Approval

Committee• Chairs of relevant Working

Parties (e.g. policy on Assessment, Academic Integrity, Curriculum and Teaching Quality Assurance, Student Charter)

• Faculty Dean• Heads of School• Assistant Deans

(Academic) • Chief Examiners• Chair of Faculty/School

Teaching and Learning Committee

• Chair of faculty or school program approval committees

• Program coordinators• Course coordinators• Teaching staff (including

casuals)• Tutors• Administration staff

Other• Members of working

parties• Academic Development

Unit staff


Different roles – different QA responsibilities


Monitor institutional processes and

outcomes

External, government

agencies

Establish policies a supporting

infrastructure

Institutional leaders

Support &monitor institutional

policies

Faculty leaders

Comply with assessment and

QA policies

Program &course

coordinators

Deliver program and course assessment

Teachers &admini.

staff

Undertake assessment with

integrityStudents

Support and assure industry standards

Professional bodies and employers

Quality assurance of assessment

Policy and infrastructure

Approval Review/evaluation

Create appropriate roles with descriptions that refer to responsibility for assessment qualityDevelop and cross-reference policy, guidelines and rules for:• Assessment•Quality assurance•Academic Integrity•Student Charter•Appointment and promotion•Academic roles

Create infrastructure required for effective implementation , e.g.•Data collection strategies•Support technology•Professional development•Staffing

Take a whole of program approach to assessment planning (e.g. map learning objectives and/or assessment across program)

Put approval processes in place for new or revised assessment

Make the approval focus appropriate for level e.g. •Course – detail of course quality•Institutional – policy compliance

Make approval consequential (i.e. a response is required when proposed assessment needs improvement)

Put in place regular and systematic review or evaluation

Collect data on assessment quality (e.g. student satisfaction, student outcomes, grade distributions, benchmarking with comparable programs) and make it available

Make evaluation formal and consequential (i.e. systematic processes are guided by checklists or templates, it is clear who is responsible, findings must be addressed and recommendations and responses are reported)

Streamline institutional QA and accreditation processes – no gaps, no duplicationsCRICOS Provider No 00025B

Assurance of the quality of judgementsCalibration – activities that develop shared understanding of standards before judgements are made

(Consensus) moderation – activities to compare, confirm, defend or adjust judgements and standards

• Develop activities to induct new academics – e.g. provide workshops to make and discuss judgements on samples of student work from previous years (video used for performance or presentation)

• Hold regular meetings of tutors to discuss expectations and agree on advice provided to students

• Have all assessors mark some samples of student work and then agree on standards before marking the rest

• Coordinators can check random samples of all assessors judgements and negotiate necessary adjustments

• Coordinators can attend oral presentations or performance and compare their judgements with those made by tutors

• Student work can be double-marked and then discussed

• Chief examiners or school committees can monitor grade distributions across courses and ask academics for an explanation where necessary

• Colleagues from other institutions can review and comment on standards


External assurance of quality of student learning outcomes• Development of discipline-specific graduate learning outcomes

(QAA in the UK: Tuning in Europe: ALTC in Australia)

• Quality audits (AUQA and now TEQSA in Australia: QAA in the UK)

• Standardised testing

– Generic (AHELO: CLA)

– Discipline specific (AHELO Engineering and Economics: Under consideration by national discipline communities – chemistry, mathematics, medicine)

• External examining or review (Well established external examining system in UK: QVS and other funded projects trialled in Australia)

• Accreditation panels of professional associations

• National student surveysCRICOS Provider No 00025B

Emerging issues

• Variable or ‘hidden’ standards

• Modular or otherwise fragmented approaches to curriculum and assessment

• Increasing class sizes, student diversity and academic workloads

• Inconsistent or limiting policies

• Debatable validity of ‘generic’ approaches to assessment – strong arguments for discipline-specific approaches and instruments for standardised testing

• Other criticisms of generic, standard instruments – e.g. Does CLA measure aptitude or learning gains?

• Doubtful value of external examining processes based on partial information

• Emphasis on QA rather than QI


Effective practice• Characteristics of effective quality assurance (QA) and improvement

(QI) of assessment quality processes– Clear standards – process and learning outcomes

– Systematic and formal processes coordinated across all administrative levels and policies, integrated with professional development and appropriately resources

– Whole of program approaches – avoidance of fragmentation

– Widespread participation – assessment quality the responsibility of all

– Balance of efficiency (e.g. automation) with meaningful involvement – avoidance of perfunctory or ritualistic compliance

– Consequential – assessment strengths are recognised and rewarded and weaknesses are addressed

– Appropriate use and interpretation of standardised processes and instruments

– Collaboration and peer review – both internal and external – as encouragement of interaction and dialogue that support QA and QI


Activity 6: Ongoing quality improvement• Thinking about your role in the university:

– What have you learned?– What workshop ideas are most relevant to your role? – How has the workshop affirmed what you are already doing?– What are the implications for your future practice?


References• AAGLO Project. (2011). AAGLO Summary 2: Assurance of graduate learning outcomes through external review.

Retrieved from http://www.itl.usyd.edu.au/projects/aaglo/pdf/AAGLO_Summary%202_Final.pdf.• Barr, R. B., & Tagg, J. (2004). From Teaching to Learning - A New Paradigm for Undergraduate Education.

Retrieved from http://ilte.ius.edu/pdf/BarrTagg.pdf• Hughes, C. (2009). Assessment as text production: drawing on systemic functional linguistics to frame the design

and analysis of assessment tasks. Assessment and Evaluation in Higher Education. 34(5), 553-563.• Krause, K., Scott, G., & Colleagues. (2011). Learning and Teaching Standards Project: Inter-university peer review

and moderation of coursework and assessment. Retrieved from http://www.uws.edu.au/LATS • Lumina Foundation. (n.d.). Tuning USA. Retrieved from http://www.luminafoundation.org/?s=tuning • OECD. (2011). Assessment of Higher Education Learning Outcomes. AHELO Project Update - May 2011 . Retrieved

from http://www.oecd.org/dataoecd/8/26/48088270.pdf • The Quality Assurance Agency for Higher Education. (2012). UK Quality Code for Higher Education Part B: Assuring

and enhancing academic quality. Retrieved from http://www.qaa.ac.uk/AssuringStandardsAndQuality/quality-code/Pages/Quality-Code-Part-B.aspx

• Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education, 30(2), 175–194

• Tertiary Education Quality and Standards Agency. (2011). Developing a Framework for Teaching and Learning Standards in Australian Higher Education and the Role of TEQSA: Discussion Paper. Retrieved from http://www.deewr.gov.au/HigherEducation/Policy/teqsa/Documents/Teaching_Learning_Discussion_Paper.pdf.

• Tuning Educational Structures in Europe. (n.d.). Tuning Educational Structures in Europe: What is Tuning? Retrieved from http://www.unideusto.org/tuningeu/