Validation & Moderation in Diverse Settings...RESEARCH REPORT VALIDATION AND MODERATION IN DIVERSE SETTINGS PAGE 2 CONTACT NQC Secretariat ... A SMALL REGIONAL PRIVATE RTO WORKING

Validation & Moderation in

Diverse Settings

FINAL RESEARCH REPORT TO THE NATIONAL QUALITY

COUNCIL

DECEMBER 2010

RESEARCH REPORT VALIDATION A ND MODER ATIO N I N DIV ERSE SET TI NG S PAGE 2

CONTACT

NQC Secretariat

TVET Australia

Level 21/390 St Kilda Road Melbourne Vic 3004

Telephone: +61 3 9832 8100

Email: [email protected]

Web: www.nqc.tvetaustralia.com.au

DISCLAIMER

This work has been produced on behalf of the National Quality Council with funding provided through the Australian Government Department of Education, Employment and Workplace Relations and state and territory governments. The views expressed herein are not necessarily those of the Australian Government or state and territory governments.

The recommendations are under consideration by the National Quality Council but it should not be assumed that they will necessarily be implemented in the future.

ACKNOWLEDGEMENT

This work has been produced by Shelley Gillis of the Work-based Education Research Centre of Victoria University in conjunction with Chloe Dyson of Chloe Dyson & Associates Pty Ltd and Andrea Bateman of Bateman & Giles Pty Ltd and has been developed as part of a project commissioned by the National Quality Council in 2010 with funding through the Australian Government Department of Education Employment and Workplace Relations and state and territory governments..

mailto:[email protected]

http://www.nqc.tvetaustralia.com.au/


TABLE OF CONTENTS

TABLE OF CONTENTS ............................................................................................................ 3

SECTION 1: BACKGROUND ................................................................................................... 4

STRUCTURE OF THE OUTPUTS ................................................................................................... 6

SECTION 2. THE METHODOLOGY .......................................................................................... 7

PROJECT ESTABLISHMENT .................................................................................................. 7 INSTRUMENT DEVELOPMENT ............................................................................................ 9 DATA COLLECTION ........................................................................................................... 10 DATA ANALYSIS ................................................................................................................ 12 REPORTING...................................................................................................................... 12

SECTION 3: CASE STUDY REPORTS ...................................................................................... 13

CASE STUDY A: TAE10 - NETWORK OF VET PRACTITIONERS ......................................................... 13 CASE STUDY B: BEAUTY TRAINING PACKAGE - A NETWORK OF ASSESSORS ....................................... 22 CASE STUDY C: A SCHOOL BASED TRAINEESHIP – PARTNERSHIP BETWEEN SCHOOL, TAFE AND AN

ENTERPRISE ........................................................................................................................ 26 CASE STUDY D: A SMALL REGIONAL PRIVATE RTO WORKING ACROSS STATES ..................................... 34 CASE STUDY E: A LARGE PUBLIC RTO ....................................................................................... 41

SECTION 4: SYNTHESIS AND RECOMMENDATIONS ............................................................. 47

SIMILARITIES AND DIFFERENCES ............................................................................................... 47 POTENTIAL BARRIERS AND FACILITATORS ................................................................................... 50 FURTHER GUIDANCE ............................................................................................................. 53

REFERENCES ....................................................................................................................... 56

APPENDIX A: CHARACTERISTICS OF NOMINATED SITES FOR REVIEW BY QA-AG ................ 57

APPENDIX B: INFORMATION TO CASE STUDY SITES. .......................................................... 58

APPENDIX C: CASE STUDY QUESTIONNAIRES ..................................................................... 64

SITE VISIT 1: NETWORK - QUALIFICATION INTERVIEW SCHEDULE .................................................... 64 SITE VISIT 1: RTOS OPERATING WITHIN DIVERSE CONTEXTS .......................................................... 72


SECTION 1: BACKGROUND

In 2009, the Work-based Education Research Centre at Victoria University (namely Shelley

Gillis and Berwyn Clayton) in collaboration with Bateman & Giles (Andrea Bateman)

conducted a research investigation for the National Quality Council to identify the role of

validation and moderation processes in the VET sector. The study was undertaken to

address concerns that had been raised by some stakeholder groups in relation to the

perceived quality and consistency of assessments being undertaken by Registered Training

Organisations (RTOs). That is, there were concerns that assessment standards in the VET

sector were often not comparable. Ensuring the comparability of standards had become

particularly pertinent in the VET sector, as assessments could be made across a range of

contexts (e.g., vocational education, educational and industrial contexts) by a diverse range

of assessors using highly contextualised performance based tasks that required professional

judgement by assessors.

Gillis, Bateman & Clayton(2009) identified a number of different quality management

processes that could be used to help achieve national comparability of standards, whilst still

maintaining flexibility at the RTO level to design and conduct assessments (NQC, 2009a).

Their study resulted in the following products being published by the NQC:

A Professional Code of Practice for Validation and Moderation (NQC, 2009a);

An Implementation Guide: Validation and Moderation (NQC, 2009b); and

A Guide to Developing Assessment Tools (NQC, 2009c).

The NQC (2009a) Code contained a set of high level principles designed to provide guidance

on how to conduct assessment validation and moderation within a vocational education and

training (VET) setting.

The NQC (2009b) Implementation Guide was designed to be a practical resource for training

organisations intending to implement and/or review validation and/or moderation involving

consensus meetings. It provided guidance on how to implement the NQC (2009a) Code

within one’s own organisation. The Guide provides practical suggestions for:

Adhering to the Principles within the (NQC, 2009a) Code of Professional Practice

(i.e., transparent, educative, equitable, tolerable, confidential and representative);

Designing assessment tools (which was subsequently published by the NQC as a

separate resource for assessment tool developers (see NQC, 2009c));

Establishing a validation and/or moderation system at the RTO level, including a set

of templates for conducting a consensus approach to validation and moderation.


The National Quality Council noted the aspirational nature of these publications, recognising

that it provided a way forward within a complex and difficult area where misunderstandings

had been common.

To assist with dissemination of the research findings and raise awareness of the products,

the NQC also funded the research team to undertake a series of interactive information

sessions in all states and territories (see NQC, 2010). The NQC noted, however, that the

focus of the NQC (2009b) Implementation Guide had been on consensus meetings as a way

of conducting moderation or validation, and that uptake would therefore be most likely

limited to larger RTOs.

Aims

Given the limitations of the NQC (2009b) Guide, in 2010 the NQC commissioned the research

team to undertake further follow up work to:

Develop additional guidelines and support materials for alternative approaches to

consensus meetings as a validation strategy as might be used by diverse RTOs which

deliver and assess in a range of contexts; and

To undertake national case studies which focus on validation of specified

qualifications as a means of enhancing consistent assessment outcomes.

The specific aims of the current study were therefore to undertake a series of case studies to

provide qualitative insights into the implementation of validation and/or moderation

processes being implemented (or intending to be implemented) within diverse RTO settings.

In particular, the case studies were undertaken to:

Explore the contexts in which validation and/or moderation processes occur;

Document and compare assessment quality management processes and procedures

currently being implemented;

Identify potential barriers and/or facilitators (e.g., structural, process and/or

personal factors) that may impact on the design and maintenance of assessment

quality management processes within diverse RTO settings; and

Identify further resources to be developed to support RTOs conducting validation

and/or moderation activities within RTOs within diverse settings.

THE RESEARCH TEAM

The investigation was undertaken by the research team comprising:

Shelley Gillis – Work-based Education Research Centre, Victoria University

Andrea Bateman – Bateman & Giles Pty Ltd

Chloe Dyson – Chloe Dyson & Associates Pty Ltd


STRUCTURE OF THE OUTPUTS

The outputs of this study have been presented within three reports.

The Research Report describes the methodology and findings of the study. It includes the following sections:

Section 1: Background

Section 2: Methodology

Section 3: Case study Reports

Section 4: Synthesis and Recommendations

An Assessor Guide: Validation and Moderation, which comprises a series of frequently asked questions for developing assessment tools as well as establishing and maintaining systematic validation and/or moderation processes within and/or across RTOs. This document was designed for VET practitioners and coordinators who conduct assessment and/or validation activities in a range of diverse settings (e.g., small RTOs, partnership arrangements, school-based traineeships, networks of assessors).

Fact Sheets (x3) which was designed to provide a quick reference to assessment tool development, systematic validation and assessor partnerships. Note that the content of these documents have been produced in accordance with the template approved by the NQC.

Assessment Fact Sheets which were designed to provide a quick reference to assessment tool development, systematic validation and assessor partnerships, in:

Assessor Partnerships

Quality assuring assessment tools

Systematic validation

These have been separately published by the NQC together with other Assessment Fact Sheets developed in 2010.


SECTION 2. THE METHODOLOGY

There were five phases to this investigation.

Project Establishment

Instrument Development

Data Collection

Data Analysis

Reporting

PROJECT ESTABLISHMENT

The research team was commissioned to undertake five case studies. Of these five case

studies, three were to focus on RTOs operating within diverse contexts and the final two

were each to focus on a network of RTOs involved in validation of a nationally recognised

VET qualification(s). The project was managed by the National Quality Council’s Quality of

Assessment Action Group (QA-AG). In an attempt to ensure diversity of the case study sites,

the QA-AG identified the following criteria for selection of the case study sites: geographic

location; size of RTO, type of RTO and partnership arrangements. The ultimate aim was to

select five case study sites that would provide representation of:

Metropolitan, regional and rural-based RTOs;

RTOs from different states/territories;

Small and medium sized RTOs;

Enterprise, community and private RTOs;

Professional and/or industry organisations; and

Delivery of VET in School programs, where possible.

Based upon such criteria, eight sites were nominated by the research team for consideration

by the QA-AG (see Appendix A). Prior to their consideration, all eight organisations had

expressed willingness to voluntarily participate in the study. Appendix A formed the basis of

discussion by the QA-AG to identify the final five cases study sites to participate in the study.

The following five case study sites were selected by the QA-AG:

Qualification Network The TAE10 – network of Victorian VET practitioners

Beauty Training Package – network of South Australian VET practitioners

RTO operating within a diverse context

School based traineeship – partnership between school, TAFE and enterprise

A small regional private RTO operating across states

A large public RTO


Table 1 displays the characteristics of the five sites that were selected by the QA-AG to form

the focus of the case study investigations. Note, to ensure confidentiality of the

participants, the names of participating organisations and individual participants have been

removed from this report.

An information letter outlining the purpose of the study, its voluntary nature, the expected

roles and responsibilities of participants and the perceived benefits of the study was then

sent to each site (see Appendix B for a copy of the written invitation). Once voluntary

consent for participation was obtained, a convenient time for site visits was determined.

Table 1: Characteristics of case study sites selected by the QA-AG

Focus Description Location Training

Package

Qualification Justification Criteria

Training

Package

Network

State –based

Network of

VET

Practitioners

VIC TAE 10 TAE40110

Certificate IV in

Training and

Assessment

Membership includes 17

TAFEs and University

TAFES in Victoria and

some private RTOs

Commenced in 2001

Professional Network

Represents private

RTOs and TAFE

Membership includes

regional, rural and

metropolitan RTOs

Training

Package

Network

State based

network of

VET

practitioners

SA Beauty

Training

Package

Various New network includes

various types of RTOs

across the state

delivering the Training

Packages.

Network of RTOs

Range of sizes

Range of geographic

locations represented

RTO

operating

within diverse

context

School based

Traineeship

NSW & QLD Health

Training

Package

Certificate III in

Health Services

Assistance

Partnership arrangement

between Schools, TAFE

and private enterprise to

deliver school-based

traineeships involving on

and off the job

assessments.

Enterprise RTO

Partnership

arrangement between

enterprise, TAFE and

Education Department

Delivery of school-

based traineeship

High risk associated

with incorrect

judgements

RTO

operating

within a

diverse

context

Small regional

private RTO

working

across states

VIC, SA, QLD General

Construction

Certificate I-III in

general

construction,

floor and wall

tiling, solid

plastering,

bricklaying/blockl

aying and

carpentry

Focus on trades in

remote regions,

apprentices and skills in

demand.

Small RTO

Regional

RTO

operating

within a

diverse

context

Large public

RTO with

multiple sites

Withheld to

maintain

confidentiality

Various Various Various campuses, small

number of assessors at

remote sites. Range of

qualifications. Includes

VET in Schools.

Large RTO

Large youth cohorts

VET in Schools

Multiple campuses

Regional and rural

campuses

The establishment phase also included a desktop analysis to review the NQC reports and

materials relevant to validation and moderation. The findings from the analysis were then

http://www.ntis.gov.au/?/Qualification/HLT32507




used to inform the development of the interview schedule for the case studies (see next

section).

INSTRUMENT DEVELOPMENT

To ensure the case studies fulfilled the research requirements and were undertaken in a

consistent manner, a semi structured interview questionnaire was developed to provide the

reporting framework for the investigation.

Questionnaire

There were four main sections to the questionnaire:

1. Characteristics of the Organisation/Network.

2. Assessment Current Practices.

3. Assessment Quality Management Strategies.

4. Feedback on the Code and Guide.

Sections 1 and 2 were designed to document the background characteristics and assessment

practices of the organisation/network. In relation to Section 3, a series of semi-structured

questions was developed by the project team that addressed the following design issues

that underpinned by the NQC (2009b) Implementation Guide: Validation and Moderation.

That is, the

Assessment Quality Management Processes (e.g., quality assurance, quality control

and/or quality review processes) that had been implemented (and/or intending to

be implemented);

Purpose of establishing and maintaining a validation/moderation system;

Staffing considerations – roles and responsibilities of key players;

Level of authority to act upon the decisions/recommendations of the

validation/moderation process;

Scheduling of activities (how is this determined?);

Strategy employed to sample units, assessment tools;

Financial considerations to establishing and maintaining the system;

Processes to handle complaints and appeals;

Records management (who, where, when and duration);

Types and processes for reporting the validation and/or moderation outcomes;

Internal review processes that have been established;

Perceived barriers and facilitators to successful implementation of the validation

and/or moderation system.


The final section of the questionnaire comprised a series of items to gather feedback on the

perceived strengths and weaknesses of the existing NQC publications on validation and

moderation. Such items were designed to identify further support materials that could be

developed, which would assist RTOs and/or networks of assessors to establish and maintain

validation and/or moderation processes within diverse VET settings.

See Appendix C for the detailed version of the questions developed for both types of case

studies (i.e., those RTOs working within diverse settings and the networks of assessors).

Professional Development Materials

For each case study site, a professional development workshop was conducted by a member

of the research team at the first face-to-face visit. The workshop was designed to provide

opportunity for the participants to become familiar with both the NQC’s (2009a) Code and

the Implementation Guide (2009b). Whilst the workshops were similar in content to the

information sessions delivered in 2009 (see NQC, 2010), they provided opportunities for

greater interaction and workshop activities. For example, participants within each workshop

were encouraged to supply copies of their assessment tools and where possible, judged

candidate evidence to enable a simulated moderation/validation activity to be undertaken

during the workshop, using the templates provided within the Guide.

DATA COLLECTION

Each case study site was coordinated by the followng members of the research team:

Type of Case Study Description of case study Case study coordinator

Assessor Network The TAE10 – network of Victorian VET Practitioners

Shelley Gillis

Beauty Training Package – network of SA VET practitioners

Chloe Dyson

RTO operating in diversity context


Shelley Gillis

A small regional private RTO operating across states

Andrea Bateman

A large public RTO

Andrea Bateman

The case study data was collected on two occassions. The first involved the case study

coordinator delivering a Professional Development workshop to the RTO/network members


to introduce the new definitions, principles and approaches that underpinned the Code and

Guide (NQC, 2009a & 2009b, respectively). In addition to the workshop, the coordinator also

conducted an interview with the key personnel from the organsiation/network on its current

approaches and perceptions of validation and/or moderation.

The second meeting occurred four to six weeks later when members of the RTO/network

had had the opportunity to review and use aspects of the Code and Guide within his/her

context. The purpose of the follow-up discussions was to obtain more in-depth feedback

about the:

Perceived usefulness and relevance of Guide;

Perceptions of potential factors that may impact on the design, implementation and

maintenance of a validation and/or modertion process within his/her context;

Recommendations for improvement to Guide; and

Suggestions for development of further support materials.

For each of the three RTOs, the second meeting was held face-to-face. However, due to

the logistical and financial costs associated with bringing the state-based network of

practitioners together, the follow up discussions with each assessor network entailed a

combination of telephone and email exchanges with the case study coordinator.

Throughout the data collection period, the case study coordinators documented the

questions being raised by participants. Such information was used to inform the

development of the support materials (see Reporting).


DATA ANALYSIS

The case studies were analysed to identify:

Similarities and differences in design and implementation of assessment quality

management strategies.

Potential barriers and facilitiators of assessment quality management strategies,

according to structural, process and personal factors.

Implications for customisation/amendments to the Implementation Guide to asssist

networks and RTOs conducting assessment and valdiationactivities within diverse

settings.

REPORTING

The findings from the case study analyses was used to produce three types of reports:

Research Report documenting the aims, methodology, case study findings and

recommendations from the study.

An Assessor Guide: Validation and Moderation which was a support document

comprising a series of frequently asked questions for developing assessment tools as

well as establishing and maintaining validation and/or moderation processes within

and/or across RTOs. This document was designed for VET practitioners and

coordinators who conduct assessment and/or validation activities in a range of

diverse settings (eg small RTOs, partnership arrangements, school-based

traineeships, networks of assessors).

Fact Sheets (x3) which provided a quick reference to assessment tool development,

systematic validation and assessor partnerships and were designed to provide a

quick reference to assessment tool development, systematic validation and assessor

partnerships, in:

Assessor Partnerships

Quality assuring assessment tools

Systematic validation

These have been separately published by the NQC together with other Assessment Fact Sheets developed in 2010.


SECTION 3: CASE STUDY REPORTS

CASE STUDY A:

TAE10 - NETWORK OF VET PRACTITIONERS

THE TRAINING PACKAGE

The TAE10 Training Package specifies the competency standards and nationally recognised

qualifications for those working within the VET sector. In 2010, it replaced the TAA04

Training and Assessment Training Package (with the exception of the TAA50104 Diploma of

Training and Assessment), which was itself a replacement for the BSZ98 Assessment and

Workplace Training Package.

THE NETWORK

In Victoria in 2001, a peak network for VET practitioners delivering accredited qualifications

in training and assessment was established. Members were initially drawn from all Victorian

TAFE institutes and Universities with TAFE Divisions. Initially, the main objective of the

network was to provide collegiate support for the public TAFE providers in their delivery of

the Diploma of VET and the BSZ98 Training Package. This objective was subsequently

extended to include the Diploma of VET Practice and the TAA04 Training Package by

providing a forum for:

discussion around the development of the teaching skills of TAFE teachers;

the sharing of resources;

dissemination of information about delivery strategies; and

validation of assessment tools.

Although the network members share teaching and assessment resources within these

qualifications, provision of such qualifications is at the individual RTO level, often supported

by employers at worksites.

Whilst the establishment of the network has provided a valuable source of shared

information and resources for delivery and assessment of VET teacher education

qualifications, the introduction of the AQTF audit requirements for conducting validation

across institutions has been another major driving force behind its continual operation.

Other benefits of forming a collective group of assessor practitioners has included increased

opportunities to have input into the development of new teacher education qualifications as

well as discuss the issues of resources and validation of assessments for the related

qualifications. For example, the network has been used as a forum to review the content

validity of using the TAA04 assessment tools for application to the TAE10 using the


competency mapping template provided in the NQC (2009b) Implementation Guide:

Validation and Moderation (see Appendix A.2).

MEMBERSHIP

There are approximately 100 members within the network, representing 17 RTOs. When the

network was first established, it was designed for TAFE providers only, but it has recently

increased its scope to include membership from within private and community-based RTOs.

Membership has been via invitation. All members must have had at least Certificate IV in

Training and Assessment and numerous members also possess higher level teaching

qualifications and other vocational qualifications.

THE ASSESSMENT QUALITY MANAGEMENT APPROACH

QUALITY ASSURANCE (QA)

The network has introduced a number of quality assurance processes such as establishing

minimum requirements for its members (e.g. having the TAA04 Cert IV qualification).

Another example of QA processes implemented was the development of a standardised

reporting format for the teaching practicum requirements of the Diploma of VET Practice,

including the development of guidelines for the way the various aspects should be

conducted. This was subsequently reviewed across time and implementation. The network

has also conducted a series of professional development type programs/workshops through

its conferences and at its quarterly meetings.

QUALITY CONTROL (QC)

Quality control, in the sense of making adjustments (where necessary) to students results to

bring each RTO member’s standards into alignment, has not been undertaken by the

network. To date, the network has only implemented face-to-face quality assurance and

quality review processes. The restrictions on not being able to moderate has been argued to

relate to timeliness (as each member RTO operates under different timeframes) and the

costs associated with coming together as a network in a timely manner. The network plans

to pilot a process to validate assessment tools electronically in the near future. This may also

provide an opportunity to further explore the possibility of implementing timely moderation

meetings via an electronic medium.

QUALITY REVIEW (QR)


The review and validation of assessment tools, processes and outcomes to identify future

improvements to assessment practices, is the most common activity undertaken by the

network. A range of assessment methods tend to be validated by the network with the most

frequently reported to be:

Classroom based products

Workplace based products

Simulated/classroom demonstration

Completion of activities in learning materials

Written/assignment/test/projects.

The network has yet to validate assessment tools that comprise interview schedules, group

projects and portfolio specifications. It has occasionally validated on-line assessment tasks

but some members still see this as a challenge yet to be overcome.

ROLES AND RESPONSIBILITIES

There are two major parties involved in the validation process:

The network coordinator

Individual members of the network

It is the network coordinator who is responsible for promoting and coordinating the network

validation meetings in collaboration with the host institution. This typically entails working

with the host institution (e.g. a member RTO or the Victorian TAFE Association (VTA)) to

arrange the layout of the rooms (including break out rooms), catering, advertising etc. The

network coordinator is also responsible for preparing a timetable/schedule for the units to

be validated by the network throughout a year.

In relation to the validation functions of the network, individual network members are

responsible for:

Attending and participating in regular validation meetings;

Providing samples of assessment tools for purposes of validation;

Disseminating the recommendations arising from the validation meeting to the

relevant tool developers within one’s own RTO;

Monitoring whether the actions arising from the validation meeting have been acted

upon within their own RTO, where appropriate;

Adhering to the Principles underpinning the Code of Professional Conduct for

Validation and Moderation.

FINANCIAL CONSIDERATIONS


The major financial consideration for establishing and maintaining the network has been

associated with each member RTO’s costs associated with teaching relief to allow

participants to attend a central meeting; as well as the time it takes for participants to both

travel and attend the meetings face-to-face. It was also acknowledged that the coordination

of the network would need to be financially supported in some way if all the requirements

specified within the NQC (2009b) Implementation Guide: Validation and Moderation were to

be adhered to.

The coordination role of the network has been a voluntary position, funded by the in-kind

contribution of one of the member RTOs in which the network coordinator has been

employed (note that the coordinator has been allocated 3 hours per fortnight from his/her

RTO’s workload model to coordinate the network). As there has been no formal structure or

process for succession planning, the same individual has coordinated the network since its

inception. Although the initial arrangement was for each institute to call and host meetings

when required, in the absence of external funding for supporting and maintaining the

network, the initial coordinator has retained the role by default.

The network pays a fee to the Victorian TAFE Association (VTA) to provide the

administration support for the conference. The VTA also allows the network to use their

meeting rooms and supply tea and coffee; this location has been found to be the most

convenient for regional and rural members to attend meetings via public transportation.

Members are charged a small fee to attend meetings and conferences etc to cover the costs

of venue hire, guest presenters and catering etc. Each RTO covers the cost of teaching relief,

travel and accommodation (if applicable) for its members to attend. Member RTOs,

particularly those within inner metropolitan regions, are also encouraged to host validation

meetings and conferences.

SCHEDULING OF MEETINGS

Typically, four network meetings are held each year in which validation of assessment tools

forms the focus. To encourage greater participation and reduce travel costs of its members,

one of the four validation meetings are held at the same time as the annual conference. The

network also hosts a one day conference for VET practitioners about the issues faced in the

delivery and assessment of VET practitioner qualifications. The network’s role also includes

facilitating professional development activities for its members.

THE VALIDATION PROCESS


The network has developed a Validation Checklist for reviewing the quality of Assessment

Tools using a consensus panel approach to validation. The checklist provides a series of

questions for a small number of assessors (around 5 or so members of the network) to

review as a panel. The purpose of the Checklist was to establish a shared understanding of

the assessment tasks, strategies and methods that members of the network were using for

the delivery of the Cert IV in Training & Assessment and with a particular emphasis on the

units that have been amended to include more specific reference to recognition of prior

learning. The questions tend to cover the following aspects of the tool:

Transparency

Currency

Validity

Reliability

Flexibility

Fairness

Sufficiency

Authenticity

AQF requirements

Key competencies

Dimensions of competency

It was however acknowledged by many of the network members that the templates within

the NQC (2009b) Implementation Guide (i.e., the A.1 Self Assessment and the C.3 Item

Record Form) were very useful templates for reviewing assessment tools.

Within a network meeting, a number of small panels are formed (e.g. 10 panels with 6 or so

members) which are responsible for reviewing the assessment tools that have been tabled

by the panel member. Each panel has its own table at the meeting. Note that each panel

reviews the assessment tools supplied by members from within that panel. That is, there is

currently no requirement or process for sharing or reviewing each assessment tools outside

each panel – this is largely due to the logistic challenges of:

Trying to coordinate all members to participate in a validation exercise at the same

time (e.g., in some instances 60 or so members can be present at one meeting); as

well as

Not having a process or finances in place for making duplicate copies of the tools for

each panel to review at the same time.

The panel reviews the assessment tool and makes recommendations for improvements etc.

To date, only the assessment tools have been reviewed in terms of content validity, clarity of

the assessment procedures and reviewing the evidence requirements with the ultimate aim

of the process to establish and maintain a shared understanding of assessment tool related

issues.


The network has yet to check the appropriateness of assessor judgements in accordance

with the decision making rules specified within the assessment tool. Members of the

network however considered such evidence as a valuable component of the validation

process and will be incorporated into future validation meetings, where such evidence is

available for review.

SAMPLING

Although the network does not currently have any formal policy and/or procedures for

conducting validation, it does implement a systematic approach to sampling units of

competency and uses standardised templates for validating its member’s assessment tools

etc.

AUTHORITY

Individual members of the network take responsibility for acting upon (or not acting upon)

the decision of the panel when they return to their RTO. There is neither a follow-up

process by the panel nor any level of accountability to report the outcomes/actions

undertaken by the RTO at any subsequent validation meeting.

RECORD KEEPING

Whilst the network has actively been involved in validating assessment tools, the network

itself does not retain any records of the validation process in terms of the process nor the

outcomes. The responsibility for maintaining such records is the individual member whose

RTO ‘owns’ the tools. Each member is responsible for selecting which assessment tools s/he

will bring to the validation meetings, although at times, members have been encouraged to

focus on selective units associated with either training and/or assessment. It should be

acknowledged that in most instances the assessment tools being reviewed have been

developed by the network member. As such, the assessment tool developer is also the panel

member where all members of the panel are aware of authorship of the assessment tool.

This has implications for maintaining confidentiality. There was general agreement that the

use of de-identified assessment tools was a way in which maintaining confidentiality could

be enhanced in future validation meetings.

The network does not have any reporting requirements, and again, it is up to the individual

participants to report the outcomes etc to their own institutes as part of their quality review

requirements.


COMPLAINTS AND APPEALS

As the onus is on individual participants to make any recommended changes at their own

RTO, there have not been any complaints and/or appeals about the validation/moderation

process from any network members. If the network continues to only conduct validation,

there was general agreement among the members that it was unlikely that any appeals

would arise in the near future. This is largely due to the fact that the network does not have

any authoritative power or desire to follow up on any network panel’s recommendations.

INTERNAL REVIEW

There is currently no formal process for monitoring and reviewing the effectiveness of the

network. Only anecdotal feedback has been collected from participants. Furthermore, as

attendance and follow-up actions are voluntary for members, ensuring comparability of

standards across the network RTOs cannot be assured. Instead, the network aims to achieve

continuous improvement in its assessment practices as its major aim.

BARRIERS AND FACILITATORS

The main aim of establishing the network was to create a ‘sharing atmosphere’ among its

members. That is, it was hoped that members would see the network as an opportunity to

collaborate and improve its assessment practices for all members. There was strong belief

that the network has been very successful in creating this atmosphere among its members.

Furthermore, members of the network reported that from their experience, whether one is

validating their own or others tools, participants come away saying how much they learn

each time they get together to discuss and share understandings.

However, the major barrier to successful implementation of validation and/or moderation

meetings within the network was thought to be associated with the geographic distances of

its members and opportunities to come together face-to-face. Although the network meets

four times a year to conduct validation, such days are also used to provide professional

development opportunities and forums for discussion. There was strong agreement that

more regular and timely validation meetings (which focus only on validation) are required. It

was also acknowledged that such meetings may not necessarily have to be undertaken face

to face meetings. The network plans to try to conduct validations electronically using

Elluminate which has been designed to support e-learning (see www.elluminate.com). It

was further acknowledged that moving toward an electronic process might assist with timely

moderation activities, if the network extends its functions to also include moderation

activities.


CHALLENGES

The major challenge of the network was thought to be associated with ensuring that the

equity principle within the Code was adhered to; and in particular, ensuring that all types of

RTOs are represented and part of the process. One possible approach to addressing this

issue would be via introducing electronic validation meetings and having a dedicated

website/blog/wiki for sharing resources and tools etc.

RECOMMENDATIONS FOR IMPROVEMENT TO THE GUIDE

Whilst the members of the network expressed positive feedback in relation to the NQC

(2009a, 2009b) Code and Guide as being useful resource materials, it was recommended

that any future developments include:

Summary tables and diagrams to reduce reading load for assessors who are time

poor;

Examples of completed competency mappings;

Examples of benchmark/exemplar assessment tools;

A quick reference guide that explains in non-academic language some of the key

issues that underpin the Guide and Code (e.g., Frequently asked questions and

answers);

Development of a checklist for panel members to refer to prior to the validation

meeting, outlining roles and responsibilities for selecting and tabling assessment

tools and candidates’ judged work.

Discussions were also raised as to the level of detail required in the competency mapping.

For example, the question was raised as to whether it was necessary to map all components

within the unit of competency to the sub-components of the tool (e.g., whether it should be

at the element versus performance criterion level). If an atomistic approach was desirable

by auditors under the AQTF, then concerns were raised about the workload implications,

even if it was only carried out during the development phase of the tool. At the same time,

it was appreciated by numerous members that such a mapping exercise would be crucial in

high risk assessment (e.g., which may provide safety risks to the candidate) or high stakes

assessment environments (e.g., for selection purposes in which the supply was much greater

than the demand). Consequently, it was suggested that greater guidance be given on the

mapping exercise in any future support materials to ensure the process was manageable but

at the same time, satisfying the requirements of demonstrating content validity of the tool.

Finally, a number of the network members raised concerns with the inconsistency of the

terminology and definitions used within various national and/state policy and/or support

documents. For example, concerns were expressed with the different definitions used within


the NQC publications and the Training and Education Training Package (TAE10) as well as the

Victorian Registration and Qualifications Authority (2010) Guidelines for VET providers. This

led to some confusion as to which definition should hold precedence.


CASE STUDY B:

BEAUTY TRAINING PACKAGE - A NETWORK OF ASSESSORS

BACKGROUND

Service Skills South Australia convened the Hair and Beauty Network in 2010 to assist with a

DFEEST VET Innovation project, which was designed to support the implementation of the

Beauty Training Package, endorsed in September 2010. The network members have worked

closely with Service Skills SA over a number of years; including work on a Reframing the

Future project on the AQTF in 2007 and a Workforce Development project in 2007 - 2008.

There are 11 Registered Training Organisations in the network and it is comprised of private

RTOs and a multi campus Institute of TAFE. All of the RTOs have bases in Adelaide but some

also operate in regional centres.

The network has not met to conduct assessment validation but some of the member RTOs

have met to informally review assessment in the past. Despite the network’s best intentions

to establish a validation panel, there has been little consensus as to how this may operate in

South Australia. At this stage it is more than likely that Service Skills South Australia will work

with the Hair and Beauty Association to determine if this could eventuate.

The purpose of the network is to maintain a collaborative network of Beauty Registered

Training Organisations that are prepared to invest time and effort into ensuring that the

standards of the delivery and assessment of Beauty competencies meets the needs of the

Beauty industry in South Australia. Services Skills SA is fully supportive of this process.

Network members felt that the network could have a very strong role in increasing the way

in which the Beauty industry is viewed in South Australia, particularly in relation to its

professionalism. The recent introduction of new technologies such as Intense Pulsed Light

(IPL) and laser technologies has exposed the Beauty industry to substantial risk as the use of

these technologies is not regulated. Members felt that this risk could be reduced by RTOs

sharing resources to collaboratively develop their knowledge of these technologies and to

develop robust assessment tools. A higher level of quality assurance may also have the

effect of reducing what the network felt were burdensome insurance requirements placed

by insurance companies on RTOs delivering training in the new technologies.

The Network members considered that the Hair and Beauty Association and Service Skills

South Australia had a pivotal role in facilitating and supporting the process of establishing a

validation network and that this role required further exploration. This could result in the

establishment of satellite validation groups, in which the authority for validation rested,


while the Hair and Beauty Association and Service Skills South Australia provided

overarching support and coordination, facilitating adherence to timelines and the sharing of

good practice, where it was warranted. However, while this model is the preferred one, the

ability of the network to sustain this activity when Service Skills South Australia withdraws its

support is questionable. Systemic support is required to facilitate this.


THE NQC ASSESSMENT VALIDATION MATERIALS

Most of the network members were aware of the NQC suite of validation resources

published in 2009. Network members were satisfied with the design of the NQC materials in

their current form if they were to conduct consensus validation face-to-face. However, the

members also felt that as it was very difficult to meet that they would be well served by

support and resources to assist them to conduct validation online. In order to develop and

sustain an online validation network, the network members felt that a number of issues

would need to be addressed. The network suggested that a guide to online validation or an

addendum to the existing NQC validation and moderation materials could be developed.

ONLINE VALIDATION MATERIALS

The network members felt that the current NQC validation materials could be adapted for

use online but that some specific online materials, as addenda to the existing materials,

would be welcome.

TEMPLATE

A template for a code of conduct/rules for operation for an online network would be

required to develop an E network. This template would guide members in articulating the

central purpose for the network and the validation processes to be used. The template

would assist E network members to determine how the NQC principles of validation and

moderation would be operationalised. For example, with the principle ‘Confidentiality’ the

members discussed the need to assign codes to RTOs in order to facilitate open feedback.

It would assist members to determine the roles and functions of each of the network

members and provide guidance on how external validators would be used by the network.

For the Beauty Industry, this might include manufacturers of new technologies such as IPL

and laser technologies.

The template would assist in the development of meeting protocols, including guidelines for

individuals or small groups making responses via the Internet; and timeframes for responses.

The network members stressed the need for guidance regarding how to confirm each

member’s commitment to the validation process.

GUIDE


The network also suggested the development of a guide on how to develop and sustain an E

network. This guide would include suggestions for how to establish an E network, including

development of a promotional strategy. Suggestions could be included about the technology

that could be exploited to develop and conduct an online network. For example, network

members felt that Moodle might be insufficient and that an E –network using an electronic

platform might be more effective. Options for systems design could also be included in the

guide, including options for log on and enrolling in the network. It was also suggested that

the guide should provide options for structuring an E network. These options could include,

for example: setting up buddy systems using an E platform; creating an E portfolio

environment to form partnerships or small groups which could then post back their findings

to a digital drop box.

The information in the NQC (2009b) Implementation Guide could be adapted to describe the

role of the Chair in an Electronic environment. Members stated that validation is often not

successful where the assessor does not know how to prepare adequately and as a result

feels some antipathy to the process. It was felt that while the NQC Implementation Guide

provided information about the assessor’s role in validation and how to prepare for

validation that it could be re-stated in plain English and adapted to an E environment.

The guide could also include information about how to close the cycle of quality review and

to build this into an online validation process. Members felt that validation consensus

meetings often lead to suggestions for improvements in assessment tools but that the

implementation of these suggestions is seldom tested. This could lead to guidance about

how to create an online resource of exemplar assessment tools as well as protocols for

accessing and using these materials.

CONCLUSION

There is a great deal of impetus for the Hair and Beauty Network to formalise its assessment

validation processes but the Network is keen to develop a model where members can

collaborate online to share good practice. This model relies on the development of

strategies to promote the sustainability of the network when external support is withdrawn.


CASE STUDY C:

A SCHOOL BASED TRAINEESHIP – PARTNERSHIP BETWEEN SCHOOL,

TAFE AND AN ENTERPRISE

THE SCHOOL-BASED TRAINEESHIP

In NSW, the Certificate III in Health Services Assistance can be offered as either a school

based traineeship (SBT) or a VET course. In relation to the school based traineeship, the

Certificate III is part of the Human Services Curriculum Framework within the NSW HSC.

School based trainees are provided with the opportunity to gain a national VET qualification

as well as the NSW HSC.

The Certificate III in Health Services Assistance is from the Health Training Package

(HLT32507). There are 15 units of competency that must be completed over the two year

HSC period to achieve the qualification. Students also undertake 100 days of a clinical work

placement within an acute health care facility (i.e., hospital) which is also spread over the

two year period.

In addition to achieving the Certificate III in Health Services Assistance, students undertaking

the SBT gain credit toward the HSC for the School-based formal training component of their

traineeship. That is, 6 of the 22 units required for the HSC are achieved by the SBT.

Furthermore, students may elect to complete the Board Endorsed Stage 6 Industry-based

Learning Course enabling them to gain HSC credit for the on-the-job training component of

the school-based traineeship. This is equivalent to an additional 4 units within the HSC. As

such, those students who elect to have their work placement formerly recognised as part of

their HSC can use the SBT to contribute toward 10 of the 22 units required by the HSC.

In 2011, a new cohort of students in NSW will begin the school-based traineeship in which

they will have the option to undertake an HSC examination in 2012. Students who elect to

undertake the HSC examination can have the results of the examination contribute toward

their Australian Tertiary Entrance Rank Score (ATAR score).

THE PARTNERSHIP

In 2008, a partnership was formed between the NSW Department of Education, TAFE NSW

and a private hospital to deliver a school-based traineeship as part of the NSW HSC. Three

cohorts of school-based traineeships have undertaken the program (with 7 in each cohort

since 2008) and another new cohort of 9 students is to commence in 2011. To date the


partnership arrangement has largely concentrated on the design of the program and the

delivery of training (eg in terms of sequencing of units, selection of students, determining

pre-requisite units for the work placement).

The TAFE and the hospital would like to expand their partnership to include the monitoring

of assessments, standards and outcomes of students across the two institutions.


EXISTING ROLES AND RESPONSIBILITIES

THE NSW DEPT OF EDUCATION

The NSW Education Department (via the local schools) is responsible for promoting the

school-based traineeships to its Year 10 cohort and for handling application forms to be

reviewed by both the TAFE and the hospital. Potential students receive details about the

program through information sessions which are organised with the schools. The Education

Department, via the Board of Studies, is also responsible for designing, administering and

assessing the HSC examination. The HSC examination will be designed to assess the HSC

content of the Health Services Curriculum Framework (i.e., health and wellbeing; industry

context; safety; and work) as well as the employability skills for the Certificate III

qualification. It should be acknowledged that the HSC examination will be independent of

the competency based assessments undertaken as part of the school-based traineeships by

the TAFE and the student’s performance on the exam will have no impact on his/her

eligibility for the AQF VET qualification. In relation to those students who elect to have their

workplacements formerly contributing toward their HSC (via enrolling in the Board Endorsed

Stage 6 Industry-based learning course), the school is responsible for reviewing the portfolio

and reflective journals maintained by the students whilst undertaking the clinical placement.

THE TAFE

The Certificate III in Health Services Assistance is under the scope of registration of the TAFE.

TAFE delivers all of the 15 units of competency that comprise the qualification. It is also

responsible for

Selecting students into the program (but it performs this role in collaboration with

the hospital);

Designing the off-the-job assessment tools;

Conducting competency based assessments against the entire set of units within the

Cert III;

Designing the assessment logbooks that are used to record the on-the-job

assessments conducted by the hospital staff;

Delivering the HSC Content (focus areas) which comprise of a number of units within

the Health Training Package; and

Satisfying the assessment requirements specified within the AQTF standards.

THE HOSPITAL

The hospital is a large private RTO with multiple sites both nationally and internationally. It

delivers and assesses against a range of qualifications from Certificate III to Graduate


Diploma levels. The work placement provided by the hospital has been designed to provide

opportunities for the trainees to enhance the learning acquired at TAFE. In addition to using

the assessment logbooks developed by the TAFE, the hospital also designs its own

assessment activities and record sheets to monitor the students’ progress throughout the

two year work placement. The hospital has a responsibility – duty of care (to patients, other

staff and to the students) to ensure that the school-based traineeship students are clinically

sound and clinically safe. This is an important responsibility as it was acknowledged that the

students have varying levels of competence when coming from a simulated/TAFE

environment into the workplace.

THE STUDENTS

The school-based trainees spend one day per week at the hospital, 4 hrs at TAFE and 3.5

days at school. Note that whilst at school, they are completing the remaining 18 units

required for the HSC which are outside the school-based traineeship. In 2011, the students

will be required to sign a learning contract, to be established jointly by the hospital and

TAFE, which will specify the two organisations requirements and expectations of the

students whilst undertaking the work placement. The Learning Contract will be designed to

clearly articulate that the clinical work placement is a significant component of the course

and if not carried out successfully, the students will not receive the Cert III qualification

(even if they have completed all the 15 units at TAFE). That is, it will be explained that the

placement will require them to exercise their theoretical knowledge in the

workplace/clinical setting to the standard specified in the industry competency standards.

ASSESSMENT

The competency based assessments undertaken as part of the school-based traineeship are

conducted by the TAFE. Although the hospital conducts its own assessments, assessment

against the units of competency for certification purposes is the responsibility of TAFE. In

the past, the assessment tools have been established centrally by NSW TAFE through the

form of student log books. In December 2012, the development of such log books and other

assessment tools is to be decentralised, providing greater autonomy for both the local TAFE

institute and the hospital to design and implement their own assessment tools.

Both the TAFE and hospital used a range of assessment methods to assess the trainees.


Table 2: Assessment methods utilised across TAFE and the hospital.

Method Hospital TAFE

Work-place observation Yes

Work-place products Yes

Simulation/classroom demonstration Yes

Classroom based products Yes

Completion of activities in learning materials Yes

Role plays Yes Yes

Oral questioning Yes Yes

Oral Presentations Yes

Written assignment/test/projects Yes Yes

Group projects Yes Yes

Portfolio Yes

To date, the assessments conducted on and off-the-job are separate activities. Both

organisations would like to establish a formal partnership arrangement to:

Formally review assessment tools used in the two different contexts; and

Establish assessor panels and/or validation partnership arrangements comprising

exchange of staff across both contexts to conduct joint assessments.

ESTABLISHING A FORMAL VALIDATION AND MODERATION PROCESS:

Although each organisation currently conducts a number of internal quality assurance and

quality review processes, establishing a formal collaborative process for conducting

consensus validation and moderation would be the ideal approach to continuously

improving the assessment practices within the school-based traineeship. To achieve this, it

was agreed that a formal Quality Assessment Management Group (QAMG) be formed to:

Panel new assessment tools being developed in either contexts;

Validate existing tools and/or any customisations using a consensus approach;

Identify ‘at risk’ students to provide early intervention (monitor adequate student

progress);

Monitor the students’ adherence to the Learning Contract specifying the work

placement requirement and expectations;

Facilitate and coordinate the establishment of assessment panels across the two

organisations (i.e., via staff exchange); and

Conduct consensus moderation to resolve discrepancies between the on and off the

job assessments in relation to competence prior to the finalisation of results.

STAFFING CONSIDERATIONS


It was agreed by the two organisations that the QAMG would comprise:

A small number (i.e. 4) of nurse educators who are employed within the education

unit of the hospital (all of whom have Cert IV within the TAA04); and responsible for

managing the learning and assessment of the school-based trainees;

The TAFE teachers who are teaching the 15 units of competency (in this case, there

are only two TAFE teachers across two campuses);

The TAFE Course coordinator; and

An independent Chair, who was employed within TAFE but not directly involved in

health but had assessment tool design expertise.

There was strong belief that a chair was required for the successful coordination and

organisation of the meetings.

As the HSC exam was seen to be a totally separate activity to the RTO based assessments,

there was an agreement that there was no need to include the NSW Education Dept as a

member of the QAMG.

AUTHORITY

The QAMG would adopt a consensus approach to decision making in which formal minutes

of decisions including action plans would be recorded. Subsequent meetings would then

include the review of evidence of the actions undertaken within the given timeframes etc.

SCHEDULING AND SAMPLING

It was thought that scheduling was more about the timing of the meetings than the sampling

of students, tools and/or units because the cohort (n=16) and course units (15 units) was so

small within the qualification. However, it was acknowledged that there was a considerable

amount of development work to be undertaken to establish the QAMG and as such, it was

recommended that six weekly meetings be scheduled in the first instance. It was also

thought that regular meetings would assist with identifying students are risk which would

enable the QAMG to adopt a preventative model as opposed to a reactive model for

monitoring student progress.

It was however acknowledged that a sampling strategy may need to be employed to manage

the selection of units, assessment tools and/or students to be undertaken as a joint panel of


assessors, given the human resource costs associated with staff exchange/relief across the

two organisations.


Maintaining comparability of standards across the two contexts would require a

commitment of both institutions to:

Attend and prepare for the QAMG meetings;

Participate in staff exchange and joint assessment panels;

Develop and/or customise assessment tools; as well as

Design validation processes and products.

Despite such human resource costs, particularly in the establishment phase, the costs were

seen to be justified by all parties as there was strong belief that it would improve the

program’s processes and outcomes for all.


There was strong agreement from all parties that there would not be a need to develop a

specific complaints and/or appeals process for the operations of the QAMG, as any

complaints/grievances could be accommodated under the RTO’s existing assessment

appeals policy. It should also be noted that both institutions thought that it would be

unlikely that there would be a need for such a process as the QAMG would:

Make decisions through a consensus approach;

Maintain formal minutes about students outcomes and progress;

Ensure all processes and outcomes are documented to ensure transparency;

Include membership of the assessment tool developers;

Be chaired by an independent member of the RTO; and

Provide mechanisms for feedback/comment by the tool developers on any

recommendations arising from previous meetings

INTERNAL REVIEW

It was agreed that the internal process would be a continuous process where the principles

underpinning the Professional Code of Practice would be an agenda item at all meetings in

which evidence of adherence would be reviewed.



The major facilitator for ensuring the success of the operations of the QAMG is that there is

close proximity between the two organisations, which have shared students and will have

shared tools.

At the same time, it was acknowledged that the major challenge would be to improve the

communication channels to receive timely reports/updates of student

achievement/progress. This was thought to be achieved via scheduling regular meetings

(i.e., 6 weekly)

It was also acknowledged by the hospital that it will be a challenge to bring other workplace

colleagues on board to provide greater opportunities for students to acquire and

demonstrate a greater breadth and depth of skills and knowledge.

RECOMMENDATIONS FOR IMPROVEMENT TO GUIDE

Whist there was strong agreement that the NQC’s Guide and Code were both useful

materials for assessment quality managers, there was an acknowledgement that such

materials may be a little daunting for VET practitioners and workplace assessors. As such, it

was recommended that any future developments include

simple guidelines and examples of the ideal characteristics of an assessment tool (in

particular the competency mapping with guidelines on how not to be too atomistic

but at the same time, establish content validity);

a template for establishing partnership arrangements (eg a Term of Reference that

features the Design features in Table 7 of the NQC (2009b) Guide; and

case study examples of application in diverse settings.


CASE STUDY D:

A SMALL REGIONAL PRIVATE RTO WORKING ACROSS STATES

BACKGROUND

This small regional private RTO has been registered for 10 years, and has focused its training

services on trade apprentices in rural and regional Victoria, South Australia and Queensland.

The RTO manages a scope across 3 states, ranging from Certificate I to Certificate III in the

areas of carpentry, joinery, wall and floor tiling, roof tiling, painting and decorating, solid

plastering and wall and ceiling lining.

The RTO is managed by two Directors and employs 18 sessional trainers, who are all

practising tradesmen and have been selected from within their region to provide training

services. All tradesmen have their own business and hence training is not their core

business.

Not all trainers have the requisite Certificate IV in Training and Assessment but supervision

and co-assessment strategies are deployed. Additional quality assurance arrangements also

include provision of developed training materials and standardised assessment tools.

The trainers that have the Certificate IV in Training and Assessment (9) play a more

significant role as trainers and assessors and provide the generic units of competency as well

as the specialist trade units. Trainers who do not have the Certificate IV are confined to

specific skills units in their trade.

Trainers are encouraged to undertake their Certificate IV qualification and two have done so

in the last six months. They are all aware of the need to upgrade to TAE10 within the next

two years and it is likely that this organisation will provide some supplementary funding to

achieve this.

ASSESSMENT

The competency based assessments usually occur in a simulated work environment in

conjunction with the classroom as well as in the workplace. The assessment methods

utilised include a mix of theory assessment which is generally an assignment or a test,

simulated practical assessment conducted at the RTO’s various premises and workplace

observation and workplace products as well as third party confirmation.

ROLES AND RESPONSIBILITIES/AUTHORITY


The two directors take responsibility for the management of the RTO. The first director

(Training Manager) coordinates the training and assessment and provides the supervision

and co-assessment presence for subject matter experts as required under the AQTF. The

second director (Resources Manager) is responsible for administration; ensuring sufficient

resources are in place for training and assessment; and ensuring compliance with

apprenticeship contracts in three states.

The Director (Training Manager) in conjunction with second director is responsible for

ensuring the development, monitoring and reviewing of the assessment tools. The

management team aims to minimise the work impact on the tradesmen and utilise them to

confirm the assessment tools or review key aspects of the assessment tool.

ASSESSMENT QUALITY MANAGEMENT APPROACH

The RTO takes a mixed approach to validation:

Once a year the training teams meets as a team building exercise and includes

professional development, operational business and also review of training

materials and assessment tools

An informal assessor partnership approach to validation is utilised amongst delivery

teams (only if distance allows) but it is not formally recorded nor managed by the

leadership team.

Currently the two directors are undertaking the bulk of the work to ensure that assessment

tools are well documented and set out (in other words ‘user friendly’ for a tradesman trainer

and assessor) and then undertake consultation with assessors for subject matter input.

Given the number of units of competency on the scope, the management team makes the

decision as to whether the changes to be made to the assessment tools are significant or

critical enough to be altered and also make the decision which assessment tools have

priority. Currently the focus of any validation activity is the assessment tool and there has

been no focus on the assessment evidence provided by candidates. There is no process in

place to monitor the effectiveness of validation processes.

The management team considers that validation activities not only inform continuous

improvement but also encourage collegiality amongst trainers/assessors who are separated

by distance.

The management team considers that given that the assessors are scattered across various

regional town and cities, that the assessor partnership model should become more

formalised, and be managed by the directors. It is envisaged that the assessors would meet


in pairs in conjunction with one of the directors to ensure that the validation activity is both

productive and recorded to inform continuous improvement.

The management team also considered that including an external party for validation would

be suitable; however it would depend on the suitability of the external validator, ensuring

confidentiality and commercial in confidence information and that it provides a positive

process for the assessment team.


The Resources Manager ensures that the assessment tools are scheduled for validation

(consensus approach), and the focus recently has been on the revised Training Package and

ensuring that the current assessment tools meet the changes created by the transition.

Given this there has been a focus on reviewing all assessment tools rather than a sampling

approach being taken. The management team sees the introduction of the new Training

Package to be an excellent opportunity to review all assessment tools. The involvement

with this National Quality Council exercise enhances this opportunity.

The two managers noted that a sampling approach and scheduling units of competency for

review over a period of time once the initial review of all assessment tools would be

appropriate. This sampling and scheduling could occur for both the consensus approach and

assessor partnership.


The RTO currently funds the annual meeting of the delivery team, including families, and

validation of assessment forms part of this meeting. Any other professional development or

assessor partnership arrangements (regardless how informal) are also funded by the RTO.

For this RTO a one day meeting can require up to three days allocation to an assessor to

travel, additional costs include travel, accommodation and salary. For the assessors, as they

all run their own businesses, then there is a significant time commitment from them as well.

The management team however acknowledges the time commitment of the assessors, and

currently fund the annual meeting and are prepared to fund the assessor partnership

approach or external validation approach if it could be easily managed and improve the

training services provided by the RTO.


It was suggested by the management team that this financial burden for the process of

assessment validation is one that is particular to rural-based RTO’s delivering over a large

geographical area.


The RTO has a brief policy and procedure related to validation which has recently been

adjusted (as a consequence of this project) to incorporate assessor partnerships. It does not

have a complaints and appeals process or procedure specifically related to validation, and

did not consider it particularly relevant to validation, only to moderation.

Both directors considered that there was very little need for a specific complaints and

appeals process as:

Assessment tools were developed by the management team from pre-existing

public domain assessment tools, and reviewed by the subject matter experts.

Decisions to make changes to assessment tools would be made by the directors in

conjunction with the assessors.

RECORDS MANAGEMENT

The RTO manages its student data using an AVETMISS compliant database and keep all hard

copy records of apprentices. However, validation activities have not been well recorded, as

many discussions and changes made are between the two managers and hence not formally

recorded.

Formal review and validation of assessment tools at the annual meeting are recorded, but

not on any specific form or template. All RTO records are maintained indefinitely.

REPORTING

Given the size of the RTO reporting requirements are not a focus. It is one of the directors

who makes notes from meetings, and reporting is for continuous improvement purposes

only.


The most significant barriers for the RTO are:

Time


Distance and geographic spread

Encouraging development of assessment knowledge of assessors.

The management acknowledged the significant time allocated to undertake an annual

meeting (that includes validation) and the time for all assessors to travel.

The geographic spread also poses its own problems, with assessors in similar fields being

separated by significant distances.

Finally, as all assessors conduct their own businesses training and assessment is not their

core business; hence demands of their businesses can affect the level of their participation.

As some are not qualified assessors, the level of professional development required to

implement the strategies suggested in the Implementation Guide should not be ignored.

The managers indicated that key facilitators would include:

Revised and simplified Implementation Guide, with forms and templates

Financial assistance

Development of a culture within the RTO to accept the importance of validation and its

impact on quality assessment and services.



The RTO had a significant number of comments to make regarding both the Code and the

Implementation Guide.

THE CODE

The Principles had variable applicability:

Transparent: Noted that the managers would provide verbal communication to inform

assessors

Representative: Once transition processes were completed, the managers stated that

unit selection would be based on risk factors

Confidential: The managers noted that samples of candidates’ work used in Validation

would be de-identified, and that should any sensitive circumstances arise confidentiality

will be maintained as appropriate. However, it had little focus for assessment tool

developers. Intellectual property would also be of concern if external validator approach

or other parties were utilised.

Educative: Validation/moderation processes would support the professional

development of trainers/assessors and the RTO as a whole.

Equitable: Although the principle was considered important it was considered that this

had little applicability to validation.

Tolerable: This pre-supposes quality of training and assessing staff and internal

processes. Professional development opportunities made available through

Validation/Moderation will highlight quality versus ‘potential to improve’ outcomes.

IMPLEMENTATION GUIDE

It was considered that the Guide was confusing, and difficult to digest for basic assessors.

The managers, noted that the tradesmen were not experts in assessment and were

confused with the language, it was considered that the words such as ‘exemplar’ were too

‘TAFEish’ and required revision. Finally, the RTO suggested that the information be

simplified, and the number of forms lessened.

It was considered that the most useful item was the basic information pertaining to

assessment tools (not the samples for each model as they provided too much information)

and the template for self-assessing the assessment tools. The Item Record Form was also

considered useful but the RTO suggested that the text be revised to make the language

more accessible for assessors.


The RTO suggested that if an additional Guide was developed that this include information

only on how to develop assessment tools and how to participate in validation – this would

focus the material for the assessors rather than the system managers.

The RTO considered that assessor partnerships would become a major strategy and that

resources to facilitate this process would be useful; as well as resources for an external

validator approach.


CASE STUDY E:

A LARGE PUBLIC RTO

BACKGROUND

The focus of the case study was on a particular assessing group within a large public RTO

that has had significant experience in conducting consensus validation over a number of

years, both as a network and now as an internal team. The field of study was horticulture

with a particular focus on Certificate III in Horticulture. However, the case study also

incorporated the views of personnel responsible for quality assurance and professional

development including teaching and learning development.

The large state public RTO has undergone significant change over the last 19 months. The

RTO includes up to 22 campuses state-wide with about 1,000 staff and over 315

qualifications on scope (not including senior secondary courses). As an organisation the

quality assurance team considers that teams across campuses need to have a robust

validation process that it is clear and provides sufficient direction for teams; it is critical to

the quality of assessment services.

The RTO has in place a range of strategies to quality assure and quality review assessment.

The RTO:

Requires assessing staff to have a minimum qualification as per NQC requirements and

aim to develop a set of professional teaching standards and develop a graduate diploma

in teaching that will embed lesser qualifications i.e. the TAE10.

Has developed a policy and procedure related to assessment.

Has developed draft validation guidelines and forms.

Has a self-assessment and review cycle, to be undertaken by each teaching team both at

the beginning of the year (self-assessment) and end of year (review). This review informs

a team specific continuous improvement action plan which is facilitated by a quality

assurance team.

The horticulture team noted that strategies employed to quality assure assessment in

addition to validation included professional development and the returning to industry

program.

ASSESSMENT


The program relies on employers for practical placement. The competency based

assessments where practicable occur in the workplace, supplemented by a simulated

environment. Not all of the RTO’s courses include a workplace component. Assessment

methods include workplace observation or products, simulated work assessment and

written assignments, test and projects. Finally, testimonials from workplaces are used to

confirm competence in the workplace.

The teams have developed assessment tools for their units of competency and have been

revising them over the years. A focus for this group would be on comparability of

assessment tools across teams.

ROLES AND RESPONSIBILITIES/AUTHORITY

The RTO assumes a consensus validation approach and considers that this should be a

formal process. The RTO has developed validation guidelines which outline the process to be

undertaken for the teams and links to the schedule to be used and the validation record

form. The policy and procedure at this stage does not provide guidance on how to conduct

consensus validation, and it is up to each team how these meetings are conducted.

The horticulture staff noted that for validation meetings there is a chair nominated but that

the decision making processes include all staff present and that they agree to the validation

decisions and what changes (if any) need to be made to assessment tools. The team

indicated that there is a report generated from the meeting and passed to all participants. In

most instances the focus of validation is on the assessment tool, however over the years

candidate generated evidence has also been used in the validation process.

Both the staff responsible for quality assurance and the horticulture team noted that

assessor partnership approach is also utilised but is undertaken on an informal basis. It was

considered by quality assurance personnel that the assessor partnership approach reflects

reality as to how assessors work together and would be a practical way of managing

coverage of training package units and the schedule. It was considered that although

assessor partnerships should be encouraged, that given the size of the RTO that the formal

process should be consensus validation approach. The interviewees did not consider that the

external validator approach would be suitable for their RTO.


The RTO does not provide specific guidance to staff as to sampling and scheduling of units of

competency, however it is an expectation that there is a schedule for validation (a sample is

provided to teams), that validation is undertaken annually and that findings are recorded


and inform continuous improvement of assessment. The guidelines state that validation

should be integrated with the continuous improvement cycle so that improvements made to

assessment as a result of validation are subject to review in order to maintain quality. The

main focus for the staff responsible for quality assurance is ensuring that quality assessment

outcomes are achieved, good practice assessment is promoted, assessment meets training

package requirements and validation is undertaken ‘systematically’.

The horticulture group undertake validation activities twice a year and address 4 -5 units of

competency each time, however given the level of discussion it may be that not all units are

fully addressed in the validation process. The team undertakes a sampling approach

generally based on units of competency that are high risk e.g. chemicals, use of machinery.

The institute quality assurance personnel noted that they aim to revise the validation

guidelines to provide groups with clear guidance as to ‘what they have to do’ so to facilitate

implementation.


The RTO acknowledged that some staff have in the past and/or may view the major driver

for validation as compliance. The quality assurance personnel believe the focus of validation

is on achieving quality outcomes, demonstrating continuous improvement, quality assuring

assessment services and in doing so meet compliance requirements. The horticultural team

considered that the main driver was consistency of assessment decision across the state.

The RTO has established semi autonomous teams and it is their responsibility to allocate

funds accordingly. Funds are provided to the RTO teams for professional development. For

the horticultural team, validation is generally conducted on student free days, or teacher

time is ‘backfilled’. Assessing staff may not attend if there are classes being conducted.

Other teams may undertake validation as part of dott (duties other than teaching) time and

this would vary according to teaching teams.

The RTO acknowledged that with any activity that involves staff, travel and logistics has

financial considerations, and the cost of validation must be costed into team expenditure.


The RTO has an assessment policy, a student complaint and appeals policy as well as a staff

complaints and appeals policy. There is also a validation policy.

RECORDS MANAGEMENT


The RTO manages its student data using an AVETMISS compliant database. The horticulture

team stated that it keeps all hard copy records of students for 7 years, and approximately

10% of assessment evidence is copied for audit purposes. Initially these records were kept in

hard copy but are now scanned. The horticulture team stated that it has used the

assessment evidence for ‘post-assessment validation’.

REPORTING

The validation process is linked with the RTO’s self-assessment and review cycle and the

yearly review/continuous improvement processes, hence reporting processes are currently

in place. However, it was noted by the quality assurance staff that it is not sufficient just to

record the required actions but that as part of the continuous improvement cycle it is

important that they are actioned.


The interviewees noted that key facilitators to successful implementation of validation

would be:

Additional resources to be developed to support the process (e.g., a guide with

minimum expectations of the teams).

Attitudinal change to encourage continuous improvement focus not a compliance

focus, encouraging professionalism and the importance of validation and good

practice assessment.

In terms of barriers the interviewees indicated that

Financing the activity across the organisation was a major issue.

There was a need to change attitude of teams, that validation is not something that

they have to do but that it is worthwhile and is important for team building and

team cohesiveness.


The RTO had a number of comments to make regarding both the Code and the

Implementation Guide.

Code


The Principles had variable applicability:

Transparent: The RTO considered that they model this principle quite well by

establishing assessor networks, quality assessment community of practice, and a

common understanding through policy documents and guidelines. However,

interviewees stated that it could do more about publicising information about teams

who have had success with validation and make information and strategies available to

other teams; and another suggestion was to engage story tellers to work with teams to

train and transfer knowledge.

Representative: The RTO considered that the processes currently established and the

requirement to utilise a validation schedule drives the notion of sampling. However it

was suggested that the RTO validation guidelines could be adjusted to include how to

undertake sampling based on risk factors. The interviewees also considered that such

factors as new assessors, new team leader and new training package would also affect

sampling.

Confidential: The interviewees indicated that this is a single organisation so the need for

confidentiality is less important; however they noted that if validation is to occur with

other external parties then confidentiality would take on greater importance and also

that protection of intellectual property would be an issue.

Educative: The interviewees considered that the validation process must be educative

and that it is important that it has a continuous improvement focus. One interviewee

also noted that it is important to make it clear that validation is not necessarily

confirming the assessment tool but a ‘really tough process’ and people need to have the

skills to manage conflict.

Equitable: The personnel responsible for quality assurance noted that this principle was

the least applicable as it seemed to take a moderation focus and hence not relevant to

validation.

Tolerable: The interviewees noted that the RTO does not address this requirement

particularly well and the requirement could be built into the current validation checklist.

One of the interviewees noted that a range of factors come into play when considering

tolerability including lack of meeting critical aspects of evidence or other unit

requirements as well as the level of risk attributed to the unit. It was considered that the

team needs to decide the priority of changes; and one interviewee proposed a decision

making tree. However, the horticulture team stated that as it is moving to a new

training package that all assessment tools require review.

Implementation Guide

Aspects of the Implementation Guide which the team thought were useful were the:

Glossary (although the definitions of the forms of validity and reliability were considered

inaccessible).

Distinction made between moderation and validation.


Definition of an assessment tool and the notion that the information does not need to

appear in one document.

Assessment Tool Self Assessment form, the Item Record form and the Summary

Validation Record form.

Advice that is provided to quality assurance and management teams to establish

validation processes, but is not relevant assessing teams.

One interviewee noted that the Implementation Guide would be useful as teaching

materials.

The RTO considered that for the Implementation Guide:

The language was dense which made it difficult to access information.

That the assessment tool information (Table 1) was useful but that the additional

method samples were not needed.

That the Guide needs to be ‘stripped back’ to provide clear guidance for the assessor

team.


SECTION 4: SYNTHESIS AND RECOMMENDATIONS

This section of the report synthesises the major findings from the case studies to:

Determine the similarities and differences in the design of the assessment quality

management system being implemented (or intending to be implemented).

Identify potential barriers and/or facilitators (e.g. structural, process and/or

personal factors); and to

Identify further resources to be developed to support RTOs conducting validation

and/or moderation activities within RTOs within diverse settings.

SIMILARITIES AND DIFFERENCES

Whilst ensuring AQTF compliance was a major incentive behind the establishment and/or

maintenance of systematic validation processes for each of the case study sites, there was

also a strong appreciation of the following potential benefits:

Promoting collegiality among staff;

Continuous improvement to the assessment processes and outcomes;

Raising the assessment literacy and self efficacy of staff;

Enhancing the comparability of standards within the network/RTO; and

Minimizing the costs associated with tool development (as there was potential for

sharing of resources).

Although all five sites had some form of quality assurance processes (e.g., minimum

requirements for assessors, establishment of an assessment policy, development of

assessment tools) and quality review processes in place (e.g., assessor partnership models,

consensus moderation, participation in AQTF audits), only one of the five case study sites

(i.e., the school-based traineeship) was intending to implement a moderation process as

part of its assessment quality management system. The rationale behind the need for such

quality control processes was to have a formal process in place to resolve any discrepancies

between the on and off-the-job assessments. Furthermore, regular moderation meetings

were also seen to provide valuable opportunities to identify ‘at risk’ students in a timely

manner. Unlike the remaining four case study sites reviewed, the school based traineeship

example involved three partner organisations who were all:

within close geographic proximity

shared the same, small cohort of students (i.e. less than 10 students per year level)

Such circumstances were thought to facilitate the process. Furthermore, it was argued that

there were potential financial as well as health and safety risks to the student, patients


and/or hospital staff if an incorrect judgement of competence was made (i.e., assessing

someone as competent when in actual fact they were not yet competent). Consequently,

the costs associated with maintaining a consensus moderation process was seen to be

justified. Whereas, with the remaining four case study sites, the additional costs associated

with establishing and maintaining an effective moderation process could not be justified,

and therefore validation processes was seen as the more viable long term solution.

It should also be noted that of the five case studies explored, only two had existing

systematic processes in place for conducting consensus validation activities on a regular

basis. That is, the large public RTO and the TAE10 state-based network of assessors. Whilst

the small private regional based RTO hosted an annual internal validation meeting with its

assessors, it relied on an informal assessor partnership approach to undertake more timely

validation activities. This was due to the perception that the consensus approach was too

expensive to undertake more than once a year across three states, and that the assessor

partnership approach would complement its annual consensus meeting approach. A

challenge for this small RTO was to make its assessor partnership processes more

formalized, systematic and transparent so that it could satisfy AQTF requirements. It also

reported willingness to explore the possibility of external participation in its validation

processes if an enterprise and/or professional association would be willing and capable of

taking on that role and financing its involvement.

The Beauty Network of Practitioners and the school-based traineeship case study example

were both in the process of developing systematic validation processes that could be applied

across their network and/or partner organisations, respectively. As such, many of their

responses were in relation to future practices and processes to be implemented at the

network/partnership level.

Although some of the sites did not have formalized, regular processes in place at the time in

which this study was conducted, all sites had a number of other activities in place for

assuring and reviewing the quality of their assessments. For example, all reported using

panels of assessors within their own to review their assessments (i.e., assessor partnerships

approach).

Table 3 displays a summary of the assessment practices, quality assurance, quality control

and quality review processes being implemented (or intending to be implemented) for each

of the five case study sites.


Table 3: Comparison of the assessment quality management processes currently being implemented (or proposed)

Case study Assessment Roles and Responsibilities Quality Assurance Quality Control Quality Review Approach

The TAE10 – network of VET Practitioners

Assessments are conducted at the individual RTOs. Role of network is to simply provide opportunities for continuous improvement by providing opportunities for sharing of resources and good practice, validating tools and disseminating information across RTOs within the state delivering the same qualification.

Minimum requirements for network members

Establishment of a standardised checklist for tool review

Establishment of a network validation coordinator Delivery of professional development on assessment and

validation

No processes in place due to logistical challenges and costs, would however like to explore e-moderation options

Establishment of a validation network across RTOs (using consensus approach)

Meetings quarterly

At this point in time, only the tools have been reviewed, not evidence of judged candidate’s work – network is looking to expand its role to include this activity

At network level, no authority to follow up on any recommendations nor any requirement to report back to the network any actions undertaken, onus is entirely on individual members to implement changes to the tool in its own RTO

No formal process for internal review of the validation network

Beauty Training Package –network of practitioners

Assessments are conducted at the individual RTOs, where there is no joint delivery of assessment or training across the member RTOs. Role of network is to provide opportunities for continuous improvement by providing opportunities for sharing of resources/good practice, validating tools and disseminating information, using a collaborative approach to raise perceptions of the Beauty Industry in SA.

Minimum requirements for network members

Establishment of a Network Coordinator, a role currently undertaken by the SA Industry Skills Council

Accessing professional development opportunities for assessment and validation

Proposed establishment of terms of reference and operation for the network

No processes in place due to logistical challenges and costs

Establishment proposed of a process for quality review of assessment online

Proposed participation in validation by industry experts, not connected to an RTO Only informal assessment validation between network members currently


Each partner organisation conducts its own assessments of the same cohort of students against the same units of competency (or a subset of the units) but come together to validate tools and outcomes.

Planning to introduce assessment panels through staff exchange

Establishment of an Assessment Quality Management Group (QAMG) with representation of all partner organisations.

Minimum requirements for partnership members

Introduction of assessor exchange to facilitate the establishment of assessment panels across the workplace and classroom contexts.

Establishment of a term of reference between partner organisations

Panelling of new tools being developed in either contexts prior to use

Establishment of a Learning Contract for the student outlining roles and responsibilities in both workplace and classroom contexts.

Using a consensus approach to moderation, the panel will:

Identify at risk students to provide early intervention (monitor adequate student progress)

Monitor the students’ adherence to their Learning Contracts

Resolve discrepancies between the on and off the job assessments in relation to competence prior to finalisation of results.

In the establishment phase of the QAMG, meetings will be monthly, with the long term view of expanding the timeframe throughout the maintenance phase.

Validate existing tools and/or any customization using a consensus approach

Minutes recorded of both moderation and validation activities and outcomes with subsequent meetings requiring the review of evidence of the actions undertaken within the given timeframes.

Introduction of an internal review process in which the principles underpinning the NQC (2009a) Code would form the focus of the review.

A small regional private RTO across three states

Assessments are conducted by a small number of sessional trainers who are all practicing tradesmen and have been selected from their own region to provide training &/or assessment services. All have their own business and hence training & assessment is not their core business. The RTO uses a mix of simulated and workplace assessments, underpinning knowledge assessment and third party reports.

The Director, if required, provides supervision and co-assessment processes for those trainers/assessor without the requisite training and assessing competencies.

Assessment Panels formed to meet minimum requirements as a collective group (eg trade and/or Cert IV TAA04)

Procedure for validation

Procedure for assessment and record keeping

Complaints and appeals procedure

Standardized assessment tools


Informal Assessor Partnerships among delivery teams where distance allows (RTO is aiming to formalize this process to satisfy AQTF requirements)

Annual validation meeting to deliver professional development and review training and assessment tools/materials internally.

A large public RTO

Employers participate in practical assessment where there is a workplace component

RTO conducts simulated assessments

Minimum requirements for assessors Intends to develop a set of professional teaching standards

and develop a graduate diploma in teaching that will embed lesser qualifications (eg TAE10)

Policy and procedures on assessment

Draft validation guidelines and forms

Development of standardized assessment tools


Formal Consensus validation approach as well as informal assessor partnerships In past, focus has largely been on the assessment tool, but moving toward

reviewing judged candidate evidence.

Self assessment and review cycle to be undertaken by each teaching team both at the beginning of the year (self-assessment) and end of the year (review)

.


POTENTIAL BARRIERS AND FACILITATORS

The study revealed a number of potential barriers and facilitators that could impact on the

design and maintenance of an assessment quality management system within a diverse

setting. These have been classified according to whether they were structural (i.e., the

organizational and resource aspects), process (i.e., that practices and activities that take

place) or personal factors (i.e., the attitudinal, assessment literacy and expectations of the

key players). Each has been dealt with next.

STRUCTURAL FACTORS

The financial implications of establishing and maintaining a consensus approach to

assessment validation/moderation was acknowledged by all sites as a potential structural

barrier. This was particularly an issue when the RTO/network had assessors who were

geographically dispersed as the costs for travel, accommodation and staff replacement/relief

could be quite high if participants had to attend face-to-face meetings. This was found to be

an issue irrespective of the size of the organisation. The need for further research and

development into alternative modes of delivery (e.g., distance, e-validation) was raised by

both networks of assessors (i.e., the Beauty and TAE10), as well as the small private provider

who had staff based within regional areas across three states.

Recommendation 1: Further research and development activities be undertaken

into e-validation in terms of :

Appropriate platforms for delivery

Mechanisms for managing the logistics of e - validation

Ensuring security and confidentiality of processes and outcomes

Alternative models for design and maintenance

Another structural barrier that was identified by participants within this study was

associated with the inconsistencies in definitions used throughout the sector at both the

national and state/territory levels. For example, a number of the definitions used in the NQC

(2009a; 2009b) Code and Guide were different to that of the Training and Education Training

Package (TAE10) and the Victorian Registration and Qualifications Authority (VRQA 2010)

Guidelines for VET Providers.

In relation to the TAE10, the Training Package has yet to incorporate the revised definition of assessment tool into its assessment related units of competency. In an attempt to differentiate between designing simple versus complex tools, the TAE10 Training Package has separated the functions of designing assessment procedures from designing assessment


instruments. For example, in the TAEASS401A Plan Assessment Activities and Processes, the unit refers to only designing assessment instruments; whereas in the TAEASS502A Design and Develop Assessment Tools, there is reference to both assessment instruments and procedures. It is however uncertain as to how an assessment instrument (even a simple instrument) could be designing and implemented in a standardised way (to enhance validity and reliability) when there are no accompanying instructions to the assessor for its use (i.e., the procedures). This has caused some confusion and misunderstandings among the participants as to which components within the NQC definition of an assessment tool refer to the instruments and which refer to the procedures. Rather than making arbitrary distinctions, the definition within the NQC (2009c) A Guide for Developing Assessment Tools has tried to encourage a more systematic approach to assessment tool development across all stages of the assessment and reporting process, which will assist with enhancing its validity and reliability.

The Victorian Registration Qualifications Authority’s (VRQA, 2010) Guidelines for VET

Providers specifies that RTOs must demonstrate evidence of external moderation of student

performance (e.g. Guideline 2.1). The term ‘external moderation’ has also caused confusion

among some of the VET practitioners within the Victorian network as it has a specific

meaning within the NQC (2009a) Code of Professional Practice for Validation and

Moderation. For example, some people were of the opinion that industry representation on

a moderation panel would constitute external moderation. However, in accordance with the

Code (NQC, 2009), an external moderation approach would typically be managed by an

industry and/or professional association who would have the power to make changes to the

assessment tools and judgements of the participating RTOs if deemed necessary. Under

such circumstances, the authoritative person/panel (e.g., from the professional

association/industry group) would not only review the way in which candidates’ evidence

was collected and judged, but if necessary, it would make adjustments to the RTO’s

assessment decisions if its standards were found not to be in alignment with those of the

external agency. In contrast, having an external representative on a panel that has been

coordinated/managed by an RTO or group of RTOs, would not constitute an external

moderation process in the true sense. Instead, it would typically be classified as a consensus

panelling approach to validation which has external representation.

Hence, despite the development of the Code and Guide (NQC, 2009) to help clarify the

language and understanding of validation and moderation in the Australian VET sector, there

still appears to be some confusion among policy makers, regulators, professional/industry

associations and RTOs on the terms ‘assessment tools’, ‘moderation versus validation’ and

‘external moderation versus external representation.’

Recommendation 2: IBSA be asked to update the assessment related units of

competency and accompanying assessment guidelines within the TAE10 to reflect


the definition of an assessment tool used within the NQC (2009 & 2010)

publications.

Recommendation 3: The Victorian Qualifications and Regulations Authority (VRQA)

be advised of the report and requested to clarify its intent of the term ‘external

moderation’ in accordance with the definitions used within the NQC (2009a) Code of

Professional Practice for Validation and Moderation and the AQTF (2010) Users

Guide to the Essential Conditions and Standards for RTOs.

It is also interesting to note that Skills Australia’s (2010) latest discussion paper canvasses

the notion of introducing an external moderation approach in the VET sector, whilst the

OECD (2008) Review of VET in Australia recommends the introduction of common

assessment procedures which would enable statistical moderation to occur. Although there

appeared to be little support for external and statistical moderation processes among the

participants within this study, further research on the feasibility of introducing alternative

models within the VET sector should be undertaken.

Recommendation 4: A feasibility study be undertaken to examine the

measurement, human resource, financial, industrial relations and logistical

implications of establishing and maintaining external and/or statistical moderation

processes within the VET sector, in particular for those qualifications deemed to be

‘high risk’.

PROCESS

In relation to moderation, a number of process factors were identified as potential barriers

to implementing a consensus approach. For example, the type of enrolment was found to

impact on the timing and scheduling of meetings. Where there were rolling enrolments, the

logistical challenge of coordinating timely meetings was seen to be a major barrier to

introducing moderation processes. In such instances, validation was seen as the more

favourable alternative in which the primary purpose of the exercise would be continuous

improvement. It was also acknowledged among those RTOs who had partnership

arrangements that effective moderation would be dependent upon establishing and

maintaining clear and timely communication channels with its partner organisations.


Another issue raised was the perceived difficulties associated with managing the assessment

workloads of staff to participate in assessment and validation activities when their core

business was not training and assessment.

PERSONAL

The need to raise the assessment literacy of assessors was seen as a major priority by many

of the participants interviewed. It was thought that having a thorough understanding of the

technical implications of assessment (e.g., validity and reliability) was necessary for effective

validation and therefore, access to more resources and professional development

opportunities in assessment tool design was required.

Recommendation 5: An on-line database of ‘quality checked’ assessment tools be

developed that demonstrates the ideal features within the NQC (2009c) Guide for

Developing Assessment Tools and which should be made available free on the web

for VET practitioners.

In addition to raising the assessment literacy of assessors and validators, it was also reported

that there needs to be an attitudinal change across the sector to encourage:

A continuous improvement focus to validation as opposed to a compliance focus;

and

Professionalism and the acknowledgement of the importance of validation and good

practice.

This would entail the development of a culture within the RTO to embrace the importance of

validation and its impact on quality assessment and services. This would then serve as a

facilitator for success.

FURTHER GUIDANCE

The current investigation revealed a strong need to develop further support material in

establishing systematic validation processes, particular for VET practitioners and workplace

assessors. All five case sites agreed to participate in the study as it was seen as an

opportunity to gain assistance with establishing and/or improving their assessment quality

management system in accordance with the processes specified within the NQC (2009)

publications. For example, despite the fact that the TAE10 state-based network of assessors

had been conducting validation activities for nearly 10 years, their process was limited to

reviewing the assessment tools without considering evidence of judged candidates’ work.


That is, the network had not established any formal procedures to check the

appropriateness of assessor judgements against the decision making rules specified within

the assessment tools. With the introduction of the NQC (2009) Code and Guide, the network

will be expanding its roles and responsibilities in the near future to include such evidence

into its consensus validation processes. Interestingly, the network however had no intention

to expand its roles and responsibilities to include individual member accountability for

actions arising from the validation meetings. The onus was (and will remain) entirely on

individual members to implement changes to his/her tools in their own RTO. Whereas, with

the remaining four sites, it was seen as important and necessary to ensure some level of

accountability for monitoring actions arising from the validation process.

In addition to seeking guidance on ways in which to improve current validation processes, all

five case studies sought greater guidance on how to design and review assessment tools,

particularly in terms of the ideal components of an assessment tool such as:

Competency mapping;

Establishing decision making rules;

Making reasonable adjustments; and

Documenting evidence of validity and reliability

For example, a number of participants raised the issue of the level of detail required in the

competency mapping. That is, questions were raised such as “was it necessary to map all

components within the unit of competency to the sub-components of the tool (e.g. whether

it should be at the element versus performance criterion level)?” Some argued that if an

atomistic approach was desired by auditors under the AQTF, then concerns were raised

about the workload implications, even if it was only carried out during the development

phase of the tool. At the same time, it was appreciated by numerous participants that such

a mapping exercise would be crucial in high risk assessment (e.g. which may provide safety

risks to the candidate) or high stakes assessment environments (e.g. for selection purposes

in which the supply was much greater than the demand). Consequently, it was suggested

that greater guidance be given on the mapping exercise in any future support materials to

ensure the process was manageable but at the same time, satisfying the requirements of

demonstrating content validity of the tool.

There were also some concerns with the technical language used within the NQC (2009)

Code and Guide and many participants suggested that whilst the materials were valuable for

system managers and assessor trainers etc, the language used and amount of reading

required would prohibit access by many VET practitioners and workplace assessors. Hence,

there was general agreement that further resources be developed which provide short and

simple guidance on issues that were commonly raised throughout the case studies such as:


Designing assessment tools in accordance with the ideal characteristics specified

within the NQC (2009b) Guide to Developing Assessment Tools (e.g., level of detail

required for competency mapping).

Establishing systematic processes for consensus validation (e.g. level of

accountability)

External validation (e.g., how does this differ to external representation on a

consensus panel?).

Formalizing assessor partnerships (e.g., in accordance with AQTF requirements).

How to satisfy AQTF auditing requirements of systematic validation when operating

within diverse contexts (e.g., small RTOs, geographically dispersed assessors).

Whilst the need to provide additional support material was raised by a number of

participants to provide greater uptake among VET practitioners and workplace assessors,

the importance of maintaining the integrity and technical soundness of the Code and Guide

was also acknowledged. As such, there was general agreement among the participants of

the study that any additional material should complement the Code and Guide rather than

replace the original documents. It was decided by the research team that the support

resources should be a complement to the original documents and users would be

encouraged to refer to the original sources for further clarification.

Recommendation 6: To ensure greater access and up-take of the NQC (2009)

publications on validation and moderation, it is recommended that the National

Quality Council publish the following support materials that have been produced

from the current investigation:

A short, easy to read guide to validation and moderation, comprising a

series of Frequently Asked Questions and Answers designed for VET

practitioners and workplace assessors.

Three Fact Sheets on Developing Assessment Tools, Systematic Validation

and Assessor Partnerships in accordance with the NQC template.


REFERENCES

Gillis, S., Bateman, A., Clayton, B. (2009). Volume 1: Research Report Validation and Moderation. Report submitted to the National Quality Council, Melbourne.

Gillis, S., Bateman, A., Dyson, C. (2010). Final Report: Validation and Moderation in Diverse Settings, Report submitted to the National Quality Council.

IBSA (2010). Training and Education Training Package (TAE10), Innovation and Business Skills Australia, Melbourne.

NQC (2009a). Code of Professional Practice for Validation and Moderation, National Quality Council, Melbourne.

NQC (2009b). Implementation Guide: Validation and Moderation, National Quality Council, Melbourne.

NQC (2009c). Developing Assessment Tools, National Quality Council, Melbourne.

NQC (2010). Ensuring comparability of standards across RTOs, National Quality Council, Melbourne.

VRQA (2010). Guidelines for VET Providers, Victorian Regulation and Qualifications Authority, Melbourne.


APPENDIX A: CHARACTERISTICS OF NOMINATED SITES

FOR REVIEW BY QA-AG

Type Location Training

Package Qualification Justification Criteria

Private RTO NSW, VIC Hairdressing Certificate III in Hairdressing/Diploma of Salon Management

Key sites in Melbourne and Sydney. Target group is apprentices and fee for service.

Small RTO

Two campus in different states

Combination of work-based and classroom based

Small regional private RTO working across states

Vic, SA, QLD General Construction

Certificate I-III in general construction, floor and wall tiling, solid plastering, bricklaying/blocklaying and carpentry

Focus on trades in remote regions, apprentices and skills in demand.

Small RTO

Regional

Industry / Professional Organisation

National Public Safety Certificate II in Public Safety (Aquatic Rescue))

Certificate III in Public Safety (Aquatic Search and Rescue)

Certificate IV in Public Safety (Leadership)

Certificate IV in Training and Assessment

National network with each state surf life saving club an RTO. Within each member organisation, staffing based on volunteers and thus has particular issues in terms of quality assurance burden on individual.

Industry/Professional Association

National membership

Staffing based on volunteers

Partnership arrangement (enterprise, TAFE and Education Department)

NSW & QLD Health Training Package

Certificate III in Health Services Assistance

Network of hospitals in NSW and QLD, auspiced by Sydney Institute of TAFE to deliver Cert III and are involved in the delivery of school-based traineeships. High risk associated with incorrect judgement.

Enterprise RTO

Partnership arrangement between enterprise, TAFE and Education Department

Delivery of school-based traineeship

High risk associated with incorrect judgements

Community Based RTO

VIC Range of training packages and curriculum

Various (large scope of registration)

Community Provider Network with a large scope including business, community services (aged care, children’s services, HACC etc), Sport and Recreation, as well as also curriculum based.

Small RT0

Community Provider

Diverse range of qualifications delivered

Large public RTO with multiple sites Not disclosed

due to confidentiality issues

Various Various Various campuses, small number of assessors at remote sites. Range of qualifications. Includes VET in Schools.

Large RTO

Large youth cohorts

VET in Schools

Multiple campuses

Regional and rural campuses

State based Network of assessors

VIC TAE 10 Cert IV

Dip o f VET Practice

Membership includes 17 TAFEs and University TAFES in Victoria and some private RTOs

Professional Network

Represents private RTOs and TAFE

Membership includes regional, rural and metropolitan RTOs

State based Network of assessors

SA Beauty Training Package

Network includes various types of RTOs across the state delivering the Training Package.

Network of RTOs

Range of sizes

Range of geographic locations represented

http://www.ntis.gov.au/?/Qualification/PUA21004







http://www.ntis.gov.au/?/Qualification/TAA40104

http://www.ntis.gov.au/?/Qualification/TAA40104




APPENDIX B: INFORMATION TO CASE STUDY SITES.

INFORMATION TO PARTICIPATING RTOS OPERATING WITHIN DIVERSE CONTEXTS

Thank you for agreeing to participate in the National Quality Council funded project entitled “Best Practice in Validation and Moderation within diverse settings.” The project is being conducted by a research team led by Shelley Gillis at Victoria University, in collaboration with Andrea Bateman and Chloe Dyson.

Background to the Project

In 2009, the research team undertook a project for the National Quality Council designed to identify the role of validation and moderation processes in the VET sector. This resulted in the following products being published by the NQC:

The Professional Code of Practice The Guide for Developing Assessment Tools The Implementation Guide for Validation and Moderation

These products may be downloaded from http://www.nqc.tvetaustralia.com.au/nqc_publications

The National Quality Council noted the aspirational nature of this work and that it provided a way forward within a complex and difficult area where misunderstandings have been common. Late last year, the NQC funded the research team to undertake a series of interactive information sessions in all states and territories to enable dissemination of the findings.

The NQC also noted, however, that the focus of the work has been on consensus meetings as a way of conducting moderation or validation, and that uptake would therefore be most likely to be limited to larger RTOs.

In 2010, the NQC has commissioned the research team to undertake further follow up work to:

Develop additional guidelines and support materials for alternative approaches to consensus meetings as a validation strategy as might be used by diverse RTOs which deliver and assess in a range of contexts; and

To undertake national case studies which focus on validation of specified qualifications as a means of enhancing consistent assessment outcomes.

The aim of this project is therefore to seek feedback from RTOs delivering VET within diverse contexts on the perceived appropriateness, flexibility and usefulness of the Guide for conducting moderation and validation of assessments.


To assist with reviewing the Implementation Guide and developing new support material, we would like to visit your organsiation on two occassions at a convenient time to be negotiated.

The first visit would involve one member of the research team delivering a Professional Development workshop to your staff and colleagues to introduce the new definitions, principles and approaches that underpin the Code and Guide. It would also involve the researcher conducting a small number of interviews with key personnal (from both within and across your partnering organisations) on current approaches and perceptions of validation and/or moderation.

The second visit would be scheduled four to six weeks later when you and your colleagues have had the opportunity to review and use aspects of the Guide within your context. The purpose of the follow-up visit would be to obtain more in-depth feedback about the

Perceived usefulness and relevance of Guide; Perceptions of potential factors that may impact on the design, implementation and

maintenance of a validation and/or modertion process within your context; Recommendations for improvement to Guide; and Suggestions for development of further support materials.

Please note that confidentiality of responses will be assured at all times and each individual will have the freedom to withdraw from the study at any time.

What will you be asked to do?

Prior to the Visit

Organise a convenient time with the researcher to deliver the professional development workshop (approximately 4 hours)

Organise a venue for the workshop Organise a room to conduct interviews (approximately 2 hours) Advertise and promote the workshop to your assessors and validators and any of

your partner organisations to attend Liaise with the researcher in relation to venue, attendance etc

At the First Visit

Attend the workshop along with your colleagues (i.e., assessors and/or validators both internal and external to you organisation)

Participate in an one hour interview with the researcher Provide copies to the researcher on any relevant policies, procedures, templates etc

that your organisation uses to conduct either validation and/or moderation


Organise for some of your partner organisations (if any) to also attend the workshop and undertake a short interview with the researcher (eg employers, VET in School Teachers, other RTOs) on the same day as the workshop

Between Visit 1 and Visit 2

Review the Guide with your colleagues, make suggestions for improvements Apply the Guide within your context Maintain notes/records of what worked, what didn’t work etc Arrange a convenient time for follow up visit with researcher and partner

organisations

At the Second Visit

Participate in a follow-up interview with the researcher to provide feedback on the Guide

Arrange for follow-up interviews with your partnering organisations. What will my organisation gain from participating?

The findings of the research will be used to inform future developments of support material for applying the Professional Code of Practice in Diverse VET Settings. All participating sites will receive a copy of the support materials for use within their own organisation. Your organisation will also receive $2200 for arranging the workshop and interviews with your colleagues.

Who is the research team?

The following individuals are conducted the study. If you have any queries or concerns, please do not hesitate to contact any one of them.

Associate Professor Shelley Gillis Deputy Director, Work-based Education Research Centre (WERC) Victoria University Mobile 0432 756 638 Email: [email protected]

Andrea Bateman Director, Bateman & Giles Pty Ltd Mobile: 0418 585 754 Email: [email protected]

Chloe Dyson Director, Chloe Dyson & Associates Pty Ltd Mobile: 0408124825 Email: [email protected]



INFORMATION TO THE PARTICIPATING NETWORKS

Thank you for agreeing to participate in the National Quality Council funded project entitled “Best Practice in Validation and Moderation within diverse settings.” The project is being conducted by a research team led by Shelley Gillis at Victoria University, in collaboration with Andrea Bateman and Chloe Dyson.

Background to the Project

In 2009, the research team undertook a project for the National Quality Council designed to identify the role of validation and moderation processes in the VET sector. This resulted in the following products being published by the NQC:

The Professional Code of Practice The Guide for Developing Assessment Tools The Implementation Guide for Validation and Moderation

These products may be downloaded from http://www.nqc.tvetaustralia.com.au/nqc_publications

The National Quality Council noted the aspirational nature of this work and that it provided a way forward within a complex and difficult area where misunderstandings have been common. Late last year, the NQC funded the research team to undertake a series of interactive information sessions in all states and territories to enable dissemination of the findings.

The NQC also noted, however, that the focus of the work has been on consensus meetings as a way of conducting moderation or validation, and that uptake would therefore be most likely to be limited to larger RTOs.

In 2010, the NQC has commissioned the research team to undertake further follow up work to:

Develop additional guidelines and support materials for alternative approaches to consensus meetings as a validation strategy as might be used by diverse RTOs which deliver and assess in a range of contexts; and

To undertake national case studies which focus on validation of specified qualifications as a means of enhancing consistent assessment outcomes.

The aim of this project is therefore to seek feedback from RTOs delivering VET within diverse contexts on the perceived appropriateness, flexibility and usefulness of the Guide for conducting moderation and validation of assessments. As part of this study, we are also looking at networks of RTOs who are responsible for validating specific qualifications.


To assist with reviewing the Implementation Guide and developing new support material, we would like to meet with members of your network at a convenient time to be negotiated.

The first meeting would involve one member of the research team delivering a Professional Development workshop to your members to introduce the new definitions, principles and approaches that underpin the Code and Guide. It would also involve the researcher conducting a small number of interviews with key personnal from your network on current approaches and perceptions of validation and/or moderation.

The second meeting would be undertaken by telephone and/or email and would be scheduled four to six weeks later when you and your colleagues have had the opportunity to review and use aspects of the Guide within your context. The purpose of the follow-up discussions would be to obtain more in-depth feedback about the

Perceived usefulness and relevance of Guide; Perceptions of potential factors that may impact on the design, implementation and

maintenance of a validation and/or modertion process within your context; Recommendations for improvement to Guide; and Suggestions for development of further support materials.

Please note that confidentiality of responses will be assured at all times and each individual will have the freedom to withdraw from the study at any time.

What will you be asked to do?

Prior to the face to face meeting

Organise a convenient time with the researcher to deliver the professional development workshop (approximately 4 hours)

Organise a venue for the workshop Organise a room to conduct interviews (approximately 2 hours) Select a qualification to form the focus of the workshop Advertise and promote the workshop to your members Liaise with the researcher in relation to venue, attendance etc

At the face to face meeting

Attend the workshop along with your network members Participate in an one hour interview with the researcher Provide copies to the researcher on any relevant policies, procedures, templates etc

that your network uses to conduct either validation and/or moderation

Between Meeting 1 and Meeting 2

Review the Guide with your members, make suggestions for improvements


Each member to apply the Guide within his/her context Each member to maintain notes/records of what worked, what didn’t work etc Provide telephone contact details to the researcher of members willing to

participate in the follow-up meeting

At the Second Follow-up Meeting

Participate in a follow-up interview with the researcher to provide feedback on the Guide

What will my network gain from participating?

The findings of the research will be used to inform future developments of support material for applying the Professional Code of Practice in Diverse VET Settings. All participating networks will receive a copy of the support materials for use by their own members. Your network will also receive $2200 for arranging the workshop and interviews with your members.

Who is the research team?

The following individuals are conducted the investigation. If you have any queries or concerns, please do not hesitate to contact any one of them.

Associate Professor Shelley Gillis Deputy Director, Work-based Education Research Centre (WERC) Victoria University Mobile 0432 756 638 Email: [email protected]

Andrea Bateman Director, Bateman & Giles Pty Ltd Mobile: 0418 585 754 Email: [email protected]

Chloe Dyson Director, Chloe Dyson & Associates Pty Ltd Mobile: 0408124825 Email: [email protected]



APPENDIX C: CASE STUDY QUESTIONNAIRES

SITE VISIT 1: NETWORK - QUALIFICATION INTERVIEW SCHEDULE

Section 1: Network Characteristics

1. How many members are there in your network?

2. Approximately how many RTOs do they

represent? 3. What types of RTOs belong to your network?

Tick all applicable boxes

Private

Public

Enterprise

Community

Other, please specify

_____________

4. Which of the following AQF qualification types do your members validate as part of the network?

Senior Secondary Certificate of Education

Certificate I

Certificate II

Certificate III

Certificate IV

Diploma

Advanced Diploma

5. What is the breadth of your membership’s delivery?

National

TAS

VIC

NSW

QLD

ACT

NT

SA

WA

_____________ 6. What provision arrangements for delivering

qualifications does your membership have?

Provision provided solely by RTO

Provision includes support by employers

at work sites

Provision includes partnering

arrangements with other organisations

Provision includes auspicing

arrangements with schools



Section 2: Assessment practices

1. For each summative assessment method listed below, indicate how often they are validated or moderated within your network.

Method Never = 1, Sometimes =2, Often =3, Always = 4

Work-place observation

1----2-----3----4

Work-place products

1----2-----3----4

Simulation/classroom demonstration

1----2-----3----4


1----2-----3----4


1----2-----3----4

Role plays

1----2-----3----4

Interviews

1----2-----3----4

Oral Presentations

1----2-----3----4

Written assignment/test/projects

1----2-----3----4

Group projects 1----2-----3----4

Portfolio

1----2-----3----4

2. Are there any other assessment methods not listed, that are predominantly validated or moderated by your membership? If so, please list.

3. To what extent does your network validate member-based assessments that have been administered on-line, in the workplace and/or within the classroom setting?

4. What challenges does each of these different delivery modes bring to your network?


5. What type of validation/moderation records are kept by your network? How long are they kept for and what are they used for? Who is responsible for maintaining such records and where are they stored?

Types of records Frequency

Never = 1, Sometimes =2,

Often =3, Always = 4

Location Storage time Retrieval purposes

Sampling framework

employed

1----2-----3----4

Coding system used to

maintain confidentiality

1----2-----3----4

Instructions to network

members

1----2-----3----4

Attendance forms

1----2-----3----4

Validation outcomes

and decisions

1----2-----3----4

Samples of assessment

tools reviewed

1----2-----3----4

Samples of judged

candidates work

1----2-----3----4

6. Please describe any other records that are kept and maintained by the network?

7. What are the reporting requirements of the network in relation to validation and/or moderation?

8. Does your network have an assessment protocol or guide? If so, can we have a copy?


Section 3: Assessment Quality Management

9. Given the definitions of quality assurance, quality control and quality review within the Code of Professional Practice for Validation and Moderation (NQC, 2009 p.4 & p.5), what strategies does your network implement to assure quality assessments amongst its member organisations?

Approach Strategies Implemented

Quality Assurance

Quality Control

Quality Review

10. Given the definitions of validation & moderation in Code (NQC, 2009, p.7 & 8), which of the two best describes your network’s current approach? Please describe this approach.

11. To what extent does your network use the following type of approaches to validation and/or moderation?

Validation Frequency

Never = 1, Sometimes =2, Often =3, Always = 4

Assessor Partnerships 1----2-----3----4

Consensus Meetings 1----2-----3----4

External (validators or panels) 1----2-----3----4

Moderation Frequency



External (moderators or panels) 1----2-----3----4

Statistical 1----2-----3----4


12. What barriers are there within your network, for successful implementation of validation and/or moderation? What factors are likely to facilitate its success?

13. Is there a formal network policy and procedure for validation and/or moderation? If so, can we please have a copy? Do you have any templates or supporting documents that you use? If so, can we also have a copy? How helpful are these? What improvements can you recommend to such documents?

14. If validation/moderation processes are in place, what was the main driving force behind establishing the processes?

15. What do you see as the key purpose of conducting validation and/or moderation within your network?

16. Who participates in the validation and/or moderation process in your network? How are these people selected? What qualifications, training and/or experience must they have? What are their roles and responsibilities?

17. Who manages the validation and/or moderation process in your network? What are their responsibilities?

18. How are the final decisions/recommendations/outcomes from the process determined? Who makes the decision and how much power do they have? What records does the network keep of validation and/or moderation outcomes and subsequent action?

19. What level of accountability should be associated with the process? For example, should the outcomes of the process be discretionary for the network member (eg are the members able to take responsibility for their own continuous improvement of their assessment tools) or should there be some corrective action to be monitored (if so, by whom?) or should it be mandatory and accountable at the system level? Please explain why.

20. How often does the validation and/or moderation process occur? When does it occur? What recommendations do you have for improving this process? Do you use a sampling


methodology? If so, what are they key factors that assisted you in selecting the sample? If not, how do you manage the number of units you moderate/validate at one time?

21. How often do you carry out the validation and/or moderation process? How is this determined? How do you determine what to validate and moderate and when?

22. Do you validate the assessment tools as well as the assessor judgements? What are the strengths and limitations of your approach?

23. What are the financial considerations associated with establishing and maintaining the process?

24. Have there ever been any complaints and/or appeals about the validation/moderation process from the network membership? How have you handled such complaints/appeals? Do you have any documentation that relates to this? If so, can we have a copy, without identifiers? If you have never had any complaints, how would you handle any in the future?

25. What are your internal processes for monitoring and reviewing the effectiveness of the validation and/or moderation process?


Section 4: The NQC (2009) Code and Guide

26. Within your context, what processes and/or procedures could be (or have been) implemented to ensure that each principle specified in the Code of Professional Practice for Validation and Moderation (NQC, 2009, p.16-18) can be met?

Principle Strategy Transparent

Representative

Confidential

Educative

Equitable

Tolerable

27. What are the strengths and limitations of using the following different types of moderation and/or validation processes in your context? Refer to Code of Professional Practice for Validation and Moderation (NQC, 2009, p.11-14)

Approach Strength Limitation Partnership

Consensus

External

Statistical

28. What components within the Implementation Guide are useful for your context?


29. What were the strengths of the Implementation Guide?

30. What were the weaknesses of the Implementation Guide for use in your context? How could these be overcome?

31. How useful was Section 1: Assessment Tools (p. 9-23)? What changes would you recommend?

32. For your particular context, how useful was the guidelines for establishing a consensus approach to validation and/or moderation (refer to pages 24-43)? What changes would you recommend?

33. To what extent would you find useful a set of additional guidelines for establishing external and/or partnership arrangements for validation and/or moderation? What issues would need to be addressed in such guidelines?

34. How useful were the Templates provided in the Appendix? What changes and/or additions would you suggest?

35. What recommendations do you have for future improvements to the Implementation Guide to meet the needs of diverse contexts?

36. To what extent could the Code and Guide be applied to your context? What would facilitate this? What would be some of the obstacles that would need to be overcome? Are there any structural (eg resources, equipment, staffing, storage requirements), processes (eg partnership arrangements, assessment methods, record keeping) or attitudinal factors (of staff, stakeholder expectations) that would need to be considered?

37. Could you please list three (3) critical issues for you with implementing the Code and/or Guide within your context that would need to be addressed?


SITE VISIT 1: RTOS OPERATING WITHIN DIVERSE CONTEXTS

Section 1: Organisational Characteristics

1. What is your RTO type?

Private

Public

Enterprise

Community


2. How many students does your organisation

enrol annually?

<200

200-499

500-999

1000-4999

5000-10,000

>10,000

3. Approximately many teaching/assessing staff does your organisation employ (including contractors)?

_____________ 4. Which fields of study does your organisation

offer qualifications?

Natural and Physical Sciences

Information Technology

Engineering and Related Technologies

Architecture and Building

Agriculture, Environment and Related Studies

Health

Education

Management and Commerce

Society and Culture

Creative Arts

Food, Hospitality and Personal Services

Mixed Field Programmes

5. Which of the following AQF qualification types does your organisation conduct

assessments toward?

Senior Secondary Certificate of Education

Certificate I

Certificate II

Certificate III

Certificate IV

Diploma

Advanced Diploma

Associate Degree

Bachelor Degree

Graduate Certificate

Graduate Diploma

6. What is the breadth of your organisation’s

delivery?

National

TAS

VIC

NSW

QLD

ACT

NT

SA

WA

Internationally (off-shore)

7. How many campuses does your organisation

have?_________________ 8. What provision arrangements for delivering

qualifications does your organisation have?

Provision provided solely by RTO

Provision includes support by employers

at work sites

Provision includes partnering

arrangements with other organisations

Provision includes auspicing

arrangements with schools



Section 2: Assessment practices

9. For each assessment method listed, indicate how often they are typically used for summative assessments within your organisation.

Method Never = 1, Sometimes =2, Often =3, Always = 4

Work-place observation

1----2-----3----4

Work-place products

1----2-----3----4

Simulation/classroom demonstration

1----2-----3----4


1----2-----3----4


1----2-----3----4

Role plays

1----2-----3----4

Interviews

1----2-----3----4

Oral Presentations

1----2-----3----4

Written assignment/test/projects

1----2-----3----4

Group projects 1----2-----3----4

Portfolio

1----2-----3----4

10. Is there any other assessment methods not listed, that is predominantly used for summative assessments within your RTO? If so, please list.

11. Where are your assessments mostly conducted?

Mode Never = 1, Sometimes =2, Often =3, Always = 4

On-line assessment

1----2-----3----4

Classroom based assessment

1----2-----3----4

Work-based assessment

1----2-----3----4


12. What type of assessment records do you keep within your organisation? How are they stored and how long for? For what purposes are they typically retrieved at a later time?

Types of assessment

records

Storage Options (please

tick)

Length Retrieval purposes

Statistical (eg grades,

marks) Electronic

Paper-based

Anecdotal (eg

logbooks, diaries,

critical incident

reports, qualitative

reports)

Paper-based

Electronically

Folios (portfolios,

products, work

samples etc)

Manually filed

Electronically

13. Do you have an assessment policy and procedure? If so, can we have a copy?

Section 3: Assessment Quality Management

14. Given the definitions of quality assurance, quality control and quality review within the Code of Professional Practice for Validation and Moderation (NQC, 2009 p.4 & p.5), what strategies does your organisation/association/network implement to assure quality assessments?

Approach Strategies Implemented

Quality Assurance

Quality Control

Quality Review


15. Given the definitions of validation & moderation in Code (NQC, 2009, p.7 & 8), which of the two best describes your current approach? Please describe your current approach?

16. To what extent do you use the following type of approaches to validation and/or moderation?

Validation Frequency


Assessor Partnerships 1----2-----3----4


External (validators or panels) 1----2-----3----4

Moderation Frequency



External (moderators or panels) 1----2-----3----4

Statistical 1----2-----3----4

17. What barriers are there within your context, for successful implementation of validation and/or moderation? What factors are likely to facilitate its success?

18. Is there a formal policy and procedure for validation and/or moderation? If so, can we please have a copy? Do you have any templates or supporting documents that you use? If so, can we also have a copy? How helpful are these? What improvements can you recommend to such documents?

19. If validation/moderation processes are in place, what was the main driving force behind establishing the processes?

20. What do you see as the key purpose of conducting validation and/or moderation within your context?


21. Who participates in the validation and/or moderation process? How are these people selected? What qualifications, training and/or experience must they have? What are their roles and responsibilities?

22. How are the final decisions/recommendations/outcomes from the process determined? Who makes the decision and how much power do they have?

23. What level of accountability should be associated with the process? For example, should the outcomes of the process be discretionary for the assessor (eg are the assessors able to take responsibility for their own continuous improvement of their assessment tools) or should there be some corrective action to be monitored (if so, by whom?) or should it be mandatory and accountable at the system level? Please explain why.

24. How often does the process occur? When does it occur? What recommendations do you have for improving this process? Do you use a sampling methodology? If so, what are they key factors that assisted you in selecting the sample? If not, how do you manage the number of units you moderate/validate at one time?

25. How often do you carry out the process? How is this determined? How do you determine what to validate and moderate and when?

26. Do you validate the assessment tools as well as the assessor judgements? What are the strengths and limitations of your approach?

27. What are the financial considerations associated with establishing and maintaining the process?

28. Have there ever been any complaints and/or appeals about the validation/moderation process from students and/or staff? How have you handled such complaints/appeals? Do you have any documentation that relates to this? If so, can we have a copy without identifiers? If you have never had any complaints, how would you handle any in the future?


29. What validation/moderation records are kept and how long for? How do you use these

records?

30. What are your internal processes for monitoring and reviewing the effectiveness of the validation and/or moderation process?

Section 4: The NQC (2009) Code and Guide

38. Within your context, what processes and/or procedures could be (or have been) implemented to ensure that each principle specified in the Code of Professional Practice for Validation and Moderation (NQC, 2009, p.16-18) can be met?

Principle Strategy Transparent

Representative

Confidential

Educative

Equitable

Tolerable

39. What are the strengths and limitations of using the following different types of moderation and/or validation processes in your context? Refer to Code of Professional Practice for Validation and Moderation (NQC, 2009, p.11-14)


Approach Strength Limitation Partnership

Consensus

External

Statistical

40. What components within the Implementation Guide are useful for your context?

41. What were the strengths of the Implementation Guide?

42. What were the weaknesses of the Implementation Guide for use in your context? How could these be overcome?

43. How useful was Section 1: Assessment Tools (p. 9-23)? What changes would you recommend?

44. For your particular context, how useful was the guidelines for establishing a consensus approach to validation and/or moderation (refer to pages 24-43)? What changes would you recommend?

45. To what extent would you find useful a set of additional guidelines for establishing external and/or partnership arrangements for validation and/or moderation? What issues would need to be addressed in such guidelines?

46. How useful were the Templates provided in the Appendix? What changes and/or additions would you suggest?

47. What recommendations do you have for future improvements to the Implementation Guide to meet the needs of diverse contexts?

48. To what extent could the Code and Guide be applied to your context? What would facilitate this? What would be some of the obstacles that would need to be overcome? Are there any structural (eg resources, equipment, staffing, storage requirements), processes (eg partnership arrangements, assessment methods, record keeping) or attitudinal factors (of staff, stakeholder expectations) that would need to be considered?

49. Could you please list three (3) critical issues for you with implementing the Code and/or Guide within your context that would need to be addressed?

Documents

Validation & Moderation in Diverse Settings...RESEARCH REPORT VALIDATION AND MODERATION IN DIVERSE SETTINGS PAGE 2 CONTACT NQC Secretariat ... A SMALL REGIONAL PRIVATE RTO WORKING