Disciplinary standards and Assessment

Peter CullenHead of Standards, Research and Policy Development Unit

HETACwww.hetac.ie

Bologna Expert Colloquium

Supporting the Design of Discipline-Specific Learning Outcomes

6 February 2009

HETAC’s mandate includes

• to determine “standards” of knowledge, skill or competence …

• “validation” means the process by which an awarding body shall satisfy itself that a learner may attain knowledge, skill or competence for the purpose of an award made by the awarding body;

• Review of the effectiveness of agreed quality assurance procedures review of delegation of authority…

What is a standard?

• It’s an educational goal• …a required or specified level of excellence,

attainment, … (from Oxford English Dictionary NSOED 1997)

• knowledge, skill or competence to be acquired by learners before a higher education and training award may be made (Qualifications Act)

What is it for?• For learners!• For the provider to use when designing and reviewing

intended programme learning outcomes• For the awarding body to use when validating programmes• For internal QA to use when reviewing performance• For external examiner to use along side the intended

programme learning outcome• For the external quality assurance agency to use when

auditing/assessing/accrediting• For the employer when interpreting the significance of

qualifications

Why not rely on tacit standards?• Traditional approaches to assessment in higher education

typically place heavy reliance on tacit understandings of standards and can be strained in new or rapidly changing contexts.

• Examples of changing contexts which have encouraged practitioners to look for innovative approaches include – massification, new kinds of teaching and learning, modularisation,

plagiarism, learner retention, computer-aided assessment, new kinds of intended learning outcomes, availability of study time, and declining resources.

• Tacit element is ineluctable– connection with community of practice/learning

Types of standards and settersNQF Award-typedescriptor

Named award orbroad Field of Learningstandard

Narrow disciplineStandard orRegulated Profession

Intended programmeLearning outcome

Honours Bachelor Degree

Nursing

Engineering

Science

Programme X

Programme Y

Generic

HETAC approach

Tuning/QAA

IOT and other HEI

NQAI

FETAC approach

Universities and DIT

General Particular

Standards and precision• Precision needs depend on how the standard is to be used. Is it to check that the

mean performance level is on target? Or is it to check that the vast majority of students graduate attain or exceed the standard? Is it used for summative assessment of learners?

• Note that the QAA benchmark statements contain threshold levels as well as other performance levels)

• Standards can be representative (describing characteristics to be possessed by most)

• Standards can be threshold (characteristics to be possessed by all)• Standards can be typical (the meaning of this varies)• Can standardsstandards that describe the average (mean performance) be used for

summative assessment of learners?• What is important is clarity and precision but not too much precision because

flexibility is necessary. The nearer the standard is to the ‘programme’ the more precise it should be– the next slide illustrates

Learning environment (including people interacting)

Synthesis of the intended programme learning outcomes

Entrants

Module

Module

Module

Module

Module

Module

Module

Graduates

…

…

Note that stages emerge naturally from the network of learning opportunities in a learning environment

Intended programme learning outcomes (the provider’s standard)

• Intended programme learning outcomes describe the minimum achievement (in terms of knowledge, skill and competence) that the learner is certified to have attained if he/she successfully completes a particular programme (i.e. passes all the required assessments).

• The intended programme learning outcomes are the principal educational goal of the programme and effective assessment helps learners to attain that goal.

• Teachers and learners may strive for additional learning outcomes beyond the minimum as expressed by the intended programme learning outcomes.

• Intended programme learning outcomes are developed and maintained by providers.

• Programmes are designed to enable learners to achieve intended programme learning outcomes.

• Intended learning outcomes are specified for each of a programme’s constituent modules.

• The number of learning outcomes in a statement of intended learning outcomes is variable (depending, for example, on the semantics and the level of explicitness used) and it is not a proxy for credit.

Implementing standards• Note the implementation gap: remember that tacit interpretation operates on all

standards• Note the implementation lag: It will take time for providers to align to new

standards (unless you choose the least as the standard) and assessment is key• A necessary condition for implementation is being assessable: for if something can

be assessed it can be used to control the learning environment appropriately• What does assessable mean

– Test for certification related (examine learners)– Test for qualification—how does this differ from certification?– Doing longitudinal research (learners and graduates and employers)– Other…

• Student assessment procedures are expected to be designed to measure the achievement of the intended learning outcomes and other programme objectives. (ESG)

• Student assessment procedures are expected to have clear and published criteria for marking (ESG)

Six assessment principles• Learners are responsible for demonstrating their attainment• Assessment supports standards based on learning outcomes• Assessment promotes and supports effective learning and

teaching• Assessment procedures are credible• Assessment methods are reviewed and renewed as necessary

to adapt to evolving requirements• Learners are well informed about how and why they are

assessed• Please have a look at the ‘Assessment and Standards’

consultation document on the HETAC website

Writing standards• HETAC uses the following process

– Background research– Standards group (recognized experts)– Formal public consultation– Review (not yet done)

• Vague aspirations and hyperbole don’t sit well with standards

• Lots of literature to refer to • Note the warnings about over-dependence on

taxonomies

Generic standards

• The award-type descriptors of the NFQ are being used by HETAC as generic awards standards.

• Generic standards are the kernel of the awards standards

HETAC Awards Standards• Statutory: determine the standard to be attained before an

award may be made• Overall standards set by the NQAI through the NFQ• NFQ award-type descriptors are kernels • Broad• Flexible (not all individual outcomes mandatory)• Pairs of standards can be used together• Silent on the curriculum, assessment, duration, except by

what is implied by learning • Based on the disciplines of the NCEA boards of study with

some additions including standards for regulated professions

Pilot standards• Fields of Learning (ISCED 1997)

– Art & Design– Business– Computing– Engineering– Nursing– Science– Complementary therapies

• Currently looking at (or considering looking at)– Architecture– Social care– Education– Others???– Have been requested to look at Humanities!?

Why broad• Standards should be sufficiently flexible to allow a range of approaches, appropriate to

different sub-fields within fields of learning and different providers.• Flexible

– Innovative and creative programme design – Avoids mainstreaming– Allows programmes to adapt to changing needs and opps – Tolerates diversity– Facilitates multi-disciplinary programmes

• Practical for a small country – Can be reviewed periodically– Reduced likelihood of obsolescence – QAA currently have 55 benchmark statements at honours bachelor degree level alone (there is one

for engineering…– Providers can supplement the HETAC broad standards with Tuning and QAA descriptors as well as

professional body statements of learning outcomes• Does it work?

– HETAC (standards have been in use for four years) – Research will tell– Anecdotal evidence is definitely positive but some modifications are necessary

www.hetac.ie

• Thank you• Please visit our website for more information