1

Quality Assurance and the

question of Validity

Crossfields Institute

The first school merged Rudolf Steiner’s philosophical, spiritual concepts with Emil Molt’seconomic reality

Waldorf School Associations have trademarked “Waldorf” and “Steiner” to ensure quality control

Validity

• What does it mean?

Two things you need to know about validity

“If one were to select a sample of psychometricians from each of the last five to ten decades and gather them together in, say, a bar, it is quite likely that all would drink a toast to validity as the paramount concept in the field of testing.”

Fast, E. and Hebbler, S. with ASR-CAS Joint Study Group on Validity in Accountability Systems. (2004). A Framework for Examining Validity in State Accountability Systems. Washington, DC: Council of Chief State School Officers.

Two things you need to know about validity

“If one were to select a sample of psychometricians from each of the last five to ten decades and gather them together in, say, a bar, it is quite likely that all would drink a toast to validity as the paramount concept in the field of testing. However, a mêlée would ensue if they were asked to define what validity is.”

Fast, E. and Hebbler, S. with ASR-CAS Joint Study Group on Validity in Accountability Systems. (2004). A Framework for Examining Validity in State Accountability Systems. Washington, DC: Council of Chief State School Officers.

But don’t worry about that because…

The general idea of validity is basically simple

• it’s whether we’re assessing the right thing, in the right way, to produce accurate and usefulassessment results

1. Validity(which is the water in your bucket)

2. Threats to validity(that risk piercing your bucket)

This session will focus upon

Illustration by Sal Murdocca

1. VALIDITY

The elixir of assessmentCredit: Joe Dyer

So this is what Ofqual and Crossfields Institute mean by validity• The validity of a particular qualification or certification is

• the degree to which it is possible to measure• whatever that qualification or certification needs to measure• by implementing its assessment procedure.

• The validity of a particular qualification or certification is determined by• ALL of the features and processes that are put in place to

ensure that results are as accurate as possible and as usefulas possible.

This version of validity is pretty broad

VALIDITY

Authenticity

Reliability Sufficiency

Currency

And other

stuff too!

VALIDITY

Comparability

Reliability Minimal Bias

Manageability

And other

stuff too!

Five critical steps in the lifecycle of any assessment

1. Measurement objectives (LOs) are clarified

2. Multiple performances (evidence) are elicited from each student (via tasks) to provide evidence of achievement or proficiency

3. All evidence is evaluated in terms of what it implies about student achievement or proficiency

4. The evaluations, for each student is combined into an overall result

5. The result is interpreted by those for whom it has been provided

A qualification specification

A set of tasks for each candidate

A set of evaluations for each candidate

An overall result for each candidate

An interpretation of the result for each candidate

O

U

T

P

U

T

S

Step 1: learning objectives are clarified

■ Manual Metal Arc Welding

□ Welding skills typically found in industry and associated underpinning knowledge to a level that will enable them to complete welded joints in simple welding positions.

■ Learning outcome 1

□ The learner will:

▪ 1. produce beads on a plate in the PA flat position

■ Assessment criteria 1

□ The learner can:

▪ 1.1 use manual metal arc welding techniques safely to produce beads on a plate in simple welding positions

▪ 1.2 check joints are aligned and welds are sound and of uniform appearance

▪ 1.3 identify defects in the weld using visual checks

■ Etc.

Step 2: Multiple performances are elicited from each student (via assessment tasks) to provide evidence of proficiency

Credit: TechShop

Multiple welds Answers to multiple questions

Credit: Alberto G.

Step 3: Each task completed by the student is evaluated in terms of what it implies about their proficiency

Good products? Safe processes?

Correct answers?

Credit: garycycles7 Credit: genebrooks

Credit: Oliver Tacke

Step 4: The IQA moderates a sample from across all assessment tasks, assessors and students

• A range of evidence from assessment tasks is compared to ensure that assessment decisions are valid

Step 5: The measurement result is interpreted by those for whom it has been provided

Credit: Alan Cleaver

Level 1 Award in

Introductory

Manual Metal Arc

(MMA) Welding

… that’s a good

foundation for our

apprenticeship

A validity argument, expressed fairly informally

1. IF the learning objective is properly specified, &

2. IF evidence of achievement for that outcome is properly elicited, &

3. IF the elicited evidence is properly evaluated, &

4. IF the assessment is properly moderated, &

5. IF the results are properly interpreted,

THEN measurement interpretations will be both accurate and useful.

2. THREATS TO VALIDITY

All bucket photos credited to: Paul and Eliza

■ Manual Metal Arc Welding

□ Welding skills typically found in industry and associated underpinning knowledge to a level that will enable them to complete welded joints in simple welding positions.

■ Learning outcome 1

□ The learner will:

▪ 1. produce beads on plate in the PA flat position

■ Assessment criteria 1

□ The learner can:

▪ 1.1 use manual metal arc welding techniques safely to produce beads on a plate in simple welding positions

▪ 1.2 check joints are aligned and welds are sound and of uniform appearance

▪ 1.3 identify defects in the weld using visual checks

■ Etc.

Step 1: We need to specify the proficiency that needs to be measured

• But how faithfully does our proficiency specification represent the ‘target proficiency’ (the skillset that reallyneeds to be acquired)?

Is this LO up-to-date?

Have we omitted any

critical LOs?

Two examples of many

potential validity threats

Step 2: We need to generate and capture evidence

• But how faithfully does the evidence in each student’s portfolio represent their ‘true proficiency’?

Credit: TechShop

Credit: Alberto G.

Do students

interpret our questions as

we intend them to be

interpreted?

Is our simulated task

sufficiently authentic to test

real-world proficiency

(as opposed to

test-world proficiency)?

Two examples of many

potential validity threats

Step 3: We need to assess the evidence

• But how faithfully does the assessment feedback in each student’s portfolio represent the ‘true quality’ of their task performances?

Credit: garycycles7

Credit: Oliver Tacke

How susceptible are

assessor judgements to

proficiency-irrelevant

factors?

Does the m/c scoring ‘key’

identify the correct answer

for all questions?

Two examples of many

potential validity threats

Credit: genebrooks

Step 4: We need to moderate (IQA) the assessments

• How faithfully does each student’s overall result represent the ‘true significance’ of their evaluation profile?

Is it justifiable to award a

pass (or higher) when

certain ACs have not

been achieved?

Is it justifiable to award

grades purely on the basis

of written components?

LO1

AC1: P, AC2: P, AC3: P, AC4: F

LO2

AC1: P, AC2: P, AC3: P

LO3

AC1: P, AC2: P, AC3: P, AC4: P

LO4 and LO5

71/80 = DISTINCTION

OVERALL RESULT

DISTINCTION

Two examples of many

potential validity threats

Step 5: They need to interpret our results

• But how faithfully does each interpretation of a student’s result represent the ‘true meaning’ of their result?

Is the qualification

title sufficiently clear?

Has sufficient information

been provided to users on

how to interpret grades?

Credit: Alan Cleaver

Two examples of many

potential validity threats

And across all five steps Did you establish the credentials (expertise/integrity)

teachers, assessors and IQAs before appointing them?

Have those staff been given

the right training, guidance and supervision?

Have safeguards been put in place to prevent

human error and deliberate malpractice?

Have you satisfactorily ensured the

accuracy and security of the data

that you record and store?

And so on…

CLARIFICATION

STEP

The proficiency

specification

ELICITATION

STEP

A set of task

performances for

each candidate

EVALUATION

STEP

A set of

evaluations for

each candidate

COMBINATION

STEP

An overall result

for each candidate

INTERPRETATION

STEP

An interpretation

of the result for

each candidate

TARGET PROFICIENCY

Candidate

Assessor

Aggregator

Qual. User

Qual. Designer

Four examples of many

potential validity threats

CLARIFICATION

STEP

The proficiency

specification

ELICITATION

STEP

A set of task

performances for

each candidate

EVALUATION

STEP

A set of

evaluations for

each candidate

COMBINATION

STEP

An overall result

for each candidate

INTERPRETATION

STEP

An interpretation

of the result for

each candidate

TARGET PROFICIENCY

Candidate

Assessor

Aggregator

Qual. User

Qual. Designer

You probably are going

to lose a little trickle of

validity here…

CLARIFICATION

STEP

The proficiency

specification

ELICITATION

STEP

A set of task

performances for

each candidate

EVALUATION

STEP

A set of

evaluations for

each candidate

COMBINATION

STEP

An overall result

for each candidate

INTERPRETATION

STEP

An interpretation

of the result for

each candidate

TARGET PROFICIENCY

Candidate

Assessor

Aggregator

Qual. User

Qual. Designer … and

here…

CLARIFICATION

STEP

The proficiency

specification

ELICITATION

STEP

A set of task

performances for

each candidate

EVALUATION

STEP

A set of

evaluations for

each candidate

COMBINATION

STEP

An overall result

for each candidate

INTERPRETATION

STEP

An interpretation

of the result for

each candidate

TARGET PROFICIENCY

Candidate

Assessor

Aggregator

Qual. User

Qual. Designer

… and

so on!

CLARIFICATION

STEP

The proficiency

specification

ELICITATION

STEP

A set of task

performances for

each candidate

EVALUATION

STEP

A set of

evaluations for

each candidate

COMBINATION

STEP

An overall result

for each candidate

INTERPRETATION

STEP

An interpretation

of the result for

each candidate

TARGET PROFICIENCY

Candidate

Assessor

Aggregator

Qual. User

Qual. Designer

But you can’t afford to

lose a big load of

validity here…

CLARIFICATION

STEP

The proficiency

specification

ELICITATION

STEP

A set of task

performances for

each candidate

EVALUATION

STEP

A set of

evaluations for

each candidate

COMBINATION

STEP

An overall result

for each candidate

INTERPRETATION

STEP

An interpretation

of the result for

each candidate

TARGET PROFICIENCY

Candidate

Assessor

Aggregator

Qual. User

Qual. Designer

… or

here!

CLARIFICATION

STEP

The proficiency

specification

ELICITATION

STEP

A set of task

performances for

each candidate

EVALUATION

STEP

A set of

evaluations for

each candidate

COMBINATION

STEP

An overall result

for each candidate

INTERPRETATION

STEP

An interpretation

of the result for

each candidate

TARGET PROFICIENCY

Candidate

Assessor

Aggregator

Qual. User

Qual. Designer

And you can’t afford to

lose a constant stream

of validity throughout…

This is the Assessment and Quality Assurance challenge!