Ofqual’s reliability of results programme

Dennis Opposs(Office of the Qualifications and Examinations Regulator)

A health check

“As the regulator of qualifications in England I believe that it is essential for all of us to understand better the reliability of assessments in our national systems […] I can, therefore, tell you today that Ofqual will undertake an in-depth programme of work – call it a health check – on the reliability of tests, examinations and teacher assessments, in this country.”Kathleen Tattersall, 16 May, 2008. Ofqual Launch Event. National Motorcycle Museum. Solihull.

Myths of perfection?

“There’s a broad expectation that assessment should be absolutely perfect and accurate, that a mark of 50 is a mark of 50, regardless of who marks, the time at which it is marked and so on. We need to explore whether that sort of expectation is well founded...”Kathleen Tattersall, 14 May, 2008. Interview with Tim Ross of PA.

Educational assessment error

What factors contribute to educational assessment error?

You… if you don’t present sufficient evidence

Instruments… if they’re poorly calibrated, or have design limitations

Your assessors… if they misinterpret the evidence before them

Student misclassification

“… it is likely that the proportion of students awarded a level higher or lower than they should be because of the unreliability of the tests is at least 30% at key stage 2”Wiliam, D. (2001). Level best? London: ATL.

“Professors Black, Gardner and Wiliam argued […] that up to 30% of candidates in any public examination in the UK will receive the wrong level or grade”House of Commons Children, Schools and Families Committee. (2008a). Testing and Assessment. Third Report of Session 2007–08. Volume I. HC 169-I. London: TSO.

Programme objectives

To generate evidence of reliability of results from a number of major national tests, examinations and qualifications

To stimulate, capture and synthesise technical debate on the interpretation of reliability evidence generated from reliability studies

To investigate how results and the associated errors are reported and communicated

To explore public understanding of, and attitudes towards, assessment inconsistency

Programme objectives

To stimulate national debate on the significance of the reliability evidence generated by this programme and other studies

To help improve public understanding of the concept of reliability

To develop Ofqual policy on reliability of results

Our programme of work

Strand 1

Generating evidence on reliability

Strand 2

Interpreting and communicating evidence on reliability

Strand 3

Exploring public understanding of reliability and developing Ofqual policy on reliability

Strand 1 – Generating evidence

Synthesising pre-existing evidence

Literature reviews

Generating new evidence

Monitoring existing practices

Experimental studies

Specifications to support experimental studies

Literature reviews: sources of unreliability; procedures used to produce reliability measures; how to report results and associated errors; how to interpret and evaluate reliability evidence

Partial estimates of reliability: estimating reliability in relative isolation

Overall estimates of reliability: all major factors considered

Our Technical Advisory Group

Strand 2 – Interpreting and communicating evidence

How do we interpret our findings?

How do we communicate our findings?

Our communication challenge

“results on a six or seven point grading scale are accurate to about one grade either side of that awarded.”Schools Council. (1980). Focus on examinations. Pamphlet 5. London: Schools Council.

Strand 3 – Developing policy

Exploring public understanding of, and attitudes towards, assessment error

Stimulating national debate on the significance of the reliability evidence generated by the programme

Developing Ofqual’s policy on reliability

Some questions for discussion

What are your experiences of exam error?

Would you be surprised if a substantial proportion of students were misclassified?

How much error is too much error?

Is it right to apportion blame when students are misclassified (if so, to whom and why)?

What about the different causes of error?

Does misclassification undermine the effective use of results (why, to what extent)?