Reliability in Testing

Is the test or assessment tool consistent and dependable?

• Student-related reliability• Rater reliability• Test administration reliability• Test reliability (Brown, 2004, 20-22)

Student-related reliability

• Give students consistent materials/time for test preparation

• Plenty of time for studying• Consistent test time/conditions (always on

Wednesdays)

Rater reliability

• For subjective scoring in high stakes tests:– Have more than one rater– Use rubric and have traning/norming sessions– Have outside periodic oversight– Keep tests anonymous

• For low stakes tests (quizzes):– Use rubric– Read through all tests before rating

Test administration reliability

• Consistent rules for all classes/teachers:– Dictionary use– Notes/books– Strict time limits

How to measure test reliability?

• One way is through test-retest method:– The same group of students takes the same test

twice. (drawback- motivation and washback)

• What else could we do to test whether the same students (or very similar students) score the same on the same test twice?

Split half method

• Have the test split into two halves, equal in tasks and difficulty.

• Then measure the scores for each half• This is a good way to look at reliability, but

it requires that the split tests are really equal

Split half method – Your task

• What can you tell about the reliability for the following split sections on my test?

Joanna 88 95Jenny 90 87Asuka 97 95David 92 88Annick 74 78Mercedes 84 89

Checklist for Reliability (1)

• Have many independent items• Delete items that do not discriminate

between weaker/stronger students• Limit choices – restrict student responses• Keep items clear and unambiguous• Provide clear instructions and examples• Keep type large enough and cleanly copied

Checklist for Reliability (2)• Provide practice with testing format• Keep administration uniform• Have clear answers for all items (as possible)• Provide detailed answer key• Train raters as necessary• Determine acceptable responses before scoring

starts• Keep test-takers anonymousJohn Bunting (2004) presentation in the Course: Testing, Assessment and Teaching- A program for

EFL Teachers at UABC. Facultad de Idiomas, UABC