Robert Kirkpatrick

Shinawatra International University, Thailand

Editor@languagetestingasia.com

IELTS Writing Test: Reliability and Raters

IELTS writing moduleWhat this presentation covers:

• Validity and reliability of writing tests

• Difficulties in scoring writing tasks

• How IELTS organization attempts

– objective and reliable rating

• What raters (examiners) are looking for

University academic life

1Carson, 2001; Hale et al., 1996; Eblen, 1983; Moore & Morton, 1999; Horowitz, 1986; Cansecon & Byrd, 1989

• Requires the writing of essays, reports and theses 1

• University administrators may require writing samples

For native speakers

– „personal‟ narratives, (also used as indicators of character and

motivation)

For non-native speakers

– a score from a large scale proficiency test (IELTS, TOEFL)

Validity of a test

• A test is valid if it “reflects the psychological reality of

behavior in the area being tested.” 2

• Great interest among researchers of ways to improve

validation methods and concerns about the test‟s validity

and fairness. 3

2 Hamp-Lyons, 1990 3Ewell, 2004

Validity of Writing Tests

• Multiple choice grammar/vocabulary tests [Indirect]

May have

– Predictive validity

– Concurrent validity (compares the results with a validated test)

• Writing essays [Direct]

(Extra costs and problematic rating issues)

– Face validity

– Content validity

– Construct validity

Direct tests and validity

• Even direct writing tests are not perfect 4

– the time restrictions (2 essays in one hour)

– snapshot approach

• Students writing reports or theses 5

– may take days or weeks

– students have access to books and references

4 Hamp-Lyons and Kroll, 1996 5 Kirkpatrick, 1997

Direct tests and validity

• Researcher noted the problems of independent tasks 6

• Can be problematic for students

– Have little background knowledge on a set topic

– Some „luck out‟ with a previously researched topic.

• Areas where IELTS is known to ask

– environment, internet and so on.

4 Hamp-Lyons and Kroll, 1996 5 Kirkpatrick, 1997 6 Gebril & Plakans, 2009; Plakans, 2007; Gebril, 2006; Weigle, 2002, 2004; Cho, 2003;

Cumming, Kantor, Powers, Santos, & Taylor, 2000; Leki & Carson, 1997

Reliability and validity

• IELTS Task 1

– Typically a graph or process

– IELTS prompt supplies all necessary information

• Task 2 is independent 7

• Many factors cause score variance in writing assessment

• tasks, topics, genre, raters, test conditions, time limits,

writing mode (e.g. online or handwritten) or rating

scales.

• Predictable tasks lead to formulaic writing

7 Schoonen, 2005

Reliability and validity

• Rater reliability is a major issue

• Tension between validity and reliability is unresolved

• For direct writing tests, raters are central to the entire

assessment.

• If raters cannot supply scores which are appropriate and

consistent, then the whole enterprise is defunct.

8 Shaw and Weir, 2007

Trained raters

• No matter how high the construct validity if reliability of

rating is questionable then one is losing.

• The process of training 9

– Familiarization activities, discussion, and practice rating

• In comparisons between untrained raters' ratings,

trained raters were more reliable than the untrained raters 11

9 Lane & Stone (2006) 10 Shohamy, Gordon, and Kraemer, 1992 11 Weigle, 1998

Raters training experience

• Full training about 5 years ago

• Four-day intensive course, Chiang Mai

– for speaking and writing

• Discussions about marking

• some raters

– not fully understanding

– getting lost in the details of each section of the band scores.

Raters training experience

• Given full registration but not confident

• Often referred to the self access material

– Samples of writing for benchmark

• The first day raters scripts were taken and checked.

• Plus regular evaluations where random samples are

checked

• Every two years have to endure the training again

• Jagged profile where if a candidate has especially high

or low values on the writing section (compared with

other sections) another examiner checks the script.

Raters training experience

• Now I have rated hundreds and hundreds of scripts

– My confidence has grown

– Rarely see more than a 0.5 band score difference (between my score

and a senior rater)

• 0.5 variation is considered acceptable by IELTS

• There were differences in rater abilities 11

– Some raters get more experienced

– but develop bad habits

• Professional, educational backgrounds are important

• Only degreed, experienced EFL teachers can apply to be IELTS

raters.

9 Wolfe, Kao, and Ranne, 1998; Huot, 1993; Barkaoui, 2010

The band scores

This last section explains the IELTS band scores

– Looks at what good academic writing is

– And what students should be looking to do

Band scores for writing

• From late 2005 IELTS changed its policy (a little)

• Previously the rating scales were hidden but now

information is available to the general public

– how band scores are reached

– what raters are looking at.

• Some aspects remain confidential

Band scores for writing

• The astute student and IELTS tutor now has enough

material to understand what IELTS (and hence what

raters) expect at different band levels.

• They have four sections

1. Task Response

2. Coherence and Cohesion

3. Lexical Resource

4. Grammar

What I look for

• Focused on topic

• Evidence of English beyond scripted sentences

e.g. “this is a topic that is hotly debated at this time and I will talk

about both sides”.

• Complexity of sentence structure.

• Clear overall structure.

Thank You