1

ACHIEVING POSITIVE WASHBACK WITH COMMUNICATIVE LANGUAGE TESTS

Dr Roger HawkeyVisiting Professor, Centre for Research in English Learning and AssessmentUniversity of Bedfordshire

3rd

Annual IELTS ConferenceBeijing, 25 November 2010

2

Which means looking at:what people mean by washback and

related conceptshow these relate to communicative

language testsSome evidence from related researchhow positive outcomes might be

achieved from the relationships

3

and my argument will be ….that test washback is a complex matter but needs pursuing because it’s key to test validitythat valid communicative language tests facilitate, but cannot ensure positive washback ….

4

Washback (Backwash)?Washback refers to an exam’s influences on teaching, teachers, learning, curriculum and materials (eg Alderson and Wall 1993, Hamp-Lyons 2000, Hawkey, 2006, 9), Shohamy, Donitsa-Schmidt and Ferman, 1996).Hughes (e.g. 2003) and Green (2007) call the same phenomenon ‘backwash’

5

Samuel Messick, test validity guru, refers to washback as ‘. . . the extent to which the introduction and use of a test influences language teachers and learners to do things they would not otherwise do that promote or inhibit language learning’ (1996: 241)

6

ImpactImpact is concerned with wider influences, broader social contexts of tests; washback with micro contexts of the classroom and the school (Hawkey 2006 and Hamp-Lyons 2000). Impact generally agreed to cover ‘the total effect of a test on the educational process and on the wider community’(McNamara 2000: 133).

7

Washback : Impact overlapThe impact : washback distinction is useful – but does not mean the two are distinct (Saville 2009)Complex relationships between individuals, their institutions and society more generally of crucial importance in understanding how impact works

8

i.e. the stakeholders…Impact at macro-level: positive

developments for all stakeholders affected by the introduction of an assessment: learners, teachers, school management, and receiving institutions and organisations, education authorities governments…..

9

…

all the stakeholders in all their roles

Context

Stakeholders in the Testing Community

Government agencies

Professional bodies

Learners

Teachers, Heads

School owners

Test writers

Consultants

Examiners

Test centre administrators

Materials writers

Publisher

inter alia

Learners

Parents/carers

Teachers, Heads

School owners

Receiving institutions

Government agencies

Professional bodies

Employers

Academic researchers

inter alia)

Input to test design Context of test use -

provided by stakeholders where decisions are made by stakeholders using test scores

Cambridge ESOL

Test construct Test format Test conditions Test assessment criteria

Test scores

Testing system

10

Backwash, Washback, Impact – and ‘consequential validity’?

The washback

of the test on the learning and teaching that precedes it, as well as its impact on institutions and society more broadly’

(Taylor (Ed) forthcoming) are key

aspects of the consequential validity of the test (Weir 2005)

An assessment is valid if:it has the intended positive impactit does not have unintended negative impacts

11

So, test washback and impact are key aspects of test validity, validation

Impact at micro-level (washback): positive changes in the classroom to content, teaching, and learning outcomes. Impact at macro-level: on stakeholders affected by the introduction of an assessment: learners, teachers, school management, education authorities and government.

12

Impact seen as separate category

?

Cambridge ESOL VRIP = o

Validity

o

+ Reliability o

+ Impact

o

+ Practicality (now termed VRIP Q = VRIP + Quality Assurance)

13

…or an element in an integrated model of a test’s ‘usefulness, ‘fitness for purpose’?

‘Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.’ (Messick 1989)Test validity seen as including the useand consequences of test results.

14

So, how about impact by design?

‘Both the construct validity

of our score- based inferences and the impact, or

consequences, of test use need to be considered from the very beginning of test design, with the test developer and test users working together to prioritise the relative importance of these qualities’

emphasises Bachman

(2005).

15

So, it’s important to study impact

…

but by no means straightforward…Alderson agrees ‘test consequences are

important and may relate to validity issues …’

However, the ‘myriad factors’

impacting on a test for example, for example teacher’s linguistic ability, training, motivation, course hours, class size, extra lessons and so on’

Alderson and

Banerjee (1996)

16

But complex as they are, impact studies are needed, or how do we know what the impacts, part of the validity of a test, are and how they compare with what they should be?

If we don’t try to find out about the likely impacts, how can we plan for achieving positive washback with communicative language tests?

17

New / Revised test

Washback of test on

teaching, learning, materials

Washback of test on learner and candidate performance

Impact on receiving

institutions, employers,

policy makers, testers

Reactions to washback/

impact study, decisions on

programme or testImpact of test

on other stakeholders, institutions

WHERE DOES IMPACT STUDY FIT INTO THE PROCESS OF

EXAMINATION DEVELOPMENT?

18

Two-way washback?

Washback is not necessarily unidirectional, i.e. from exam to textbook and teaching rather than bi-directional, i.e. also from textbook and teaching to exam (Wall 2005, and Hawkey 2009)

Changes in approaches to teaching and assessment also have washback on exams.

19

Some washback / impact implications from an IELTS study?

A smallish study (see Hawkey 2006), following up on previous joint initiatives between Cambridge ESOL and Lancaster University

But with some interesting food for thought on the IELTS washback and impact fronts….

20

Impact Study Participants?572 IELTS candidates, pre-

and post-test

(largest subgroup from East Asia and Pacific)83 teachers completing the teacher

questionnaire43 teachers completing the instrument for

the evaluation of textbooks120 students, 21 teachers and 15 receiving

institution administrators in face-to-face interviews, focus groups

21

KEY QUESTION : IS IELTS A FAIR TESTOF PROFICIENCY?

Yes 72% No

28%If “No”, why not?

1 Opposition to all tests2 Pressure, especially time3 Topics4 Rating of writing and speaking5 No grammar test

22

CANDIDATES’ (post-IELTS) TOP LIKES (n=132)...

1 ‘VALIDITY’?

fair (17), 4-skills/ comprehensiveness (15) recognition (7),

language and study skills (2)

41

2 SPEAKING 17

3 STRUCTURE, ORGANISTION, FORMAT 16

4 WRITING 15

5 INCENTIVE, CHALLENGE, INTEREST, VARIETY 14

6 LISTENING 13

23

…AND DISLIKES? (n=138)

1 TIME PRESSURE 502 READING 413 LISTENING 184 WRITING 165 ‘COMPLICATED QUESTIONS’ 96 LANGUAGE DIFFICULTY 87 SPEAKING 7

24

WHAT IS MOST DIFFICULT ABOUT IELTS?

Most difficult IELTS module (%)

Reading 49 45Writing 24 26Listening 18 20Speaking 9 9

Learners Teachers

25

IELTS SKILLS: STUDENTS’

PERCEIVED DIFFICULTY AND PREP COURSE TIMINGS:

49

28

24

17

18

20

9

19 7 8

0 20 40 60 80 100 120

Difficulty

Time

Reading Writing Listening Speaking Grammar Vocab

26

DID YOUR IELTS PREP COURSE PROVIDE YOU WITH THE LANGUAGE KNOWLEDGE

AND SKILLS YOU NEED?

83%

17%

Yes

No

27

STUDENTS WORRY AT TAKING IELTS

Very much 41%

Quite a lot 31%

A bit 19%

V little 9%

28

AND TO THE PREP COURSE TEACHERS?Does the IELTS test cause stress

for your

students?

Yes

44 No

27

Don't

know

11 No

answer

1

Does the IELTS test provide motivation

for your students?

Yes

70No

8

Don't

know

4No

answer

1

29

Conditions that need to be in place for positive washback

include:making the test tasks direct and criterion-referencedoptimising overlap between test and target language domain demandsemphasising the importance, demandingness, but attainability of the testensuring the test-takers and teachers are familiar with the test providing optimal support for teachers.

But the effect of the many intervening variables always remains….

30

Food for thought ….

that test washback is a complex matter but needs pursuing because it’s key to test validitythat valid communicative language tests facilitate, but cannot ensure positive washback ….

31

Thank you.

Roger Hawkey

rogerhawkey@btinternet.com

Hawkey.r@cambridgeesol.org

32

ReferencesAlderson J C & Wall D (1993) Does washback exist? Applied Linguistics 14(2):

115-129Alderson J C and Banerjee J (1996) How might impact study instruments be

validated? Paper commissioned by the University of Cambridge Local Examinations Syndicate (UCLES) as part of the IELTS Impact Study

Bachman L (1990) Fundamental Considerations in Language Testing, Oxford: Oxford University Press.

Bachman (2005) Building and supporting a case for test use, Language Assessment Quarterly 2, 1, 1-34

Bachman L and Palmer A (1996) Language Testing in Practice, Oxford: Oxford University Press.

Green A (2007) IELTS Washback in Context: preparation for academic writing in higher education, Cambridge: Cambridge University Press

Hamp-Lyons, L (2000) Social, professional and individual responsibility in language testing, System 28 (4), 579-591.

Hawkey, R (2006) Impact theory and practice: studies of the IELTS test and Progetto Lingue 2000, Cambridge: Cambridge University Press

Hawkey R (forthcoming) Consequential Validity, in Taylor, L (Ed)Hughes A (2003) Testing for language teachers, Cambridge: Cambridge

University Press.

33

McNamara, T. 2000. Language Testing. Oxford: Oxford University Press.Messick, S (1989) Validity, in Linn, R L (Ed.) Educational Measurement (3rd ed.), New

York: Macmillan, 13-103.Messick, S (1994) The interplay of evidence and consequences in the validation of

performance assessments, Educational Researcher 23 (2), 13-23. Messick S (1996) Validity and washback in language testing, Language Testing

November 1996 vol. 13 no. 3 241-256Saville N (2009) Developing a model for investigating the impact of language assessment

within educational contexts by a public examination provider, unpublished PhD thesis. Saville N., & Hawkey, R (2004). The IELTS Impact Study: Investigating Washback on

Teaching Materials. In L. Cheng & Y Watanabe (Eds.) 2004: 97-112. Shohamy, E, 1999 Language Testing: Impact in B. Spolsky, ed., Concise Encyclopaedia of Educational Linguistics ((oxford 1999) 711-714

Shohamy E (2001) The Power of Tests: A critical perspective on the uses of language tests, Harlow: Pearson Education.

Shohamy, E. Donitsa-Schmidt S & Ferman I.(1996). Test impact revisited: Washback effect over time. Language Testing, 13 (3) 298-317

Taylor L Ed (forthcoming) Examining Speaking, Cambridge: Cambridge University Press.Wall D (2005) The Impact of a High-Stakes Examination on Classroom Teaching: A Case

Study Using Insights from Testing and Innovation Theory Studies in Language Testing 22 UCLES Cambridge: Cambridge University Press.

Weir C (2005) Language Testing and Validity Evidence: Oxford:. Palgrave.