View
61
Download
4
Category
Preview:
Citation preview
1
ACHIEVING POSITIVE WASHBACK WITH COMMUNICATIVE LANGUAGE TESTS
Dr Roger HawkeyVisiting Professor, Centre for Research in English Learning and AssessmentUniversity of Bedfordshire
3rd
Annual IELTS ConferenceBeijing, 25 November 2010
2
Which means looking at:what people mean by washback and
related conceptshow these relate to communicative
language testsSome evidence from related researchhow positive outcomes might be
achieved from the relationships
3
and my argument will be ….that test washback is a complex matter but needs pursuing because it’s key to test validitythat valid communicative language tests facilitate, but cannot ensure positive washback ….
4
Washback (Backwash)?Washback refers to an exam’s influences on teaching, teachers, learning, curriculum and materials (eg Alderson and Wall 1993, Hamp-Lyons 2000, Hawkey, 2006, 9), Shohamy, Donitsa-Schmidt and Ferman, 1996).Hughes (e.g. 2003) and Green (2007) call the same phenomenon ‘backwash’
5
Samuel Messick, test validity guru, refers to washback as ‘. . . the extent to which the introduction and use of a test influences language teachers and learners to do things they would not otherwise do that promote or inhibit language learning’ (1996: 241)
6
ImpactImpact is concerned with wider influences, broader social contexts of tests; washback with micro contexts of the classroom and the school (Hawkey 2006 and Hamp-Lyons 2000). Impact generally agreed to cover ‘the total effect of a test on the educational process and on the wider community’(McNamara 2000: 133).
7
Washback : Impact overlapThe impact : washback distinction is useful – but does not mean the two are distinct (Saville 2009)Complex relationships between individuals, their institutions and society more generally of crucial importance in understanding how impact works
8
i.e. the stakeholders…Impact at macro-level: positive
developments for all stakeholders affected by the introduction of an assessment: learners, teachers, school management, and receiving institutions and organisations, education authorities governments…..
9
…
all the stakeholders in all their roles
Context
Stakeholders in the Testing Community
Government agencies
Professional bodies
Learners
Teachers, Heads
School owners
Test writers
Consultants
Examiners
Test centre administrators
Materials writers
Publisher
inter alia
Learners
Parents/carers
Teachers, Heads
School owners
Receiving institutions
Government agencies
Professional bodies
Employers
Academic researchers
inter alia)
Input to test design Context of test use -
provided by stakeholders where decisions are made by stakeholders using test scores
Cambridge ESOL
Test construct Test format Test conditions Test assessment criteria
Test scores
Testing system
10
Backwash, Washback, Impact – and ‘consequential validity’?
The washback
of the test on the learning and teaching that precedes it, as well as its impact on institutions and society more broadly’
(Taylor (Ed) forthcoming) are key
aspects of the consequential validity of the test (Weir 2005)
An assessment is valid if:it has the intended positive impactit does not have unintended negative impacts
11
So, test washback and impact are key aspects of test validity, validation
Impact at micro-level (washback): positive changes in the classroom to content, teaching, and learning outcomes. Impact at macro-level: on stakeholders affected by the introduction of an assessment: learners, teachers, school management, education authorities and government.
12
Impact seen as separate category
?
Cambridge ESOL VRIP = o
Validity
o
+ Reliability o
+ Impact
o
+ Practicality (now termed VRIP Q = VRIP + Quality Assurance)
13
…or an element in an integrated model of a test’s ‘usefulness, ‘fitness for purpose’?
‘Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.’ (Messick 1989)Test validity seen as including the useand consequences of test results.
14
So, how about impact by design?
‘Both the construct validity
of our score- based inferences and the impact, or
consequences, of test use need to be considered from the very beginning of test design, with the test developer and test users working together to prioritise the relative importance of these qualities’
emphasises Bachman
(2005).
15
So, it’s important to study impact
…
but by no means straightforward…Alderson agrees ‘test consequences are
important and may relate to validity issues …’
However, the ‘myriad factors’
impacting on a test for example, for example teacher’s linguistic ability, training, motivation, course hours, class size, extra lessons and so on’
Alderson and
Banerjee (1996)
16
But complex as they are, impact studies are needed, or how do we know what the impacts, part of the validity of a test, are and how they compare with what they should be?
If we don’t try to find out about the likely impacts, how can we plan for achieving positive washback with communicative language tests?
17
New / Revised test
Washback of test on
teaching, learning, materials
Washback of test on learner and candidate performance
Impact on receiving
institutions, employers,
policy makers, testers
Reactions to washback/
impact study, decisions on
programme or testImpact of test
on other stakeholders, institutions
WHERE DOES IMPACT STUDY FIT INTO THE PROCESS OF
EXAMINATION DEVELOPMENT?
18
Two-way washback?
Washback is not necessarily unidirectional, i.e. from exam to textbook and teaching rather than bi-directional, i.e. also from textbook and teaching to exam (Wall 2005, and Hawkey 2009)
Changes in approaches to teaching and assessment also have washback on exams.
19
Some washback / impact implications from an IELTS study?
A smallish study (see Hawkey 2006), following up on previous joint initiatives between Cambridge ESOL and Lancaster University
But with some interesting food for thought on the IELTS washback and impact fronts….
20
Impact Study Participants?572 IELTS candidates, pre-
and post-test
(largest subgroup from East Asia and Pacific)83 teachers completing the teacher
questionnaire43 teachers completing the instrument for
the evaluation of textbooks120 students, 21 teachers and 15 receiving
institution administrators in face-to-face interviews, focus groups
21
KEY QUESTION : IS IELTS A FAIR TESTOF PROFICIENCY?
Yes 72% No
28%If “No”, why not?
1 Opposition to all tests2 Pressure, especially time3 Topics4 Rating of writing and speaking5 No grammar test
22
CANDIDATES’ (post-IELTS) TOP LIKES (n=132)...
1 ‘VALIDITY’?
fair (17), 4-skills/ comprehensiveness (15) recognition (7),
language and study skills (2)
41
2 SPEAKING 17
3 STRUCTURE, ORGANISTION, FORMAT 16
4 WRITING 15
5 INCENTIVE, CHALLENGE, INTEREST, VARIETY 14
6 LISTENING 13
23
…AND DISLIKES? (n=138)
1 TIME PRESSURE 502 READING 413 LISTENING 184 WRITING 165 ‘COMPLICATED QUESTIONS’ 96 LANGUAGE DIFFICULTY 87 SPEAKING 7
24
WHAT IS MOST DIFFICULT ABOUT IELTS?
Most difficult IELTS module (%)
Reading 49 45Writing 24 26Listening 18 20Speaking 9 9
Learners Teachers
25
IELTS SKILLS: STUDENTS’
PERCEIVED DIFFICULTY AND PREP COURSE TIMINGS:
49
28
24
17
18
20
9
19 7 8
0 20 40 60 80 100 120
Difficulty
Time
Reading Writing Listening Speaking Grammar Vocab
26
DID YOUR IELTS PREP COURSE PROVIDE YOU WITH THE LANGUAGE KNOWLEDGE
AND SKILLS YOU NEED?
83%
17%
Yes
No
27
STUDENTS WORRY AT TAKING IELTS
Very much 41%
Quite a lot 31%
A bit 19%
V little 9%
28
AND TO THE PREP COURSE TEACHERS?Does the IELTS test cause stress
for your
students?
Yes
44 No
27
Don't
know
11 No
answer
1
Does the IELTS test provide motivation
for your students?
Yes
70No
8
Don't
know
4No
answer
1
29
Conditions that need to be in place for positive washback
include:making the test tasks direct and criterion-referencedoptimising overlap between test and target language domain demandsemphasising the importance, demandingness, but attainability of the testensuring the test-takers and teachers are familiar with the test providing optimal support for teachers.
But the effect of the many intervening variables always remains….
30
Food for thought ….
that test washback is a complex matter but needs pursuing because it’s key to test validitythat valid communicative language tests facilitate, but cannot ensure positive washback ….
31
Thank you.
Roger Hawkey
rogerhawkey@btinternet.com
Hawkey.r@cambridgeesol.org
32
ReferencesAlderson J C & Wall D (1993) Does washback exist? Applied Linguistics 14(2):
115-129Alderson J C and Banerjee J (1996) How might impact study instruments be
validated? Paper commissioned by the University of Cambridge Local Examinations Syndicate (UCLES) as part of the IELTS Impact Study
Bachman L (1990) Fundamental Considerations in Language Testing, Oxford: Oxford University Press.
Bachman (2005) Building and supporting a case for test use, Language Assessment Quarterly 2, 1, 1-34
Bachman L and Palmer A (1996) Language Testing in Practice, Oxford: Oxford University Press.
Green A (2007) IELTS Washback in Context: preparation for academic writing in higher education, Cambridge: Cambridge University Press
Hamp-Lyons, L (2000) Social, professional and individual responsibility in language testing, System 28 (4), 579-591.
Hawkey, R (2006) Impact theory and practice: studies of the IELTS test and Progetto Lingue 2000, Cambridge: Cambridge University Press
Hawkey R (forthcoming) Consequential Validity, in Taylor, L (Ed)Hughes A (2003) Testing for language teachers, Cambridge: Cambridge
University Press.
33
McNamara, T. 2000. Language Testing. Oxford: Oxford University Press.Messick, S (1989) Validity, in Linn, R L (Ed.) Educational Measurement (3rd ed.), New
York: Macmillan, 13-103.Messick, S (1994) The interplay of evidence and consequences in the validation of
performance assessments, Educational Researcher 23 (2), 13-23. Messick S (1996) Validity and washback in language testing, Language Testing
November 1996 vol. 13 no. 3 241-256Saville N (2009) Developing a model for investigating the impact of language assessment
within educational contexts by a public examination provider, unpublished PhD thesis. Saville N., & Hawkey, R (2004). The IELTS Impact Study: Investigating Washback on
Teaching Materials. In L. Cheng & Y Watanabe (Eds.) 2004: 97-112. Shohamy, E, 1999 Language Testing: Impact in B. Spolsky, ed., Concise Encyclopaedia of Educational Linguistics ((oxford 1999) 711-714
Shohamy E (2001) The Power of Tests: A critical perspective on the uses of language tests, Harlow: Pearson Education.
Shohamy, E. Donitsa-Schmidt S & Ferman I.(1996). Test impact revisited: Washback effect over time. Language Testing, 13 (3) 298-317
Taylor L Ed (forthcoming) Examining Speaking, Cambridge: Cambridge University Press.Wall D (2005) The Impact of a High-Stakes Examination on Classroom Teaching: A Case
Study Using Insights from Testing and Innovation Theory Studies in Language Testing 22 UCLES Cambridge: Cambridge University Press.
Weir C (2005) Language Testing and Validity Evidence: Oxford:. Palgrave.
Recommended