Language testing and the use of the common european framework of reference for languages

Language testing and the use of the Common European Framework of Reference for Languages (CEFR)

J Charles Alderson, Department of Linguistics and English

Language, Lancaster University

Universal Principles of Language Testing

Validity

Reliability

Washback

Practicality

Test Principles and Grammatical Tense: The Simple Past?

In many countries in Europe: • Teacher knew best • Having a degree in a language meant you were  an  ‘Expert’

• Experience was all • But 20 years experience may be one year

repeated twenty times and is never checked

Past (?) European tradition • Quality of important examinations not monitored • No obligation to show that exams are fair, unbiased,

reliable, and measure relevant skills • University degree in a foreign language qualifies one

to examine language competence, despite lack of training in language testing

• In many circumstances merely being a native speaker qualifies one to assess language competence.

• Teachers  assess  students’  ability    without  having  been  trained in assessment.

Past (?) European tradition

Teacher-centred Teacher develops the questions Teacher's opinion the only one that counts Teacher-examiners are not standardised Assumption that the teacher-examiner makes

reliable and valid judgements Authority, professionalism, reliability and validity of

teacher rarely questioned Rare for students to fail

Psychometric tradition: Perfect?

Tests externally developed and administered National or regional agencies responsible for

development, following accepted standards Tests centrally constructed, piloted and revised Difficulty levels empirically determined External, trained assessors Empirical equating to known standards or levels

of proficiency

Validity

• My parents think the test looks good. • The test measures what I have been taught. • My teachers tell me that the test is

communicative and authentic. • If I take the X test instead of the Cambridge

FCE, I will get the same result. • I got a good English test result, and I had no

difficulty studying in English at university.

Validity

Note: a test that is not reliable cannot, by definition, be valid

• All tests should be piloted, and the results analysed to see if the test performed as predicted

• A  test’s  items  should  work  well:  they  should  be of suitable difficulty, and good students should get them right, whilst weak students are expected to get them wrong.

Reliability

• If I take the test again tomorrow, will I get the same result?

• If I take a different version of the test, will I get the same result?

• If the test had had different items, would I have got the same result?

• Do all markers agree on the mark I got? • If the same marker marks my test paper again

tomorrow, will I get the same result?

Practicality

• Number of tests to be produced

• Length of test in time

• Cost of test

• Cost of training

• Cost of monitoring

• Difficulty in piloting/ pre-testing

• Time to report results

Washback

• Test can have positive or negative effects

• Test can affect content of teaching

• Test can affect method of teaching

• Test can affect attitudes and motivation

• Test can affect all teachers and students in same way, or individuals differently

• Importance of test will affect washback

WASHBACK

Testing is too important to be left to the teacher

Testing is too important to be left to the tester

Both are needed, to reflect and influence teaching, validly and reliably.

Present Perfect?

Present Tense / Tension: Practice vs. Principles

Teacher-based assessment vs central development Internal vs external assessment Quality control of exams vs. no quality control Piloting or not Test analysis and the role of the expert The existence of test specifications – or not Guidance and training for test developers and

markers – or not

Exam Reform in Europe

(mainly school-leaving exams)

• Slovenia • The Baltic States • Hungary • Russia • Slovakia • Czech Republic • Poland • Germany • Austria

Hungarian English Exams Reform Teacher Support Project

• Project philosophy:

“The  ultimate  goal  of  examination  reform  is  to  encourage, to foster and to bring about change in the way language is taught and learned  in  Hungary.”  

Achievements of English Exam Reform Teacher Support Project

– Trained item writers, including class teachers

– Trained teacher trainers and disseminators

–Developed, refined and published Item Writer Guidelines and Test Specifications

–Developed a sophisticated item production system

Achievements of English Exam Reform Teacher Support Project

• In-service courses for teachers in modern test philosophy and exam preparation – Modern Examinations Teacher Training (60 hrs)

– Assessing Speaking at A2/B1 (30 hrs)

– Assessing Speaking at B2 (30 hrs)

– Assessing Writing at A2/B1 (30 hrs)

– Assessing Writing at B2 (30 hrs)

– Assessing Receptive Skills (30hrs)

Achievements of English Exam Reform Teacher Support Project

–Developed sets of rating scales and trained markers

–Developed Interlocutor Frame for speaking tests and trained interlocutors

– Items / tasks piloted, IRT-calibrated and standard set to CEFR using DIALANG procedures

Achievements of English Exam Reform Teacher Support Project

• Into Europe series: textbook series for test preparation:

–many calibrated tasks

–explanations of rationale for task design

–explanations of correct answers

–CDs of listening tasks

–DVDs of speaking performances

Into Europe

Reading + Use of English

Writing Handbook

Listening + CDs

Speaking Handbook + DVD All downloadable for free from

http://www.lancs.ac.uk/fass/projects/examreform

22

Item Writer Training

Test specification

Text mapping

Task development

Peer review

Standard setting

Statistical Analysis

Revision - Rejection

Statistical Analysis

Central Correction

Trial 1

Trial 2

Live administration

Central Correction

Banking - Rejection

Expert review

Marking support

Post test analysis

Testing cycle

Good tests and assessment, following professional practice, cost

money and time

But

Bad tests and assessment,

ignoring professional practice, waste money, time and LIVES

Use and abuse of

the Common European Framework of Reference for Languages: Learning, teaching and

assessment (CEFR)

Hands up!

• Who owns a copy of the CEFR – the Blue Book?

• Who has read it?

• Who is familiar with its contents?

• Who has already heard of the CEFR?

Outline

• Background

• Uses in various contexts

• Advantages

• Limitations

• Misuse

• Improvement and development

Background

• 1970s work encouraged by the Council of Europe

• Notional-functional syllabus (Wilkins, Morrow) – Threshold – Waystage – Vantage – Learning target specifications

• 1996 • 2001

CEFR: comprehensive, non-prescriptive, reflection tool

Common reference points + Common metalanguage Relevant to objectives + progress + outcomes Descriptive scheme / chapters + Common reference

levels / scales Tool for reflection

CEFR: comprehensive, non-prescriptive, reflection tool • Guides for users

• Compendium of case studies • CEFR Tool kit

• CDs for Reading and Listening • DVDs for Speaking • Dutch Grid for Reading and Listening • Grids for Writing and Speaking

• Manual for relating exams to the CEFR 2003, 2009 (standard-setting)

Descriptive  scheme:  ‘action-oriented’

Users as social agents: «members of society who have tasks to accomplish in a given set of circumstances in a specific environment and within a particular field of action»

General competences (knowledge, skills, existential competence; ability to learn)

Communicative language competences (linguistic, pragmatic, sociolinguistic and sociocultural)

Descriptive  scheme:  ‘action-oriented’

• Dimensions of communicative language competence: – general linguistic range, vocabulary range,

vocabulary control, grammatical accuracy, phonological control, sociolinguistic appropriateness, flexibility, turn-taking, thematic development, coherence and cohesion, spoken fluency, propositional precision

Uses in various contexts

• Case studies 2002 and 2004

• Intergovernmental Language Policy Forum, 2007: – “The  clear  success  of  the  CEFR  has  significantly  

changed the context in which language teaching and assessment of language learning outcomes now  take  place  in  Europe”

• Martyniuk and Noijons Survey, 2006

Uses in various contexts

• The usefulness of the CEFR rated at 2,44 on a 0-3 scale

• The CEFR most useful in the domains of testing /assessment /certification (2,70 on a 0-3 scale) and curriculum/ syllabus development (2,66 on a 0-3 scale)

• Institutionally, the CEFR most useful for examination providers (2,88 on a 0-3 scale)

Uses in various contexts

• Curriculum development – Varying impact

• Teacher education/training – Wide spectrum of use – Useful for defining proficiency of teachers

• Testing and assessment – Support for a common reference – CEFR-based examinations attempted in most

countries

EALTA’s  Guidelines  for  Good  Practice

1. What evidence is there of the quality of the process followed to link tests and examinations to the Common European Framework?

2. Have the procedures recommended in the Manual and the Reference Supplement been applied appropriately?

3. Is there a publicly available report on the linking process?

Example use of CEFR: DIALANG

A European System

for

On-line

Diagnostic Language Assessment

What is DIALANG?

• Computer-based diagnostic language testing system

• 14 European languages

• Delivers tests across the Internet

• Supports language learners

• Institutional or private use, free of charge

• Still widely used throughout Europe and beyond, 8 years after launch

COUNCIL OF EUROPE

• DIALANG is an application of the Common European Framework of reference

• DIALANG uses – Common European Framework – scales – self-assessment statements (modified)

• DIALANG provides some evidence of their validity

PURPOSE

• to provide language users and learners with diagnostic information about their strengths and weaknesses and to help them to find ways of improving their proficiency

INNOVATIVE ASPECTS

• first large-scale system for diagnosis / feedback rather than certification

• on-line, Internet-delivered, universally available, not restricted to a particular place or time

• first implementation of CEFR in tests • first attempt at standard-setting – empirically

relating test items and sections to the CEFR

Vocabulary Size

Placement Test

reading writing listening structures vocabulary

Client enters D I A L A N G

Selection of section:

1 2 3

ASSESSMENT PROCEDURE

Self- assess- ment

Respond- ing to tasks

F e e d b a c k

Selection EXIT

Another section/ language

Goodbye!

4 5 6 7

ASSESSMENT PROCEDURE

SECTIONS

• Reading Comprehension (CEFR) • Listening Comprehension (CEFR) • Writing (CEFR) • Structures • Vocabulary • no overall section (nor grade & feedback) • from beginners to advanced

LANGUAGES

• Danish • Dutch • English • Finnish • French • German • Greek

• Icelandic • Irish • Italian • Norwegian • Portuguese • Spanish • Swedish

Feedback

• VSPT – score band and description

• results (and self-assessment) – CEFR scales and report on self assessment

• explanatory feedback – Why self-assessment may not match test result

• advisory feedback – What you can do and how to progress, based on CEFR

• item review

Example use of CEFR: Standardisierte Reifeprüfung

The current Austrian Matura: – Only one examiner: the class teacher – Teachers set tasks for their own students – Teachers mark the essays with whatever criteria

they wish – No central training, no central monitoring – No piloting – No post-test analysis

The Reform • Began in 2007, obligatory use by law in 2014/15 • Parallel reforms, coordinated by University of

Innsbruck, in English, French, Spanish, Italian, Latin and Greek.

• First foreign language (English) aims at CEFR B2 in Listening, Reading and Language Use (The Written Examination)

• Second foreign languages (French, Italian, Spanish) 6-year and 4-year courses, targeted next (for 6-year courses, B2 except for Listening and Writing = B1. For 4-year courses, target is B1).

The Reform

• Rolling reform, first with 59 schools in 2008, gradually spreading as schools or teachers volunteer for the new standardised Written Exam tasks.

• Spring 2011, 300+ gymnasia volunteered for tests in Reading, Listening and Language in Use in English, French, Italian or Spanish

• Standardised Written Exam obligatory for all gymnasia in 2014 and for all vocational schools in 2015

• See http://uibk.ac.at/srp/

Advantages of the CEFR • European: not American, Australian or British

• Relevant to much more than testing and assessment

• Widely accepted

• Levels frequently cited: A common currency

Advantages of the CEFR • The CEFR claims to be comprehensive; • “...it should attempt to specify as full a range

of language knowledge, skills and use as possible…and all users should be able to describe their objectives, etc., by reference to it”. (Council of Europe, 2001: 7).

Advantages of the CEFR • Research-based:  teachers’  perceptions  of  

levels and progression, Rasch-scaled

• Descriptive Scheme and Illustrative Scales

• Intended to enhance transparency in language education, mutual understanding and thus to encourage mobility

Advantages of the CEFR • Point of reference, not an instrument of

coercion, nor for accountability

• Nevertheless, a force for change and innovation, especially in testing and assessment

• e.g. European Language Portfolio, DIALANG, school-leaving exam reforms

Limitations of the CEFR

• Not enough information for test development – DIALANG experience

• Lack of specificity as to how language proficiency develops

• No reference to specific languages - but see reference level descriptions: www.coe.int/t/dg4/linguistic/DNR_EN.asp

Limitations

• Limited empirical research to underpin • Based  on  teachers’  opinions  /  perceptions  

about the level of the descriptors and on that of some of their learners

• No theoretical basis • Draws on Waystage, Threshold, Vantage, etc

but these documents are barely different from each other

Limitations

• All too frequently couched in language that is not easy to understand, often vague, undefined and imprecise

• Has needed a plethora of accompanying documents to help users: The Manual, now in revised form; The Reference Supplement; Guidance on conducting case studies, the Tool Kit CDs and DVDs, and still users request more teacher training, simpler versions, more illustrative performances, etc, etc

USE and MISUSE

• CEFR

• Yet politicians legislate levels for school-leaving (A2, B1, B2), for University graduation (C2!), for migration (A1 minus to B1), for citizenship (A1 to B2)

• How to establish the appropriacy of a level?

• How  to  engage  politicians  in  a  debate  about  “levels”?

‘Destination  B2 is the ideal grammar and vocabulary practice book for all students preparing to take a B2 level exam, for example the Cambridge FCE examination.

Key Features: A well researched

grammatical and lexical syllabus based on the B2 (Vantage) level of the Council  of  Europe’s  Common European Framework’

Claims about links with the CEFR and reality

• Importance of CEFR in testing, training, publishing and curricula

• Many claims of links to CEFR • How many claims are empirically based? • Who monitors the quality of the claims?

– Council of Europe? – ALTE? – Self-monitoring?

Results of 2006 Survey Curriculum development

Need for further dissemination, guidance and training Need to develop additional level specifications, descriptors

and scales Need for plans to relate curricula and/or textbooks to the

CEFR empirically Teacher education/training

Need for more dissemination, guidance and training Need for co-operation at international level

Testing and assessment Complexity of relating tests to the CEFR levels Need for more guidance and training

Dutch CEFR Construct Project

Web-based Grid for content analysis

www.ling.lancs.ac.uk/cefgrid

Problems with the CEFR

• Terminology problems: synonymy or not?

• Inconsistency?

• Lack of definition

• Gaps

Terminology problems: synonymy ?

Operations at A2

• Understand

• Take

• Get

• Follow

• Identify

• Infer

Operations at B2

• Understand

• Scan

• Monitor

• Obtain

• Select

• Evaluate

• Locate

• Identify

Inconsistency?

• I can understand familiar names, words and very simple sentences, for example on notices and posters or in catalogues”  (page  26)

• “Can recognise familiar names, words and very basic phrases on simple notices in the most common everyday situations”  (page  70)

Lack of definitions

• Simple, the most common, everyday, familiar, concrete, predictable, straightforward, factual, complex, specialised, highly colloquial, short, long

• Is  a  short  text  necessarily  “easier”  than  a  longer text?

Gaps in the CEFR

• The Task: what is it that candidates have to do with text?

• Test methods and the processing demands they create

• CEFR is NOT a test specification

Gaps: Processes of comprehension

• Focus on and retrieve explicitly stated information

• Make straightforward inferences

• Interpret and integrate ideas and information

• Examine and evaluate content, language and textual elements

Intergovernmental Forum • Language of CEFR needs simplifying • Training essential to avoid oversimplifications • Need to ensure the quality of the

implementation of the CEFR • How to avoid prescriptive use of CEFR and the

scales? • Need for international networks and training

to ensure proper application in assessment and curricula

• Importance of national, regional and local contexts and their needs when applying the CEFR

Improvement and development

More research needed into the development of language proficiency as learners progress through the levels of the CEFR

Design and construction of learner language corpora linked to the CEFR, based on standardised tasks

Investigation of instruction aimed at the different CEFR levels

Diagnosis of learner strengths and weaknesses at the different CEFR levels

Revision and (further) supplementation of the CEFR

Some issues

• How does L2 proficiency develop? • What are the linguistic features that

characterise CEFR levels? • How are the abstract constructs in the CEFR to

be operationalised? • What and how do teachers teach at the

various CEFR levels?

Some issues

The design of tasks to measure development of language proficiency

1. How can we ensure that we elicit target language features?

2. How can we check both what the learners are able to do and also what they freely choose to do?

3. How can we ensure that tasks at a given CEFR level are parallel? Is my B1 your B1?

4. We need banks of validated reading and listening tasks to illustrate CEFR levels

Will the future be perfect?

There will probably always be misuse of the CEFR

Politicians will probably always lack assessment literacy

Governments will always want simple (simplistic) solutions to complex problems

But relevant research is ongoing The CEFR can be improved The Council of Europe might publish a revised

second edition of the CEFR

Thank you for your attention!

c.alderson@lancaster.ac.uk