Incorporating the CEFR into language test development ... · 1 Jamie Dunlea, British Council Incorporating the CEFR into language test development: Using an international framework

www.britishcouncil.org 1

Jamie Dunlea, British Council

Incorporating the CEFR into language

test development: Using an international

framework in local contexts

E-merging Forum 4

March 13-14

Moscow

Language Assessment

Research

To summarize there is no gold standard, there is no

true cut-off score, there is no best standard setting

method, there is no perfect training, there is no

flawless implementation of any standard setting

method on any occasion and there is never

sufficiently strong validity evidence. In three words,

nothing is perfect. (Kaftandjieva, 2004)


What is the CEFR?

What is the CEFR?

Common

European

Framework

of Reference for Language:

learning, teaching and assessment


Published by the Council of Europe in 2001

“Formal origins of the CEFR date back to 1991” (Morrow,2004)

40 years of research in language education in Europe (Morrow,2004; Trim, 2010)

Waystage, Threshold, Vantage


What is the CEFR?

What the blue book says…

Provides a common basis for the elaboration of language syllabuses, curriculum guidelines, examinations, textbooks, etc. across Europe.

defines levels of proficiency which allow learners’ progress to be measured at each stage of learning and on a life-long basis.

will facilitate the mutual recognition of qualifications gained in different learning contexts, and accordingly will aid European mobility.


What is the CEFR?

“At the heart of the CEF are the

Common Reference levels.”

(Morrow, 2004)


What is the CEFR?


C2

C1 6

5

4

3

2

1

B2+

B2

B1+

B1

A2+

A2

C2

C1

B2

B1

C2

C1

B2

B1

A2

A1

Proficient

User

Independent

User

Basic

User

What is the CEFR?

The Global Scale summarizes “the

proposed Common Reference Levels

in single holistic paragraphs”

(Council of Europe, 2001, p. 24)


What is the CEFR?

Can understand with ease virtually everything heard or read. Can summarise information from different spoken

and written sources, reconstructing arguments and accounts in a coherent presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating finer shades of meaning even in more complex situations.

Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can express

him/herself fluently and spontaneously without much obvious searching for expressions. Can use language

flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured,

detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive

devices.

Can understand the main points of clear standard input on familiar matters regularly encountered in work, school,

leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is

spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe

experiences and events, dreams, hopes and ambitions and briefly give reasons and explanations for opinions and

plans.

Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g.

very basic personal and family information, shopping, local geography, employment). Can communicate in simple

and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can

describe in simple terms aspects of his/her background, immediate environment and matters in areas of

immediate need.

Can understand the main ideas of complex text on both concrete and abstract topics, including technical

discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that

makes regular interaction with native speakers quite possible without strain for either party. Can produce clear,

detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and

disadvantages of various options.

Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs

of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal

details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way

provided the other person talks slowly and clearly and is prepared to help.

C2

C1

B2

B1

A2

A1

Can understand the main points of clear standard input on familiar matters regularly encountered in work, school,

leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is

spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe

experiences and events, dreams, hopes and ambitions and briefly give reasons and explanations for opinions and

plans.

Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g.

very basic personal and family information, shopping, local geography, employment). Can communicate in simple

and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can

describe in simple terms aspects of his/her background, immediate environment and matters in areas of

immediate need.

Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs

of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal

details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way

provided the other person talks slowly and clearly and is prepared to help.

Can understand the main ideas of complex text on both concrete and abstract topics, including technical

discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that

makes regular interaction with native speakers quite possible without strain for either party. Can produce clear,

detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and

disadvantages of various options.

9

Can understand a wide range of demanding,

longer texts, and recognise implicit meaning.

Can express him/herself fluently and

spontaneously without much obvious searching

for expressions. Can use language flexibly and

effectively for social, academic and professional

purposes. Can produce clear, well-structured,

detailed text on complex subjects, showing

controlled use of organisational patterns,

connectors and cohesive devices.

C2

C1

B2

B1

A2

A1 10

C2

C1

B2

B1

A2

A1

Can understand the main points of clear

standard input on familiar matters regularly

encountered in work, school, leisure, etc. Can

deal with most situations likely to arise whilst

travelling in an area where the language is

spoken. Can produce simple connected text on

topics which are familiar or of personal interest.

Can describe experiences and events, dreams,

hopes and ambitions and briefly give reasons

and explanations for opinions and plans. 11

C2

C1

B2

B1

A2

A1

Can understand and use familiar everyday

expressions and very basic phrases aimed at

the satisfaction of needs of a concrete type. Can

introduce him/herself and others and can ask

and answer questions about personal details

such as where he/she lives, people he/she

knows and things he/she has. Can interact in a

simple way provided the other person talks

slowly and clearly and is prepared to help. 12

The global scale is “just the tip of the iceberg” (Morrow, 2004)

Illustrative descriptors in 54 scales

o Communicative activities

o Strategies

o Communicative language competences


What is the CEFR?

Overall Listening Comprehension

Understanding Interaction between Native

Speakers.

Listening as a Member of a Live Audience

Listening to Announcements & Instructions

Overall Reading Comprehension

Reading Correspondence

Reading for Orientation

Reading for Information and Argument

Reading Instructions www.britishcouncil.org 14

What is the CEFR?

CEFR

LEVEL

C2

C1

B2

B1

A2

A1

Cambridge ESOL

IELTS TOEFL PBT TOEFL iBT

CPE 8

CAE 6.5 560 110-120

FCE 5 87-109

PET 4 457 57-86

KET


What is the CEFR?

Based on information presented by individual exam boards

IIELTS scores are borderline band scores for that level (for more, see

www.ielts.org/researchers/common_european_framework)

Morrow (2004): notes ambiguity in terminology: “what are main points?”; “How many is most?”

Alderson et al (2006): problems for designing tests: Inconsistencies; Terminology problems; Lack of definition; Gaps.

O’Sullivan & Weir (2011): “lacks the theoretical rigor, coverage and explicitness necessary…to develop tests”

Davidson & Fulcher (2007): “does not detail particular contexts in which it is to be used, and so lacks the necessary detail on which to build test specifications.”


Cautions, criticisms…

The CEFR is purely descriptive – not normative

The CEFR is language neutral – it needs to be applied with regard to each specific language.

The CEFR is context neutral – it needs to be applied and interpreted with regard to each specific educational context in accordance with the needs and priorities of that context.

The CEFR attempts to be comprehensive. It cannot, of course, claim to be exhaustive. Further elaboration and developments are welcomed.


Principles for users of the CEFR

(North, Martyniuk, & Panthier, 2010)

City & Guilds Communicator IESOL Examination (O’sullivan, 2008)

Dutch state foreign language examinations (Berger, Kuiper, & Maris, 2009; Noijons & Kuipers, 2010)

TestDAF (Kecker & Eckes, 2010)

Trinity College Examinations (Papageorgio, 2007; Papageorgio, 2009)

The European language portfolio (Lenz, 2004)

The English Profile project (Trim, 2010)


Applications to testing

TOEFL PBT (Tannenbaum & Wylie, 2005)

TOEFL iBT (Tannenbaum & Wylie, 2008)

GEPT, Taiwan (Wu & Wu, 2010; Wu ,

2012)

EIKEN, Japan (Dunlea & Figueras, 2012)


Applications to testing: Outside Europe

It has problems, but…

It was not designed to be a completed, prescriptive document

It was designed to facilitate communication and collaboration amongst language educators

It is a work in progress

Attempts to link or relate exams to the CEFR should be seen as validation projects of the CEFR itself

Describing problems encountered in the “linking” process can lead to more extensive descriptions being added to the CEFR tool kit


Is it useful?

A three-way classification of proficiency scales (Alderson, 1991) is often used :

user oriented assessor oriented constructor oriented

The CEFR is a user oriented scale (North, 2000)

It was not designed or intended for use as a rating scale or in test development without adaption and modification suitable for those purposes


Is it useful?

Davidson & Fulcher (2007) encourage test developers to see the framework as a “series of guidelines from which tests (and teaching materials) can be built to suit local contextualized needs.”


Is it useful?

The CEFR can be a

springboard to task and test

development

Socio-cognitive framework for language test development and validation (Weir, 2005; O’Sullivan & Weir, 2011)

Reading: Khalifa & Weir (2007)

Listening: Geranpayeh & Taylor (2013)

Speaking: Taylor, L. (2012).

Writing: Shaw & Weir (2009)

Test design & development: O’Sullivan (2012)


Filling the gaps: validation

CONTEXT VALIDITY COGNITIVE VALIDITY

RESPONSE

SCORING VALIDITY

TEST-TAKER CHARACTERISTICS

CONSEQUENTIAL VALIDITY CRITERION –RELATED

VALIDITY



Are the characteristics of the test tasks and their

administration fair to the candidates who are taking them?

(Context validity)

Are the cognitive processes required to complete the

tasks appropriate? Are candidates likely to use the same

cognitive processes as they would if performing the task

in a ‘real world’ context? (Cognitive validity)

To what extent can we depend on the scores on the test?

What do the numbers or grades mean? (Scoring validity)

What effects does the test have on its various

stakeholders? (Consequential validity)

What external evidence is there outside of the test scores

themselves that the test is doing a good job? (Criterion-

related validity)



Filling the gaps: cognitive

(Khalifa & Weir, 2007)

Types of

reading

(goal

setting)

Expeditious reading: local Careful reading: local

Expeditious reading: global Careful reading: global

Levels

of

reading


Filling the gaps: cognitive

(Khalifa & Weir, 2007)

Types of

reading

(goal

setting)



Levels

of

reading

Word recognition

Lexical access

Syntactic parsing

Establishing propositional meaning

Inferencing

Building a mental model

Creating a text level representation

Creating an intertextual representation


Filling the gaps: cognitive OVERALL READING COMPREHENSION

B2 Can read with a large degree of independence, adapting style and

speed of reading to different texts and purposes, and using

appropriate reference sources selectively. Has a broad active reading

vocabulary, but may experience some difficulty with low-frequency

idioms. B1 Can read straightforward factual texts on subjects related to his/her

field and interest with a satisfactory level of comprehension. Can understand short, simple texts on familiar matters of a concrete

type which consist of high frequency everyday or job-related language

A2 Can understand short, simple texts containing the highest frequency

vocabulary, including a proportion of shared international vocabulary

items. A1 Can understand very short, simple texts a single phrase at a time,

picking up familiar names, words and basic phrases and rereading as

required.

OVERALL READING COMPREHENSION

B2 Can read with a large degree of independence, adapting style and

speed of reading to different texts and purposes, and using

appropriate reference sources selectively. Has a broad active reading

vocabulary, but may experience some difficulty with low-frequency

idioms. B1 Can read straightforward factual texts on subjects related to his/her

field and interest with a satisfactory level of comprehension. Can understand short, simple texts on familiar matters of a concrete

type which consist of high frequency everyday or job-related language

A2 Can understand short, simple texts containing the highest frequency

vocabulary, including a proportion of shared international vocabulary

items. A1 Can understand very short, simple texts a single phrase at a time,

picking up familiar names, words and basic phrases and rereading as

required.


Filling the gaps: Test specification for

an A1 Reading task in the Aptis test

Types of

reading

(goal

setting)



Levels

of

reading

Word recognition

Lexical access

Syntactic parsing

Establishing propositional meaning

Inferencing


Creating a text level representation

Creating an intertextual representation


Filling the gaps: a test spec Test

Aptis

General Component Reading Task Multiple Choice Gap-Fill

Features of the Task

Skill focus Reading comprehension up to the sentence level

Task Level A1 A2 B1 B2 C1 C2

task

description

Multiple-choice gap fill. A short text of 6 sentences is presented. Each sentence

contains one gap. Test takers choose the best option from a pull-down menu for

each gap to complete the sentence. The first sentence is an example with the

gap completed. Each gap can be filled by reading within the sentence.

Cognitive

processing

Goal

setting

Expeditious reading: local

(scan/search for specifics)

Careful reading: local

(understanding sentence)

Expeditious reading: global

(skim for gist/search for key

ideas/detail)

Careful reading: global

(comprehend main idea(s)/overall

text(s))

Cognitive

processing

Levels of

reading

Word recognition

Lexical access

Syntactic parsing

Establishing propositional meaning (cl./sent. level)

Inferencing


Creating a text level representation (disc. structure)

Creating an intertextual representation (multi-text)


Features of the Input Text

Words 40-50 words (including target words for gaps)

Domain Public Occupational Educational Personal

Discourse

mode

Descriptive Narrative Expository Argument

ative

Instructive

Content

knowledge

General Specific

Cultural

specificity

Neutral Specific

Nature of

information Only concrete Mostly concrete Fairly abstract Mainly abstract

Lexical

Level

K1 K2 K3 K4 K5 K6 K7 K8 K9 K10

Text genre E-mails, letters, notes, postcards

Features of the Response

Target Length 1 word Lexical K1

Part of

Speech Noun, verb, adjective

Distractors Length 1 word Lexical K1

Part of

Speech Noun, verb, adjective

Key Within sentence Across sentences Across paragraphs

Filling the gaps: a test spec


Filling the gaps: a test task


Skill focus Lvl Task description Types and levels of

reading

Sentence level

meaning

A1 A short text with 5 gaps. Filling each gap

only requires comprehension of the

sentence containing the gap. Text-level

comprehension is not required.

• Careful local reading

• Syntactic parsing

• Understanding

propositional meaning

Inter-sentence

cohesion

A2 Reorder jumbled sentences to form a

cohesive text

• Careful global reading

• Inferencing

• Building a mental

model

Text-level

comprehension

of short texts

B1 A short text with 7 gaps. Requires

comprehension of text across

sentences.

• Careful global reading

• Building a mental

model

Integrating

macro-

propositions and

understanding

important ideas

in longer texts

B2 Matching the most appropriate heading

to paragraphs. Requires integration of

micro- and macro-propositions within

and across paragraphs, and

comprehension of discourse structure of

more complex and abstract texts.

• Expeditious global

reading

• Creating a text level

representation

Filling the gaps: Test spec overview

for the Aptis reading test


Filling the gap: a test spec

http://www.britishcouncil.org/aptis

More on the Aptis Test is

available online:

In three words, nothing is

perfect.

But that is part of the fun

Thank you!


And remember…

References Alderson, J.C. (1991). Bands and scores. In Alderson, J.C. & North, B. (eds.), Language Testing in the 1990s. London. Macmillan.

Alderson, J., Figueras, N., Kuijper, H., Nold, G., Takala, S., Tardieu, C., (2006). Analysing tests of reading and listening in relation to the Common European Framework of Reference: the experience of the Dutch CEFR construct project. Language Assessment Quarterly, 3(1), 3-30

Bechger, T., Kujper, H., & Maris, G., (2009). Standard setting in relation to the Common European Framework of Reference for Languages: the case of the State Examination of Dutch as a Second Language. Language Assessment Quarterly. 6, 126-150.

Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge: Cambridge University Press.

Davidson, F., & Fulcher, G. (2007). The Common European Framework of Reference (CEFR) and the design of language testes: A matter of effect. Language Teaching, 40, 231-241.

Geranpayeh, A., & Taylor, L.(Eds.) (2013). Examining Listening: Research and practice in assessing second language listening. Studies in Language Testing 35. Cambridge: Cambridge University Press.

Dunlea, J., & Figueras, N. (2012). Replicating Results from a CEFR Test Comparison Project Across Continents. In D. Tsagari & I. Csepes (Eds.), Collaboration in Language Testing and Assessment. New York: Peter Lang.

Kaftandjieva, F. (2004). Standard Setting. In S. Takala (Ed.), Reference Supplement to the Preliminary Version of the Manual for Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF). Strasbourg: Language Policy Division.

Khalifa, H., & Weir, C J. (2009). Examining Reading: Research and practice in assessing second language reading.

Studies in Language Testing 29. Cambridge: Cambridge University Press.

Lenz, P. (2004). The European language portfolio . In K. Morrow (Ed.). (2004). Insights from the Common European

Framework. Oxford, England: Oxford University Press.


References Morrow, K. (Ed.). (2004). Insights from the Common European Framework. Oxford, England: Oxford University Press.

North, B. (2004). Relating assessments , examinations, and courses to the CEF. In K. Morrow (Ed.) Insights from the

Common European Framework. Oxford, England: Oxford University Press.

North, B., (2001) The development of a common framework scale of language proficiency. New York:Lang.

North, B., Martyniuk, W., & Panthier, J. (2010) Introduction: The manual for relating examinations to the Common

European Framework of Reference for Languages in the context of the Council of Europe’s work on language Education.

In Martyniuk, W. (Ed) Aligning Tests with the CEFR. Cambridge: Cambridge University Press.

O’Sullivan, B. (2012). Aptis Test Development Approach. Aptis Technical Report ATR-1. London: British Council.

O’Sullivan, B. (2008). City & Guilds Communicator IESOL Examination (B2) CEFR Linking Project: Case study Retrieved

from: reporthttp://www.cityandguilds.com/documents/ind_general_learning_esol/CG_Communicator_Report_BOS.pdf.

O’Sullivan, B., & Weir, C. (2011). Test development and validation. In B. O’Sullivan (Ed), Language Testing: Theories and

Practice. Oxford: Palgrave Macmillan.

Papageorgio, S. (2007). Relating the Trinity College London GESE and ISE exams to the Common European Framework of Reference: Piloting of the Council of Europe draft manual, final project report. London: Trinity College London. Retreived from http://www.trinitycollege.it/accreditamenti/cefr-report.pdf

Papageorgio, S. (2009). Linking international examinations to the CEFR: the Trinity College London Experience. In W. Martyniuk (Ed). Aligning Tests with the CEFR. Cambridge: Cambridge University Press.

Shaw, S., & Weir, C J. (2007). Examining writing: Research and practice in assessing second language writing, Studies in Language Testing 26. Cambridge: Cambridge University Press and Cambridge ESOL.


http://www.trinitycollege.it/accreditamenti/cefr-report.pdf



References Tannenbaum, R. J., & Wylie, E. C. (2008). Linking English-Language Test Scores onto the Common European

Framework of Reference: An Application of Standard Setting Methodology (TOEFL iBT Research Report). Princeton, NJ:

ETS.

Tannenbaum, R. J., & Wylie, E. C. (2005). Mapping English language proficiency test scores onto the Common European

Framework (ETS Research Rep. No. RR-05-18; TOEFL Research Rep. No. RR–80). Princeton, NJ: ETS.

Taylor, L.(ed.) (2012). Examining Speaking: Research and Practice in Assessing Second Language Speaking. Studies in

Language Testing 35. Cambridge: Cambridge University Press.

Trim, J. (2010). The modern languages programme of the Council of Europe as a background to the English Profile

Project. English Profile Journal, 1(1), 1-12.

Weir, C. J. (2005). Limitations of the Common European Framework for developing comparable examinations and tests.

Language Testing, 22, 281–300.

Wu, J. R. W., & Wu, R.Y.F. (2010). Relating the GEPT reading comprehension tests to the CEFR. In W. Martyniuk (Ed).

Aligning Tests with the CEFR. Cambridge: Cambridge University Press.

Wu, R. Y. F. (2012). Establishing the validity of the General English Proficiency Test Reading Component through a

critical evaluation on alignment with the Common European Framework of Reference. Unpublished PhD thesis: University

of Bedfordshire, Bedfordshire.

Weir, C. J. (2005). Language Testing and Validation: an evidenced-based approach. Oxford: Palgrave.


Documents

Incorporating the CEFR into language test development ... · 1 Jamie Dunlea, British Council Incorporating the CEFR into language test development: Using an international framework