Incorporating the CEFR into language test development

www.britishcouncil.org 1

Jamie Dunlea, British Council

Incorporating the CEFR into language

test development: Using an international

framework in local contexts

E-merging Forum 4 March 13-14

Moscow

Language Assessment Research

To summarize there is no gold standard, there is no true cut-off score, there is no best standard setting method, there is no perfect training, there is no flawless implementation of any standard setting method on any occasion and there is never sufficiently strong validity evidence. In three words, nothing is perfect. (Kaftandjieva, 2004)


What is the CEFR?

What is the CEFR? Common European Framework of Reference for Language: learning, teaching and assessment


Published by the Council of Europe in 2001

“Formal origins of the CEFR date back to 1991” (Morrow,2004)

40 years of research in language education in Europe (Morrow,2004; Trim, 2010)

Waystage, Threshold, Vantage


What is the CEFR?

What the blue book says…

Provides a common basis for the elaboration of language syllabuses, curriculum guidelines, examinations, textbooks, etc. across Europe. defines levels of proficiency which allow learners’ progress to be measured at each stage of learning and on a life-long basis. will facilitate the mutual recognition of qualifications gained in different learning contexts, and accordingly will aid European mobility.


What is the CEFR?

“At the heart of the CEF are the Common Reference levels.” (Morrow, 2004)


What is the CEFR?


C2 C1

6 5 4 3 2 1

B2+

B2

B1+

B1 A2+ A2

C2 C1 B2 B1

C2 C1 B2 B1 A2 A1

Proficient

User

Independent

User

Basic

User

What is the CEFR?

The Global Scale summarizes “the proposed Common Reference Levels in single holistic paragraphs” (Council of Europe, 2001, p. 24)


What is the CEFR?

Can understand with ease virtually everything heard or read. Can summarise information from different spoken and written sources, reconstructing arguments and accounts in a coherent presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating finer shades of meaning even in more complex situations.

Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can express him/herself fluently and spontaneously without much obvious searching for expressions. Can use language flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured, detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive devices.

Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes and ambitions and briefly give reasons and explanations for opinions and plans.

Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g. very basic personal and family information, shopping, local geography, employment). Can communicate in simple and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can describe in simple terms aspects of his/her background, immediate environment and matters in areas of immediate need.

Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible without strain for either party. Can produce clear, detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and disadvantages of various options.

Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help.

C2

C1

B2

B1

A2

A1

Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes and ambitions and briefly give reasons and explanations for opinions and plans.

Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g. very basic personal and family information, shopping, local geography, employment). Can communicate in simple and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can describe in simple terms aspects of his/her background, immediate environment and matters in areas of immediate need.

Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help.

Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible without strain for either party. Can produce clear, detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and disadvantages of various options.

9

Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can express him/herself fluently and spontaneously without much obvious searching for expressions. Can use language flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured, detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive devices.

C2

C1

B2

B1

A2

A1 10

C2

C1

B2

B1

A2

A1

Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes and ambitions and briefly give reasons and explanations for opinions and plans. 11

C2

C1

B2

B1

A2

A1

Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help. 12

The global scale is “just the tip of the iceberg” (Morrow, 2004)

Illustrative descriptors in 54 scales

o Communicative activities

o Strategies

o Communicative language competences


What is the CEFR?

Overall Listening Comprehension Understanding Interaction between Native Speakers. Listening as a Member of a Live Audience Listening to Announcements & Instructions

Overall Reading Comprehension Reading Correspondence Reading for Orientation Reading for Information and Argument Reading Instructions


What is the CEFR?

CEFR LEVEL

C2

C1

B2

B1

A2

A1

Cambridge ESOL IELTS TOEFL PBT TOEFL iBT

CPE 8

CAE 6.5 560 110-120

FCE 5 87-109

PET 4 457 57-86

KET


What is the CEFR?

Based on information presented by individual exam boards IIELTS scores are borderline band scores for that level (for more, see

www.ielts.org/researchers/common_european_framework)

Morrow (2004): notes ambiguity in terminology: “what are main points?”;; “How many is most?”

Alderson et al (2006): problems for designing tests: Inconsistencies; Terminology problems; Lack of definition; Gaps.

O’Sullivan & Weir (2011): “lacks the theoretical rigor, coverage and explicitness necessary…to develop tests”

Davidson & Fulcher (2007): “does not detail particular contexts in which it is to be used, and so lacks the necessary detail on which to build test specifications.”


Cautions, criticisms…

The CEFR is purely descriptive – not normative The CEFR is language neutral – it needs to be applied

with regard to each specific language. The CEFR is context neutral – it needs to be applied and

interpreted with regard to each specific educational context in accordance with the needs and priorities of that context.

The CEFR attempts to be comprehensive. It cannot, of course, claim to be exhaustive. Further elaboration and developments are welcomed.


Principles for users of the CEFR (North, Martyniuk, & Panthier, 2010)

City & Guilds Communicator IESOL Examination (O’sullivan, 2008)

Dutch state foreign language examinations (Berger, Kuiper, & Maris, 2009; Noijons & Kuipers, 2010)

TestDAF (Kecker & Eckes, 2010) Trinity College Examinations (Papageorgio, 2007;

Papageorgio, 2009) The European language portfolio (Lenz, 2004) The English Profile project (Trim, 2010)


Applications to testing

TOEFL PBT (Tannenbaum & Wylie, 2005) TOEFL iBT (Tannenbaum & Wylie, 2008) GEPT, Taiwan (Wu & Wu, 2010; Wu ,

2012) EIKEN, Japan (Dunlea & Figueras, 2012)


Applications to testing: Outside Europe

It has problems, but… It was not designed to be a completed, prescriptive

document

It was designed to facilitate communication and collaboration amongst language educators

It is a work in progress Attempts to link or relate exams to the CEFR should be

seen as validation projects of the CEFR itself

Describing problems encountered in the “linking” process can lead to more extensive descriptions being added to the CEFR tool kit


Is it useful?

A three-way classification of proficiency scales (Alderson, 1991) is often used : ¾ user oriented ¾ assessor oriented ¾ constructor oriented

The CEFR is a user oriented scale (North, 2000) It was not designed or intended for use as a

rating scale or in test development without adaption and modification suitable for those purposes


Is it useful?

Davidson & Fulcher (2007) encourage test developers to see the framework as a “series of guidelines from which tests (and teaching materials) can be built to suit local contextualized needs.”


Is it useful?

The CEFR can be a springboard to task and test development

Socio-cognitive framework for language test development and validation (Weir, 2005;; O’Sullivan & Weir, 2011) Reading: Khalifa & Weir (2007) Listening: Geranpayeh & Taylor (2013) Speaking: Taylor, L. (2012). Writing: Shaw & Weir (2009) Test design & development: O’Sullivan (2012)


Filling the gaps: validation

CONTEXT VALIDITY COGNITIVE VALIDITY

RESPONSE

SCORING VALIDITY

TEST-TAKER CHARACTERISTICS

CONSEQUENTIAL VALIDITY CRITERION –RELATED VALIDITY



Are the characteristics of the test tasks and their administration fair to the candidates who are taking them? (Context validity)

Are the cognitive processes required to complete the tasks appropriate? Are candidates likely to use the same cognitive processes as they would if performing the task in a ‘real world’ context? (Cognitive validity)

To what extent can we depend on the scores on the test? What do the numbers or grades mean? (Scoring validity)

What effects does the test have on its various stakeholders? (Consequential validity)

What external evidence is there outside of the test scores themselves that the test is doing a good job? (Criterion-related validity)



Filling the gaps: cognitive (Khalifa & Weir, 2007) Types of reading

(goal setting)

Expeditious reading: local Careful reading: local

Expeditious reading: global Careful reading: global

Levels of

reading


Filling the gaps: cognitive (Khalifa & Weir, 2007) Types of reading

(goal setting)



Levels of

reading

Word recognition Lexical access Syntactic parsing Establishing propositional meaning Inferencing Building a mental model Creating a text level representation Creating an intertextual representation


Filling the gaps: cognitive OVERALL READING COMPREHENSION

B2 Can read with a large degree of independence, adapting style and speed of reading to different texts and purposes, and using appropriate reference sources selectively. Has a broad active reading vocabulary, but may experience some difficulty with low-frequency idioms.

B1 Can read straightforward factual texts on subjects related to his/her field and interest with a satisfactory level of comprehension.

Can understand short, simple texts on familiar matters of a concrete type which consist of high frequency everyday or job-related language

A2 Can understand short, simple texts containing the highest frequency vocabulary, including a proportion of shared international vocabulary items.

A1 Can understand very short, simple texts a single phrase at a time, picking up familiar names, words and basic phrases and rereading as required.

OVERALL READING COMPREHENSION B2 Can read with a large degree of independence, adapting style and

speed of reading to different texts and purposes, and using appropriate reference sources selectively. Has a broad active reading vocabulary, but may experience some difficulty with low-frequency idioms.

B1 Can read straightforward factual texts on subjects related to his/her field and interest with a satisfactory level of comprehension.

Can understand short, simple texts on familiar matters of a concrete type which consist of high frequency everyday or job-related language

A2 Can understand short, simple texts containing the highest frequency vocabulary, including a proportion of shared international vocabulary items.

A1 Can understand very short, simple texts a single phrase at a time, picking up familiar names, words and basic phrases and rereading as required.


Filling the gaps: Test specification for an A1 Reading task in the Aptis test Types of reading

(goal setting)



Levels of

reading

Word recognition Lexical access Syntactic parsing Establishing propositional meaning Inferencing Building a mental model Creating a text level representation Creating an intertextual representation


Filling the gaps: a test spec Test Aptis

General Component Reading Task Multiple Choice Gap-Fill

Features of the Task Skill focus Reading comprehension up to the sentence level Task Level A1 A2 B1 B2 C1 C2

task description

Multiple-choice gap fill. A short text of 6 sentences is presented. Each sentence contains one gap. Test takers choose the best option from a pull-down menu for each gap to complete the sentence. The first sentence is an example with the gap completed. Each gap can be filled by reading within the sentence.

Cognitive processing

Goal setting

Expeditious reading: local (scan/search for specifics)

Careful reading: local (understanding sentence)

Expeditious reading: global (skim for gist/search for key ideas/detail)

Careful reading: global (comprehend main idea(s)/overall text(s))

Cognitive processing Levels of reading

Word recognition Lexical access Syntactic parsing Establishing propositional meaning (cl./sent. level) Inferencing Building a mental model Creating a text level representation (disc. structure) Creating an intertextual representation (multi-text)


Features of the Input Text Words 40-50 words (including target words for gaps) Domain Public Occupational Educational Personal

Discourse mode

Descriptive Narrative Expository Argumentative

Instructive

Content knowledge

General Specific

Cultural specificity

Neutral Specific

Nature of information

Only concrete Mostly concrete Fairly abstract Mainly abstract

Lexical Level

K1 K2 K3 K4 K5 K6 K7 K8 K9 K10

Text genre E-mails, letters, notes, postcards Features of the Response

Target Length 1 word Lexical K1 Part of Speech Noun, verb, adjective

Distractors Length 1 word Lexical K1 Part of Speech Noun, verb, adjective

Key Within sentence Across sentences Across paragraphs

Filling the gaps: a test spec


Filling the gaps: a test task


Skill focus Lvl Task description Types and levels of reading

Sentence level meaning

A1 A short text with 5 gaps. Filling each gap only requires comprehension of the sentence containing the gap. Text-level comprehension is not required.

• Careful local reading • Syntactic parsing • Understanding

propositional meaning Inter-sentence cohesion

A2 Reorder jumbled sentences to form a cohesive text

• Careful global reading • Inferencing • Building a mental

model Text-level comprehension of short texts

B1 A short text with 7 gaps. Requires comprehension of text across sentences.

• Careful global reading • Building a mental

model Integrating macro-propositions and understanding important ideas in longer texts

B2 Matching the most appropriate heading to paragraphs. Requires integration of micro- and macro-propositions within and across paragraphs, and comprehension of discourse structure of more complex and abstract texts.

• Expeditious global reading

• Creating a text level representation

Filling the gaps: Test spec overview for the Aptis reading test


Filling the gap: a test spec

http://www.britishcouncil.org/aptis

More on the Aptis Test is available online:

In three words, nothing is perfect. But that is part of the fun

Thank you! www.britishcouncil.org 35

And remember…

References Alderson, J.C. (1991). Bands and scores. In Alderson, J.C. & North, B. (eds.), Language Testing in the 1990s. London. Macmillan.

Alderson, J., Figueras, N., Kuijper, H., Nold, G., Takala, S., Tardieu, C., (2006). Analysing tests of reading and listening in relation to the Common European Framework of Reference: the experience of the Dutch CEFR construct project. Language Assessment Quarterly, 3(1), 3-30

Bechger, T., Kujper, H., & Maris, G., (2009). Standard setting in relation to the Common European Framework of Reference for Languages: the case of the State Examination of Dutch as a Second Language. Language Assessment Quarterly. 6, 126-150.

Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge: Cambridge University Press.

Davidson, F., & Fulcher, G. (2007). The Common European Framework of Reference (CEFR) and the design of language testes: A matter of effect. Language Teaching, 40, 231-241.

Geranpayeh, A., & Taylor, L.(Eds.) (2013). Examining Listening: Research and practice in assessing second language listening. Studies in Language Testing 35. Cambridge: Cambridge University Press.

Dunlea, J., & Figueras, N. (2012). Replicating Results from a CEFR Test Comparison Project Across Continents. In D. Tsagari & I. Csepes (Eds.), Collaboration in Language Testing and Assessment. New York: Peter Lang.

Kaftandjieva, F. (2004). Standard Setting. In S. Takala (Ed.), Reference Supplement to the Preliminary Version of the Manual for Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF). Strasbourg: Language Policy Division.

Khalifa, H., & Weir, C J. (2009). Examining Reading: Research and practice in assessing second language reading. Studies in Language Testing 29. Cambridge: Cambridge University Press.

Lenz, P. (2004). The European language portfolio . In K. Morrow (Ed.). (2004). Insights from the Common European Framework. Oxford, England: Oxford University Press.


References Morrow, K. (Ed.). (2004). Insights from the Common European Framework. Oxford, England: Oxford University Press.

North, B. (2004). Relating assessments , examinations, and courses to the CEF. In K. Morrow (Ed.) Insights from the Common European Framework. Oxford, England: Oxford University Press.

North, B., (2001) The development of a common framework scale of language proficiency. New York:Lang.

North, B., Martyniuk, W., & Panthier, J. (2010) Introduction: The manual for relating examinations to the Common European Framework of Reference for Languages in the context of the Council of Europe’s work on language Education. In Martyniuk, W. (Ed) Aligning Tests with the CEFR. Cambridge: Cambridge University Press.

O’Sullivan, B. (2012). Aptis Test Development Approach. Aptis Technical Report ATR-1. London: British Council.

O’Sullivan, B. (2008). City & Guilds Communicator IESOL Examination (B2) CEFR Linking Project: Case study Retrieved from: reporthttp://www.cityandguilds.com/documents/ind_general_learning_esol/CG_Communicator_Report_BOS.pdf.

O’Sullivan, B., & Weir, C. (2011). Test development and validation. In B. O’Sullivan (Ed), Language Testing: Theories and Practice. Oxford: Palgrave Macmillan.

Papageorgio, S. (2007). Relating the Trinity College London GESE and ISE exams to the Common European Framework of Reference: Piloting of the Council of Europe draft manual, final project report. London: Trinity College London. Retreived from http://www.trinitycollege.it/accreditamenti/cefr-report.pdf

Papageorgio, S. (2009). Linking international examinations to the CEFR: the Trinity College London Experience. In W. Martyniuk (Ed). Aligning Tests with the CEFR. Cambridge: Cambridge University Press.

Shaw, S., & Weir, C J. (2007). Examining writing: Research and practice in assessing second language writing, Studies in Language Testing 26. Cambridge: Cambridge University Press and Cambridge ESOL.


http://www.trinitycollege.it/accreditamenti/cefr-report.pdf



References Tannenbaum, R. J., & Wylie, E. C. (2008). Linking English-Language Test Scores onto the Common European Framework of Reference: An Application of Standard Setting Methodology (TOEFL iBT Research Report). Princeton, NJ: ETS.

Tannenbaum, R. J., & Wylie, E. C. (2005). Mapping English language proficiency test scores onto the Common European Framework (ETS Research Rep. No. RR-05-18; TOEFL Research Rep. No. RR–80). Princeton, NJ: ETS.

Taylor, L.(ed.) (2012). Examining Speaking: Research and Practice in Assessing Second Language Speaking. Studies in Language Testing 35. Cambridge: Cambridge University Press.

Trim, J. (2010). The modern languages programme of the Council of Europe as a background to the English Profile Project. English Profile Journal, 1(1), 1-12.

Weir, C. J. (2005). Limitations of the Common European Framework for developing comparable examinations and tests. Language Testing, 22, 281–300.

Wu, J. R. W., & Wu, R.Y.F. (2010). Relating the GEPT reading comprehension tests to the CEFR. In W. Martyniuk (Ed). Aligning Tests with the CEFR. Cambridge: Cambridge University Press.

Wu, R. Y. F. (2012). Establishing the validity of the General English Proficiency Test Reading Component through a critical evaluation on alignment with the Common European Framework of Reference. Unpublished PhD thesis: University of Bedfordshire, Bedfordshire.

Weir, C. J. (2005). Language Testing and Validation: an evidenced-based approach. Oxford: Palgrave.


Documents

Incorporating the CEFR into language test development