The assessment of speaking and the CEFR - · PDF fileThe assessment of speaking and the CEFR ... credible claims of any linkage of its standard(s) tthCEFRb it td t tto the CEFR because

Generalitat de CatalunyaDepartament d’Ensenyament

The assessment of speaking and the CEFR

Neus Figueras

The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011

Th i i d h l d iThe action-oriented approach to language use proposed in the CEFR (2001) reflected, to a considerable extent, theie s of spoken disco rse anal sts hich describedviews of spoken discourse analysts, which described

spoken language as a picture of “dynamism, fluidity,variability mixing and negotiation” (McCarthy:1998)variability, mixing and negotiation (McCarthy:1998).However, and despite the obvious impact of the CEFR in language policies after its publication the changes in thelanguage policies after its publication, the changes in theteaching, the learning and the assessment of the spokenlanguage are yet to be documented. This session willlanguage are yet to be documented. This session willrevise and problematise issues that may not yet have beenaddressed in the field of teaching and assessing speaking,g g p gand will explore possible ways of doing so.


The widespread movement to establish educational standards hasl d t k bl i i ff t t l t lled to a remarkable increase in efforts to evaluate learnerperformance through various types of objective testing procedures.At universities it is the CEFR that has primarily set new standardsAt universities, it is the CEFR that has primarily set new standardsin language teaching. In this context, language teachers are beingincreasingly confronted with the task of testing languageincreasingly confronted with the task of testing languageaccomplishments. The 3rd Bremen Symposium 2011 will respond to the demand forThe 3rd Bremen Symposium 2011 will respond to the demand for information, dialogue and exchange on these issues. Thesymposium will be concerned with all aspects surrounding they p p gcentral question of which areas of language competence can bemeasured by which procedures (as put into practice in languageclassrooms), as well as which areas of language competence canbe related to self-imposed or predetermined standards.


• Language competenceg g• Learner performance

CEFR Standards

University (context)U e s ty (co te t)

• Assessment constructs• Assessment methods


Assessment methods

Generalitat de CatalunyaDepartament d’Ensenyament

L ( ki ) d h CEFRLanguage assessment (speaking) and the CEFR, ten years later. y

Neus Figueras


Policy Definitiony

Elaborated Description

Test Content

Cut Score on Test

R k 2008


Reckase 2008

Policy Definitiony


Test Content

Cut Score on Test

R k 2008


Reckase 2008

Revisiting issues

• The nature of speaking*• The nature of speaking• Teaching and learning speakingg g p g• Assessing speaking • Standards and cutscores


What is speaking?From Austin (1955) …To say something is in the full normal sense to “do” something,

To Bachman (1990) …language competence (organisational, pragmatic) strategic competence (assessment, planning, execution)

h h i l i l h ipsychophysiological mechanisms

To the CEFR (2001)To the CEFR (2001)…views users and learners of a language primarily as “socialagents” i e members of society who have tasks to accomplishagents , i.e. members of society who have tasks to accomplish (not exclusively target-related) in a given set of circumstances, ina specific environment and within a particular field of action


a specific environment and within a particular field of action.

The CEFR action oriented approachThe CEFR action oriented approach

4 1 C f4.1. Context of use4.2. Communication themes4.2. Communication themes4.3. Communicative tasks and purposes4.4. Communicative language activities and

strategies g4.5. Communicative language processes4 6 T t4.6. Texts


Oral production Sustained monologuePublic announcementsAddressing audiencesg

Understanding a native speaker interlocutorConversation

Oral interactionInformal discussionFormal discussion and meetingsGoal-oriented cooperationTransactions to obtain goods and servicesInformation exchangeInterviewing and being interviewed

Interaction strategies TurntakingCo-operatingAsking for clarification

Production strategiesAsking for clarification

PlanningCompensating

Mediation* No illustrative scales are yet available

CompensatingMonitoring and repair


Mediation No illustrative scales are yet available

Teaching and learning speakingTeaching and learning speaking

(Speaking) is the vehicle par excellence of social solidarity, of social ranking, of professionaladvancement and of business. It is also a mediumthrough which much language is learnt, and which formany is particularly conductive for learning Perhapsmany is particularly conductive for learning. Perhaps,then, the teaching of speaking merits more thought.

Bygate, 1987


.. And the car driver analogy:

The job we do when we speak is similar: we do notThe job we do when we speak is similar: we do notmerely know how to assemble sentences in the

b t t h t d th d d tabstract: we have to produce them and adaptthem to the circumstances. This means makingdecisions rapidly, implementing them smoothly, andadjusting our conversation as unexpected problemsadjusting our conversation as unexpected problemsappear in our path.


It is the feeling that things are still out of kilter with regard to our ambitions to teach “real language”,g g g ,“language for communication” and “spoken skills” onthe one hand, and our readiness (or otherwise) to( )accept revised descriptions of target languagesbased on what their speakers actually say on thep y yother hand

McCarthy, 1998


McCarthy’s proposal

“What should and can be taught in the spoken language”What should and can be taught in the spoken language (1988:2):

The three I’sIllustration InteractionInteraction Induction


TensionsLanguage Learning

Language UsingLanguage Using

Teaching

Assessment

Testing

ValidityValidity

Reliability


Assessment specificationsAssessment specifications(Luoma, 2004)

Construct specificationConstruct specification

Task specification

Assessment specification


• Assessment of learningg• Assessment as learning• Assessment for learning

(Assessment Reform Group, 2002)


• Assessment of learningg• Assessment as learning• Assessment for learning

(Assessment Reform Group, 2002)


Effective test development(Downing and Haladyna 2006)

1. Overall plan2 Content definitionQ 2. Content definition 3. Test specifications4 It d l t

Q

U4. Item development5. Test design and assemblyA

L 6. Test production7. Test administration

L

I8. Scoring test responses9. Passing scores

I

T10. Reporting test results11. Item banking

Y


g12. Test technical report

McNamara’s Model 1996Rater

Rating (score) Marking schemeMarking scheme

Performance

Interlocutor Instrument/Task

Candidate (Underlying Competencies,


Skehan)

• A score is a number or a letter that representsA score is a number or a letter that represents how well an examinee perfoms along a continuum.continuum.

A t d d i t t t b t h th• A standard is a statement about whether an examination performance is good enough for a

ti lparticular purpose.-The special score that serves as the boundary p ybetween passing and failing- The numerical answer to the questionThe numerical answer to the question“How much is enough?”


77

Pass DistinctionGoodPass Good


A2 B2B1A2 B1


What is your standard?What is your standard?

How do I know that my B1 is your B1?


Policy Definitiony


Test Content

Cut Score on Test

R k 2008


Reckase 2008

January 2009

Relating Language Examinations to the CommonEuropean Framework of Reference for Languages:European Framework of Reference for Languages:Learning, Teaching, Assessment (CEFR)

A Manual

Language Policy Division, Strasbourg


Building an argument

Familiarisation with the CEFRFamiliarisation with the CEFR



“Linking of a test to the CEFR cannot be validunless the examination or test that is the subject of the linking can demonstrate validityin its own right”in its own right …

“A test that is not appropriate to context willA test that is not appropriate to context willnot be made more appropriate by linking to the CEFR; “


“An examination that has no procedures forAn examination that has no procedures forensuring that standards applied by interviewersor markers are equivalent in severity, or thatsuccessive forms of tests administered insuccessive forms of tests administered indifferent sessions are equivalent,cannot makecredible claims of any linkage of its standard(s)t th CEFR b it t d t tto the CEFR because it cannot demonstrateInternal consistency in the operationalisation ofy pIts standard(s).” (2009:7)


Where to start?Where to start?






What is Specification? D

• Content analysis of an

What is Specification?T

R

G

O

D

O

C• Content analysis of an examination

R

A

N

O

D

C

U

M• Relating to the CEFR content

and scalesS

PP

M

E

N

• Profiling to CEFR levelsA

R

R

AT

A• Claim of linkageE

N

C

T

I

T

IC

Y

I

C

E

O

N


E

“The emphasis in the procedures presented inthi h t li b th d tthis chapter lies on both process and outcome.Users are encouraged to go through a processof content analysing and linking. It is stronglyadvised to reconsider every interim claim thatad sed to eco s de e e y te c a t athas been made during the process. It is quitepossible that the initial estimation of thepossible that the initial estimation of therelationship to the CEFR that was given in FormA8 will need to be revised. The user shouldrevisit the analysis and make a consideredyjudgement. The estimation (Form A8) is confirmedor corrected in Form A24 ” (2009:41)The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011

or corrected in Form A24. (2009:41)

What is standard-setting?g“The process of establishing one or more cutscores on examinations. The cut scores divide thedistribution of examinee’s test performances intoptwo or more categories”

“…it is perhaps the branch of psychometrics thatblends more artistic political and cultural ingredientsblends more artistic, political, and cultural ingredientsinto the mix of its products than any other”

Cizek and Bunch 2007:5


• Standardisation of judgementsReceptive skillsProductive skillsProductive skills

• Judgement sessionsTasks and itemsScripts and Performances

• Setting cutscores How good is good enough



Text characteristicsText characteristicsAbstract/concreteGeneral/specialisedSimple/complex words &structuresL / h tLong/shortReadability measuresTest taker

Topical knowledgeReading experienceStrategic abilities

Response Empirical difficulty

Task

Strategic abilities Test wiseness Text + item demands

Retrieve factual informationMake straightforward

Pilot/test

?JudgeTopical knowledge

ginferencesInterpret, integrate and evaluate meaning and style

Difficulty

?

Reading experienceStrategic abilities+Beliefs

Lexical relations/bondsy

estimationJudgement

Item(s)characteristicsType + NumberGridding

Testing experience


McNamara’s model, 1996,Rater

Rating (Score) Scale

PerformancePerformance

Interlocutor Instrument/Task

Candidate (Underlying Competencies, P. Skehan)



Stepsp

Wh d H M• Selecting the Panel • Familiarisation

Who and How ManyWhatFamiliarisation

• Selection of approach How

• Completion of procedures• Reporting documentation

How much

• Reporting, documentation and dissemination


What is validation?Validation concerns the body of evidence put a da o co ce s e body o e de ce puforward to convince the test users that the wholeprocess and its outcomes are trustworthy.

• Test

• Standard settingStandard setting



Standard Setting ValidationStandard Setting Validation

Procedural Explicitness

Standard

Practicability

Standard Setting

ValidationInternal Implementation

Feedback

External Documentation

The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011(Cizek & Bunch, 2007:60)



Available literature• Council of Europe Manual and Reference supplement

• Cizek, G.J. and M.B.Bunch, Standard Setting. A guide t t bli hi d l ti f t d dto establishing and evaluating performance standards on tests. Sage

• Figueras, N. and Noijons, J. (eds) (2008) Linking to th CEFR L l R h ti C il fthe CEFR Levels: Research perspectives. Council of Europe – CITO. Available on line www.ealta.eu.org

• Martinyuk, W. (ed) (2010) Aligning tests to the CEFR. St di i l t ti CUP


Studies in language testing. CUP

Available exemplar items & performancesAvailable exemplar items & performances

• Council of Europe item CD• Council of Europe item CD • Ceftrainhttp://www.helsinki.fi/project/ceftrain/index.php.35.html• Dutch CEFR construct projectp jhttp://www.lancs.ac.uk/fss/projects/grid/• Council of Europe scripts• Council of Europe scriptswww.coe.int• CIEP performanceswww.ciep.fr


p

"Spoken performances illustrating the sixlevels of the CEFR“ (www ciep fr)levels of the CEFR (www.ciep.fr)

Examples of oral productions of young people from 13 to 18 years illustratingpeople from 13 to 18 years, illustrating in English, French, German, Italian and Spanish the 6 levels of the CEFR. p

P d t f i i d i 2008Product of a seminar organized in 2008 by CIEP with the assistance of the Language Policy Division of theLanguage Policy Division of the Council of Europe with Cambridge ESOL, Cervantes Institute,


Eurocentres Foundation, Goethe Institute, University of Perugia

Conclusions?Conclusions?• A system is available (Manual relevantA system is available (Manual, relevant

literature, toolkit) G id li (EALTA ILTA ALTE )• Guidelines (EALTA, ILTA, ALTE,…)

• Hard work is neededHard work is needed• No easy answers• Systematicity• DocumentationDocumentation• On going process



AND, HOPEFULLY, SOME FUN!


Thank you for your attention!


Documents

The assessment of speaking and the CEFR - · PDF fileThe assessment of speaking and the CEFR ... credible claims of any linkage of its standard(s) tthCEFRb it td t tto the CEFR because