Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency

Eaquals International Conference, Lisbon, 21 – 23 April 2016

Assessing Oral Proficiency

Ines Paland-Riedmüller,Deutsch-Uni Online

Claudia Pop,TestDaF-Institut,

g.a.s.t. e.V. Germany

www.eaquals.org

Content of the Workshop

Specifications of speaking The basics of assessment: Purposes,

methods, formats The subjectivity of assessment Quality management tools What is viable for your institution?


articulating (phonetic)

phrasing / formulating (linguistic)

planning & organising (cognitive)

Specifications of speaking

Complexity Pronun-ciation

Time


The basics of assessment: Purposes

• Summative assessment• used at end of a certain unit of the learning process• ”assessment of learning”

• Formative assessment• used continuously, during the learning process• ”assessment for learning”


The basics of assessment: Purposes

• Reference group orientated assessment• comparing the test outcome with the group

outcome• ranking, e.g. course

• Criteria orientated assessment• comparing the outcome directly with the aspects

intended to be measured


The basics of assessment: Methods

• Holistic/global scoring• assessing the performance as a whole• one general mark

• Analytic/profile scoring• decomposing the performance according to

several aspects of interest• several marks


The basics of assessment: Formats

Direct assessment face-to-face interview one or more candidates one or more raters

Semi-direct assessment recorded tasks recorded performances


The basics of assessment: Formats


situation face to face, media transmittedreal communication, simulated communication

partners interlocutor, assessor, co-candiate(s)real person, avatar

activities production, interaction, spontaneous, reflected

number of partners

monologue, dialogue, multilogue

input material

aural, written, visual, graphical

rating simultaneous, posteriorhuman, (semi-) automated

Which aspects actually influence

our decision making?

An example


Is the grey surface of object

X in shape and size identical with the grey

surface of object A, object B or

object C?

X


X




object C?


X




object C?


X




object C?


X

Is the grey

surface of object X in shape and

size identical with the grey


object C?


We have built-in perceptions of

geometrical forms.

How much stronger might this effect be in assessing learners´ proficiency?


The subjectivity of assessment

• Rudolf Weiss, 1965• main question: How large is the variability in

teachers´ assessment?• rating of one student´s essay by 92 teachers

independently, criteria: • correctness• style• content• general mark




Mark Correctness Style Content

1 7% 24% 26%

2 28% 41% 47%

3 39% 19% 20%

4 22% 14% 7%

5 4% 2% 0%

Weiss, 1965


1 2 3 4 5

-10%

0%

10%

20%

30%

40%

50%

60%

10%

45%

35%

10%

0%

General mark

Weiss, 1965


The subjectivity of assessmentReplication of the Weiss-study: Birkel & Birkel, 2002

1 2 to 3+ 3 to 4+ 4 to 5+ 50

10

20

30

40

50

60

1

18

51

16

2

Birkel & Birkel, 2002


Assessment “errors”:• halo effect• contamination effect• serial effect• central/ extreme tendency• personal dispositions


Serious reliability problem – what can we do?

Standardise!


Quality management tools

trainings calibration material monitoring system rating grids


Tools: Trainings

• Face-to-Face-Interview • standardise the procedure• discuss appropriate behaviour• analyse recorded material• practise: Role play


Tools: Trainings

• Assessment• recollect the goal (construct)• introduce criteria • operationalise the criteria• assess in groups and discuss the

results • raise error awareness• discuss problems


Tools: Calibration material

• Guidelines• manual, script, interlocutor frame • best practice example of a face to face

interview• Bench marks

• typical performances for each level• calibrated assessment (expert group)• framework of expectation


Tools: Monitoring

• two or more raters for the same performance

• (peer) observation• video analysis

constructive feedback


Tools: Rating grids

• “dare” to modify the CEFR according to your context and aim of assessment

• e.g. ALTE-Materials for the Guidance of Test Item Writers (module 4)


Descriptors should be

• positive • short • accurate/ precise• independent of each other

Tools: Rating grids

• “dare” to modify the CEFR according to your context and aim of assessment

• e.g. ALTE-Materials for the Guidance of Test Item Writers (module 4)

• evaluate and modify your grid before introducing it


Feasible?

• What is already done in your institution?

• Which additional tools do you think might be suitable?


Thank you

[email protected]@deutsch-uni.com


Education

Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency