Upload
m-b
View
290
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Welcome!
The Grid - Reloaded Competence-based test design and
assessment
The theory: what we need for a...
Good test
Validity
Reliability
Feasibility
Authenticity
Validity
= The degree to which the test actually
measures what it intends to measure.
Reliability
= The consistency of results that a particular test produces.
Reliability
Test reliability Reliability of
test conditions
Student-related reliability
Scorer reliability
Feasibility
= is the test practical... ◦ to set up? ◦ to administer? ◦ to assess/mark? ◦ to finance?
Authenticity
àHow convincingly does the test simulate “real-world” tasks?
� Can be increased through: ◦ Contextualisation � “Authentic” topics, situations � Linked sentences, not isolated ones à thematic organisation (story, episodes...)
◦ Layout and instructions of test tasks
Keep in mind:
The importance of specifying: - Form
- review, story, letter...
- Audience - familiarity, distance
- Topic - Purpose
- entertain, persuade, describe...
Any remaining questions?
Good, because now it’s YOUR turn...
What you need to do now:
1. Use the marking grid you are given to assess and mark the three student productions.
2. Use the Think Aloud Protocol to record any reflections, justifications... that underpin your decision(s).
3. Compare and discuss your findings with the other members of your group.
Conclusion: Why work with marking grids?
� Criterion-referenced assessment ◦ Systematic, nuanced ◦ Qualitative and quantitative considerations ◦ Accountability
� Avoiding dangers of ‘gut-feeling’ assessment � Increasing scorer reliability � Standardising assessment procedure � Chance for focused, detailed feedback � A positive look at the “bigger picture”
BAND 0 BAND 1 BAND
2 BAND 3 BAND 4 BAND 5
CONTENT • Task achievement • Effect on reader • Format (if required)
No
rate
able
lang
uage
. Tot
ally
inco
mpr
ehen
sibl
e. T
otal
ly ir
rele
vant
Only 1/3 of the content elements dealt with and / or hardly any content elements dealt with successfully. Message only partly communicated to the reader and / or considerable effort required on behalf of the reader. Format only partly respected.
Feat
ures
of b
ands
1 a
nd 3
. S
erio
us e
ffort
by re
ader
requ
ired.
Only 2/3 of the content elements dealt with: message only partly communicated to the reader; and /or not all content elements dealt with successfully. Support data might be missing or there might be irrelevant detail. Some effort required of the reader. Awareness of format, but occasional inappropriacies.
Feat
ures
of b
ands
3 a
nd 5
. Li
ttle
effo
rt by
read
er re
quire
d.
All content elements covered appropriately: message clearly communicated to reader. The desired effect on reader is fully achieved. Overall, no effort required of the reader. The format is appropriate
ORGANISATION: COHERENCE AND COHESION • Logic • Fluency • Control of linking devices,
referencing... • Paragraphing • Punctuation
Response may contain some inconsistencies. Only short simple sentences. At times limited control of simple sentences. The sentences are simply listed rather than connected and presented as a text.
Overall coherent response. Mostly good control of simple sentences. Complex sentence forms attempted, but they tend to be less accurate than simple sentences. Use of simple linking devices such as ‘and’, ‘or’, ‘so’, ‘but’ and ‘because’. Information presented with some organisation.
Coherent response. Good control of simple sentences and in general also of complex sentences. Coherence is achieved by use of simple linking devices. Response is well organised.
LEXIS • Appropriacy • Range • Accuracy
Limited range which is minimally adequate for the task and which may lead to repetition. Limited control of spelling and / or word formation.
In general, appropriate and adequate range for the task. Errors, which do not reduce communication and are mainly due to inattention or risk taking.
Confident and ambitious use of a wide range of vocabulary within the task set. Few minor errors which are non-impeding, perhaps due to ambitious attempts at more complex language.
GRAMMATICAL STRUCTURES • Appropriacy • Range • Accuracy
Limited range of structures. Frequent grammatical errors and faulty punctuation, which cause difficulty in terms of communication.
Sufficient range of structures for the task. Errors, which do not reduce communication and are mainly due to inattention or risk taking.
Confident and ambitious use of a wide range of structures within the task set. Few minor errors which are non-impeding, perhaps due to ambitious attempts at more complex language.
BAND 0 BAND 1 BAND 2 BAND 3 BAND 4 BAND 5
CONTENT • Task response • Relevance • Effect on listener
No
rate
able
lang
uage
. Tot
ally
inco
mpr
ehen
sibl
e. T
otal
ly ir
rele
vant
.
Meaning only conveyed in very familiar everyday situations. Response is only partly relevant. Serious effort on behalf of the listener required.
Feat
ures
of b
ands
1 a
nd 3
. S
erio
us e
ffort
by li
sten
er re
quire
d.
Communication handled in everyday situations. In general, response is relevant. Generally meaning is conveyed successfully, occasional effort
Feat
ures
of b
ands
3 a
nd 5
. Li
ttle
effo
rt by
list
ener
requ
ired.
Communication on familiar topics is handled well. Communication on unfamiliar topics is limited. Contributions are relevant despite some repetition. Meaning is conveyed successfully
PRONUNCIATION AND DISCOURSE MANAGEMENT • Pronunciation • Fluency (speech rate and
continuity) • Effort to link ideas and language
so as to form a coherent, connected speech
• Prompting and support • Interaction
Speech is mostly intelligible, despite limited control of phonological features. Considerable effort on behalf of the listener required. Noticeable pauses and slow speech, frequent repetition and self-correction. Very short basic utterances, which are juxtaposed rather than connected or linked through repetitious use of simple connectives. Requires prompting and support.
Generally intelligible though mispronunciation may occasionally cause momentary strain for the listener. Produces simple speech fluently. Usually maintains flow of speech but uses repetition, self-correction and / or slow speech to keep going. May over-use or under-use certain connectives and discourse markers. Requires little prompting and support.
Generally intelligible. Intonation is generally appropriate. L1 is perceptible but does not interfere with intelligibility. Produces extended stretches of language but may lose coherence at times due to occasional repetition, self-correction or hesitation. Uses connectives and discourse markers but not always appropriately. Requires no prompting and support. Initiates and responds appropriately.
LEXIS • Appropriacy • Range • Accuracy • Paraphrase
Limited range of vocabulary which is minimally adequate for the task and which may lead to repetition. Paraphrase rarely attempted.
In general, adequate range of vocabulary for familiar topics. Attempt to vary expressions, but with inaccuracy. Paraphrase attempted, but with mixed success.
Has a wide enough range of vocabulary to discuss familiar topics at length and make meaning clear in spite of some inappropriacies. Unfamiliar topics may provoke inaccuracies and errors. Generally paraphrases successfully.
GRAMMATICAL STRUCTURES • Appropriacy • Range • Accuracy
Only a limited range of structures is used. Limited control of basic forms and sentences. Subordinate structures are rare and tend to lack accuracy. Frequent grammatical errors.
In general, adequate range for familiar topics. Good degree of control of simple grammatical forms and sentences. Both simple and complex forms are used. Minor and occasional errors.
Uses simple and complex structures, but with limited flexibility. May make mistakes with complex structures, though these rarely cause comprehension problems.
Thank you for attending!