Download ppt - Hoffmann magno testing_apiba_27_10_12

IN HONOUR OF ANA MARÍA

ARMENDÁRIZ

Third Generation Tests

Cristina [email protected]

Marisa [email protected]

What is a test?

A test is a sample of an individual’s behavior/ performance , i.e., a series of tasks (e.g., items) used to obtain systematic observations presumed to represent attributes or characteristics.

A language test refers to ….

Any kind of measurement, examination or technique which intends to describe the testee’s foreign language proficiency

WHEN DO YOU TEST?

BEFORE A COURSE STARTS?

DURING THE COURSE?

WHEN THE COURSE FINISHES?

WHY DO YOU TEST?

TO CHECK THE LEARNERS ARE STUDYING?

TO CHECK HOW EFFECTIVE THE TEACHING HAS BEEN?

AS BACKWARD OR FORWARD TOOL?

BACKWARDAND FORWARD

PURPOSES

To gain an overview of what students bring to new instruction and thus plan and identify potential difficulties to be faced

BACKWARD AND FORWARD PURPOSES

to diagnose what individual learners know

to determine the pace of classroom instruction


to take better decisions about grouping students in the class


to share information with boards of education, parents, and the general public through the media


to measure the effectiveness of instruction and learning

to help make promotion and retention decisions

PRINCIPLES OF MEASUREMENT EVIDENCE AND EVALUATION

measurement evidence:

differentiating degrees of a trait by

description or by assigning scores

Evaluation: interpretation of the

description or scores

“The systematic analysis of evidence”

( Shepard, 2000)

Teachers and administrators need to have analysis skills to effectively interpret evidence and make value judgments about the meaning of the results.

DIFFERENT WAYS OF MEASURING

EVALUATIONASSESSMENT

TESTING

EVALUATION

looking at all the factors that influence the learning process, ex: syllabus objectives, course design, materials, methodology, teacher performance and assessment

ASSESSMENT

It involves measuring the performance of students and the progress that they are making. It helps us to be able to diagnose the problems they have and to provide them with useful feedback.

ASSESSMENT

1) Informal assessment

2) Formal assessment (testing)

3) Self assessment

INFORMAL ASSESSMENT

observation of everyday performance

collecting data about our students’ performance

in normal classroom conditions

intuitive assessment in all activities performed in the classroom

FORMAL ASSESSMENT

At the end of the course

Or

Along the course

FORMAL ASSESSMENT

commonly administered in class by the teacher, in order to assess learning.

not so formal limited to the context for purposes internal to the class assess a narrow range of language assess either objectively or subjectively to assist teaching often backward looking.

FORMAL ASSESSMENT

synonymous of “testing”

1) external examinations (KET, PET, FCE, etc)

2) administered to many students

3) under standardized conditions

4) assess a broad range of language

5) marked objectively or under standardized subjective marking schemes

6) administered at the end of a course.

FORMAL ASSESSMENT

Formative to evaluate the effectiveness of learning at a time during the course forward looking, concerned with future language learning

Summative on several contents marked as ‘pass’ or ‘fail’

Scriven (1967:43)

SELF ASSESSMENT

the students themselves assess their own progress.a) as a complement to self instructionb) to build autonomous learnersc) to give learners an opportunity to reflect on their learning.

Dickinson (1997)

What we test is related to our purpose in testing

Before course Aptitude test

During course

Placement test

Diagnostic test

Progress test

Achievement test

After Course Proficiency test

APTITUDE TESTS

to determine an individual’s ability to acquire a second or foreign language.

large scale tests taking a long time to administer

different facet of language

also forward-looking tests,

PLACEMENT TESTS

to decide the students’ placement into appropriate groups

quick to administer and to mark

administered at the start of a new phase or language course

DIAGNOSTIC TESTS

usually syllabus based

to determine the students’ areas of strength and weaknesses in relation to

the contents to be covered in the course.

PROGRESS TEST

forms part of the formative evaluation of the courses

provides continuous feedback to both the teacher and the learner

are usually written and administered by a class teacher

look back over recent work

ACHIEVEMENT TESTS

at the end of a relatively long period of learning

the content derives from the syllabus that has been taught over the period of time

large scale tests, covering a wide range of language and skills

PROFICIENCY TESTS

based on a theory of language proficiency and the specific language abilities to constitute language proficiencyoften related to specific academic or professional situations where English is needed. (TOEFL, KET, PET, FCE, CAE, IELTS, etc)

DESIRABLE FEATURES IN TESTS

VALIDITY

RELIABILITY

UTILITY

DISCRIMINATION

PRACTICALITY

VALIDITY

The extent to which a test measures what it intends to measure and nothing else

RELIABILITY

To provide consistent results when administered under similar conditions.

it shows the consistency of the scoring of the test,

both between different raters, and between the same rater on different occasions

UTILITY

To provide a lot of feedback to assist in the planning of the rest of a course or future courses.

DISCRIMINATION

The ability to distinguish between stronger and weaker students.

PRACTICALITY

How efficient the test is

in physical terms.

in equipment required

in time to set, administer

in time mark

…

THE LINGUISTIC APPROACH WE TAKE ON LANGUAGE LEARNING

WILL DETERMINE THE GENERATION OF THE TEST

FIRST GENERATION TESTS

REFLECT THE

GRAMMAR TRANSLATION APPROACH

FOCUS ON ACCURACY

DEVOID OF CONTEXT

SUBJECTIVE SCORING

SECOND GENERATION TESTS

REFLECT THE

STRUCTURALIST APPROACH

FOCUS ON DISCRETE LANGUAGE

THEMATIC ORIENTED

OBJECTIVE ASSESSMENT

Are these samples suitable for communicative purposes?

Do you feel any areas are missing in these examples? If so, which?

OLLER (1973)

The primary function of language is

COMMUNICATION

Thus, language should be tested in terms of communicative abilities

BACHMAN (1990)Communicative ability

Learners´competence +

capacity for implementing +

executing that competence in CONTEXTUALIZED communicative

language use.

Candlin (1985)

Creating meaning

“a coming together of organized knowledge structures with a set of procedures for adapting this knowledge to solve new problems of communication that do not have ready-made and tailored solutions.”

Bachman & Palmer (1996)

Language is multicomponential

Factors other than language should be put to test employing both quantitative and qualitative methodologies in interrelated areas.

Metacognitive strategies

Grammatical knowledge

Textual knowledge

Pragmatic knowledge

Topical knowledge

Background knowledge

THIRD GENERATION TESTS

REFLECT THE

COMMUNICATIVE APPROACH

Authentic materialContextualizedGlobal and discrete items

WEST (1990)Tension between principles of language testing

COMPETENCE vs. PERFORMANCE

USE vs. USAGE

INDIRECT TESTING vs. DIRECT TESTING

RECEPTIVE SKILLS vs. PRODUCTIVE SKILLS

DISEMBODIED LANGUAGE

vs. CONTEXTUALIZED LANGUAGE

WHEN CREATING

TESTING MATERIAL

BEAR IN MIND

BACKWARD LOOKING ASSESSMENT

vs.

FORWARD LOOKING ASSESSMENT

NORM REFERENCED ASSESSMENT

vs.

CRITERION REFERENCED ASSESSMENT

.

THE DEVELOPMENT AND

IMPLEMENTATION OF BENCHMARK

SYSTEMS AND FRAMEWORKS THAT TAKE

AN ASSET-BASED APPROACH TO

LANGUAGE ASSESSMENT FOCUSES

MAINLY ON WHAT THE LEARNER “CAN DO”

Canadian Language Benchmarks (CLB)

Common European Framework of Reference for Languages

3 G TESTS COMBINE

DISCRETE POINT APPROACHES

+

INTEGRATIVE APPROACHES

AUTHENTIC DISCOURSE

+

RECEPTIVE SKILLS

SO AS TO COMPLY WITH THE SYLLABUS

INCLUDE DISCRETE POINT TENCHNIQUES

(GR + VOC) To measure the range and degree of mastery

ESTABLISH PRODUCTIVE SKILLS related to objectives and contents

HUERTA-MACIAS (1995) HMAYAN (1995)

Through techniques on a daily classroom routine basis

emulate real-life tasks which aim at evaluating what learners CAN produce

integrate rather than recall and repeat

Pappamihiel & Walser (2009)

“Traditional ways of assessing language acquisition are inadequate in today’s world”

LANGUAGE ACQUISITION

IS

A COMPLEX, NONLINEAR AND COMMUNICATIVE ENDEAVOUR

NEW TRENDS

AS TEACHERS

WE NEED TO ASK OURSELVES

How do I help them grow from where they are now?”

HOW C

AN I

HELP MY

PUPILS ?

It means recognizing students’ abilities to understand, use and produce language in a variety of forms, for a variety of purposes

IT MEANS GIVING GRAMMATICAL ITEMS, VERBS AND VOCABULARY LISTS LESS IMPORTANCE THAN THEY HAVE BEEN HOLDING SO FAR.

IT MEANS BALANCING STRUCTURE AND MEANING

Non communicative learning

Pre-communication language practice

Communicative language practice

Structured communication

Authentic communication

Focus on gramma-tical accuracy

Focus on meaning

SUGGESTION OF A THIRD

GENERATION

ACHIEVEMENT TESTING

THE TEXT

published in December 2009 by the World Health Organization which is available at http://www.who.int/features/factfiles/climate_change/en/ as the triggering element

THE SUBJECTS

a group of young adults at upper intermediate course in reading.

Designed to be taken in a situation

where the learners have completed an

eighty hour course where the

following topics and grammatical

items have been covered.

TOPIC GRAMMATICAL ITEMS

GLOBALIZATION Revision of tenses and

verb forms

Narrative tenses: past

simple and continuous;

Past perfect simple and

continuous; The future in

the past.


EnviroHealth Connections: Food/Nutrition

Noun phrases. Plural expressions in compound adjectives, compound nouns; prepositional phrases, participial clauses.Modals to express probability, obligation, permission, ability and willingness. Abstract nouns and relative clauses.


Mind, body and spirit Adjectives and adverbs.

Patterns with

comparatives and

superlatives. As: as versus

like; as…as for

comparisons.Use and

non-use of passive forms.

Particles which add

meaning to verbs.

TOPIC GRAMMATICAL ITEMS21st Century lifestyles. Adjective order. The position

of adverbs. Adverbs and

expressions of opinion. Verb

patterns: reported speech.

Verb patterns. Conditional

sentences: mixed

conditionals; tense usage

for fact versus non-fact.

Hypothesizing (I wish/ if

only; as if/though; It’s

time); Present subjunctive.

Strategies to understand the gist of the passage and/or get detailed understanding such as:

predicting and anticipating, skimming and scanning, locating key words and topic sentences,

recognizing grammatical and lexical cohesive devices,

inferring meaning of unknown words from context and relating them to their morphology and their position in the sentence.

MEASURING CRITERIA for the reading section

1. Reading for comprehension and application of reading strategies

2. Use of reading strategies

3. Critical reading

MEASURING BANDS

This correction and marking criterion makes the instrument more trustworthy especially in terms of inter- and intra- rater reliability and, on the other hand, gives the task the necessary validity component to make sure we are testing what we intend to test

Band scales

Include a number of divisions, or “bands”, of possible student performance, and descriptors for each band

They assure some degree of reliability in assessments of student performance

Reading for comprehension and application of reading

strategiesBand 1

1-2 point

Can get the gist of a multipurpose authentic text

Band 2

3-4 points

Can understand main ideas, and some important details of a multipurpose authentic text.

Band 3

5-6 points

Can understand main ideas, and most important details of a multipurpose authentic text.

Band 4

7-8 points

Can understand main ideas, secondary ideas and almost all details of a multipurpose authentic text.

Band 5

9-10 points

Can understand main ideas, secondary ideas and all details of a multipurpose authentic text as well as identify the purpose of the text.

Use of reading strategies1-2 point 1-2

pointsCan do search, skimming and scanning

Band 2 3-4

points

Can use low-level inference and handle some ambiguity.

Band 3 5-6

points

Can guess the meaning of an unknown word, phrase or idiom from the context.

Band 4 7-8

points

Can use inference to locate and integrate several specific pieces of abstract information across paragraphs or sections.

Band 5 9-10

points

Can search through information and use high level inference to locate and integrate several specific pieces of abstract information (explicit and implied) from various parts of text.

CRITICAL READING

Band 1

1-2 point

Can make a general comment or give an opinion on an authentic text

Band 2

3-4 points

Can sufficiently grasp the meaning of text to paraphrase or summarize key points.

Band 3

5-6 points

Can paraphrase or summarize key points and draw conclusions.

Band 4

7-8 points

Can synthesize and critically evaluate various pieces of abstract information.

Band 5

9-10 points

Can interpret, compare and evaluate both the content and the form of written text.

MEASURING CRITERIA FOR WRITING

1. Task accomplishment and organization2. Grammar and range of structure3. Spelling, punctuation and range of vocabulary

Task accomplishment and organization

Band 1 2 point Can write full sentences related to the topic requested but does not integrate them cohesively.

Band2

4 points Can convey information related to the topic requested and link sentences to conform coherent and cohesive paragraphs but does not respect text type conventions.

Band 3 6 points Can write the text type requested accordingly responding to conventions and organization in a coherent and cohesive way

Band 4 8 points Can write an effective, stylistically complex text responding to conventions and organization in a coherent and cohesive way.

Band 5 10 points

Can write an effective, stylistically complex text responding to conventions and organization in a coherent and cohesive way and is aware of audience.

Grammar and range of structuresBand 1

2 point Can write simple sentences in which the extent of grammatical mistakes hinders comprehension and exhibits word order difficulties

Band2

4 points Can write correct though relatively simple sentences and shows difficulties regarding word order.

Band 3

6 points Can write complex sentences containing mistakes not related to earlier stages where the order is reasonable.

Band 4

8 points Can write complex sentences using the target structures appropriate to this level and to this task though not in completely accurate way

Band 5

10 points

Can write complex sentences using the target structures appropriate to this level and to this task accurately. Makes use of the appropriate tone and register.

Spelling, punctuation and range of vocabularyBand points

1 2 Can write choppy sentences very simple vocabulary which does not relate to the topic. The text presents several spelling and punctuation mistakes. Register is not appropriate.

2 4 Can write choppy sentences with simple vocabulary vaguely related to the topic. There are several spelling and punctuation mistakes.

3 6 Can write fluent sentences with vocabulary thematically related though quite simple. There are a few spelling, punctuation and register mistakes.

4 8 Can write fluent sentences with adequate and correctly used vocabulary. There are minor spelling but does not respect register. Punctuation is almost correct.

5 10 Can write fluent sentences with a fairly wide range of vocabulary appropriate to the topic, text type and register. Punctuation is correct and there are not spelling mistakes.

Description of this test

Divided into

Before reading

While reading

Global understanding

Detailed understanding

After reading

Before reading:

Have a look at the titles, the pictures and the name of the source where this text was taken from and do these exercises:

Match the pictures to the corresponding paragraph.

BEFORE READING:

State the effects global warming is having on the world. Why do you suppose the World Health Organization is worried about/involved in global warming?

WHILE READING Global understanding

State the five consequences mentionedWhere do you suppose this article was published?

What is its purpose? Is the passage complete? Yes/ No.

Why?/ Why not?

WHILE READINGDetailed understanding

Read the passage carefully

Which is the correct option? Multiple choice questions

1- The burning of fossil fuelsa) has increased in a 30%b) traps heat in the lower atmospherec) causes the temperature to increased) changes patterns of infectious diseases.

Match the meaning of these words

1. Scarcity a. Is likely to catch

2. Crop yields b. General

3. Lessen c. Economic position

4. Trapping d. Not enough

5. Overwhelming e. Organization in the house

6. Outbreaks f. Impossible to control

7. Vulnerability g. Catching

8. Livelihood h. Episode

9. Households i. Reduce

10. Overall j. Harvest collected

AFTER READINGWriting

Your school has become involved in a project to raise awareness about the dangers of global warming in health in influential business men in the city where you live. Write the speech you would give.

Bear in mind the audiencethat you would like them to react in some way that you should sound convincing

Essential elements inherent to third generation test

“The processing of realistic discourse since authenticity is of vital importance concerning the use of language in the activities proposed and in the relationship between the language which is being tested and the language of the real world” (Chalhoub-Deville,2001: 214-217 in Brown,2004: 101)


The performance of tasks which will enable the learners to make use of language in a context which resembles real life situations.


The tasks learners are expected to carry out aim at taking into account a multilayer scheme dealing with

levels of analysis related to both use

and usage of the target language:

situational,

pragmatic,

functional,

semantic levels, and

the connection with the text texture and the paratext information


and addressing syntax:

the range of structures, cohesive devices, lexical issues, special

vocabulary and lexical cohesion and the use of linguistic and non-

linguistic resources.

CONCLUSIONS

Which do you consider are some of the benefits to using third generation tests?

CONCLUSIONS

It allows learners

To view what they have learnt as a significant whole

To map their knowledge of the language in specific areas of syntax, lexis, pragmatics, pronunciation

To show an array of linguistic and paralinguistic features which they are expected to put to use in a concrete task.

CONCLUSIONS

Third generation tests enable learners to integrate these different aspects into a meaningful task following a problem solving scheme which resembles real life situations.

CONCLUSIONS

From the teachers’ standpoint

It allows viewing students’ performance and their learning process in a more holistic manner which is much more consistent with CLT techniques.

Testing is an ongoing process whereby not only the pupils show the assimilation of what they have been exposed to but also the teachers’ performance as well as the evaluator’s expertise is put to test.

ReferencesBachman, L. F., A. S. & Palmer (1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.Bachman, L. F. (1990) Fundamental Considerations in Language Testing. Oxford: OUP. Bachman, L. F., & A. S. Palmer,. (2010). Language assessment in practice (2nd ed.). Oxford: Oxford University Press.Brown, J. (2004) “Performance Assessment: Existing Literature and Directions for Research” Second Language Studies, 22(2), pp. 91-139.Canale, M. and M. Swain (1980) “Theoretical bases of communicative approaches to second language teaching and testing”. In Applied Linguistics 1. Canale, M. (1983). On some dimensions of language proficiency. In J. W. Oller (Ed.), Issues in language testing research. Rowley, MA: Newbury House.Candlin, C. N. (1985) «Explaining communicative competence: the limits of testability». En Report 21 of the IInd International TOEFL Conference, ETS. Princeton: New JerseyDouglas, D. (2000). Assessing language for specific purposes: Theory and practice. New York: Cambridge University Press.Hamayan, E.V. (1995). Approaches to alternative assessment. Annual Review of Applied Linguistics, 15, 212-226.Huerta-Macias, A. (1995). Alternative assessment: Responses to commonly asked questions. TESOL Journal, 5, 8-10.Lennon, A. (2009).Assessment and Testing in the Classroom. Course material. FUNIBER.McNamara, T. (2000). Language testing. Oxford: Oxford University Press.Oller, J. (1973). Cloze tests of second language proficiency and what they measure. Language Learning 23, 105-118.Oller, J. (1979): Language tests at school, London, Longman. SPOLSKY, the problem of validation, TESOLOller, J. (editor). 1983.Issues in language testing research. In Reprint in Candlin 1986. Rowley, MA: Newbury HouseWest, R. (1990): Introduction and Principles of Language Testing. University of Manchester SEDE.