Wiseman Facets of L2 writing ability

Facets of Testing L2 Writing Ability

Pre-Convention Institute

TESOL Conference Philadelphia 2012

Cynthia S. Wiseman, Ed.D.Borough of Manhattan Community College

City University of New York

Agenda

Introductions General features of language assessment &

L2 writing assessment Activity Language Use Argument & L2 Writing Defining the construct: L2 Writing Framework of Task Characteristics Examination of items Review: crossword

EXAMS!!!

Homework

Classroom observation

Participation

Quiz

Self-Assessment

Portfolios

Explicit•T & Ss are aware of assessment•Clearly distinct from Teaching - Exam

Implicit•T & Ss may be unaware -- organic•Continuous – graded, ungraded

free, pre-writing•Instantaneous – in-class written free wr•Cyclical – reflective practice

Assessment is…

Purpose of L2 writing assessment: to collect information about teaching & learning of L2 writing to make decisions

Teacher & Student:–Formative decisions

• To correct S errors or not – based on student draft

• To change question of inquiry – Revise prompt? description of assignment?

• To model a structure – based on student writing, do we need to model a paragraph in class?

• To go to next lesson or review?

• To go more in-depth in content area – based on thin development, do we need to do more research, discussion, reading?

• Use a different strategy to respond to write an essay or explain a genre

–Summative decisions• To place, pass, fail or promote a student

Decisions about…. Individuals

– Selection for admission/employment – screening exams, like CATW

– Placement into course of study – Department exams, multiple measures

– Certification for profession – LAST (NY Teacher Certification Exam)

– Prediction of future performance – TWE, DIALANG

Program– Formative, to make changes to improve program

– Summative, to continue existing program

Research– To decide on new research questions or methodology

– To change/modify view/understanding of language phenomenon

Uses of Language Assessments

Intended use: – To collect information for making decisions– Beneficial consequences for stakeholders

• E.g., ESL writing teacher is teaching lesson on cohesive devices in essay writing

– To make decision about instruction

– To change/improve instruction so Ss will effectively use cohesive devices to improve writing

– Short fill-in-the-blank paragraph to get feedback on Ss’ learning & effectiveness of teaching

Introductory ActivityThink of an L2 writing assessment

development situation that you are familiar with. Describe the context, participants, and the test development process. What decisions were to be made based on the results of this assessment?

What were some of the intended beneficial consequences of assessment use in this situation?

Did any problems came up as a result of this assessment development?

Steps in creation of an assessment

Assessment Use Argument (AUA) Set of claims:

– Conceptual links between TTs performance on assessment and interpretation about the ability

– Decisions to be made– Consequences

– What would the AUA of the L2 writing assignment that you just described look like?

Assessment Use Argument --Bachman & Palmer (2010)

Assessment Development Use Bachman & Palmer (2010)

Initial Planning in test development What beneficial consequences do we want to happen? Who

are the stakeholders (i.e., intended TTs, etc.)? Who will be directly affected by the use of the assessment? How?

What are the specific decisions that need to be made to reach the intended consequences?

What do we need to know about the ELL’s language ability as demonstrated in writing to make the intended decision?

What sources could we use to make that decision? Is an existing assessment available? Is it appropriate?

Do assessment tasks correspond to TLU tasks? Does the developer provide evidence justifying intended uses? Do we really need to develop our own assessment?

Case scenario Educational Context: CC, US, diverse urban population,

85% ELL/bilingual background, open admissions Decision: To place S in ENG101, 3-credit composition

class? Beneficial consequences? What do we need to know about the ELL’s language ability

as demonstrated in writing to make the intended decision? What sources could we use to make that decision? Is an

existing assessment available? Is it appropriate?– ACT Compass (Writing Sample Test)

Do assessment tasks correspond to TLU tasks? Does the developer provide evidence justifying intended

uses? Do we really need to develop our own assessment?

Steps in Creating an Assessment Identify the target population/test context Identify the type of assessment Specify the specific purpose of the test Define the construct: Describe the Target Language Use

Domain & Target Language Tasks Write specifications for the test Write items/tasks that operationalize the construct &

incorporate task characteristics that correspond to TLU tasks

Create the test of items/tasks with clear instructions Create an answer key/rubric

Test Development ModelBachman & Palmer, 1996

What is L2 writing ability?

Communicative Language Ability --Bachman (1990)

Language characteristics• Organizational Characteristics

• Grammatical (vocabulary, syntax, phonology, graphology)

• Textual (cohesion, rhetorical, conversational organization)

• Pragmatic characteristics• Functional (ideational, manipulative, heuristic, imaginative)

• Sociolinguistic (genres, dialect/variety, register, naturalness, cultural references and figurative language)

• Topical characteristics

• Metacognitive Strategies/Competence

Target Language Use (TLU) Domain “…a set of specific language use tasks that

the TT is likely to encounter outside of the test itself, and to which we want our inferences about language ability to generalize.” (p. 44)– Distinguishing characteristics of language use

tasks to describe language use domain– Inferences that generalize to specific domains in

which TT is likely to need to use the language– Inferences about TT’s ability to use language in

a target language use domain

Language Use Task

Language use task: an activity that involves individuals in using language for the purpose of achieving a particular goal or objective in a particular situation

– Specific situations– Goal-oriented– Active participation of language users

TLU Domain Language Use Settings Language Use Tasks

English for Business Communication

Managing & operating an office

•Writing memos•Preparing reports•Taking phone msgs•Writing letters•Writing emails•Texting

Negotiating with clients & customers

•Writing proposals•Responding to written offers •Writing emails•Texting

Promoting products or services

• Writing advertising copy

• Writing solicitation pitch

Characteristics of Tasks Link between tasks in the domain of test tasks

and the domain of non-test tasks – Selection or design of tests that correspond in

specific ways to language use tasks Extent and ways TTs’ language ability is

engaged Degree of correspondence between

characteristics of given test task and a particular language use task: authenticity, validity of inferences, domain to which inferences generalize

Control of characteristics of the test task through test design and development

Language Task Characteristics

Characteristics of the setting •Physical Characteristics•Participants•Time of task

Characteristics of the test rubrics •Instructions•Structure•Time allotment•Scoring method

Characteristics of the input •Format•Language of input

Characteristics of the expected response

•Format •Language of expected response

Relationship between input and response

•Reactivity•Scope of relationship•Directness of relationship

Framework for Language Task characteristics

Characteristics of the setting

•Physical characteristics•Participants•Time of task

Characteristics of the test rubrics

•Instructions •Language (L1, L2)•Channel (aural,visual)•Specification of procedures and tasks

Structure •# of parts/tasks•Salience of parts/tasks•Sequence of parts/tasks•Relative importance of parts/tasks•# of tasks/items per part

Time allotment

Scoring method •Criteria for correctness•Procedures of scoring the response•Explicitness of criteria and procedures


Characteristics of the Input

Format •Channel (aural, visual)•Form (language, non-language, both)•Language (native, target, both)•Length•Type (item, prompt)•Degree of speededness•Vehicle (live, reproduced, both)

Language of input

Language characteristics•Organizational Characteristics

•Grammatical (vocabulary, syntax, phonology, graphology)•Textual (cohesion, rhetorical, conversational organization)

•Pragmatic characteristics•Functional (ideational, manipulative, heuristic, imaginative)•Sociolinguistic (dialect/variety, register, naturalness, cultural references and figurative language)

•Topical characteristics


Characteristics of the Expected Response

Format •Channel (aural, visual)•Form (language, non-language, both)•Language (native, target, both)•Length•Type (item, prompt)•Degree of speededness•Vehicle (live, reproduced, both)

Language of input

Language characteristics•Organizational Characteristics

•Grammatical (vocabulary, syntax, phonology, graphology)•Textual (cohesion, rhetorical, conversational organization)

•Pragmatic characteristics•Functional (ideational, manipulative, heuristic, imaginative)•Sociolinguistic (dialect/variety, register, naturalness, cultural references and figurative language)

•Topical characteristics


Relationship between Input & Response

Reactivity •Reciprocal•Non-reciprocal•adaptive

Scope of relationship

•Broad•Narrow

Directness of relationship

•Direct •Indirect

Language Use Tests comprised of Tasks Language test: a procedure for eliciting

instances of language use from which inferences can be made about an individual’s language ability– Language test should consist of language use

tasks– The elemental activities and situations of

language use– Performance of a set of interrelated language

use tasks Framework of task characteristics

Test Items

Is this writing task similar to tasks that the 2nd language learner would have to do in real life?

What aspect of second language writing ability is the item testing?

Does this task require the TT to demonstrate that aspect of L2 writing ability?

Task Characteristics TLU Task Test Task

Characteristics of the Setting Physical characteristics Participants Time of task

Characteristics of the test rubrics Instructions Structure Time allotment Scoring method

Characteristics of the input Format Language of input

Characteristics of the expected response Format Language of expected response

Relationship between input and expected response Reactivity Scope of relationship Directness of relationship

EAP ESL Low Intermediate L2 writing class, 20-25 students/class, 40-minute class, 4x/week, college preparation program w/ 7 levels, level & program exit exam: basic skills writing

proficiency test: argumentative essay.

Common Core State Standards For ELA & Literacy Council of Chief State School Officers

(CCSSO) & National governors Association (NGA)– Aligned with college and work expectations

– Standard was included based on the best available evidence that its mastery was essential for college and career readiness in 21st c, globally competitive society

Language use domain: literate in a global 21st century world Close attentive reading to understand and enjoy complex

works of literature Critical reading for important points Able to handle large amounts of information Actively seek wide, deep, thoughtful engagement with

high-quality literary and information texts that builds knowledge, enlarges experience & broadens worldviews

Reflexively demos cogent reasoning and use of evidence essential to private deliberation and responsible citizenship in democracy

Writing: K-5: College & Career Readiness

anchor Standards text types and Purposes* Write arguments to support claims in an analysis of

substantive topics or texts, using valid reasoning and relevant and sufficient evidence.

Write informative/explanatory texts to examine and convey complex ideas and information clearly and accurately through the effective selection, organization, and analysis of content.

Write narratives to develop real or imagined experiences or events using effective technique, well-chosen details, and well-structured event sequences.

Anchor Standards for Writing 6-12

Text types and Purposes Write arguments to support claims in an analysis of substantive topics or texts, using valid reasoning and relevant and sufficient evidence. Write informative/explanatory texts to examine and convey complex ideas and information clearly and accurately through the effective selection, organization, and analysis of content. Write narratives to develop real or imagined experiences or events using effective technique, well-chosen details, and well-structured event sequences.

College & Career Readiness Anchor Standards for LanguageConventions of Standard English Demonstrate command of the conventions of standard English grammar and usage when writing or speaking. Demonstrate command of the conventions of standard English capitalization, punctuation, and spelling when writing.

Production & Distribtion of Writing Produce clear and coherent writing in which

the development, organization, and style are appropriate to task, purpose, and audience.

Develop and strengthen writing as needed by planning, revising, editing, rewriting, or trying a new approach.

Use technology, including the Internet, to produce and publish writing and to interact and collaborate with others.

Agenda

Rubrics: facets of L2 writing ability Comparison of rubrics measure L2 writing

ability Design of writing task Wrap-up & evaluation

Scoring

Answer key – objective scoring Rubric - subjective scoring

– Construct– Rubric– Partial credit

L2 Writing ability

Target Language Use Domain: Academic writing in a community college: Language Use Task: write narrative/persuasive essays

Control of content developmentRhetorical controlGrammatical controlControl of register & vocabularyTask fulfillment (McNamara, 1996)

This exceptionally executed essay takes a clear position and

exceptionally succeeds in expressing a point of view or telling a story. The thorough development of ideas includes at

least two outstanding points directly related to the topic, and the examples used, particularly those from personal experience, are rich, e.g., occasional citation of statistics or reference to personal readings.

The essay is clearly and logically organized with no digressions; the writer demonstrates skillful command of cohesive devices. Writer demonstrates ability to write in the

appropriate academic register and demonstrates extensive range of vocabulary for academic purposes, with few

problems in word choice or usage. A few grammatical errors are noticeable but rarely do the grammar errors interfere with meaning.

Sentence variety and complexity reflect a sufficient command of standard written English to ensure reasonable clarity of expression.

Holistic Scoring Scale: Criteria for Grading ESL Papers 6This exceptionally executed essay takes a clear position and exceptionally succeeds in expressing a point of view or telling a story. The thorough development of ideas includes at least two outstanding points directly related to the topic, and the examples used, particularly those from personal experience, are rich, e.g., occasional citation of statistics or reference to personal readings. The essay is clearly and logically organized with no digressions; the writer demonstrates skillful command of cohesive devices. Writer demonstrates ability to write in the appropriate academic register and demonstrates extensive range of vocabulary for academic purposes, with few problems in word choice or usage. A few grammatical errors are noticeable but rarely do the grammar errors interfere with meaning. Sentence variety and complexity reflect a sufficient command of standard written English to ensure reasonable clarity of expression. 5The focus of this competently executed essay is clear but there may be a few digressions. The writer provides substantial support in the development of the essay although all examples may not be entirely relevant or appropriate for the topic. The essay is effectively organized, demonstrating systematically competent use of cohesive devices. The writer demonstrates ability to use a variety of patterns of sentence construction but with some errors. Range of vocabulary for academic purposes is generally competent, and the writer demonstrates accurate and generally appropriate control of word choice, word forms and idiomatic expressions for academic writing. Some errors in language use, but errors do not generally interfere with meaning. 4In this adequately executed essay the writer’s position is clear despite some possible digressions and contradictions. The writer provides adequately detailed support of two or more points that directly relate to the topic. The essay is generally organized, demonstrating generally accurate and appropriate use of cohesive devices. The writer demonstrates some sentence variety with simple, compound, and some complex sentences though not always correctly. The essay may contain frequent errors that may occasionally interfere with meaning. Vocabulary is adequate in range, but there are some inappropriate or inaccurate word choices and word forms. 3The essay minimally succeeds in taking a position or relating a narrative with a discernable organizational pattern (introduction, body, conclusion) but may lack clear focus in development of the central idea. The writer makes an attempt at development although examples are sometimes irrelevant. The writer makes minimal use of cohesive devices and he/she demonstrates a minimal range of sentence variety and vocabulary, with some inaccurate and/or inappropriate word choices or inappropriate register. The essay demonstrates minimal control of language, with frequent errors, some of which interfere with meaning. . 2The paper represents limited success in writing a persuasive or narrative essay. The writer provides limited development of the topic with one or more points that directly or indirectly relate to the supporting argument or story. The writing shows limited evidence of organization of ideas (paragraphs are often one sentence) or accurate or appropriate use of cohesive devices. The range of vocabulary and word choice and the use of academic register is limited. The control of language is uneven, with frequent errors, many of which obscure meaning. The writing lacks sentence variety. 1The paper is a failed attempt to write an essay. The writer does not fully develop the topic, lacking related support. There is often no clear organizational pattern, lacking a clear beginning, middle and end. The writer does not use cohesive devices. The writer demonstrates a narrow range of vocabulary. There is little evidence of appropriate word choice or usage or academic register. The writer demonstrates little control, with frequent errors of all types. The errors generally obscure meaning. The writing lacks basic sentence structure and variety. In some cases, the paper may even be written in the writer’s first language.

Template for Holistic Rubrics

Score Description

5 Demonstrates complete understanding of the problem. All requirements

of task are included in response.

4 Demonstrates considerable understanding of the problem. All

requirements of task are included.

3 Demonstrates partial understanding of the problem. Most requirements

of task are included.

2 Demonstrates little understanding of the problem. Many requirements

of task are missing.

1 Demonstrates no understanding of the problem.

0 No response/task not attempted

Criteria Beginning 1 Developing 2 Accomplished 3 Exemplary 4 Score

1 Description reflecting beginning level of performance

Description reflecting movement toward mastery level of performance

Description reflecting achievement of mastery level of performance

Description reflecting highest level of performance












Description reflecting beginning level of performance

Creating a task to test L2 writing Describe the population you teach Define the purpose for an L2 writing

assessment task Define L2 writing ability: Think of a TLU

domain task in which that language ability would be demonstrated

Using the framework of task characteristics, design a task that would require L2 writing ability to accomplish

Share your task with a partner

Self-Assessment I could create an Assessment Use Argument (conceptual link between

assessment & intended decisions & consequences). I can define the construct of L2 writing ability. I can articulate a conceptual framework for designing and evaluating

L2 writing assessment tasks. I can identify the steps in constructing a language test. I can evaluate strengths and weakness of some specific L2 writing

items. I would be able to create a rubric to serve the purposes of an L2

writing assessment in my program. I practiced item/task evaluation. I created an L2 writing task that would be suitable for my program.

THANK YOU

for your participation!

Technology

Wiseman Facets of L2 writing ability