Assessing WritingAssessing Writing

Christine CoombeChristine Coombe

Presentation OutlinePresentation Outline

We explore the main practical issues We explore the main practical issues that teachers often face when that teachers often face when evaluating the writing work of their evaluating the writing work of their students. students.

BackgroundBackground The field of writing assessment has The field of writing assessment has

developed considerably over the past 50 developed considerably over the past 50 years. years.

Teacher judgment has always played an Teacher judgment has always played an important role in the assessment of writing. important role in the assessment of writing.

Ts frequently ask their Ss to write on a Ts frequently ask their Ss to write on a variety of topics and then assess them on:variety of topics and then assess them on:– the message contained in the writing sample, the message contained in the writing sample, – the clarity and organization of the messagethe clarity and organization of the message– the mechanics (spelling, capitalization and the mechanics (spelling, capitalization and

punctuation) utilized (O’Malley & Valdez Pierce, punctuation) utilized (O’Malley & Valdez Pierce, 1996). 1996).

Approaches to Writing Approaches to Writing AssessmentAssessment

Two major approaches to writing Two major approaches to writing assessment have been identified in assessment have been identified in the literature: the literature: – direct direct – indirectindirect

Indirect Writing Indirect Writing AssessmentAssessment

Indirect measures of writing ability have Indirect measures of writing ability have been very commonly used over the years. been very commonly used over the years. – basically measures correct usage in sentence basically measures correct usage in sentence

level constructions and focuses on spelling and level constructions and focuses on spelling and punctuation via objective formats like MCQs punctuation via objective formats like MCQs and cloze tests. and cloze tests.

– measures determine the S’s knowledge of measures determine the S’s knowledge of writing sub skills such as grammar and writing sub skills such as grammar and sentence construction which are assumed to sentence construction which are assumed to constitute components of writing ability constitute components of writing ability

Indirect writing assessment measures are Indirect writing assessment measures are largely concerned with accuracy rather largely concerned with accuracy rather than communication. than communication.

Direct Writing AssessmentDirect Writing Assessment

Direct writing assessment measures Direct writing assessment measures the S’s ability to communicate the S’s ability to communicate through the written mode based on through the written mode based on the production of written texts. the production of written texts. – requires Ss to come up with the content, requires Ss to come up with the content,

find a way to organize the ideas and use find a way to organize the ideas and use appropriate vocabulary, grammatical appropriate vocabulary, grammatical conventions and syntax. conventions and syntax.

Direct writing assessment integrates Direct writing assessment integrates all elements of writing.all elements of writing.

Designing Writing Designing Writing Assessment Tests/TasksAssessment Tests/Tasks

According to Hyland (2003) the design of According to Hyland (2003) the design of good writing assessment tests and tasks good writing assessment tests and tasks involves four basic elements:involves four basic elements: – Rubric: the instructionsRubric: the instructions– Prompt: the task Prompt: the task – Expected response: what the teacher intends Expected response: what the teacher intends

Ss to do with the taskSs to do with the task– Post task evaluation: assessing the Post task evaluation: assessing the

effectiveness of the writing task.effectiveness of the writing task.

RubricRubric First element of a good writing assessment. First element of a good writing assessment. The rubric is basically the instructions for The rubric is basically the instructions for

carrying out the writing task. carrying out the writing task. The rubric should include information like The rubric should include information like

– the procedures for responding the procedures for responding – the task format the task format – time allotted for completion of the test/tasktime allotted for completion of the test/task– information about how the test/task will be information about how the test/task will be

evaluated evaluated Much of the information in the rubric should Much of the information in the rubric should

come from the test specification. come from the test specification.

Writing Test SpecsWriting Test Specs

Test specifications for a writing test Test specifications for a writing test should provide the test writer with should provide the test writer with – details on the topic details on the topic – the rhetorical pattern to be tested the rhetorical pattern to be tested – the intended audience the intended audience – how much information should be included how much information should be included – the number of words the student is the number of words the student is

expected to produce and overall weighting expected to produce and overall weighting (Davidson & Lloyd, 2005).(Davidson & Lloyd, 2005).

Effective RubricsEffective Rubrics Good rubrics should:Good rubrics should:

– Specify a particular rhetorical pattern, length of Specify a particular rhetorical pattern, length of writing desired and amount of time allowed to writing desired and amount of time allowed to complete task;complete task;

– Indicate resources Ss can use (dictionaries, Indicate resources Ss can use (dictionaries, spell/grammar check etc) and method of spell/grammar check etc) and method of delivery (i.e. paper and pencil, laptop, PC)delivery (i.e. paper and pencil, laptop, PC)

– Indicate whether a draft or an outline is requiredIndicate whether a draft or an outline is required– Include overall weighting of writing task as Include overall weighting of writing task as

compared to other parts of the examcompared to other parts of the exam Rubrics should be as clear and as Rubrics should be as clear and as

comprehensive as possible. comprehensive as possible.

Writing PromptsWriting Prompts Second essential part of writing test Second essential part of writing test Hyland (2003) defines the prompt as “the stimulus Hyland (2003) defines the prompt as “the stimulus

the student must respond to” (p. 221). the student must respond to” (p. 221). Kroll and Reid (1994:233) identify three main Kroll and Reid (1994:233) identify three main

prompt formats: prompt formats: – Base = state the entire task in direct and very simple Base = state the entire task in direct and very simple

terms terms – framed = present the writer with a situation that acts as a framed = present the writer with a situation that acts as a

frame for the interpretation of the taskframe for the interpretation of the task– text-based = present writers with a text to which they text-based = present writers with a text to which they

must respond or utilize in their writing must respond or utilize in their writing

The first two are the most common in F/SL writing The first two are the most common in F/SL writing assessment. assessment.

Sample Base PromptsSample Base Prompts

Do you favor or oppose a complete Do you favor or oppose a complete ban on smoking? Why? Why not?ban on smoking? Why? Why not?

Discuss the view that women are Discuss the view that women are better drivers than men.better drivers than men.

Many say that “Money is the root of Many say that “Money is the root of all evil.” Do you agree or disagree all evil.” Do you agree or disagree with this statement?with this statement?

Sample Framed PromptsSample Framed Prompts On a recent flight back home to the UAE, Emirates On a recent flight back home to the UAE, Emirates

Airlines lost your baggage. Write a complaint letter Airlines lost your baggage. Write a complaint letter to Mr. Al-Ahli, the General Manager, telling him to Mr. Al-Ahli, the General Manager, telling him about your problem. Be sure to include the about your problem. Be sure to include the following:following:– Your flight detailsYour flight details– A description of the baggage lost and its contentsA description of the baggage lost and its contents– What you would like Mr. Al-Ahli to do for youWhat you would like Mr. Al-Ahli to do for you

An announcement has been made by the An announcement has been made by the Hollywood Academy on this year’s nominations for Hollywood Academy on this year’s nominations for the Oscar award. You notice that you and most of the Oscar award. You notice that you and most of your friends have not heard of the films nominated your friends have not heard of the films nominated for the ‘Best Picture’ award. Write a letter to the for the ‘Best Picture’ award. Write a letter to the head of the Academy Awards about this and head of the Academy Awards about this and suggest and support alternative nominees.suggest and support alternative nominees.

Sample Text-based PromptsSample Text-based Prompts You have been asked by a youth travel You have been asked by a youth travel

magazine to write an article about things to magazine to write an article about things to see and do in your hometown. Using the see and do in your hometown. Using the attached set of pictures, write a one-page attached set of pictures, write a one-page article on this topic. article on this topic.

You have been put in charge of selecting an You have been put in charge of selecting an appropriate restaurant for your senior class appropriate restaurant for your senior class party. Use the restaurant reviews below to party. Use the restaurant reviews below to select an appropriate venue and then write select an appropriate venue and then write an invitation letter to your fellow classmates an invitation letter to your fellow classmates persuading them to join you there.persuading them to join you there.

Criteria of Good Writing Criteria of Good Writing PromptsPrompts

A writing prompt defines the writing task for A writing prompt defines the writing task for students. students.

It consists of a question or a statement that Ss It consists of a question or a statement that Ss will address in their writing and the conditions will address in their writing and the conditions under which they will be asked to write. under which they will be asked to write.

Each prompt you use in the assessment of writing Each prompt you use in the assessment of writing should meet the following criteria (Davidson & should meet the following criteria (Davidson & Lloyd, 2005):Lloyd, 2005):– generate the desired type of writing, genre or rhetorical generate the desired type of writing, genre or rhetorical

pattern;pattern;– get students involved in thinking and problem-solving; get students involved in thinking and problem-solving; – be accessible, interesting and challenging to students;be accessible, interesting and challenging to students;– address topics that are meaningful, relevant, and motivating; address topics that are meaningful, relevant, and motivating; – not require specialist background knowledge;not require specialist background knowledge;– use appropriate signpost verbs;use appropriate signpost verbs;– be fair and provide equal opportunities for all students to be fair and provide equal opportunities for all students to

respondrespond– be clear, authentic, focused and unambiguous;be clear, authentic, focused and unambiguous;– specify an audience, a purpose, and a context.specify an audience, a purpose, and a context.

Appropriate Signpost VerbsAppropriate Signpost Verbs Developing a good writing prompt requires appropriate Developing a good writing prompt requires appropriate

‘signpost’ terms to match rhetorical pattern you are using. ‘signpost’ terms to match rhetorical pattern you are using. Some of the most common are listed below:Some of the most common are listed below:

– Describe:Describe: give a detailed accountgive a detailed account– Discuss:Discuss: argue a thesis, identifying pros and consargue a thesis, identifying pros and cons– Explain:Explain: state and interpretstate and interpret– Compare:Compare: show similarities between two thingsshow similarities between two things– Contrast:Contrast: show the differences between two thingsshow the differences between two things– Analyze:Analyze: identify main points and evaluate themidentify main points and evaluate them– Define: Define: provide the definition and exemplifyprovide the definition and exemplify– Summarize: produce a concise account of the main ideas, Summarize: produce a concise account of the main ideas,

omitting details and examplesomitting details and examples– Outline:Outline:provide a summary of main points and sub headingsprovide a summary of main points and sub headings– Evaluate:Evaluate: appraise the worth or value of somethingappraise the worth or value of something

Expected ResponseExpected Response

Another essential element of good Another essential element of good writing assessment. writing assessment.

A description of what the teacher A description of what the teacher intends Ss to do with the writing task. intends Ss to do with the writing task.

Before communicating information on Before communicating information on the expected response to Ss, it is the expected response to Ss, it is necessary for the teacher to have a necessary for the teacher to have a clear picture of what type of response clear picture of what type of response they want the assessment task to they want the assessment task to generate.generate.

Post Task EvaluationPost Task Evaluation

Whatever way you choose to assess writing, Whatever way you choose to assess writing, it is recommended that you it is recommended that you evaluate the evaluate the effectivenesseffectiveness of your writing tasks/tests. of your writing tasks/tests.

According to Hyland (2003), good writing According to Hyland (2003), good writing tasks are likely to produce positive tasks are likely to produce positive responses to the following questions:responses to the following questions:– Did the prompt discriminate well among my Ss?Did the prompt discriminate well among my Ss?– Were the essays easy to read and evaluate?Were the essays easy to read and evaluate?– Were Ss able to write to their potential and show Were Ss able to write to their potential and show

what they knew?what they knew?

Issues in Writing Issues in Writing AssessmentAssessment

Time allocationTime allocation Process vs. productProcess vs. product Test administration conditionsTest administration conditions Topic restrictionTopic restriction Classroom teacher as raterClassroom teacher as rater Multiple ratersMultiple raters

Time AllocationTime Allocation A commonly-asked question by teachers is A commonly-asked question by teachers is

how much time should students be given to how much time should students be given to complete writing tasks. complete writing tasks. – Depends on whether it’s process or product Depends on whether it’s process or product

orientedoriented A good rule of thumb is provided by Jacobs A good rule of thumb is provided by Jacobs

et al (1981). et al (1981). – In their research on the Michigan Composition In their research on the Michigan Composition

Test, they (1981:19) state that allowing 30 Test, they (1981:19) state that allowing 30 minutes is probably enough time for most minutes is probably enough time for most students to produce an adequate sample of students to produce an adequate sample of writing. writing.

With process oriented writing or portfolios, a With process oriented writing or portfolios, a lot more time should be allocated for lot more time should be allocated for assessment tasks.assessment tasks.

Process vs. ProductProcess vs. Product

There has been a shift towards focusing on the There has been a shift towards focusing on the process of writing rather than on the written product. process of writing rather than on the written product.

Some writing tests have focused on assessing the Some writing tests have focused on assessing the whole writing process from brainstorming activities whole writing process from brainstorming activities all the way to the final draft or finished product. all the way to the final draft or finished product.

In using this process approach, students usually have In using this process approach, students usually have to to – submit their work in a portfolio that includes all draft submit their work in a portfolio that includes all draft

material. material. A more traditional way to assess writing is through a A more traditional way to assess writing is through a

product approach. product approach. – accomplished through timed essay which usually occurs at accomplished through timed essay which usually occurs at

the mid and end point of the semester the mid and end point of the semester It is recommended that teachers use a combination It is recommended that teachers use a combination

of the two approaches in their writing assessment.of the two approaches in their writing assessment.

Test Administration Test Administration ConditionsConditions

Technology has the potential to really impact on Technology has the potential to really impact on writing assessment.writing assessment.

Should Ss be allowed to use computer, spell and Should Ss be allowed to use computer, spell and grammar check, thesaurus and online dictionaries?grammar check, thesaurus and online dictionaries?– these tools would be available to them in real-life contexts. these tools would be available to them in real-life contexts.

Technological advances bring several issues to the Technological advances bring several issues to the fore. fore. – with computers, Ss can access tools like spell and grammar with computers, Ss can access tools like spell and grammar

check. check. This access could put those who write by hand at a This access could put those who write by hand at a

disadvantage. disadvantage. – Skill contamination must also be considered Skill contamination must also be considered

electronic writing assessment is also a test of keyboarding and electronic writing assessment is also a test of keyboarding and computer skills. computer skills.

Whatever delivery mode you decide to use for your Whatever delivery mode you decide to use for your writing assessments, it is important to be consistent writing assessments, it is important to be consistent with all Ss. with all Ss.

Topic RestrictionTopic Restriction A controversial and heated issue in writing assessment. A controversial and heated issue in writing assessment. Belief that all Ss should write on the same topic and no alternatives are Belief that all Ss should write on the same topic and no alternatives are

given. given. Ts feel that Ss perform better when they can select the prompt from a Ts feel that Ss perform better when they can select the prompt from a

variety of alternative topics. variety of alternative topics. – When given choices, Ss select topics that interests them and ones they When given choices, Ss select topics that interests them and ones they

have background knowledge on. have background knowledge on. – major advantage is the reduction of student anxiety. major advantage is the reduction of student anxiety. – major disadvantage is the difficulty of writing prompts at the same level major disadvantage is the difficulty of writing prompts at the same level

of difficulty. of difficulty. Testers recommend that all Ss write on the same topic because allowing Testers recommend that all Ss write on the same topic because allowing

choice introduces too much variance into the scores. choice introduces too much variance into the scores. – marker consistency may be reduced if all papers read at a single writing marker consistency may be reduced if all papers read at a single writing

calibration session are not on the same topic. calibration session are not on the same topic. – It is the general consensus within language testing community that all Ss It is the general consensus within language testing community that all Ss

should write on the topic and preferably on more than one topic. should write on the topic and preferably on more than one topic. – Research results are mixed on whether students write better with single Research results are mixed on whether students write better with single

or with multiple prompts (Hamp-Lyons, 1990). or with multiple prompts (Hamp-Lyons, 1990). Ss who are given multiple prompts may do less well than expected Ss who are given multiple prompts may do less well than expected

because Ss waste time selecting a topic instead of spending that because Ss waste time selecting a topic instead of spending that time writing. time writing.

If you do decide to allow students to select a topic from a variety of If you do decide to allow students to select a topic from a variety of alternatives, make sure your alternative topics are the same genre and alternatives, make sure your alternative topics are the same genre and rhetorical pattern. This practice will make it easier for you to achieve inter-rhetorical pattern. This practice will make it easier for you to achieve inter-rater reliability. rater reliability.

Classroom Teacher as RaterClassroom Teacher as Rater

Should classroom teachers mark their own Should classroom teachers mark their own students’ papers? students’ papers?

Divided opinions on this issue. Divided opinions on this issue. – Those who are against warn of possibility that Those who are against warn of possibility that

Ts might show bias either for or against SsTs might show bias either for or against Ss– Those who are for state that Ts know their own Those who are for state that Ts know their own

Ss bestSs best Double blind marking is the recommended Double blind marking is the recommended

ideal where no student identifying ideal where no student identifying information appears on scripts.information appears on scripts.

Multiple RatersMultiple Raters

Do we really need more than one marker for Do we really need more than one marker for student writing samples? student writing samples? – The answer is an unequivocal ‘yes’. The answer is an unequivocal ‘yes’.

All reputable writing assessment programs All reputable writing assessment programs use more than one rater to judge essays. use more than one rater to judge essays. – the recommended number is two, with a third in the recommended number is two, with a third in

case of extreme disagreement or discrepancy. case of extreme disagreement or discrepancy. – Why? It is believed that multiple judgments lead Why? It is believed that multiple judgments lead

to a final score that is closer to a “true” score than to a final score that is closer to a “true” score than any single judgment (Hamp-Lyons, 1990).any single judgment (Hamp-Lyons, 1990).

Ways to Assess WritingWays to Assess Writing

The F/SL literature generally addresses The F/SL literature generally addresses two types of writing: two types of writing: – free writing free writing

requires Ss to read a prompt that poses a situation requires Ss to read a prompt that poses a situation and write a planned response based on a and write a planned response based on a combination of background knowledge and combination of background knowledge and knowledge learned from the course. knowledge learned from the course.

– guided writing guided writing requires Ss to manipulate content that is provided in requires Ss to manipulate content that is provided in

the prompt, usually in the form of a chart or diagram.the prompt, usually in the form of a chart or diagram.

Guided WritingGuided Writing

A bridge between objective and subjective formats. A bridge between objective and subjective formats. Requires teachers to be very clear about what they expect Ss Requires teachers to be very clear about what they expect Ss

to do. to do. Decide whether mechanical issues like spelling, punctuation Decide whether mechanical issues like spelling, punctuation

and capitalization matter when the task focuses on and capitalization matter when the task focuses on comprehension. comprehension.

Some important points to keep in mind are:Some important points to keep in mind are:– Be clear about the expected form andBe clear about the expected form and length of response (one length of response (one

paragraph, a 250-word essay, a letter etc.).paragraph, a 250-word essay, a letter etc.).– If you want particular information included, clearly specify it in If you want particular information included, clearly specify it in

the prompt (i.e. three causes and effects, two supporting details the prompt (i.e. three causes and effects, two supporting details etc.)etc.)

– Specify the discourse pattern(s) the students are expected to use Specify the discourse pattern(s) the students are expected to use (i.e. compare and contrast, cause and effect, description etc.)(i.e. compare and contrast, cause and effect, description etc.)

– Be sure to ask Ss to provide something beyond the prompt such Be sure to ask Ss to provide something beyond the prompt such as an opinion, an inference, or a prediction.as an opinion, an inference, or a prediction.

– Be amenable to revising the anticipated answer even as you Be amenable to revising the anticipated answer even as you grade.grade.

Free WritingFree Writing All GW suggestions are germane to free writing. All GW suggestions are germane to free writing. Goal for Ts is to elicit comparable products from Ss of Goal for Ts is to elicit comparable products from Ss of

different ability levels. different ability levels. – Use of multiple raters is especially important Use of multiple raters is especially important – Agree on grading criteria in advance and calibrate before the Agree on grading criteria in advance and calibrate before the

actual grading session.actual grading session.– Decide on whether to use holistic, analytical or a combination Decide on whether to use holistic, analytical or a combination

of the two as a rating scale for marking.of the two as a rating scale for marking.– If using a band scale, adjust it to the task.If using a band scale, adjust it to the task.– Acquaint Ss with marking scheme in advance by using it for Acquaint Ss with marking scheme in advance by using it for

teaching, grading homework and providing feedback.teaching, grading homework and providing feedback.– Subliminally teach good writing strategies by providing Ss with Subliminally teach good writing strategies by providing Ss with

enough space for an outline, a draft and the finished product.enough space for an outline, a draft and the finished product. In ES/FL classrooms, be aware of cultural differences and In ES/FL classrooms, be aware of cultural differences and

sensitivities among students. sensitivities among students. Avoid contentious issues that might offend or disadvantage Avoid contentious issues that might offend or disadvantage

Ss.Ss.

Authentic Writing Authentic Writing AssessmentAssessment

Student-Teacher Conferences:Student-Teacher Conferences: Teachers can learn a lot about their students’ writing habits Teachers can learn a lot about their students’ writing habits

through student-teacher conferences. These conferences through student-teacher conferences. These conferences can also provide them with important assessment can also provide them with important assessment opportunities. opportunities.

Among the questions that teachers might ask during Among the questions that teachers might ask during conferences include:conferences include:– How did you select this topic?How did you select this topic?– What did you do to generate content for this writing? What did you do to generate content for this writing? – Before you started writing, did you make a plan or an outline?Before you started writing, did you make a plan or an outline?– During the editing phase, what types of errors did you find in During the editing phase, what types of errors did you find in

your writing?your writing?– What do you feel are your strengths in writing?What do you feel are your strengths in writing?– What do you find difficult in writing? What do you find difficult in writing? – What would you like to improve about your writing?What would you like to improve about your writing?

Authentic Writing Authentic Writing AssessmentAssessment

Self Assessment:Self Assessment: There are two self-assessment techniques than There are two self-assessment techniques than

can be used in writing assessment: can be used in writing assessment: – dialog journals dialog journals – learning logs learning logs

Peer AssessmentPeer Assessment– Involves Ss in the evaluation of writing. Involves Ss in the evaluation of writing. – Eases the marking burden on the teacher. Eases the marking burden on the teacher. – Ss get regular feedback on what they produce. Ss get regular feedback on what they produce. – Ss can use checklists, scoring rubrics or simple Ss can use checklists, scoring rubrics or simple

questions questions The major rationale is that when Ss learn to The major rationale is that when Ss learn to

evaluate the work of their peers, they are evaluate the work of their peers, they are extending their own learning opportunities.extending their own learning opportunities.

Authentic Writing Authentic Writing AssessmentAssessment

Portfolio Assessment:Portfolio Assessment: Examines multiple pieces of writing written over Examines multiple pieces of writing written over

time under different constraints rather than an time under different constraints rather than an assessment of a single essay written under a assessment of a single essay written under a specified time. specified time.

Definitions vary but the general consensus is Definitions vary but the general consensus is – a purposive collection of student writing over time, which a purposive collection of student writing over time, which

shows the stages in the writing process a text has gone shows the stages in the writing process a text has gone through and the stages of the writers’ growth. through and the stages of the writers’ growth.

– Increasingly, evidence of reflection is presentIncreasingly, evidence of reflection is present Key elements of portfolios are student reflection and Key elements of portfolios are student reflection and

self-monitoring. self-monitoring. Portfolios can showcase a S’s best work or display a Portfolios can showcase a S’s best work or display a

collection of both drafts and final products to collection of both drafts and final products to demonstrate progress and continued improvement. demonstrate progress and continued improvement.

Characteristics of a Characteristics of a PortfolioPortfolio

Portfolios must include student Portfolios must include student participation in four important participation in four important areas:areas:

– the selection of portfolio contents the selection of portfolio contents – the guidelines for selectionthe guidelines for selection– the criteria for judging meritthe criteria for judging merit– evidence of student reflectionevidence of student reflection

Characteristics of a Characteristics of a PortfolioPortfolio

Five characteristics typify a model portfolio Five characteristics typify a model portfolio (Moya and (Moya and O’Malley, 1994):O’Malley, 1994):

– ComprehensivenessComprehensiveness: potential for determining depth and : potential for determining depth and breath of S’s capabilities can be realized through comprehensive breath of S’s capabilities can be realized through comprehensive data collection and analysis. data collection and analysis.

– Predetermined and systematicPredetermined and systematic: a sound portfolio is planned : a sound portfolio is planned prior to implementation. This includes information like the prior to implementation. This includes information like the purpose, contents, data collection schedule and student grading purpose, contents, data collection schedule and student grading criteria. criteria.

– InformativeInformative: the information in the portfolio must be : the information in the portfolio must be meaningful to all stakeholders in the process (i.e. teachers, meaningful to all stakeholders in the process (i.e. teachers, students, staff and parents). students, staff and parents).

– TailoredTailored: an exemplary portfolio is tailored to the purpose for : an exemplary portfolio is tailored to the purpose for which it will be used to classroom goals and objectives and to which it will be used to classroom goals and objectives and to individual student assessment needs. individual student assessment needs.

– AuthenticAuthentic: A good portfolio provides student information based : A good portfolio provides student information based on assessment tasks that reflect authentic activities used during on assessment tasks that reflect authentic activities used during classroom instructionclassroom instruction

Writing Marking Writing Marking ProceduresProcedures

Reliable writing assessment requires a carefully Reliable writing assessment requires a carefully thought-out set of procedures and a significant thought-out set of procedures and a significant amount of time needs to be devoted to the rating amount of time needs to be devoted to the rating process. process.

Steps in the writing calibration process:Steps in the writing calibration process:– First, a small team of trained and experienced raters needs First, a small team of trained and experienced raters needs

to select a number of sample benchmark scripts from to select a number of sample benchmark scripts from completed exam papers. completed exam papers.

– Benchmark scripts need to be representative of the following Benchmark scripts need to be representative of the following levels at minimum:levels at minimum: Clear pass (good piece of writing that is solidly in the A/B range)Clear pass (good piece of writing that is solidly in the A/B range) Borderline pass (a paper that is on the borderline between pass Borderline pass (a paper that is on the borderline between pass

and fail but shows enough of the requisite information to be a and fail but shows enough of the requisite information to be a pass)pass)

Borderline fail (a paper that is on the borderline between pass Borderline fail (a paper that is on the borderline between pass and fail but does not have enough of the requisite information and fail but does not have enough of the requisite information to pass)to pass)

Clear fail (a below average paper that is clearly in the D/F range)Clear fail (a below average paper that is clearly in the D/F range)

Writing Marking Writing Marking ProceduresProcedures

The team of experienced raters needs to rate the scripts The team of experienced raters needs to rate the scripts using the scoring criteria and agree on a score. using the scoring criteria and agree on a score. – note down a few reasons why the script was rated in such a way note down a few reasons why the script was rated in such a way

Next, lead arbitrator conducts a calibration session (referred Next, lead arbitrator conducts a calibration session (referred to as a standardization or norming session) where the entire to as a standardization or norming session) where the entire pool of raters rate the sample scripts and try to agree on the pool of raters rate the sample scripts and try to agree on the scores that each script should receive. scores that each script should receive. – In these calibration sessions Ts should evaluate and discuss In these calibration sessions Ts should evaluate and discuss

benchmark scripts until they arrive at a consensus score. benchmark scripts until they arrive at a consensus score. These sessions are time consuming and not very popular with These sessions are time consuming and not very popular with

Ts who want to get started on the writing marking right away. Ts who want to get started on the writing marking right away.

They are also very heated especially when raters of different They are also very heated especially when raters of different educational and cultural backgrounds are involved. educational and cultural backgrounds are involved.

Despite these disadvantages, they are an essential Despite these disadvantages, they are an essential component to standardizing writing scores. component to standardizing writing scores.

Holistic Marking ScalesHolistic Marking Scales Holistic marking is based on the marker's total impression of Holistic marking is based on the marker's total impression of

the essay as a whole. the essay as a whole. – variously termed impressionistic, global or integrative markingvariously termed impressionistic, global or integrative marking– is quick and reliable if 3 to 4 people mark each scriptis quick and reliable if 3 to 4 people mark each script– mark for two hours and then rest grading no more than 20 scripts mark for two hours and then rest grading no more than 20 scripts

per hourper hour– most successful using scales of a limited range (i.e. from 0-6).most successful using scales of a limited range (i.e. from 0-6).

FL/SL educators have identified a number of advantages to this FL/SL educators have identified a number of advantages to this type of marking. type of marking. – reliable if done under no time constraints and if Ts receive training. reliable if done under no time constraints and if Ts receive training. – generally perceived to be quicker than other types of writing generally perceived to be quicker than other types of writing

assessment assessment – enables a large number of scripts to be scored in a short period of enables a large number of scripts to be scored in a short period of

time. time. – since overall writing ability is assessed, Ss are not disadvantaged since overall writing ability is assessed, Ss are not disadvantaged

by one lower component such as poor grammar bringing down a by one lower component such as poor grammar bringing down a score. score.

– An additional advantage is that the scores tend to emphasize the An additional advantage is that the scores tend to emphasize the writer’s strengths (Cohen, 1994: 315).writer’s strengths (Cohen, 1994: 315).

Holistic Marking ScalesHolistic Marking Scales Several disadvantages of holistic marking have also been Several disadvantages of holistic marking have also been

identified identified – can be unreliable if marking is done under short time can be unreliable if marking is done under short time

constraints and with inexperienced, untrained teachers (Heaton, constraints and with inexperienced, untrained teachers (Heaton, 1990). 1990).

– longer essays often tend to receive higher marks. longer essays often tend to receive higher marks. – Testers point out that by reducing a score to one figure tends to Testers point out that by reducing a score to one figure tends to

reduce the reliability of the overall mark. reduce the reliability of the overall mark. – It is also difficult to interpret a composite score from a holistic It is also difficult to interpret a composite score from a holistic

mark. mark. – The most serious problem associated with holistic marking is The most serious problem associated with holistic marking is

the inability of this type of marking to provide washback. the inability of this type of marking to provide washback. Tendency on the part of the marker to overlook the various Tendency on the part of the marker to overlook the various

sub-skills that make up writing.sub-skills that make up writing.

Hamp-Lyons (1990) has stated that holistic marking is Hamp-Lyons (1990) has stated that holistic marking is severely limited in that it does not provide a profile of the severely limited in that it does not provide a profile of the student's writing ability. student's writing ability.

Analytic Marking ScalesAnalytic Marking Scales ““raters provide separate assessments for each of a raters provide separate assessments for each of a

number of aspects of performance” (Hamp-Lyons, 1991). number of aspects of performance” (Hamp-Lyons, 1991). – In other words, raters mark selected aspects of a piece of writing In other words, raters mark selected aspects of a piece of writing

and assign point values to quantifiable criteria. and assign point values to quantifiable criteria. – has been termed discrete point marking and focused holistic has been termed discrete point marking and focused holistic

marking. marking. Analytic marking scales are generally more effective Analytic marking scales are generally more effective

with inexperienced teachers. with inexperienced teachers. Are more reliable for scales with a larger point range. Are more reliable for scales with a larger point range. A number of advantages have been identified with A number of advantages have been identified with

analytic marking. analytic marking. – provide Ts with a "profile" of their Ss' strengths and weaknesses provide Ts with a "profile" of their Ss' strengths and weaknesses

in writing. in writing. – it is very reliable if done with a population of inexperienced Ts it is very reliable if done with a population of inexperienced Ts

who have had little training and grade under short time who have had little training and grade under short time constraints (Heaton, 1990). constraints (Heaton, 1990).

Finally, training raters is easier because the scales are Finally, training raters is easier because the scales are more explicit and detailed. more explicit and detailed.

Analytic Marking ScalesAnalytic Marking Scales Educators point out a number of disadvantages with Educators point out a number of disadvantages with

this scale this scale – perceived to be more time consuming because it requires Ts to perceived to be more time consuming because it requires Ts to

rate various aspects of a S's essay. rate various aspects of a S's essay. – necessitates a set of specific criteria to be written and for necessitates a set of specific criteria to be written and for

markers to be trained and attend frequent calibration sessions. markers to be trained and attend frequent calibration sessions.

– Also, because Ts look at specific areas in a given essay, the Also, because Ts look at specific areas in a given essay, the most common being content, organization, grammar, most common being content, organization, grammar, mechanics and vocabulary, marks are often lower mechanics and vocabulary, marks are often lower

The most well-known analytic writing scale is the ESL The most well-known analytic writing scale is the ESL Composition Profile (Jacobs et al, 1981). Composition Profile (Jacobs et al, 1981). – This scale contains five component skills, each focusing on an This scale contains five component skills, each focusing on an

important aspect of composition and weighted according to its important aspect of composition and weighted according to its approximate importance: approximate importance:

content (30 points), organization (20 points), vocabulary (20 content (30 points), organization (20 points), vocabulary (20 points), language use (25 points) and mechanics (5 points). points), language use (25 points) and mechanics (5 points).

The total weight for each component is further broken down into The total weight for each component is further broken down into numerical ranges that correspond to four levels from “very poor” to numerical ranges that correspond to four levels from “very poor” to “excellent” “excellent”

Responding to Student Responding to Student WritingWriting

An important aspect of writing marking is providing An important aspect of writing marking is providing written feedback written feedback – is essential in that it provides opportunities for Ss to learn and is essential in that it provides opportunities for Ss to learn and

make improvements to their writing. make improvements to their writing. Probably the most common type of written teacher Probably the most common type of written teacher

feedback is handwritten comments on the Ss’ papers. feedback is handwritten comments on the Ss’ papers. – usually occur at the end of the paper or in the margins. usually occur at the end of the paper or in the margins. – Some Ts like to use correction codes to provide formative Some Ts like to use correction codes to provide formative

feedback feedback – These simple correction codes facilitate marking and minimize These simple correction codes facilitate marking and minimize

the amount of “red” ink on student writings. the amount of “red” ink on student writings. Advances in technology provide us with another way of Advances in technology provide us with another way of

responding to student writing. responding to student writing. – Electronic feedback is particularly valuable because it can be Electronic feedback is particularly valuable because it can be

used to give a combination of handwritten comments and used to give a combination of handwritten comments and correction codes. correction codes.

– Can easily provide commentary and insert corrections through Can easily provide commentary and insert corrections through Microsoft Word’s track changes facility and through programs Microsoft Word’s track changes facility and through programs like Markin. like Markin.

Research on Written Research on Written FeedbackFeedback

Research indicates that teacher Research indicates that teacher written feedback is highly valued by written feedback is highly valued by second language writers second language writers – many Ss particularly value feedback on many Ss particularly value feedback on

their grammar (Leki, 1990). their grammar (Leki, 1990). – positive remarks are motivating and positive remarks are motivating and

highly valued by Ss highly valued by Ss – too much praise or positive commentary too much praise or positive commentary

early on in a writer’s development can early on in a writer’s development can make students complacent and make students complacent and discourage revisiondiscourage revision

Sample Correction CodesSample Correction Codes

spsp SpellingSpelling

vtvt Verb tenseVerb tense

wwww Wrong wordWrong word

wvwv Wrong verbWrong verb

☺☺ Nice idea/content!Nice idea/content!

₪₪ Switch placementSwitch placement

¶¶ New paragraphNew paragraph

?? I don’t understandI don’t understand

10 Things to Remember…..10 Things to Remember….. 1.1. Give students multiple writing assessment Give students multiple writing assessment

opportunitiesopportunities. . Provide plenty of opportunities for Ss at all levels to practice the Provide plenty of opportunities for Ss at all levels to practice the

type of writing that you expect them to do on the writing test. type of writing that you expect them to do on the writing test. 2.2. Test a variety of writing skills and create tasks of Test a variety of writing skills and create tasks of

varying lengths.varying lengths. Take more than one sample of a students’ writing. This reduces the Take more than one sample of a students’ writing. This reduces the

variation in performance that might occur from task to task. variation in performance that might occur from task to task. 3.3. Develop prompts that are appropriate for the Ss.Develop prompts that are appropriate for the Ss. Make sure that the prompts you select or develop invite the desired Make sure that the prompts you select or develop invite the desired

type of writing. They should be realistic, sensitive to the cultural type of writing. They should be realistic, sensitive to the cultural background of the student. Choose subjects which are within the background of the student. Choose subjects which are within the realm of experience of your Ss.realm of experience of your Ss.

4.4. Evaluate all answers to one question before going on Evaluate all answers to one question before going on to the next.to the next.

This practice prevents a shifting of standards from one question to This practice prevents a shifting of standards from one question to the next and helps the rater mark more consistently. the next and helps the rater mark more consistently.

5.5. Mark only what the student has writtenMark only what the student has written. . Don’t be influenced by other factors in addition to the quality of the Don’t be influenced by other factors in addition to the quality of the

work like the quality or legibility of the handwriting.work like the quality or legibility of the handwriting.

10 Things to Remember…..10 Things to Remember….. 6. Have a systematic approach for dealing with marking 6. Have a systematic approach for dealing with marking

discrepancies.discrepancies. One such approach might be to take the average of the two raters for One such approach might be to take the average of the two raters for

a small discrepancy and to utilize a third rater if there is a big a small discrepancy and to utilize a third rater if there is a big discrepancy.discrepancy.

7.7. Get students involved.Get students involved. Get Ss involved in both the development and marking of their writing Get Ss involved in both the development and marking of their writing

tests. tests. 8.8. Provide students with diagnostic feedback.Provide students with diagnostic feedback. Use writing assessment results to identify what Ss can and cannot do Use writing assessment results to identify what Ss can and cannot do

well and make sure to provide this information to Ss. With analytic well and make sure to provide this information to Ss. With analytic marking you will have access to a profile to give Ss feedback. With marking you will have access to a profile to give Ss feedback. With holistic marking scales, be sure to take notes on Ss’ strengths and holistic marking scales, be sure to take notes on Ss’ strengths and areas for improvement. areas for improvement.

9.9. Practice blind or double blind marking.Practice blind or double blind marking. Mark essays without looking at Ss’ names as the general impression Mark essays without looking at Ss’ names as the general impression

we have of our Ss is a potential form of bias. we have of our Ss is a potential form of bias. 10.10. Calibrate and recalibrate.Calibrate and recalibrate. The best way to achieve inter-rater reliability is to practice. Start early The best way to achieve inter-rater reliability is to practice. Start early

in the academic year by employing the marking criteria and scale in in the academic year by employing the marking criteria and scale in non-test situations. Make Ss aware from the outset of the criteria and non-test situations. Make Ss aware from the outset of the criteria and expectations for their work. expectations for their work.