Designing Writing Assessments and Rubrics - larc.sdsu.edu · Web viewAn Assignment: Before the webinar on Thursday, please read the essay below and then rate it using the. holistic

Designing Writing Assessments and Rubrics LARC/CALPER Testing & Assessment WebinarThursday, November 14, 2013

An Assignment:

Before the webinar on Thursday, please read the essay below and then rate it using theholistic rubric on page 2. During the webinar, we’ll poll the audience to see what scores

everyone gave the essay. Although I generally teach the rubric and norm raters before I ask raters to use the rubric, this exercise will provide some experience with using rubrics for those who have not used them in the past.

The essay is in response to a no-longer-used prompt on a timed-writing placement test. Students taking the test would be placed into one of three second language writing options. The assignment read as follows:

Directions: You have forty-five (45) minutes to write an essay. Imagine that you have a friend from your home country who is preparing to visit the U.S. for the first time. In a well-developed essay, describe the advice that you would offer him/her. What would you tell your friend to expect? What would you share from your own experience that you think your friend would find helpful? What do you think your friend will find most surprising?

The Essay:

When my friend preparing to visit the U.S for the first time I will tell he/she to expect that the

people in the U.S. are very busy, and the school in the U.S are long. Most of the school and work

time are spend on the week day and most of the free time are spend on weekend to cook or play

around the park. From my experience in the U.S. that younger children are very helpful stay in

the U.S. because he/she can learn English faster and new culture to know about. The teacher in

the U.S. are so friendly and have a lot of specials program for non-speak English, for that was

very helpful. Also he/she can experience that a lot of difference people from out of the country

and learn difference culture he/she didn’t know about it. Well I think the most surprising the

he/she are very happy to know the special program in the U.S. and a lot of difference people and

culture he/she never meet and know. And finally I hope he/she like what ever he/she think most

helpful for he/she in the future.

Deborah Crusan ~ Wright State University ~ Dayton, [email protected]

1


The Rubric:

This is a holistic rubric. For in-class assessment, an analytic rubric is more useful, but since the essay you have been given to assess is an essay written for placement, the best way to assess it is with a holistic rubric.

6-- Demonstrates clear competence in writing on both the rhetorical and syntactic levels, though it may have occasional errors.

A paper in this category:* effectively addresses the writing task

* is well organized and well developed * uses clearly appropriate details to support a thesis or illustrate ideas * displays consistent facility in the use of language * demonstrates syntactic variety and appropriate word choice

5-- Demonstrates competence in writing on both the rhetorical and syntactic levels, though it will probably have occasional errors.

A paper in this category:* may address some parts of the task more effectively than others

* is generally well organized and developed * uses details to support a thesis or illustrate an idea * displays facility in the use of language * demonstrates some syntactic variety and range of vocabulary

4-- Demonstrates minimal competence in writing on both the rhetorical and syntactic levels.A paper in this category may reveal one or more of the following weaknesses:

* addresses the writing topic adequately but may slight parts of the task * is adequately organized and developed * uses some details to support a thesis or illustrate an idea * demonstrates adequate but possibly inconsistent facility with syntax and usage * may contain some errors that occasionally obscure meaning

3-- Demonstrates some developing competence in writing, but it remains flawed on either the rhetorical or syntactic level, or both.

A paper in this category may reveal one or more of the following weaknesses:* inadequate organization or development

* inappropriate or insufficient details to support or illustrate generalizations * a noticeably inappropriate choice of words or word forms * an accumulation of errors in sentence structure and/or usage

2-- Suggests incompetence in writing.A paper in this category is seriously flawed by one or more of the following weaknesses:

* serious disorganization or underdevelopment * little or no detail, or irrelevant specifics * serious and frequent errors in sentence structure or usage * serious problems with focus

1-- Demonstrates incompetence in writing.A paper in this category:

* may be incoherent * may be undeveloped * may contain severe and persistent writing errors

http://www.etstechnologies.com/html/TOEFLscoringguide.htm


2


The Advantages of Using Rubrics for the Assessment of Writing

Rubrics allow for more objective and consistent evaluation.

Rubrics necessitate the clarification of teacher criteria.

Rubrics clearly show students ways in which their work will be evaluated and what is expected of them.

Rubrics raise student awareness regarding peer assessment criteria.

Rubrics provide feedback to teachers regarding instruction effectiveness.

Rubrics provide benchmarks upon which to measure and document progress.

Rubrics provide all students with an opportunity to succeed at some level.

Common Features of Rubrics

Rubrics focus on measuring stated objectives or outcomes.

Rubrics rate student performance using a range.

Rubrics include explicit performance characteristics arranged in levels indicating the degree to which a standard has been met.

Rubrics should be easy to understand and use.

How to Create Rubrics

Develop the goals for your course and daily class meetings.

Select assessment tasks that provide data aligned with your goals.

Develop performance standards for each of the goals.

Differentiate performances (categories) based on well-described criteria.

Rate (assign weight or value to) the categories.


3


Holistic rubric:

The holistic rubric is generally a five or six point scale which involves ranking a sample of a student’s writing by matching it with another piece in a benchmark series, scoring it for the prominence of certain features (coherence, development) important to that kind of writing, and assigning it a grade. Holistic rating is an impressionistic rating of essays for overall quality and is quite often used for placing students in composition classes and for assessing writing in large-scale assessments such as the TOEFL and the SAT. Interestingly, even numbered rubrics are preferred because they force the evaluator to place the work at above or below the middle.

Crusan, 2010aAdvantages:

fast evaluation relatively high inter-rater reliability scores represent ‘standards’ easily interpreted by lay persons scores tend to emphasize writer’s strengths applicable to writing across disciplines

Disadvantages:

One score masks differences across the subskills within each score No diagnostic information available (no washback potential) Scale may not apply to all genres of writing equally Raters need to be extensively trained to use scale accurately

Brown, 2004, p. 242

The rubric you used to score the essay is a classic example of a holistic rubric.


4


Analytic Rubric:

An analytic rubric includes a more detailed analysis, usually based on a scale or checklist of prominent features or characteristics of a piece of writing. The features selected for evaluation vary according to the context of the specific writing assignment, the audience, and the purpose for writing. One of the most prominent analytic rubrics is the ESL Composition Profile (Jacobs, Hartfiel, Hughey, & Wormuth, 1981). Another equally excellent analytic rubric, a more up-to-date adaptation of the ESL Composition Profile, is available in Ferris & Hedgcock (2013).

Crusan, 2010a

Advantages: Offers more washback More in-depth Offers feedback on more aspects of student writing

Disadvantages Numerical scores alone not sufficient for helping students become better

writers More time consuming to create than holistic More difficult to create an analytic rubric Genre dictates variation in scoring


5



6


Primary Trait Rubric:

A primary trait rubric analyzes certain language use and scores for one trait (e.g. organization, the correct use of the past tense). Useful for assessing L2 learners’ proficiency ad growth in writing; they are potentially more informative than the other types of rubrics because they are more focused and are particularly helpful in the classroom for diagnosing student writing ability.

Crusan, 2010aAdvantages:

Focus on one trait or skill at a time May be more informative for writers, especially at the beginning level Helpful in the classroom

Disadvantages: Probably not fair to base grades on these rubrics Cannot be applied to other tasks without revision

PRIMARY TRAIT RUBRIC EXAMPLE

Primary Trait: Vocabulary Use in Writing0 Shows no control over vocabulary taught; not enough writing to evaluate1 Shows limited control of vocabulary taught in class; appropriate use is

infrequent2 Shows control of an adequate range of the vocabulary taught in class and most

often uses this vocabulary appropriately3 Shows control of a wide range of the vocabulary taught in class and always uses

vocabulary appropriately

Adapted from http://www.carla.umn.edu/assessment/vac/Evaluation/rubrics/types/interpersonal.html

In Assessment in the second language writing classroom (Crusan, 2010a), you will find examples of assignments and their accompanying rubrics.


7

http://www.carla.umn.edu/assessment/vac/Evaluation/rubrics/types/interpersonal.html


Some Helpful Websites

Rubric Builder http://landmark-project.com/classweb/tools/rubric_builder.php3This builder allows you to enter your own criteria.

Rubric, Rubrics, Teacher Rubric Makers http://www.teach-nology.com/web_tools/rubrics/Here you will find a paragraph writing rubric maker, a persuasive writing rubric maker, and a writing rubric maker along with dozens of other rubric generators. However, you cannot plug in your own criteria. But it’s a start.

Teaching Writing http://humanities.byu.edu/elc/teacher/teaching_writing/Teaching_Writing.htmlContains links to a Writing Scoring Guide and an exercise for students in which they read “bad” essays and grade them using a teacher-designed rubric.

IMPORTANT: While these websites offer helpful tips and ease of creation, there are best used as a starting point. The best rubrics are those that have been created for and tailored to one assignment. After all, we are asking our students to do different things when they write a literacy narrative or a summary or an evaluation.


8

http://humanities.byu.edu/elc/teacher/teaching_writing/Teaching_Writing.html

http://www.teach-nology.com/web_tools/rubrics/

http://landmark-project.com/classweb/tools/rubric_builder.php3


SOME References

Broad, B. (2003). What we really value: Beyond rubrics in teaching and assessing writing. Logan, UT: Utah State University Press.

Brown, H. D. (2004). Language assessment: Principles and classroom practices. White Plains, NY: Pearson Education, Inc. (Chapter 9)

CCCC Committee on Assessment. (2006). Writing Assessment: A Position Statement. Retrieved from http://www.ncte.org/cccc/announcements/123784.htm.

CCCC Committee on Second Language Writing. (2009). CCCC statement on second language writers and writing. Retrieved from http://www.ncte.org/cccc/resources/positions/secondlangwriting.

Crusan, D. (2010a). Assessment in the second language writing classroom. Ann Arbor, MI: University of Michigan Press.

Crusan, D. (2010b). Assess thyself lest others assess thee. In P. K. Matsuda, & T. Silva (Eds.), Practicing theory in second language writing (pp. 245-262). West Lafayette, IN: Parlor Press.

Crusan, D. (2011). The promise of directed self-placement for second language writers. TESOL Quarterly, 45(4), 774–780.

Crusan, D. (2013). Assessing writing. In A. J. Kunnan (Ed.), The companion to language assessment. Volume 1: Abilities, Contexts, Learners (pp. 201-215). Hoboken, NJ: Wiley/Blackwell.

Ferris, D. (2011). Treatment of error in second language student writing, 2nd ed. Ann Arbor, MI: University of Michigan Press.

Ferris, D., and Hedgcock, J. (2013). Teaching L2 composition: Purposes, process, and practice, 3nd ed. New York: Routledge.

Hamp-Lyons, L. & Kroll, B. (1996/2001). Issues in ESL writing assessment: An overview. In T. Silva & P. K. Matsuda (Eds.), Landmark essays on ESL writing (pp. 225-240).Mahwah, NJ: Lawrence Erlbaum.

Huot, B. (2002). (Re)Articulating writing assessment for teaching and learning. Logan, UT: Utah State University Press.

Jacobs, H. L., Zingraf, S. A., Wormuth, D. R., Hartfiel, V. F., & Hughey, J. B. (1981). Testing ESL composition: A practical approach. Rowley, MA: Newbury House.

O’Malley, J. M. & Pierce, L. V. (1996). Authentic assessment for English language learners: Practical approaches for teachers. New York: Addison-Wesley.


9

http://www.ncte.org/cccc/resources/positions/secondlangwriting

http://www.ncte.org/cccc/announcements/123784.htm


Weigle, S. C. (2002). Assessing writing. Cambridge: Cambridge University Press.

Weigle, S. C. (2007). Teaching writing teachers about assessment. Journal of Second Language Writing 16, 194-209.

White, E. M. (1999). Using Scoring Guides to Assess Writing. In Richard Straub (Ed.), A Sourcebook for Responding to Student Writing (pp. 203-212). New Jersey: Hampton Press.


10

Documents

Designing Writing Assessments and Rubrics - larc.sdsu.edu · Web viewAn Assignment: Before the webinar on Thursday, please read the essay below and then rate it using the. holistic