Essays

This presentation was developed for the exclusive use of students enrolled in:

Educational Testing & Grading, Professor Gregory E. Stone.

© 2004 Gregory E. Stone. All rights reserved. This presentation may not be reproduced in any form, in part or as a whole, without the express written permission of the author.

Essays& other non-objective items

We’ll include:

Essays, Short answers andother items that can fitwithin the general parameters

Mathematical problem solving

Other partial credit items

Allow for free expressionwithout artificial restriction

and without prompting.

Extended Response

Restricted Response

Two Fundamental Types:

Restricted Response

Limit the ways in which you willpermit the students to answer.

Restricted Response

Limit the ways in which you willpermit the students to answer.

� There ARE correct answers. � We allow students to express

the answer in their own words.

Restricted Response

“Write a brief essay comparing andcontrasting the term analysis and

synthesis as they relate toconstructing (a) objective items

and (b) essay items.”

example

Restricted Response

“What is the poet’s attitudetoward literature as it isapparent in lines 1 to 8?what words in these linesmake this apparent.

example

Restricted Response

“A car traveling 50 mph leaves Chicago at 9am. A train traveling at 70 mph leaves Milwaukee at 10 am. Who will arrive in Toledo (250 miles away) first? Show your work.”

example

Short Answer

Restricted Response

Predicated on the notion that students supply the answers rather than selecting the answer from a group options.

Extended Response

Allow students to express their ownideas and interrelationships amongideas and use their own strategy fororganization.

� No “correct” answers but reasonable/un logic..

Extended Response

Sometimes people write just for the fun of it. This is a chance for you to have some fun writing. Pretend that you are a pair of tennis shoes. You’ve done all kinds of things with your owner in all kinds of weather. Now you are being picked up again by your owner. Tell what you, as the tennis shoes, think about what’s going to happen to you. Tell how you feel about your owner. Space it provided below and on the next page.

example

Extended Response

Devise a plan to determine whether the democrats or republicans are evenly distributed throughout the city, or whether the supporters of each party are concentrated in certain wards.

Design an experiment to calculate the height of a redwood tree.

Create an original story.

example

Extended Response

Because the focus is on logicalargument and reasoned answeringthe teacher must be open andaccepting of uncomfortable responses.

Extended Response

Based on the creative process of the student - often including reasoning and factual presentations.

Essays have a unique ability to assess multiple taxonomic levels at once - while

maintaining a good understanding of strengths and weaknesses.

Look at the family photo attached. Describe the mood or feeling in the photo as well as the body

language of the people. Use metaphors or similes to make these descriptions.

Analysis Knowledge

Comprehension

Is your essay a good essay?

Checklist

1. Does the item test an important aspect of the learning objective?

Essays should ideally be targeted at no less than Application!


Checklist

2. Does the item match the learning objective in terms of performance, emphasis and “number of points” (on the exam)?


Checklist

3. Does the item require the students to apply their knowledge and skill to solve a new and novel problem?Essays should ideally be targeted at no less than Application!


Checklist

4. When viewed in relation to the other items on the test, does this item measure new information covering the range of content and behavior specified in the learning objectives?


Checklist

5. Is the item focused? Does it define a task with specific directions, rather than leave the assignment so broad that almost any answer would be acceptable?


Checklist

6. Is the task defined by the item within a level of complexity that is appropriate for the intellectual maturity of the students?


Checklist

7. To get a good grade on the item, is the student required to demonstrate more than facts, ideas, lists, definitions, generalizations, etc.


Checklist

8. Is the item worded in a way that leads all students to interpret the assignment in the way you intended.


Checklist

9. Does the wording of the item make clear to the students: Length, purpose, time needed and the basis for evaluation?


Checklist

10. If the essay asks students to state and support their opinions on controversial matters, does the wording of the item make it clear that students’ assessments will be based on the logic and evidence supporting their arguments, rather than on the actual position or opinion?

Creating scoring rubrics

Essential for ensuring fair,reliable, valid and meaningfulassessments.

One of the hardest problems!

Rubrics come in 2 varieties:

Holistic (logic of argument)

Analytic (correct answersprovided in the student’s ownwords)

Holistic (logic of argument)

Best used for Extended Response Essays

Grading essays on how well the argument/essay is written in

general.

Analytic (correct answers)

Best used for Restricted Response Essays

Grading essays on whether or not the student has provided the

correct answer.

Analytic rubrics require thestudent to provide specific

information in their answerin order to receive credit.

No “bluffing” allowed!

To construct a rubric:

(1)Ask - what is your objective?(2)Ask - what is the taxonomy?(3)Ask - what pieces of the puzzle do I expect the student to provide?

Example: Comprehend Vertical Leap

Concepts: Potential/Kinetic energy

Describe the concepts of Potential and Kinetic energy. Give an example of each in your description.

Describe the concepts of Potential and Kinetic energy. Give an example of each in your description.

1. Student correctly defined terms 2 points: Answer was clear and fully correct. 1 point: One or more answers were nearly correct, but student missed key concept. 0 points: Student failed to provide correct answer.2. Examples were clear and correct 2 points: Both examples were appropriate. 1 point: One example was appropriate. 0 points: Neither example was appropriate.

Creating rubrics is essential.

Using the rubrics in a standard, systematic fashion is vital!

The loss of reliability and validity are the leading causes of essay

derailment.

May be caused by poorly written essays; or

May be caused by poor use of the created rubrics.

Inconsistent Standards

Raters of essays may vary greatly. Individuals may grade essays differently on one day versus the next.

(Reliability loss)

Rater Drift

Raters tend to either fail to pay attention to criteria or change them - midstream - over time.

(Reliability loss)

Changes in Topic and Prompt

Raters often grade essays of different topics differently. (e.g. I like this topic more so I’m easier)

(Validity loss)

Reader Bias

Halo Effect

Grading based on non-graded topics. (e.g If the student uses good grammar, grade the essay higher, even if the student doesn’t fully answer the item)

Reader Bias

Carryover Effect

If question 1 was answered poorly, the rater already has a bad feeling about the student when theybegin grading question 2.

Or vice versa!

Scoring Suggestions!

You’ve created a rubric, so USE it!

Score one item at a time.

Score essays “anonymously”

Score subject-matter correctness separately.

Provide your students with feedback!

Essays& other non-objective items