23
Do Longer Papers Mean Higher Scores? Tom Beery [email protected] and John Fallon [email protected] English Faculty Rhodes State College, Lima, OH USA Hello from Ohio!

Do Longer Papers Mean Higher Scores?

  • Upload
    lula

  • View
    42

  • Download
    0

Embed Size (px)

DESCRIPTION

Hello from Ohio!. Tom Beery [email protected] and John Fallon [email protected] English Faculty Rhodes State College, Lima, OH USA. Do Longer Papers Mean Higher Scores?. Quick Exercise: You Be the Grader!. Directions. - PowerPoint PPT Presentation

Citation preview

Page 1: Do Longer Papers Mean Higher Scores?

Do Longer Papers Mean Higher Scores?

Tom Beery [email protected] John Fallon [email protected] English Faculty Rhodes State College, Lima, OH USA

Hello from Ohio!

Page 2: Do Longer Papers Mean Higher Scores?

Quick Exercise: You Be the Grader!

Page 3: Do Longer Papers Mean Higher Scores?

3

Directions Read the prompt and take two minutes to grade the

paper

Holistic scoring: Give the paper one score on a scale 1 (very low) – 8 (very high); no half scores.

When scoring, you may generally consider such traits as:

▪ Organization▪ Development▪ Critical thinking▪ Style▪ Mechanics

Page 4: Do Longer Papers Mean Higher Scores?

4

Now for our session

In 2005, SAT and ACT add a writing sample to their tests.

Writing experts say this is no way to test writing ability.

Page 5: Do Longer Papers Mean Higher Scores?

5

One-time writing assessments• Are not representative of the writing

“process”: brainstorming, drafting, revising, editing, and proofreading

• Are not supported by the Conference on College Composition and Communication

Page 6: Do Longer Papers Mean Higher Scores?

6

CCCC Position Statement on Writing Assessment (revised 2009)

“Any individual's writing ability is a sum of a variety of skills employed in a diversity of contexts….”

“One piece of writing—even if it is generated under the most desirable conditions—can never serve as an indicator of overall writing ability, particularly for high-stakes decisions.”

Page 7: Do Longer Papers Mean Higher Scores?

7

Adding a one-time writing sample to the SAT and ACT…

Adds time to the test Fosters changes in classroom pedagogy Doesn’t equate to writing ability (CCCC) Leads to inaccurate evaluations (e.g.,

test procedures)

Page 8: Do Longer Papers Mean Higher Scores?

8

Scoring the SAT: PerelmanDr. Les Perelman found a 90% correlation between paper length and score:

“I have never found a quantifiable predictor in 25 years of grading that was anywhere as near as strong as this one is.”

“Write long, badly, and prosper!”

Page 9: Do Longer Papers Mean Higher Scores?

9

Perelman’s recommendations

1. High stakes testing is too important to leave to private testing agencies

2. Have students write two essays over the course of a day

3. Gather graders together for a weekend to grade (e.g., calibrate graders on the rubric)

Page 10: Do Longer Papers Mean Higher Scores?

10

Was Perelman Right?

Is paper length correlated to paper score?

Page 11: Do Longer Papers Mean Higher Scores?

11

Our Study•We needed someone to take the test

Regular and CP high school juniors from Ohio Hi Point Career Center in Bellefontaine, OH

•We needed someone to grade the test

20 teachers: 8 college and 12 high school

Page 12: Do Longer Papers Mean Higher Scores?

12

Next Step: Training Day • Give 12 teachers two hours of rubric training

• Divide into 6 teams – 3 high school and 3 college

• Assign each team one rubric trait

• Discuss proposed revisions

• Rewrite rubric to reflect consensus changes

Page 13: Do Longer Papers Mean Higher Scores?

13

Our Rubric vs. Theirs: Essentially, the Same

Our Rubric SAT Rubric ACT Rubric

Critical thinking Critical thinking Express judgments by taking a position on the issue in the writing prompt

Organization Organization Organize ideas in a logical way

Maintain a focus on the topic throughout the essay

Development Development Develop a position by using logical reasoning and by supporting their ideas

Style

Use of language and vocabulary Use language clearly and effectively according to the rules of standard written EnglishMechanics

Sentence structure Grammar and usage

Scored 1-8; two readers; trait scores are averaged and totaled

Scored holistically 1-6; two readers; scores are averaged

Scored holistically 1-6; two readers; scores are combined

Page 14: Do Longer Papers Mean Higher Scores?

14

Procedures

All 10 teams scored all 42 papers.

Did longer papers receive higher scores?

Yes, longer papers received higher scores.

Page 15: Do Longer Papers Mean Higher Scores?

15

Score by Word Count51-159 Words Average score 15

160-267 Words Average score 21

268-375 Words Average score 24

376-477 Words Average score 26

Page 16: Do Longer Papers Mean Higher Scores?

16

Scatterplot: Word Count & Score

The more the points cluster tightly about the line, the higher the magnitude of the correlation (between word count and score). We have a tight cluster!

Page 17: Do Longer Papers Mean Higher Scores?

17

Scoring Exercise for Paper #7

Room side A

________

Room side B

________

Page 18: Do Longer Papers Mean Higher Scores?

18

Paper #7: A Microcosm1. If a paper even looks longer, it will receive a

higher grade.

2. Trained readers (6 teams) scored this paper 17.3

3. Untrained readers (4 teams) scored this paper 23.25

4. Rubric-trained teams and rubric untrained teams scored very differently.

Page 19: Do Longer Papers Mean Higher Scores?

19

Mean Team26.4048 High School Untrained 24.0238 High School Untrained19.9881 College Untrained 19.3690 College Untrained   18.7143 High School Trained17.4881 College Trained16.3452 College Trained16.0000 High School Trained15.2381 High School Trained15.2262 College Trained

Page 20: Do Longer Papers Mean Higher Scores?

20

Trained vs. Untrained

This chart looks at the strength of the relationship between word count and paper score. A correlation of + 1 is a perfect positive correlation.

Levels of Magnitude

.1 small

.3 medium

.5 large

Page 21: Do Longer Papers Mean Higher Scores?

21

Options for Rubric Training

Rubrics may be used for two entirely separate activities: evaluation and assessment.

1.Evaluation: Use of a trait-scoring rubric would help eliminate biases in teacher evaluation (risking one trait being too prominent in how a reader scores the whole paper, e.g., the students can’t spell; therefore, they can’t write).

2.Assessment: Using a trait rubric for assessment in the classroom would promote discussion for what specifically makes “good” writing before the students actually write.

Page 22: Do Longer Papers Mean Higher Scores?

22

Final Observations

1. Grading with a rubric does not ensure standards.

2. Even just a couple hours of rubric training is enough to norm evaluators to a standard.

3. Moreover, rubrics can be used for two separate purposes: evaluation and assessment.

4. Perelman was right. Word count does influence paper score; however, so does rubric training.

Page 23: Do Longer Papers Mean Higher Scores?

Thank you!

Questions?