Assessment FOR Learning: Rubric Development

Transforming Our Teaching And Learning

Module 2

Do you hear this at your school?

Why did I get a B for this assignment?

You didn’t tell us that spelling counts.

Will this be on the test?

Why do we have to learn this?

My friend said my “A” in America History isn’t the same as his “A”.

Purpose for Module 2

•Give teachers an understanding of formative assessment

•Present a framework and tools for creating and using rubrics for assessing student learning

Introduction Professional

Development Modules Hawaii Standards Implementation Process

Six Step Model Identify Relevant

Standards Determine Acceptable evidence & criteria

Determine Learning Experiences

Teach and Collect Evidence

Assess student work/inform instruction

Evaluate/judge/ communicate learning results

#1. Standards Implementation Process

HCPS Toolkit:

Curriculum Frameworks

Instructional Guide

:

The process

6 steps

Cyclical Nature

Built on system of standards

#2 Assessment FOR Learning: Rubric Development

HCPS Toolkit:

Curriculum Framework

Instructional Guide

Learning driven by

standards

Know when

learning has been achieved

Know what learning looks like

Evidence aligned to curriculum and instruction

Use information for goal setting

#3 Curriculum Mapping

HCPS Toolkit: Pacing Guides Instructional Guide

Standards Pacing Guides as a resource

Curriculum maps

and assessment data

School and

district reform approach

Collect data

regarding actual teaching

#4. Standards-Based Grading and Reporting

HCPS Toolkit RC FAQs/Website Parent Brochure

Assessment Grades

Based on Standards

Alignment with

standards based instruction and assessment

Grading

Practices Purpose for

Grading Special

Needs

Guidelines HI Implementation

Desired Outcomes

Teachers will review their understanding of quality assessment and its critical role in instruction.

Teachers will know how to identify and develop appropriate criteria for use in rubrics and checklists.

Teachers will know a process to develop performance assessment rubrics that are valid and reliable and inform instruction.

Identify relevant content standards

Determine acceptable evidence and criteria

Determine learning experiences that will enable students to learn what they need to

know and to do

Teach and collect evidence of student learning

Assess student work to inform instruction or use data to provide feedback

Evaluate student work and make judgment on learning results and

communicate findings

Reteach, or repeat the process with the next set of relevant standards

Teach

er c

olla

bora

tion

thro

ug

hou

t the

pro

cess.

Stu

den

t involv

em

en

t thro

ug

hou

t the

pro

cess.

Adapted from WestEd’s Learning from Assessment

Assessment FOR Learning Journey

= Rest Stop, time for reflection

Key 1: Purpose

Key 3: SoundDesign

Key 4: Communicat

ionRubrics

Key 2: Targets

Key 5: Student Involvement

Key Concepts

•The best performance assessment tasks are interesting, worthwhile activities that relate to instructional outcomes and allow students to demonstrate what they know and can do.

•Sound criteria should truly reflect the nature of quality in a product or performance.•Rubrics communicate in concrete and observable terms what the school values most. •Training students to develop and use rubrics to score their own work can be a powerful instructional tool.•The validity of generalizations we can make about student learning is increased when assessments are reliable, fair, and linked to instruction.

Agenda

•Assessment FOR Learning: 5 Keys

•Purpose: Seven Strategies

•Target-Method Match

•Deconstructing Standards

•Communication: Descriptive Feedback

•Student Involvement

•Reliability/Validity

ActivityActivityPart 1 Assessment OF

Learning or

Assessment FOR

Learning?Think of every type of

assessment used in your

classroom or at your

school. Write them on

post-its (one per post-it)

and set them aside.

With a partner:

Think of a negativenegative experience you’ve had being assessed. What made it negative?What made it negative? What was the effectthe effect on you?

Now, think of a positivepositive experience. What made it positive?What made it positive?What was the effectthe effect on you?

Negative Experiences

Causes Not clear what was to be tested Feedback method embarrassing Feedback too late to do any good Incomprehensible feedback/no

feedback Results didn’t reflect knowledge Trick questions No chance to improve/

one shot do or die Personal attack Time limits

Effects Stopped trying Never talked in class

again Changed majors Never took another

class in this subject Embarrassment Anger Redoubled efforts

Positive ExperiencesCauses Clear what was to be tested Criteria for success were clear Feedback could be used to improve

performance Personalized feedback Practice similar to assessment even

if not graded Step by step learning aligned with

assessment Questions were understandable Chance to improve

Effects Felt successful Wanted to take

more classes on this subject

Felt encouraged to keep trying

Knew what it took to succeed

Motivated to learn Redoubled efforts

Assessment Philosophy

We believe that assessment can do more than merely document student achievement. We believe that assessment, if approached properly, can nurture the very achievement that we are trying to document.

Judy Arter and Bill Savard

CLASSROOM ASSESSMENT

Formative Assessment

(for Learning)

Summative Assessment

(of Learning)

Purpose

Performance Assessment

Selected Response

Personal Communicatio

nEssay

Type

Purposes: Assessment OF/FOR Learning

DVD: A Hopeful Vision of the Future

Use handout #1 to take notes

ActivityActivityAssessment OF

Learning or

Assessment FOR

Learning?View the Rick Stiggins

video and complete

Figure 2.1. After

viewing the video, place

your assessment post-its

where they belong on the

FOR and OF chart in your

handout packet

Part 2

What we choose to evaluate and how we choose to evaluate delivers powerful

messages to students about those things we value. Students view their learning

and their sense of worth through the lens we help them construct unless they

cannot bear to look through it. “

Staytor and Johnson, 1990

When to Use Each

Discuss when to use Assessment OF Learning Assessment FOR Learning

7 Strategy Handout Jigsaw Read Discuss Implications

Assessment for Learning StrategiesAssessment for Learning StrategiesKey 1: PurposeKey 1: Purpose

Where am I going?Where am I going?

1. Provide a clear statement of the learning target

2. Use examples and modelsWhere am I now?Where am I now?

3. Offer regular descriptive feedback4. Teach students to self-assess and set goals

How can I close the gap?How can I close the gap?

5. Design focused lessons6. Teach students focused revision7. Engage students in self-reflection; let them

keep track of and share their learning

Key 2: Targets

Without clear targets we can’t:Without clear targets we can’t:

Know if the assessment covers what we taught.

Make a good inference about level of student achievement.

Plan the next steps in instruction. Keep track of student learning target-

by-target or standard-by standard. Give descriptive feedback to students. Have students self-assess. Complete a standards-based report

card.

Key 2: Clear TargetsWhat’s the Target?

Is This a Target? Senior project Bird feeder Use a band saw

safely State report Diorama

A Mathematics Example

Math Decimals Pages 152 in the

book Going on a “decimal

hunt” Read decimals and

put them in order

Kinds of Achievement Targets

Master Factual and Procedural Knowledge

Use Knowledge to Reason

Demonstrate Mastery of Skills

Create Products

Acquire Dispositions

Knowledge VerbsKnowledge Verbs

ExplainUnderstandDescribe IdentifyListDefineRecallRecognizeSelect

Kinds of Achievement Targets

Master factual and procedural knowledge

Understands long-term physiological benefits of regular participation in physical activity

Explains the important characteristics of U.S. citizenship

Knows that energy can be transformed between various forms

Reasoning VerbsReasoning Verbs

Predicts Infers Classifies Hypothesizes Compares Summarizes Analyzes Evaluates Generalizes

Kinds of Achievement Targets

Use knowledge to reason and solve problems

Uses statistical methods to describe, analyze, evaluate, and make decisions

Analyzes fitness assessments to set personal fitness goals, strategizes ways to reach goals, evaluates activities

Examines data/results and proposes meaningful interpretation

Skill VerbsSkill Verbs

ObservePerformDo ConductReadSpeakOperateInvestigateCollect

Kinds of Achievement Targets

Demonstrate mastery of specific skills Measures length in metric and US units Reads aloud with fluency and expression Dribbles to keep the ball away from an

opponent Participates in civic discussion with the

aim of solving current problems Uses simple equipment and tools to

gather data

Product VerbsProduct Verbs

DesignProduceMake WriteDrawRepresentDisplayModel

Kinds of Achievement Targets

Create quality products Constructs bar graphs Develops a personal health-related

fitness plan Builds physical models of familiar

objects Creates a scripted scene based on

improvised work

Kinds of Achievement Targets

Acquire positive affect Sees self as capable of doing mathematics Chooses to read for enjoyment Intends to vote in every election Looks forward to science class Wants to participate in community theater Enjoys opportunities to converse in

Spanish

GENERAL LEARNER OUTCOMES

1. Self-Directed Learner2. Community Contributor3. Complex Thinker4. Quality Producer5. Effective Communicator6. Effective and Ethical User of

Technology

Targets: Deconstructing

Standards

Your Turn…What Kind of Target Is This?

• Interpret text(s) from multiple perspectives… (6-8)• Create and use surveys, interviews, polls, … (K-3)• Design and conduct simple investigations… (4-5)• Explain how sanitary practices, vaccinations,

medicines, … keep people healthy (K-3)• Identify and use appropriate gestures and other

forms of non-verbal communication (4-5)• Sing expressively with appropriate dynamics,

breath control, phrasing, … (6-8)

What Kind of Target Is This?

• Relate critical facts and details in narrative or information text to comprehend text (2-3)

• Conveys meaning, provides important information, makes, a point, fulfills a purpose (writing, 4-5)

• Explain and apply tools and methods drawn from political science to examine political issues and/or problems (6-8)

• Communicate and defend scientific explanations and conclusions (9-12)

• Describe and exemplify how information and communication technologies affect research and work done in the field of science (6-8)

• Sing expressively with appropriate dynamics and phrasing (K-3)

What from Key 2 can you take back to your classroom and use?What from Key 2 can you take back to your classroom and use?

Making targets for a lesson clear Making sure the mix of target

types is as wanted Putting learning targets in

student-friendly language.

Key 3:Designing an Assessment

Design Features

Select a proper methodproper methodCreate quality items/tasksquality items/tasksGather enough evidenceenough evidence Minimize biasbias

Five Standards of Assessment Quality

Standard 1Clear Targets

Standard 2Clear and

Appropriate Users and UsesStandard 3

Appropriate Assessment Methods

Standard 4Sampling

Standard 5Potential Sources of Bias and Distortion

TUMS Bias

T = Clear TargetsU = Appropriate Users/UsesM = Method of AssessmentS = SamplesBias = Minimize Bias

Key 3: Sound DesignPossible Assessment Methods

Selected responseExtended Written response (essay)Performance assessmentsPersonal communication

Task

Any activity, project, prompt, problem or assignment that is given to the student to do.

The quality of the student response to the task is what is assessed using the criteria.

Task Vs. Assessment

Assess How?—Assess How?—SoundSound Design Design

Assessment methods matched to learning methods matched to learning targetstargets

Well-writtenWell-written tasks, items, and scoring guides Adequate sampleAdequate sample Bias avoidedBias avoided

Selected Response Example

Benchmark: Explain the phases of the moon and eclipses.

A lunar eclipse occurs when:A. The moon passes between the earth and the

sun.B. The earth passes between the moon and the

sun.C. The sun passes between the earth and the

moon.D. Every month when there is a “new moon”.E. Both answers A and D.

Short Responses

Benchmark: Describe and explain the effects of multiple forces acting on an object.Prompt: There are at least two forces that affect the speed of your car, forward motion and resistance to forward motion.

Explain how these forces affect the performance of your solar car and give an example to support your answer.

Criteria for SelectingPerformance Tasks

Measure several standards or outcomes

Involve critical thinking skills Use instructional time wisely Focus on credible real world tasks Minimize bias and fair Feasible Clearly defined-- Adapted from Herman, Aschbacher and Winters (1992)

Personal CommunicationType of Target AchievementDispositions

Assessment MethodPersonal CommunicationCan talk with students about their feelings.

Fine Arts:Integrated Stds, Grade 1 Performance Indicator: Discusses personal feelings about a variety of works.

Make an animal sculpture with clay starting with a pinch pot. Tell how you feel about the creature that you made.

Key 3: Sound DesignKey 3: Sound DesignAdequate Sampling—the

Basics Can’t cover everything; need to

make inferences about student learning

Example: “Land a plane.”Example: “Write well for different

audiences and purposes.”

Sampling—What to doSampling—What to do

Gather enough evidence to draw confident conclusions—without—wasting time gathering too much

Ask: Does number of items/tasks for each target represent importance in instruction?

Ask: Do the items/tasks cover the target well?

Sound Design: Sources of Bias

AssessmentAssessorStudentEnvironment

Some generic Some method

specific

See p. 115-116 in CASL

Sound DesignSound DesignBias and Distortion Activity

Read through the potential sources of bias and distortion on CASL, pp 115-116.

Mark with a check (or list) any you have experienced—as the assessment giver or taker.

Two categories—those we control and those we don’t

Find a source of bias you checked that is one we can control

Talk with a partner about how you would go about eliminating or minimizing this source of bias and distortion—use

Analyze an Assessment for Quality: Keys 1, 2, and 3

Choose any one of the four sample assessments. Use the separate rating/checklist form to analyze it for quality.

Assessment Title Key 1: Purpose Key 2: Targets Key 3: Design

Mathematics

Fish Tank

Culminating Project

Emerson

Key 4:

Communication: Descriptive

Feedback

Key 4: CommunicationDescriptive Feedback

Describes features of work or performance

Relates directly to learning targets and/or standards of quality

Points out strengths and gives specific information about how to improve

CommunicationCommunication

Explains the strategy and processes used

Explains why what was done was done

Explains why answer works

FeedbackFeedback

Research backed Brown, 1994 Sadler, 1989 Black & Wiliam, 1998

Descriptive or Evaluative? You decide

Key 5:

Student Involveme

nt

Policy on Reporting Student Progress (4510)

Policy on Reporting Student Progress (4510)

“The involvement of the student in the evaluation process should be considered essential …”

“The progress report shall … be diagnostic and constructive …”

“Secondary students are encouraged to conduct regular parent-teacher-student conferences.”

Key 5:Student-Involved Assessment

Students … Are partners in development Describe expectations for success See how close they are now Create a clear path to success

What are the implications of involving students in assessment?

REST STOP

Key 5: Student Involvement

1. Identify an important learning goal, or one students have difficulty learning.

2. Identify word(s) needing clarification.3. Define the word(s). We use a dictionary as a

starting point.4. Rewrite the definition as an “I can” (or an “I

am learning to”) statement, in terms that your students will understand.

5. Try it out on students or a colleague and refine as needed.

6. Have students try this process for subsequent learning goals.

Converting Learning Targets to student-Friendly Language

The Process:

Standards-based Learning Goals

Summarize text, e.g.: Summarize important information in texts to demonstrate comprehension (4)

Make predictions, e.g.: Predict or hypothesize…from information in the text, … (6)

Summarize textSummarize textSummarize textSummarize text

Words to be definedWords to be defined: SUMMARIZE SUMMARIZE: to give a brief statement of

the main points, main events, or important ideas

Student-friendly language: I can summarize text. This means I can make a short statement

of the main points or the big ideas of what I read.

Make PredictionsMake Predictions

Words to be defined: prediction PREDICTION: A statement that something

will happen in the future. Student-friendly language:

I can make predictions. This means I can use information from

what I read to guess at what will happen next. (Or, to guess what the author will tell me next.)

Your Turn…Your Turn… Words to be defined: Infer

Definition: Student-friendly language: I can infer. This means I can…

Words to be defined: Hypothesize Definition: Student-friendly language: I can hypothesize. This means I can…

Make inferencesMake inferences

Word to be defined: INFERENCE INFERENCE: A conclusion drawn

from the information available Student-friendly language:

I can make inferences. This means I can use information

from what I read to draw a conclusion. (Or, I can use clues to make good guesses.)

Key 5: Student Involvement4-Step Process for Setting

Criteria with Students

Step 1: Brainstorm

Step 2: Sort & Categorize

Step 3: Create a T-chart

Step 4: Add, Revise, & Refine

Step 1: Brainstorm

Pose a question

Record all ideas on the chart

paper

Add in your own ideas to make

sure that all the important

features are included and your

goals are met

Step 2: Sort & Categorize

Have students look at the list and see if any ideas fit together. Ask “Do you see any similar items?”

Circle or mark the similar items using colored pens.

Decide on headings using big ideas.

Step 3: Create a T-Chart

Draw a T-Chart Transfer the items

from Step 2. Put the big ideas or

criteria on the left. Put the details or

specifics on the right.

Criteria Details

Step 4: Add, Revise, Refine

Have students review the T-Chart. Ask students, “ Are there any new

items or criteria that we need to add?”

Make changes as necessary.

Classroom Assessment Quiz ___Rubric

___Scoring Guide

___Criteria

A. A newly formed republic in eastern Europe

B. Your Aunt Frieda’s favorite cabbage recipe

C. Written statements that describe levels of performance

D. John Grisham’s latest novel

E. A 1950’s dating manual

Now that you have the criteria, what do you do with it??

What is a Rubric?

An assessment that evaluates student performance

Uses specific criteria for assessment

Defines levels of performance for each criterion

Performance Criteria are guidelines, rules or principles by which student responses, products or performances are judged. They describe what to look for in student performances or products to judge quality. They are useful in assessing criteria which are complex and subjective.

Rubric Development

Performance Criteria

MeetsProficiencyW/Excellence

MeetsProficiency

Approaches

ProficiencyBelowProficiency

PRODUCINGQUALITYWORK

Leaps tall buildings in a single bound

Must take a running start to leap over buildings

Can leap over short buildings

Crashes into buildings or cannot recognize them

JOBKNOWLEDGE

Always walks on water

Walks on water in emergency

Washes with water

Drinks water

COMMUNICATION

Talks with God

Talks with employees

Talks to him/herself

Loses arguments with self

Why Use Rubrics

Assessment is more objective and consistent.

The target is clearer – shows what is expected and how work will be evaluated.

It gives feedback on the effectiveness of the instruction.

It provides benchmarks against which to measure and document progress.

Common Features of Rubrics

Measures a stated objective (performance, behavior, or quality)

Uses a range to assess performance

Contains specific performance characteristics arranged in levels indicating the degree to which a criterion has been met.

What Student Involvement with Rubrics Doesn’t Look Like

Handing the rubric to students with little explanation.

No practice with the rubric before the “grading” use.

Asking students to self- or peer-assess using the rubric without teaching them how.

Using any old rubric without making sure it adheres to standards of quality.

Key 5:Student-Involved

Communication

Student-led parent/teacher conferences Greater sense of responsibility Pride in accomplishment Greater achievement

“Grades … must meet the dual criteria of validity and reliability. The test of validity is met when the grades have been based on HCPS. The test of reliability is met when (1) there is a sufficient number of grade entries … (2) records are maintained accurately.”

Policy on Grading (4501)

Validity

A good rubric must possess “validity.”Definition: the extent to which what is being

measured by an instrument is actually what is intended. Are the test and rubric actually measuring the desired performance-outcomes?

Reliability

A good rubric must possess “reliability.”

Definition: the extent to which the measuring

instrument yields responses that are consistent across time and between different scorers.

Reliability

Factors that have nothing to do with the purpose of the assessment can impact scores a student can get. Examples: Rater fatigue Rater’s mood Failing students may get more critical look than

“good” students Well-designed rubrics respond to this concern

by establishing a description of the scoring criteria in advance. Scorer should revisit the established criteria to ensure consistency.

Reliability (cont.)

Clarity of the rubric Are the scoring categories well-defined? Are the differences between scoring

categories clear? Would two independent raters arrive at the

same score for a given response? Use anchor papers. Whenever possible, share scoring rubrics

with students in advance.

Why Accuracy Is Important

Assessment information can be inaccurate in one of two ways:

We think students have learned material when they actually have not

We think students haven’t learned material when they actually have

What problems for What problems for teachers and teachers and

students would arise students would arise from each of these from each of these

situations?situations?

“We must change from a model that picks winners to one that will create winners.”Harold Hodgkinson, 1987“Michigan: The State and Its Educational System”

Observations of students in the process of learning

Communications- student interviews, journals, discussions, comments

Products- student work, tests, reports, models, performances

Triangulation of Evidence: Assessment Data from Many Sources

ActivityActivityTriangulati

onTakingTaking your post-its

from the earlier

Assessment FOR and

OF Learning chart,

make a triangulation

poster at your table.

Place the post-its

where they belong on

the poster.

What do you notice?

Reflecting on your practice: When developing assessments, what concerns or issues regarding triangulation of data need to be considered?

REST STOP

Decisions About Performance Criteria

Performance Criteria

Task Specific Generalized

Holistic (on overall Score)

Analytical Trait

Trait

(several dimensions scored separately)

Quantity or Quality

Amount of Detail

Number of Score Points

General or Task Specific?

GENERALIZED or Generic - can be used across similar performances.

TASK-SPECIFIC- each one can only be used for

a single task.

Holistic or Analytical?

HOLISTICgives a single score or rating for an entire product or performance based on an overall impression of a student’s work.

ANALYTICAL TRAITdivides a product or

performance into essential traits or dimensions so that they can be judged separately.

Holistic or Analytical Trait

HolisticUse : Quick snapshot of overall

status When speed of scoring is

important Simple products or

performances

Disadvantages: 2 students can get same

score for different reasons

Can’t identify strengths & weaknesses

Not useful for students

AnalyticalUse Planning instruction - show

relative strengths & weaknesses

Details to teach student quality Detailed feedback Precision more important that

speed:Disadvantages: Scoring is slower Takes longer to learn

Options for Selecting Rubrics

Create your own - build from scratch Adopt - use an existing rubric Adapt - Modify or combine existing

rubrics Reword parts Drop or change one more

scales Omit criteria not relevant “Mix” and Match” Divide a holistic rubric into

several scales Add a “no response” category Modify for different grade levels

Online Rubric Resources

Kathy Schrock at Discovery School has compiled an extensive list of online rubric resources and rubric builders at:http://school.discovery.com/schrockguide/

assess.html#builders

1. Individually read and score three student work samples (Practice Papers #01, #02, and #03) using the Sample Student Rubric (Handout 6).

2. Share the scores in your group.

3. Discuss the criteria on the rubric to develop common understandings. What do you notice?

REVISEREVISE-- Using the Scoring Criteria on the Item and Scoring Criteria sheet, revise the rubric to include criteria measuring Core Concepts as well as the criteria for Accessing Information.

READ and SCORE-READ and SCORE- Then each member of the group will read and score 2-3 of the remaining samples (Practice Papers #04-#08) using the revised rubric.

DISCUSS and REFINEDISCUSS and REFINE - Discuss the following questions in your group: If we were going to use this rubric again, would we want to add new criteria? Clarify existing criteria? Delete insignificant criteria?