Assessment Literacy Module - Digital Chalkboard · Assessment Literacy Module ... What action can you take to ensure that all students have ... families that speak a language other

9/26/2013

1

Unit 6: Identifying, Developing,

and Implementing Sound,

Quality Assessments

Assessment

Literacy Module

California Department of Education

Tom Torlakson, State Superintendent of Public Instruction

Welcome to Unit 6

The purpose of this unit is to help educators identify, select,

and develop high quality classroom assessments, implement

them at multiple levels to continuously monitor student

progress, and increase the accuracy of student performance

information.

2

3

Learning Objectives for Unit 6

By the end of this unit, participants will be able to:

Evaluate the quality of assessments and their

items

Recognize when linguistic complexity makes assessment

unfair

Describe a variety of checking for understanding strategies

9/26/2013

2

4

Identifying, Selecting, and Developing

Appropriate Classroom Assessments

Refer again to the teaching-assessment cycle and the assessment

literacy attribute to be covered first in this unit:

5

Identifying, Selecting, and Developing

Appropriate Classroom Assessments

Educators need accurate and relevant student performance

information from high-quality assessments to make informed

instructional decisions. The quality of assessments rests on

three key concepts: validity, reliability, and fairness.

Key Concepts in Accurately Measuring Student Performance

6

Key Concepts in Accurate Measurement: Validity

Validity is the extent to which inferences and actions made on

the basis of assessment results are appropriate and backed by

evidence (Messick, 1989). It is the use and/or interpretation of

assessment data that is key to the concept of validity.

Validity

The validity of any assessment is the extent to which test

scores accurately reflect the relevant knowledge and skills

of test takers.

Discuss:

What are the actions and decisions you make based on

students’ assessment results?

Do any of the actions/decisions call for more valid

assessments than others?

9/26/2013

3

7

Key Concepts in Accurate Measurement: Validity

For the Smarter Balanced assessments, the relevant knowledge and

skills are defined by the CCSS. These assessments are developed

using the principles of Evidence-Centered Assessment Design

(ECD). The three basic elements of ECD are:

Validity

(1) stating the claims to be made about test takers,

(2) deciding what evidence is required to support the claims, and

(3) administering test items that provide the required evidence

(Mislevy, Steinberg, & Almond 1999).

Additional information about ECD is available at

http://padi.sri.com/downloads/TR9_ECD.pdf

8

Key Concepts in Accurate Measurement: Reliability Reliability

Reliability is the degree of confidence that both scores

and student performance are repeatable over time and

across different circumstances.

Repeatability of scores means that different scorers or the same

scorer at different times should assign the same score(s) to the

same piece of student work (O’Neill & Stansbury 2000).

Discuss:

How can teachers ensure that their classroom

assessments are reliable?

9

Key Concepts in Accurate Measurement: Fairness Fairness

“Test scores are fair when they yield score interpretations

that are valid and reliable for all students taking the test.

Regardless of race, national origin, gender, or disability,

academic tests must measure the same knowledge of

content for all students who take the test. Test scores must

not systematically underestimate or overestimate the

knowledge of students of a particular group.”

‒California Education Code Section 60208




9/26/2013

4

10

Key Concepts in Accurate Measurement: Fairness Fairness

For the Smarter Balanced assessments, the Bias and

Sensitivity Guidelines helps to ensure that the evidence

provided by the items means the same thing for various

groups of test takers and allows the ECD to work as intended

(Educational Testing Service 2010).

For additional information, download the Smarter Balanced

Assessment Consortium: Bias and Sensitivity Guidelines at

http://www.smarterbalanced.org/wordpress/wp-content/

uploads/2012/05/TaskItemSpecifications/Guidelines/Biasan

dSensitivity/BiasandSensitivityGuidelines.pdf

11

Assessment Fairness and Bias

Educators have the greatest opportunity to improve

their assessments and directly impact students by

increasing the fairness of their classroom assessments.

“Assessment bias occurs whenever test items offend or

unfairly penalize students for reasons related to students’

personal characteristics, such as their race, gender,

ethnicity, religion, or socioeconomic status” (Popham 2003).

12


Fairness includes:

1. Instructional Learning Targets: Communicating clear, specific learning targets to students ensures they know what will and will not be assessed, what method will be used, and how the assessment will be scored.

2. Opportunity to Learn: Providing all students equally with adequate time and appropriate instruction will enable them to obtain mastery.

3. Prerequisite Skills and Knowledge: Ensuring students have the prerequisite skills and knowledge is necessary to complete a task, but are not the learning target itself. If learning targets are NOT related to writing or reading comprehension, this source of unfairness must be reduced by:

a. Identifying the prerequisite skills and knowledge of the learning target

b. Pre-assessing students’ abilities in the prerequisite skills and knowledge, either formally or informally through assignments, observations, or questioning

Source: McMillan 2000

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/05/TaskItemSpecifications/Guidelines/BiasandSensitivity/BiasandSensitivityGuidelines.pdf












9/26/2013

5

13


Respond to the following questions:

2. What groups of students are most impacted when

assessments are not fair?

3. What action can you take to ensure that all students have

adequate time and appropriate instruction to enable them to

achieve mastery?

4. What steps can you take to minimize bias in your

assessment?

1. How does bias, opportunity to learn, and prerequisite skills

and knowledge impact students and the way they are assessed?

14

Assessment Accessibility

To ensure that assessments are fair and accessible, it is

essential to allow all students to show what they know and

can do, particularly students with special learning needs such

as students with disabilities and English learners.

In California classrooms, a Universal

Design approach is being used to

modify instruction according to IEPs,

504 plans, and language needs.

Applying some of these processes

and approaches to assessment

takes time and practice.

15

Assessment Accessibility

Smarter Balanced uses an “Access by Design” approach,

which includes an array of universal digital tools and features

available to all students.

Additional information about Universal Design is

available on The National Center on Universal Design for

Learning Web site at http://www.udlcenter.org/aboutudl.

Examples include: Text-to-speech administration of the

mathematics test and ELA items, screen magnification, guided

line readers, and color options designed to increase contrast

and readability.

IEPs or 504 plans may specify tools and features such as

keyword translation glossaries, auditory calming, audio

captioning, administration in American Sign Language, or braille

administration and response formats.

http://www.udlcenter.org/aboutudl



9/26/2013

6

16

Linguistic Accommodation for English Learners

“Linguistic accommodation” is a research-based approach in

which the language in test items, directions, and/or selected

response choices is changed in ways that clarify and simplify

the text without simplifying or significantly altering the learning

target.

It is especially useful with assessments in content areas such

as math, science, social science, and technical subjects when

language may be irrelevant to the learning target.

To facilitate comprehension, the complexity of text is reduced

by shortening sentence length and complexity, using common

or familiar words and concrete language.

Source: Abedi, Courtney, Mirocha, Leon, & Goldberg 2005;

Abedi, Lord, & Plummer 1997; Rivera & Stansfield 2001

17


The Linguistic Accommodation approach is aligned with the

Universal Design used in Smarter Balanced assessments.

Considering that 37 percent of California’s students come from

families that speak a language other than English, it is

important for educators to know the impact of linguistics on

classroom assessment (CalEdFacts 2011).

18


Watch Kenji Hakuta of Stanford University discuss

how the needs of English learners are being considered

in the CCSS and the creation of new assessments:

“Kenji Hakuta on ELLs and the Common Core Standards”

http://www.youtube.com/watch?

eature=player_embedded&v=PY2fhP8Q0OY

http://www.youtube.com/watch?feature=player_embedded&v=PY2fhP8Q0OY








9/26/2013

7

19

Limited English Proficiency Affects Learning and Assessment

“While ELL students are struggling to learn English, learning

content-based knowledge cannot occur at the same rate as for a

native speaker of English when that instruction is offered only in

English. Limited English proficiency may also make it difficult for

ELL students to benefit fully from the teacher's instructions and to

understand assessment questions. Therefore, limited English

proficiency affects both learning and assessment. To help close the

performance gap between ELL and non-ELL students both learning

and assessment conditions must be addressed."

‒Abedi 2007

Key question in deciding when to use linguistic accommodations in

creating tests:

Is the learning target in ELA or another content area?

20


If the learning target is not in language, assessment-

literate educators use strategies to reduce the

interference of unnecessarily complex language in test

directions and test items.

Even when language is the learning target,

assessment-literate educators make test directions

clearer and simpler.

21


Refer to the table of interferences and

accommodation strategies handout and complete the

steps on the next slide:

Analyzing Linguistic Interferences and Accommodations

Handout

9/26/2013

8

22


1. Look at the Examples of Interferences on the next slide. If an

example matches an “Interference” in Column 1 of your handout,

write the letter of the example into the adjacent cell in Column 2.

Some examples will be used more than once.

2. In Column 4, use the corresponding “Accommodation Strategy”

from Column 3 and write in your idea to fix the Interference.

3. Compare your answers to the Completed Table handout.

23


Examples of Interferences:

A. Alba needed to know about how much the sum of 19.6, 23.8,

and 38.4 is. She correctly rounded each of these numbers to

the nearest whole number. What three numbers did she use?

B. Ted can no longer drive over 40 mph in his truck.

C. The weights of two objects were measured.

D. As long as you bring your own bedding, you can stay with us.

E. According to the article, what role did some Navajo speakers

play during World War II?

24


Compare your responses to the Completed Table

Discuss findings and comparisons within your group.

Completed Table: Analyzing Linguistic Interferences and Accommodations

Handout

9/26/2013

9

25


Important: Linguistic accommodation is not in conflict with the

CCSS’ emphasis on text complexity and academic language

because it is NOT about “dumbing down” texts. It IS about access to

the CCSS for ALL students. The challenges for assessment-literate

educators are:

1. To be clear about when language is the learning target and

when it is not;

2. To recognize and eliminate unnecessarily complex language.

By using linguistic accommodation strategies to develop and

improve assessments, assessment-literate educators allow

students to dedicate their attention to show what they know

instead of translating text (Abedi, Lord, & Plummer 1997).

26


To explore accommodations options for your own

classroom, use a new copy of the Interferences and

Accommodations handout, and try the following:

1. Choose an assessment you are currently using. Look at the

directions, questions, and answer choices (if applicable). Using

the list of Interferences, how many can you spot? Use the list

of strategies to edit the assessment. Be clear about what is

being assessed — language or another content area?

2. For students with communication disorders and/or reading

disabilities, which linguistic accommodations are appropriate

for making assessments more accessible? Discuss how to

address linguistic accommodations in IEPs or 504 Plans. This

is especially critical for English learners with disabilities.

Interferences and Accommodation Strategies Template

Handout

27

Variations, Accommodations, or Modifications

Becoming familiar with the principles of UDL, the diverse

needs of students, and the various accessibility options

available is the first step that educators can take to choose

fair and accessible assessments — so that ALL students have

the opportunity to show what they know and understand.

The choice of variations, accommodations, or modifications in

classroom assessments must be made thoughtfully, based on

the students’ individual needs.

If students with disabilities are entitled to variations in state

testing, these variations should be used routinely in the

classroom. Assessment-literate educators understand the test variations

indicated for their students AND are able to carry out these

variations in their classroom instruction and assessment.

9/26/2013

10

28

Selecting and Developing Appropriate

Classroom Assessments: Assessment Methods

Now that you understand the key concepts behind measuring

student learning, you are ready to develop sound, quality

assessments.

In Unit 2, you learned about two basic types of

assessment methods that might be used in your

classroom to gather information about student learning:

selected-response and constructed-response.

29



Selected-Response Methods

In this short video, W. James Popham explains

relevant features of selected-response assessment

methods: “Selected-Response Assessment”

http://www.k-state.edu/ksde/alp/module6/

Selected-response (SR) methods prompt students to select

one or more correct responses from a set of choices. Carefully

constructed SR items allow students to demonstrate complex

thinking skills such as developing comparisons or contrasts;

identifying cause and effect; identifying patterns or conflicting

points of view; categorizing, summarizing, or interpreting

information.

30



Constructed-Response Methods

In this short video, W. James Popham explains

relevant features of constructed-response assessment

methods: “Constructed-Response Assessment”


Constructed-response (CR) methods prompt students to

generate a text or numerical response in order to collect

evidence about their knowledge or understanding of a given

assessment target. CR items can be both short and extended.

Short items may require test takers to enter a single word,

phrase, sentence, number, or set of numbers, whereas

extended items require more elaborated answers and

explanations of reasoning. These kinds of CR items allow

students to demonstrate their use of complex thinking skills.











9/26/2013

11

31

Selected-Response vs. Constructed-Response Assessment Methods

Mr. Bumble is a hard-working “every

teacher.” He struggles with how to

assess his students and often shows you

what not to do. You will learn about Mr.

Bumble’s exploits throughout this unit.

Meet Mr. Bumble.

Mr. Bumble realizes that he needs to improve his classroom

assessments. He thinks the way to do this is to choose between

selected- or constructed-response methods. So, he compares the

two to find out which one is better for classroom assessments.

32


Selected-response (e.g., multiple choice, true/false, matching)

Constructed-response (e.g., short answer, essay)

Advantages • Easier to score

• Can be answered quickly

• Covers a broader range of

curriculum in a shorter time

• Allows students to demonstrate

complex, in-depth understanding

• Less likelihood of guessing correct

answer

• Motivates students to learn in a way

that stresses the organization of

information, principles, and

application

Disadvantages • Constrains students to single

appropriate answer

• Encourages students to learn by

recognition

• Subject to guessing correct answer

• More time-consuming to score

• More time-consuming to answer

EUREKA! Mr. Bumble realizes there are advantages and disadvantages to

both methods. Now he’s confused about when he should use which one...

33


If Mr. Bumble asked you which method to use, how

would you answer him?

Refer to the Target-Assessment Method Match table handout from

Unit 4. Recall that the key determination in selecting the type of

assessment method is how well it measures the type of learning

target.

Target-Assessment Method Match (from Unit 4)

Handout

9/26/2013

12

34

Developing Better Assessments

Assessment-literate educators develop assessments that are

appropriate and yield accurate information on which they can

make sound instructional decisions.

Popham (2003) alerts educators to:

These roadblocks apply to any type of item.

35

Developing Better Assessments

Now let’s look at the directions and some of the items from

one of Mr. Bumble’s science tests. As you review them,

identify how many roadblocks you find.

Read sections of Mr. Bumble's exam on the following

slides and then learn about the roadblocks they contain, along

with tips for overcoming them.

To review the roadblocks and tips after the workshop, refer to

the online module at

http://www.myboe.org/portal/default/Content/Viewer/Content?ac

tion=2&scId=505706&sciId=15350

36

Mr. Bumble’s Science Exam: African Elephant Unit

Directions:

"This exam consists of four parts. Each part contains different

types of items (for example: true/false, multiple-choice, and

essay). Each of the exam’s four parts covers one of the major

units you have studied. Work your way through the test

efficiently because there is a time limit. Don’t write on the test.

Good luck!"

Discuss the roadblocks that you encounter.

http://www.myboe.org/portal/default/Content/Viewer/Content?action=2&scId=505706&sciId=15350

http://www.myboe.org/portal/default/Content/Viewer/Content?action=2&scId=505706&sciId=15350

9/26/2013

13

37

Mr. Bumble’s Science Exam:

African Elephant Unit

True/False:

T F "Several research studies show that adult elephants

always become domineering toward younger elephants

because of their inherited characteristics."

Multiple Choice:

The commonly recognized example of a pachyderm is an

a. elephant

b. turtle

c. lion

d. pigeon

38


Apply the item-writing rules you have learned so far to fix

Mr. Bumble’s essay question. Then, compare your response

to how the expert would fix it.

Essay Prompt:

“Discuss elephants in Africa."

39


To develop high quality assessments, assessment-

literate educators review all items and directions from the

perspective of students. Answering essay questions

mentally or in writing can help to uncover problems in

advance.

For more information on developing assessments,

including matching learning objectives with test items and

writing test questions aligned with Bloom’s Taxonomy, see

“Is This a Trick Question? A Short Guide to Writing

Effective Test Questions” at http://www.k-state.edu/ksde/

alp/resources/Handout-Module6.pdf.

http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf









9/26/2013

14

40

Instructionally Embedded Formative Assessment:

Checking for Understanding Strategies

Here is where we are in the teaching-assessment cycle and

the assessment literacy attribute that will be covered.

41

Sources of Assessment Data

The graphic below shows the varying frequency and duration of

teaching-assessment cycles. This section addresses an effective

assessment strategy for shorter-cycle, daily, and minute-by-minute

formative assessment. Checking for understanding provides useful

assessment information for adjusting instruction in real time.

Source: Heritage 2010

Sources of Assessment Data

42

The Benefits of Checking for Understanding

Checking for understanding is a formative assessment

practice where educators ask questions or give short

assignments and students select or construct their responses.

Assessment-literate educators use “end of class”

formative assessment results to guide planning for the

next day’s lesson, or embed the assessment seamlessly

into instruction to make adjustments “in the moment.”

9/26/2013

15

43


The benefits of checking for understanding when responses

are obtained from all students:

Gives educators information quickly about the need to

differentiate instruction

Helps educators understand students’ background knowledge,

skills, and misconceptions

Models for students how to monitor their own understanding

The biggest payoff is the positive effect on student

achievement. Research findings support the boost in student

achievement when educators pause to check for understanding and

use the information to adjust their teaching (Fisher and Frey 2007).

44


Watch this scene in Mr. Bumble's math class right after

he completes an explanation of the statistical concept of the

mean. Is he really checking for understanding?

“Mr. Bumble Checks for Understanding”

http://goanimate.com/videos/08KkNc8Zt5_E?utm_source=linkshare

Later, when Mr. Bumble reviews the results of the unit test, he is

surprised to see that only one or two students understood the

concept of the statistical mean. Where did Mr. Bumble go wrong?

What assumptions did he make?

45


Nonverbal checks for understanding in the formative

assessment process include observing facial expressions and

body language, as well as establish classroom routines such

as hand signals, colored cards, or answers to selected-

response questions.

What nonverbal cues can students give spontaneously that

indicate whether they understand?

http://goanimate.com/videos/08KkNc8Zt5_E





9/26/2013

16

46


Watch how an assessment-literate educator

checks for understanding nonverbally on a minute-by-

minute basis. “Show your Cards”

http://successatthecore.com/teacher-development/featured-

video.aspx?v=41

How is the data gathered from colored cards used to adjust

instruction?

47

Checking for Understanding on a Daily Basis

The next four videos show how assessment-literate educators

can check for understanding on a daily basis where students

respond verbally or in writing.

As you watch, think about:

• How does the teacher ensure that checking for

understanding is a low-stress formative assessment?

• How can the data gathered from the assessment be used

to adjust instruction?

• How are students benefiting from the strategy? What

feedback are they getting?

• How would you use the strategy?

48


1. Stoplight

Watch an ELA teacher use exit slips to assess learning and

how students self-assess their learning:

“The Stoplight Method: An End-of-Lesson Assessment”

https://www.teachingchannel.org/videos/daily-lesson-assessment

2. Exit Cards

How does this strategy help teachers assess learning and

plan future math lessons?

“Assess and Plan with Exit Tickets”

https://www.teachingchannel.org/videos/teacher-assessment-

strategy?fd=1

http://successatthecore.com/teacher-development/featured-video.aspx?v=41













https://www.teachingchannel.org/videos/teacher-assessment-strategy?fd=1







9/26/2013

17

49


3. Tiered Exit Cards

Watch the video to see how a secondary math teacher uses

differentiated exit cards to group students for the next day.

“Daily Assessment with Tiered Exit Cards”

https://www.teachingchannel.org/videos/student-daily-assessment

4. Text What You Learned

A Secondary English teacher uses texting to check for

understanding.

“Text What You Learned: Using Technology to Assess”

https://www.teachingchannel.org/videos/texting-to-assess-learning

50

Using Think-Pair-Share to Check for Understanding

“Think-Pair-Share” is a peer-to-peer discussion strategy where students

discuss their constructed responses to a prompt or question before

sharing with the whole class. It is a strategy that engages every student

so that checking the understanding of the whole class can be done

efficiently. The following steps outline the strategy (Fisher & Frey 2007).

1. Pose a question, prompt, reading, visual, or observation.

2. Allow a few minutes to individually THINK about the question.

3. PAIR up students with designated partners to discuss their

respective responses, comparing thoughts and agreeing on the

best responses.

4. After an appropriate length of time, ask the pairs to SHARE

their thinking with the whole class. Students need specific

feedback on speaking and listening especially since these skills

do not always have products for reviewing.

51

Using Think-Pair-Share to Check for Understanding

View an example of a student rubric that elementary students

can use to assess and improve their Think-Pair-Share skills at http://www.rcampus.com/rubricshowc.cfm?code=M3X599&sp=yes&

View an example of a rubric that teachers can use to assess

and improve Think-Pair-Share activities in their classrooms at http://www.rcampus.com/rubricshowc.cfm?code=H46482&sp=true

More information about rubrics will be covered in Unit 7.
















http://www.rcampus.com/rubricshowc.cfm?code=M3X599&sp=yes&




http://www.rcampus.com/rubricshowc.cfm?code=H46482&sp=true

http://www.rcampus.com/rubricshowc.cfm?code=H46482&sp=true

9/26/2013

18

52

Set the Stage for Student Self-Assessment of Understanding

Many of these strategies ask students to self-assess their

level of understanding. When students overestimate their

understanding, assessment-literate educators:

1. Set a comfortable climate where honest self-assessment has

no consequences

2. Clearly communicate learning targets and criteria for mastery

3. Make learning public and welcome mistakes as opportunities

for everyone to learn

53

Reflect on Checking for Understanding

Respond to the questions below and include other

thoughts you have about checking for understanding.

1. How often do you check for understanding? What strategy

do you use most often?

2. Are your students comfortable with publicly self-reporting

when they don’t understand? If so, how did you create the

appropriate classroom climate? If not, what questions do

you have for improving your classroom climate?

3. What ideas for your classroom did you get from watching

the videos?

54

Apply What You Learned

Using a learning target you have developed for

your grade level and/or content area, develop multiple

methods for assessing the target.

For assessments you can use formatively, develop a

selected-response item, a constructed-response item

or prompt (written or performance as appropriate), and

one check-for-understanding strategy you will try for

monitoring student progress in real time during instruction.

Then, develop a summative assessment you can use to

assess mastery of the learning target.

9/26/2013

19

55

Summary of Unit 6

Let’s review the learning targets for Unit 6. Can you:

• Evaluate the quality of assessments and assessment

items?

There are three key concepts that distinguish quality

assessments -- reliability, validity, and fairness.

Assessment-literate educators are aware of these concepts and

select or develop classroom assessments of the highest

possible quality. Their assessments are fair and accessible to a

wide range of students while yielding the most accurate picture

of student achievement.

56

Summary of Unit 6

Can you:

• Recognize when linguistic complexity makes assessment

unfair?

When developing assessments it is important to identify

unnecessary linguistic features that slow down the reader, add

to their cognitive load, and interfere with the intended

assessment task.

There are strategies that can be used to mitigate these

interferences and improve the quality of assessments. Use of

these strategies will improve the fairness and validity of

classroom assessments.

57

Summary of Unit 6

Can you:

• Describe a variety of check-for-understanding strategies for

classroom formative use?

Checking for understanding is a formative assessment process

that provides information that educators can use to make

instructional adjustments on a minute-by-minute basis or on a

daily basis.

Checking for understanding can be done using a wide variety of

methods – verbal, nonverbal, constructed response, selected

response, etc.

9/26/2013

20

58

Summary of Unit 6

Review the assessment item Mr. Bumble

developed to measure CCSS Math standard 1.0A.1

(using subtraction within 20 to solve word problems) and

respond to the questions:

1. Is this item fair? To what groups of students might it be biased

against? To what groups of students might it be biased in favor?

2. How would you improve the quality of this item?

3. Describe one strategy to check for understanding nonverbally.

4. Describe one strategy to check for understanding verbally.

Three alpacas were sold at auction by Stephan, and he went home

with five of the beasts. Therefore, with how many alpacas did

Stephan start?

Documents

Assessment Literacy Module - Digital Chalkboard · Assessment Literacy Module ... What action can you take to ensure that all students have ... families that speak a language other