Collaboration on Formative Assessment Quality and Assessment Literacy Emily R. Lai

Collaboration on Formative Assessment Quality and Assessment Literacy

Emily R. Lai

Presentation Overview

• Background/rationale• Goals of the project• What is a Theory of Action?• What is Principled Design for Efficacy (PDE)?• How did we implement PDE in a district?

Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved. 2

Background/Rationale

• Comeback of performance assessment• Potential of formative assessment to improve student

learning (Black & Wiliam, 1998)• But teachers may have trouble closing the formative loop

(Heritage et al., 2009)• Only successful under certain conditions (Kingston & Nash,

2011)– Formative assessment more successful in ELA than in other

domains– Implementation through PD more effective than other models

• Value of Principled Assessment Design processes (e.g, Evidence-Centered Design)– But no examples of implementation of these approaches within

a school district

3

Goals of the project

• Create a theory of action to guide implementation of formative performance assessment in BCPS

• Deliver training and support to BCPS staff and educators in using principled assessment design methods

• Develop and pilot-test performance assessments in ELA, Grades 8 and 9, that could be used formatively in the classroom

4

What is a Theory of Action?

• Identifies specific processes, sequences of actions

• Specifies how early and intermediate outcomes relate to longer-term outcomes

• How or why will the system produce the desired change?

5

What is a Theory of Action?

• Potential mechanisms of change represent hypotheses

• They can be empirically tested

• Evidence can be used to fine-tune or revise the theory of action

• Evidence can be used to identify areas for improving processes

6


What is Principled Design for Efficacy (PDE)?

• Principled approach to assessment design and development

• One of several related approaches (related to Evidence-Centered Design)

• Begin with clearly-defined assessment targets

• Reason backwards from desired inferences/claims to evidence, and from evidence to task features

• Creation of reusable templates


PDE Process: Potential Benefits

• Offers a practical way to implement principled assessment design

• Helps assessment developers make design decisions thoughtfully and explicitly

• May make assessment development more efficient and effective

• Translates research into practice


PDE Process: 3 Driving Questions

1. What knowledge, skills, and abilities (KSAs) are you trying to assess? (constructs)

2. What could a student do, say, or make to convince you that they possess those KSAs? (evidence)

3. How can you design assessment tasks to elicit that evidence? (content features)


PDE: Four Stages of Design and Development

• Identify relevant learning sciences research to define and clarify KSAs, evidence, content features

• Create assessment prototype

• Construct reusable templates

• Use reusable templates to generate additional assessment materials


Define Create Generalize Re-create

PDE: 6 Design Concepts• Construct: What are you trying to assess?

• Theory of Change: What do you expect will happen when you assess? What are the mechanisms you believe will cause those changes?

• Content: How do you design assessments to elicit evidence of the assessment targets?

• Evidence: What student behaviors will provide evidence of those assessment targets? How will you evaluate that evidence?

• Communication: How do you talk with students and parents about what you are assessing?

• Implementation: How do you work within practical constraints?


How did we implement PDE in Baltimore County?• Focus on “unpacking” standards

– Defining constructs at a fine grain size

– But not treating them as discrete

• Distinguishing between “focal” and “prerequisite” KSAs

– Focal KSAs=grade-level assessment targets

– Prerequisite KSAs=concepts and skills that should be addressed during prior instruction so that students are prepared

– Assume students have mastered prior grade’s standards


How did we implement PDE in Baltimore County?• Articulating evidence

– PARCC evidence statements

– But response mode fits within constraints of testing/scoring time available

• Identifying text features

– Text is primary complexity driver

– PARCC text complexity rubric and worksheet

– Sources of text complexity should match focal KSAs


How did we implement PDE in Baltimore County?• Learning sciences research to support

– Research on text structure and comprehension

– Progressions for argumentation skills

• Use of templates

– Structure decision making of task writers

– Increase alignment to assessment targets (content and complexity)

– Document assessment quality


How did we implement PDE in Baltimore County?• Flexibility/responsiveness to district constraints

– Re-arrange some of the steps in the process

– Better fit the style and preferences of task writers

• BCPS staff participation in the process

– Increase staff capacity for formative performance assessment

– Allow classroom considerations to inform design of PBAs

– Build educator “buy-in”


Example: CCSS ELA Informational Text

Let’s take a closer look at one particular standard:

CCSS.ELA-LITERACY. RI.9-10.5

Analyze in detail how an author’s ideas or claims are developed and refined by particular sentences, paragraphs, or larger portions of a text (e.g., a section or chapter).


Example: CCSS.ELA-LITERACY. RI.9-10.5

Focal KSAs:

• Analyze the author’s development of ideas and concepts within informational texts

• Analyze the structure of an informational text and explain how it contributes to meaning and/or purpose

• Analyze the structure of the relationships among concepts in a text, including relationships among key terms

• Analyze how a text uses structure to emphasize key points or advance an explanation or analysis



Prerequisite KSAs:

• Analyze the structure of a specific paragraph in a text, including the role of particular sentences in developing and refining a key concept

• Analyze the structure an author uses to organize a text, including how the major sections contribute to the whole and to an understanding of the topic

• Describe how a text presents information (e.g. sequentially, comparatively, causally)



KSA research sources & strategies

• Journal of Educational Psychology

• Reading Research Quarterly

• The Reading Teacher

• Journal of Literacy Research

• Journal of Experimental Psychology: Learning, Memory, and Cognition

• Journal of the Learning Sciences

• Google Scholar

• Search terms: “text structure,” “expository structure”



A few sample papers we reviewed:

1. How to teach expository text structure to facilitate reading comprehension (Akhondi et al., 2011)

2. Teaching and assessing understanding of text structure across grades (Hess, 2008)

3. Individual differences in children’s knowledge of expository text structures: A review of the literature (Ray & Meyer, 2011)


Example: CCSS.ELA-LITERACY.RI.5.5

Potential content features:

• Different types of expository structure (e.g., description, comparison, causation, problem-solution)

• Explicit versus implicit structure

• Structure conforms to genre conventions versus disciplinary conventions

• Effective signaling of text structure


Assessment Features Important for TOA

• Cognitively complex

– Require active construction of meaning

– But accessible to students across the performance spectrum

• Engaging for students

– Opportunities for student collaboration, inquiry, choice

– Incorporation of multimedia texts, where appropriate

– Varied response modes



• Flexible use

– Teachers can administer within a broad window

– Prerequisite KSAs provide indication of when students are ready

• Authentic

– Integrate reading, writing, speaking, listening

– Use authentic texts

– Across the year, opportunities for full writing process



• Support valid interpretations

– Require at least one individual accountability component

– Balance text/writing types

• Produce actionable feedback for students and teachers

– Scored using PARCC rubrics

– Separate scores on reading comprehension, writing, and speaking and listening

– Refer back to evidence



Sample Task Template

Thanks!

Emily R. LaiLead, Center for Product Design Research & EfficacyResearch & Innovation [email protected]


mailto:[email protected]

Documents

Collaboration on Formative Assessment Quality and Assessment Literacy Emily R. Lai