C R E S S T / U C L A Model-Based Assessment: Why, What, How, How Good, and What Next? Eva L. Baker National Research Council, Board on Testing and Assessment

C R E S S T / U C L A

Model-Based Assessment: Why, What, How, How Good,

and What Next?

Eva L. Baker

National Research Council, Board on Testing and AssessmentBridging the Gap Between Classroom and Large-Scale Assessment Workshop

January 23-24, 2003Washington, DC

UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation

National Center for Research on Evaluation, Standards, and Student Testing


Why? Assessment Knowledge: Usable and

UsefulUsable Knowledge

In a form that can be understood

In a form that can be applied

Timed appropriately

May cause rethinking of the problem

Useful Knowledge

Rethinking indicates a new solution path

Adapted to situation

Sufficient to guide solution

Improved outcomes occur as a result


Why Are Some Schools Successful in Using

Assessment Knowledge?

Focus on learning (students and adults)

Constant use of appropriate information (formal and informal)

Focus on feedback and change

Public display and exchange

Community pride in outcomes of students and place


Goals for CRESST Model-Based Assessment (MBA)

Assessment components share a common framework. MBA starts with thinking skills and applies them to content domains to support

Coherent, sustained learning

Spiral teaching-common language

Transfer (application to new situations)

Multipurpose

Learning organization


CRESST Model-Based Assessments (MBA)

Features Research based

Focus on cognition and learning

Abstracted in models based on key learning elements—principles guiding test design and instruction

Operationalized in templates

Reusable and cost-sensitive design/training/scoring


Model-Based AssessmentCognitive Families

ContentUnderstanding

ProblemSolving

Teamwork andCollaboration

MetacognitionCommunication

Learning


Model-Based Assessment Design

Models to templates (to specification) to tests

Template contains domain-independent (transfer) and domain-specific (strategy and knowledge) components

Templates that allow common domain-specific design approaches to be used, e.g., primary sources in history

Scoring requirements


Expert Model—Deep Understanding of Content

(Domain Independent)

Principles or themes (big ideas)

Use of prior knowledge

Explicit relationships

Avoiding misconceptions

Expert performance-based scoring


Specifications forLarge-Scale Use

Standards reference

Place in sequence

Content domain (what’s in and out)

Proportion of effort

Format options

Interpretation rules

Time


Template

Task(s)

Format(s)

Prompt(s) and requirements

Scoring

Directions

Sample


Three Templates for the Model of Deep Content

Understanding

Explanation

Explanation with explicit knowledge

Graphical representation of relationships


Deep Content Understanding

Primary source materials in each domain

Student required to integrate prior knowledge and principles to succeed

Scored by using expert model in subject matter


Content UnderstandingTemplate #1 Explanation

An array of primary source materials

A prompt that asks for an explanation in context

Constructed (written) answer

Evaluated by means of a scoring rubric that operationalizes learning model


Hawaiian History Writing

Assignment: BayonetConstitution

Be sure to show the relationships among your ideas and facts.

Your essay should be based on two major sources:

1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution.

2. What you have learned from the readings yesterday.

Imagine you are in a class that has been studying Hawaiian history. One ofyour friends, who is a new student in the class, has missed all the classes.Recently, your class began studying the Bayonet Constitution. Your friend isvery interested in this topic and asks you to explain everything that you havelearned about it.

Write an essay explaining the most important ideas you want your friend tounderstand. Include what you have already learned in class about Hawaiianhistory, and what you have learned from the texts you have just read. Whileyou write, think about what Thurston and Liliuokalani said about the BayonetConstitution, and what is shown in the other materials.


EXCERPTS from HAWAIIAN HISTORYPRIMARY SOURCE DOCUMENTS

LILIUOKALANI

For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they becamewealthy, their greed and their love of power increased. Although settledamong us, and drawing their wealth from resources, they were alien to usin their customs and ideas, and desired above all things to secure their own personal benefit.

Kalakaua valued the commercial and industrial prosperity of his kingdomhighly. He sought honestly to secure it for every class of people, alien ornative. Kalakaua’s highest desire was to be a true sovereign, the chiefservant of a happy, prosperous, and progressive people.

And now, without any provocation on the part of the king, having maturedtheir plans in secret, the men of foreign birth rose one day en masse, calleda public meeting, and forced the king to sign a constitution of their ownpreparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.


ExplanationScoring Rubric

General impression of content quality

Principles or concepts

Prior knowledge

Examples

Misconceptions

Argumentation


Template #2Prior Knowledge and

Explanation

Explicit measurement of knowledge domain before explanation

Uses short answer or selected response

Helps interprets explanation performance


Template #3Knowledge Representation

Same prompts

Key aspects of ideas, supporting facts and views and their relationships

Relationship is explicit

Organizational options

Core and peripheral Hierarchical Cause-and-effect Chronological

Expert scoring


History


Genetics


Bicycle Pump


Brief History of MBA in LAUSD

Content understanding and problem-solving models

Explanation templates

4 subjects, 3 grade levels, 2 languages

Purposes: (1) to clarify expectations; (2) to provided instructionally embedded assessment; (3) to get a measure of school performance

CRESST-managed teacher involvement


LAUSD Process

Teacher design teams

LAUSD standards first

Adapted to success standards

Training cadre of scorers

Training trainers

Supervising scoring


LAUSD Process (cont’d)

Shift in four-topic focus (capacity based) to two and then to one, now back to two

Continual assaults

Curriculum mandates

Accountability pressure (API)

Long-term embedded approach resurfacing


Present LA Situation

Administered in 2, 3, 4, 5, 6, 7, 8, 9

Purpose added regarding promotion

Teacher scored with an audit reported to school

Local sub-districts managing activity

Ongoing validity studies

District review of alternative assessments



CRESST Validation Studies

Score reliability

Task and rater generalizability

Stability of student performance over time

Relationships among measures

Instructional sensitivity

Opportunity to Learn (OTL)

Effect of school composition on performance

Cut-score modeling


LAUSD Grade 7 Students’ Achievement Levels: Comparison of 2002 California Standards Test and Performance

Assignment Scores

Evidence of Predictive Validity

73.7%

49.1%

25.1%

9.3%

21.4%

36.3%41.2%

31.7%

4.9%

14.5%

33.8%

59.0%

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Not Proficient PartiallyProficient

Proficient Advanced

2001 Performance Assignment Scores

% o

f S

tud

ents

in

Dif

fere

nt

Cat

ego

ries

o

f P

erfo

rman

ce i

n C

A S

tan

dar

ds

Tes

t

Below Bas ic

Bas ic

Above Bas ic


LA Scale-Up

Cost and time driven

Maintained by board and union support

Transfer of responsibility

Reduction in technical quality

Reduction in range of measures

Positive evaluation from independent group focusing on changing teaching practices


Continuing R&D Areas

New contexts

Trade-offs (limited number of templates vs. wide range of formats)

Performance over time

Scalability in the long run

Authoring systems to support teacher-developed assessments linked to large- scale assessment


Summary of Assessment Knowledge Requirements

Knowing why

Knowing what to assess: content plus cognitive demands (problem solving, communication, learning to learn, teamwork, content knowledge)

Knowing how: transfer (application to other topics and situations)

Reflecting: applying MBA to teaching


Model-Based AssessmentCognitive Families

ContentUnderstanding

ProblemSolving

Teamwork andCollaboration

MetacognitionCommunication

Learning


Context for Success of Knowledge-Based Reform

Local ownership of knowledge

Infrastructure and stability

Capacity to investigate

Learning

Congruence or peace with external mandates


Usable Knowledge and Support May Get to Useful

Knowledge

For assessment knowledge to be useful, it depends upon the context, capacity, and communication of the teaching system

For assessment knowledge to be useful to students, it must go to the heart of why, what, and how they learn

Documents

C R E S S T / U C L A Model-Based Assessment: Why, What, How, How Good, and What Next? Eva L. Baker National Research Council, Board on Testing and Assessment