Slide 1, 9-9-04

Moving From Challenge To Action:

Accountability Supporting Student Learning

Joan L. Herman

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

CRESST ConferenceUCLA, Los Angeles, CA

September 9, 2004

Slide 2, 9-9-04

Overview

• Role of assessment in improving student learning

• Evidence on how things are working

• Advice on moving ahead

Slide 3, 9-9-04

Theory of Action: Motivation

• Establish standards

• Develop measures

• Set performance goals

• Leaven with incentives/sanctions

• Schools will be pay attention• Attend to standards/assessment results

• Work to improve student performance/learning

Slide 4, 9-9-04

Theory of Action: Technical System

• Assessment results will provide accurate/valid information at multiple levels

• Data will be well used to inform planning and decision making

• Educational system will well use data to engage in continuous improvement

Slide 5, 9-9-04

Is the Motivation System Working?

• The good news:• Districts and schools taking action to align

curriculum, etc with standards

• Teachers listen to the signal and focus instruction accordingly

• The not so good news• Curriculum narrowed

• Possibility of dual curriculum

• Concern for teacher’/principals’ professionalism and morale

Slide 6, 9-9-04

Some Critical Questions:

• How can we design standards and assessment systems that optimally focus instruction but don’t stifle it?

• What happens to motivation when schools hit the wall and performance levels out?

Slide 7, 9-9-04

Is the Technical System Working?

• Feds are asking for right kinds of validity evidence

• Alignment

• Validity for EL and SWD

• Accuracy: 75% probability that CST correctly classifies students’ proficiency level (Rogosa)

• Reliability and year to year score fluctuations a continuing challenge

Slide 8, 9-9-04

Is the Technical System Working?(cont.)

• Validity of gains suspect

• Feasibility of AYP targets problematic

• Thorny technical/psychometric issues in dealing with 40+ possible indicators of AYP

Slide 9, 9-9-04

Critical Questions

• Validity of AYP designations: Do they identify the right schools?

• Is there a better way to establish and operationalize improvement targets?

Slide 10, 9-9-04

Alignment: A Critical Lynchpin

• Are we clear enough about what we want:

• Teachers to teach?

• Test developers to test?

• Is face validity of content and cognitive demand enough?

• Research suggests difficulties

Slide 11, 9-9-04

One Alignment Example

• Study used 20 content experts

• Considered each item and rated

• Topic addressed

• Depth of knowledge (DOK)

• Content centrality

• Results - Clear majority agreed on

• Topic for 35/42 items

• Topic AND DOK for 14/42 items

• Topic, DOK and centrality for 5 items

Slide 12, 9-9-04

Implications

• How can teachers teach to “standards” if experts don’t agree on what topics and levels of DOK mean?

• Current reality: alignment is a moving target

Slide 13, 9-9-04

A Moving Target Example from NY(Alan Tucker)

• Standard: Apply formulas to find measures such as length, area, volume, weight, time, and angle in real world situations

• Representation of Math A Test, June, 02 - June, 03 (4 tests)

• # items: 1-3

• Range of content and complexity

Slide 14, 9-9-04

Critical Questions

• If we want teachers to teach to the standards rather than the test, how can we be sure that developers and teachers share the same understandings?

• How can we specify standards and assessments in ways that clearly communicate expectations without unacceptable curriculum narrowing?

• Can we frame expectations in ways that are feasible for all students?

Slide 15, 9-9-04

Teacher Assessment: The Neglected Key

• If we believe in the power of assessment to support learning, we must put that power where learning occurs

• Many capacity challenges

• Some realistic remedies:

• Teacher pre-service education

• Requirements for materials developers

Slide 16, 9-9-04

A Full Agenda for Deliberation and Action