1 Oregon Content Standards and Assessment System Evaluation Prepared for the Oregon Department of Education by WestEd Dr. Stanley Rabinowitz Dr. Edynn

1

Oregon Content Standards and Assessment System Evaluation

Prepared for the Oregon Department of Education

by WestEd

Dr. Stanley Rabinowitz

Dr. Edynn Sato

June 22, 2007

2

Background: Overview

One of several key projects funded by the Gates Foundation

Independent evaluation by WestEd, a non-profit educational research, development, and service agency

3

Content Standards and Assessment System Evaluation: What?

WestEd will: Review the content standards

(all grades, academic content areas) Evaluate the structure of the content

standards Evaluate alignment between the state

assessments and the content standards Make recommendations for improvement of

structures and systems

4

Key Deliverables

Preliminary report of an initial review of selected content standards and grades using initial protocol/criteria

Final report for the comprehensive review of the content standards

Final report for the alignment of assessments to content standards

Final report on the structure and quality of Oregon’s content standards and assessments

5

Comprehensive Evaluation of Content Standards

• English language arts• Mathematics• Science• Social sciences• Arts• Second languages• Physical education• Health education• Technology• English language proficiency

6

Standards Evaluation Studies

The analyses aimed to address the following key questions:

1. Do Oregon’s content standards adequately represent the knowledge and skills that all students should know and be able to do?

2. Do Oregon’s content standards reflect the appropriate breadth and depth of the content area?

3. Do Oregon’s content standards have the clarity and consistency needed to adequately guide instruction and assessment?

7

External Referents

CONTENT AREA LEVELS STATE REFERENT NATIONAL REFERENT

English Language Arts

10 GradesK-8 and CIM

Indiana Draft 2009 NAEP Reading Framework and 2011 NAEP Writing Framework (grades 4, 8, and 12), McREL Speaking and Listening (grades K-3, 5-7, CIM)

Mathematics 10 GradesK-8 and CIM

Indiana NCTM Principals and Standards for School Mathematics (2000) and Curriculum Focal Points for K-8 Mathematics (2006)

Science 10 GradesK-8 and CIM

Indiana AAAS benchmarks (2001)

8

External Referents (continued)


Social Sciences: Geography

Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM

Washington NAEP Geography Framework (2001)

Social Sciences: History

Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM

Washington NAEP History Framework (2006)

Social Sciences: Civics

Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM

Washington NAEP Civics Framework (2006)

Social Sciences: Geography

Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM

Washington NAEP Economics Framework (2006)

9



Arts Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM

Washington Compendium Arts Standards developed by McREL (2006)

Physical Education Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM

Indiana Moving into the Future: National Standards for Physical Education (2006)

Health Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM

Indiana National Health Education Standards: Achieving Health Literacy

Educational Technology

1 Level (CCGs) Washington ISTE Standards for Students (2000)

10



Second Languages Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM

Indiana American Council on the Teaching of

Foreign Languages (1999)

English Language Proficiency: Forms and Functions

5 Levels of Language Forms and Functions

NA Current National Research on Forms

and Functions

11

Standards Evaluation Studies (continued)

Overall Quality Oregon’s English Language Arts (ELA) standards overall are of good

quality and provide breadth and depth of coverage of Reading, Literature, Writing, Speaking, and Listening. The standards generally are clearly written, focus on important skills/concepts for instruction, and are assessable.

Oregon’s mathematics standards overall are of good quality and reflect a breadth and depth of content coverage. The standards generally are clearly written, focus on important skills/concepts for instruction, and are assessable.

Oregon’s science standards overall are of good quality in terms of clarity and importance for instruction. However, in all grades except for grade 8, analysts determined that the standards did not reflect a range of depth of knowledge. For all grades except for grade 1, analysts determined that the standards did not reflect a breadth of knowledge. With regard to consistency, analysts determined that standards were inconsistent (language, skills, knowledge) in grades 4, 8, and CIM. And, in all grades except for grades 3, 5, 8, and CIM, there were issues of assessability.

12


Overall Quality (continued) Oregon’s social sciences standards overall are of good quality

in terms of depth, breadth, consistency, importance for instruction, and assessability. However, overall these standards lack clarity.

Oregon’s arts standards overall are of good quality and cover a breadth and depth of content. The standards generally are clearly written, focus on important skills/concepts for instruction and are assessable.

Oregon's physical education standards overall are of good quality in terms of breadth and depth of content coverage, consistency across benchmark grades, importance for instruction, and degree of assessability. However, analysts determined that these standards are not easily understood due to lack of clarity and precision of language.

13


Overall Quality (continued) Oregon’s health standards overall are of good quality in terms

of depth, consistency, importance for instruction, and assessability. For all benchmark levels except for Benchmark 2, the standards overall are of good quality in terms of breadth of content. And, overall these standards lack clarity.

Oregon’s educational technology standards overall are of good quality in terms of depth, breadth, consistency, importance for instruction, and assessability. However, overall these standards lack clarity.

Oregon’s second languages standards overall are of good quality in terms of depth, breadth, clarity, consistency, importance for instruction, and assessability.

14


Overall Quality (continued) Generally, Oregon’s English language proficiency (ELP)

forms and functions are presented in a clear and useful format. The proficiency level descriptors clearly describe and differentiate the language skills of students at each level. However, the definitions of language function and forms of language appear incomplete. All the language functions except one are appropriate—literary analysis is typically considered English-Language Arts content rather than a language function. All the language forms except one are appropriate—the language of propaganda is not a form consistent with other elements of form presented by the state; rather it appears to be a genre.

15


Alignment to External Referents Oregon’s ELA standards generally aligned to the Indiana, NAEP

and McREL standards in terms of overall content order/sequence and depth of content. However, while the breadth of the Oregon standards appeared comparable to the NAEP and McREL standards, they were not as comparable to the Indiana standards.

Oregon’s mathematics standards were compared to Indiana’s mathematics standards as well as to the NCTM Principles and Standards for School Mathematics (2000) and Curriculum Focal Points for K–8 Mathematics (2006). Oregon’s mathematics standards generally aligned to the NCTM standards and to the Indiana standards with regard to overall depth and breadth of content coverage. However, there was not overall congruence between Oregon and Indiana’s content ordering and sequencing.

16


Alignment to External Referents (continued) Oregon’s science standards were compared to Indiana’s science

standards and the AAAS benchmarks (2001). – Generally, the Earth and Space Science standards aligned in

terms of overall depth of content; however, they did not align in terms of content order/sequence or overall breadth of content.

– Generally, the Physical Science standards aligned to the Indiana standards in terms of content order/sequence, overall depth, and overall breadth of content. The Oregon standards also aligned to the AAAS standards in terms of overall depth and breadth of content; however, they did not align in terms of content order/sequence.

– Generally, the Life Science standards aligned to the Indiana standards in terms of content order/sequence, overall depth, and overall breadth of content. However, they did not align to the AAAS standards in terms of content order/sequence, overall breadth, or overall depth of content.

17


Alignment to External Referents (continued) Oregon’s social sciences standards were compared to Washington’s

social science standards and the NAEP frameworks.

– Generally, the Geography standards aligned in terms of overall content order/sequence with Washington’s standards. However, they did not align in terms of content order/sequence with the NAEP framework or in terms of the overall depth or breadth of content with the Washington standards or the NAEP framework.

– Generally, the History standards did not align to the NAEP framework in terms of content order/sequence, depth or breadth of content. Nor did these standards align in terms of overall breadth with the Washington U.S. History and World History standards. However, the Oregon standards aligned to the Washington standards in terms of content order/sequence and overall depth.

18


Alignment to External Referents (continued) Oregon’s social sciences standards (continued).

– Generally, the Civics standards aligned in terms of overall content order/sequence with Washington’s standards and the NAEP framework. However, they did not align in terms of overall depth or breadth of content with either the Washington standards or the NAEP framework.

– Generally, the Economics standards aligned to the Washington standards and the NAEP framework in terms of content order/sequence, depth and breadth of content.

At the broad level of the content standard, there was a high degree of alignment between Oregon’s arts standards and those of Washington and McREL. In terms of content order / sequence, Washington’s standards spiral content across different benchmarks in a manner similar to Oregon; many of McREL’s benchmark level standards are not spiraled across different benchmarks. Both external referents show similar depth of content to Oregon’s standards.

19


Alignment to External Referents (continued) Generally, Oregon's physical education standards, the National

Association of Sports and Physical Education Standards (NASPE), and the Indiana standards share a common purpose. In broad terms, the nature of the content knowledge/cognitive information and the types of skills students are expected to learn and be able to demonstrate are comparable across the three sets of standards. The Oregon differs in depth of content coverage from the NASPE and Indiana State standards.

Oregon’s health standards were compared to Indiana’s health standards as well as to the National Health Education Standards (NHES). Oregon’s standards generally aligned to the Indiana and NHES standards with regard to overall content order/sequence and breadth of content coverage. However, there was not overall congruence in terms of depth of content.

20


Alignment to External Referents (continued) Oregon’s educational technology standards generally aligned to the

Washington and NETS standards in terms of overall content order/sequence, depth of content and breadth of content.

While Oregon and Indiana both have the same number of proficiency levels, a comparison of the depth and breadth of the second languages standards across levels in both states shows that the Indiana standards start at a slightly higher level than those in Oregon, and reach a higher level. Oregon’s proficiency levels appear to be based on ACTFL levels Novice-Low through Intermediate-Mid. The standards statements differ in level of detail. Generally, Oregon standards are written more broadly, and Indiana’s are more detailed and specific. Oregon’s set of Functions Supporting Standards describe a similar level of detail as Indiana’s standards.

21


Alignment to External Referents (continued) Oregon’s English language proficiency (ELP) forms and

functions were compared to current national research and literature on forms and functions (e.g., TESOL, CALLA, Butler, et al.). All but one of Oregon’s language functions (literary analysis) are appropriate and relevant for ELD instruction. The external referents include functions that are currently not present among those in the Oregon ELP standards.

22

Standards Evaluation StudiesGeneral Recommendations

Evaluate standards for redundancy within a grade. Evaluate standards for level of detail—some standards

have elaborate detail and others do not. Differentiate between detail that describes the curriculum

to be taught and the intended learning outcomes (content skills, knowledge).

Evaluate coherence across levels of specificity for a given standard—e.g., strand, common curriculum goals (CCGs), grade-level standards—in terms of appropriateness of progression of specificity (general/broad to specific) and clarity of relationships.

23

General Recommendations(continued)

Verify that expansion/extension of content reflected in standards (e.g., references that include student’s knowledge of the community, knowledge of historical periods) is reasonable and intentional.

Evaluate the clarity and effectiveness of small variations in the wording/language of specific standards across the grade levels—verify that the differences are intentional and reflect clear, real differences in expectations of students vis-à-vis the content/skill.

24

General Recommendations(continued)

Embed definitions of key content terminology, if interpretations of terminology can be varied AND the intent of the state is to promote consistency

Make sure the language clearly conveys expected skills and complexity

Evaluate coherence and consistency of content emphases across grades

25

Core Standards “Content Standards” define the knowledge and skills that a

state’s students are supposed to learn The definition of “Core Standards” includes/reflects:

– Knowledge and skills central to a content area– Significant/essential content/skill “targets” in a content

area– State-specific priorities in a content area– Learning expectations and performance goals for all

students Core standards subsume other standards and/or provide a

super-structure for a coherent system of content standards Some states identify core standards in order to ensure that

these standards are addressed in each grade and developed across grades—local curricula are expected to align to these standards

26

Core Standards (continued)Considerations: Will the standard provide students with

knowledge and skills that will be of value beyond a single test date? (What endures?)

Will the standard provide knowledge and skills that will be of value in multiple disciplines? (What has leverage?)

Will the standard provide students with essential knowledge and skills that are necessary for success in the next level of instruction? Beyond school? (What is really necessary for “success”?)

[Based on Reeves, 2004]

27

Essential Skills “Essential Skills” refer to the core abilities/process

skills that all students are expected to develop and apply across disciplines

Essential skills typically span content areas and facilitate the effective delivery of content knowledge across disciplines

Essential skills typically are expected to be demonstrated in a variety of ways and in different contexts

Essential skills reflect state’s priorities Local curricula are expected to incorporate these

skills—the skills are frequently embedded within existing learning standards, curricula, and assessments.

28

Essential Skills (continued)Considerations: Will the skills be of value beyond a single

test date? (What endures?) Will the skills be of value in multiple

disciplines? (What has leverage?) Will the skills be necessary for success in

the next level of instruction? Beyond school? (What is really necessary for “success”?)

[Adapted from Reeves, 2004]

29

Next Steps Additional alignment analyses and

information Additional information regarding

assessments and standards (ODE questions)

Analyses of “core standards” Analyses of “essential skills” Continued consideration of

feedback/information provided by key stakeholders through surveys, meetings, etc.

30

For more information about the standards and assessment evaluation:

ode.state.or.us/go/real

For more information about WestEd:

www.wested.org

Documents

1 Oregon Content Standards and Assessment System Evaluation Prepared for the Oregon Department of Education by WestEd Dr. Stanley Rabinowitz Dr. Edynn