STUDENT LEARNING OBJECTIVES: An Emerging Framework for Design

Lessons from the Teacher and Leader Evaluation Multi-State Network

April 26, 2012

DRAFT

1

PurposeAs increasing numbers of states begin reforming their systems of educator evaluation to reflect a significant emphasis on student outcomes, there is increasing demand to codify sound frameworks of Student Learning Objectives (SLOs), for measuring growth beyond the traditional growth or value-added models. Several leading states are following the practice of early

adopter districts and implementing SLOs as a measure of student growth. From that work, key elements of a framework have emerged.

This document seeks to outline an early framework of key decision points and questions states should address in designing SLO systems. Moving forward, states will also need to address early and continuously districts needs with regard to capacity building and training to ensure successful implementation.

2

Overview and Rationale

Recognizing that student learning is an important part of improving educator practice, SLOs will play a critical role in the design of new systems of educator evaluation in many states. Promising practice suggests that multiple, meaningful measures of

student growth are needed for strong evaluation systems, which can be particularly difficult for teachers who do not have data from state assessments and growth models.

The use of SLOs in statewide systems of evaluation is a new and quickly evolving practice that will be refined and improved over time, drawing from the lessons-learned and promising practices of early-adopters.

Recognizing that present focus must be on establishing a basic system to meet immediate needs, states should not lose sight of the long-term vision in which SLOs can drive and significantly improve teaching practice.

Meaningful educator evaluations and the use of SLOs can and should support broader state college and career ready reforms, e.g. Common Core implementation.

3

Opportunities and Challenges

SLOs provide opportunities for collaboration among teachers and evaluators, and provide a system of evaluation based on alignment among instruction, standards, assessment, and evaluation. Conversely, in order to ensure validity, reliability, and comparability of measures among teachers, states must address key challenges. Opportunities: SLOs take best practices in instruction and translate those up to inform policy. SLOs provide an opportunity for instructionally focused conversations between evaluators and teachers, use

of baseline data to set meaningful targets and tailor instruction, and a system for monitoring and evaluating progress throughout the year.

Use of SLOs can help drive professional learning for educators. SLOs provide an opportunity for greater authenticity and broader evidence in educator evaluation. SLOs allow for a direct connection to measures of student college and career-readiness including rigorous

knowledge and higher order skills.

Challenges: Lack of existing assessment coverage for many grades and subjects. Significant capacity building required to ensure the knowledge base necessary for principals and teachers to

utilize high quality assessments and targets. Intensive and sustained training needed on an ongoing basis to ensure high quality implementation and

continuous improvement, with a long-term focus on improving teaching practice4

Student Learning Objectives Framework

Designing a process for student learning objectives requires states to address three core areas, and continuously improve over time.

I. Assessment Selection

II. Target Selection

III. Reviewing Results and Scoring

Training and district capacity-building

Evaluation and

continuous improvement

Foundational Issues

5

Early Adopter Districts

Along with several other large school districts, District of Columbia Public Schools (DCPS) was an early adopter of SLO processes as part of their evaluation system. These districts provide important lessons learned and insights to states as they develop systems for SLOs.

DCPS: The district is in its third year of using an SLO process as part of teacher evaluation. Evaluation outcomes are also linked to compensation. Teachers in non-tested grades and subjects develop SLOs that count as 10% of their overall evaluation rating. The district provides a list of suggested assessments and targets for most grades and subjects, and all assessments must be rigorous, aligned to standards, and appropriate for the teacher's students. Targets can be based on either growth or mastery and are developed collaboratively by the teacher and the principal. When SLOs are set, a clearly defined rubric is developed to determine a teacher's SLO score based on student performance.

6

Emerging State Models

Several leading states (DE, GA, IN, NY, RI) have recently begun piloting the use of SLOs as part of statewide systems of educator evaluation. These states provide an initial framework for designing systems of SLOs and offer early lessons learned as they continue to refine and improve SLO implementation. This presentation takes a closer look at two of those models:Indiana: SLOs are used for all teachers in the state default model, while districts developing their own model can choose to use them. In the default model, SLOs comprise 10%, 15%, or 20% of the overall evaluation, depending on the availability of growth model data. State guidance outlines criteria for assessment selection and specifies an order of preference that requires teachers use state assessments if available. Teachers must select a learning objective that applies to all students in a class and a targeted objective focused on low-performing students. SLOs are set collaboratively among teachers and approved by principals.

7

Emerging State Models, continued

Rhode Island: All teachers must set between two and four SLOs. For teachers in non-tested grades and subjects, SLOs will comprise 51% of their evaluation. SLOs will be combined with value-added data for teachers in tested areas. Teacher and school-developed assessments must be reviewed on a quality rubric. Student learning targets must be aligned with school-level goals and be horizontally and vertically consistent across a school. Targets should be rigorous but attainable and if met, indicate that students have the essential knowledge and skills for the next level of instruction. Teachers and evaluators agree on SLOs at a beginning-of-year conference and meet mid-year to discuss progress and determine if any adjustments are needed.

8

Foundational Issues

There are several foundational issues that must contemplated as states design systems of SLOs.

Key questions include:

What are the state's values and core objectives for use of SLOs?

What should be measured through SLOs? Growth of all students? Subgroup

growth? Both?

How can comparability of measures and process be ensured across teachers,

schools, and districts?

How is sufficient validity and reliability of the overall system ensured to

support meaningful results and potentially high-stakes use?

What is the role of the state (regulations, systems of support, monitoring)?

What is the capacity at the local level? 9

The first step in measuring student learning is to select appropriate assessments or sources of evidence. Assessments may vary in format and design, but should meet minimum standards.Key criteria include:• Meaningful: Reflects knowledge and skills that are valuable to student and

school context, including alignment to state standards • Stretch: Assesses student growth across a wide range of performance levels• Precision: Addresses validity, accuracy, and specificity • Rigor: Captures true mastery of skills, includes higher order thinking skills • Attribution: Provides data that can be assigned to individual teacher efforts• Basis for Measurement: Includes baseline data to benchmark progress• Timely: Promptly provides data for use in evaluations • Adaptable: Includes the potential for accommodations and modifications

I. Assessment Selection

10

Key issues to consider: Determine what types of assessments qualify, for example:

common assessments aligned to state standards, assessments purchased or created by schools or districts, classroom assessments, which may include teacher-created assessments, project-based assessments, performance-based assessments.

Establish and clearly articulate an order of preference for use of different assessment types.

Consider how the state will address the issue of assessment cost.

Consider how teachers, schools, and districts can work collaboratively to develop assessments.

Determine a process for approval of assessment selection. (Will the state play a supportive role by providing guidance and training to districts, or will it exercise stronger authority?)

I. Assessment Selection, continued

11

DCPS Assessment Selection

DCPS provides a list of suggested assessments for most grades and subjects, and outlines key criteria for assessment selection, including: • Alignment: Must be aligned to state standards and appropriate for

teacher's class and students; exemplar/recommended SLOs developed by district for most subjects/grades.

• Stretch: Not addressed, though district provides guidance on setting appropriate targets for students at various performance levels.

• Precision: Not addressed• Rigor: Must include multiple question types including higher levels of

Bloom's taxonomy, real work application of problem-solving, and brief constructed response.

• Attribution: Required.• Basis for Measurement: Performance tasks must have a pre-test

component.• Timely: Required.• Adaptable: Required.

12

Indiana Assessment Selection

State guidance for the default model specifies that teachers use state assessments if available, or, if not, common school or district assessments, or individual classroom assessments, if there are no other teachers in the content area. The state provides a pre-approved list of assessments, and requires evaluators approve any additional assessments. State guidance outlines key criteria for assessment selection:• Alignment: Must cover most key content standards; the state requires

completion of the Alignment and Coverage Check Chart.• Stretch: Must assess pre-requisite objectives from prior years and objectives

from the next year's course• Precision: Must adhere to state established order of preference.• Rigor: Required – the state provides an Assessment Rigor Analysis chart.• Attribution: Required.• Basis for Measurement: Required in target-setting guidance. • Timely: Must provide score reporting in time for evaluation• Adaptable: State provides guidance for local decision-making.

13

Rhode Island Assessment Selection

Rhode Island requires all teachers to set between 2-4 SLOs. Teacher and school-developed assessments must be reviewed on a quality rubric reflecting the following criteria:• Alignment: Must be aligned to state standards (and show evidence of this

alignment); must be paced with curriculum; common assessments must be used if available.

• Stretch: Must include varied response types (e.g. multiple choice, constructed response, etc.) to assess complexity of content

• Precision: To ensure validity, must include at least three versions of a test if given multiple times throughout year.

• Rigor: Must include some evidence of higher-order thinking• Attribution: Required.• Basis for Measurement: Required in target-setting guidance. • Timely: Must provide score reporting in time for evaluation• Adaptable: Must make accommodations available to ensure equitable testing

14

II. Selection of Learning Objective Targets

After assessments are selected, it is essential to develop strong student learning targets.Key considerations include: Measurable: Do targets distinguish different levels of performance mapped to a

clear rubric? Breadth: Do targets address the performance of all students or a subset or both?

Do they address all content standards or a subset or both? Rigor: How do targets reflect the nature and various levels of learning? How

should the state define and articulate levels of rigor for setting targets? Meaningful: How are student learning targets specific and valuable to student and

school context? Consistency: Do teachers of the same grade or subject have the same or similar

targets? (Are SLOs vertically and/or horizontally aligned?) Comparability: How are student learning targets comparable to other measures of

student growth used for other teachers? Timing: By what point in the school year must targets be set?

15

II. Selection of Learning Objective Targets

A clear process for setting and approving SLOs must be developed that articulates requirements and areas of flexibility.

What is the process for setting objective targets?

Who sets them?

Who approves them, and what's the process for ensuring quality?

How are teachers involved?

How clearly articulated are different levels of performance? (Will the state provide standards for each level? Exemplars?)

What is the process for tracking teacher progress toward targets and adjusting instruction as necessary?

Is there a process for revising learning objectives mid-year if it becomes clear they can be improved? In what circumstances is this appropriate?

16

District of Columbia Targets

DCPS requires principals to review and approve all SLOs to ensure that they meet quality criteria and conducts a central office review of feasibility. The district provides sample SLOs for most grades/subjects. Measurable: Targets can be mastery or growth and must be anchored in

baseline data. Breadth: Targets must cover all students that a teacher instructs and cover

the majority of content for a given course. Rigor: Targets must represent an ambitious but achievable level of

performance and should set a goal of 1.25 years of academic growth. Meaningful: Targets should be based on the specific needs of the students a

teacher instructs. Consistency: Sample targets are provided for most subjects and grades. ELL

targets must be based on state AMOs. Standard rubrics are available for some subjects.

Timing: Targets must be set at initial conferences (by October) with data available in time to inform a final evaluation (in June). 17

Indiana Targets

Indiana provides teachers and evaluators with a checklist of key steps for determining targets. It also provides exemplar and non-exemplar SLOs. Measurable: Targets must be measurable and based on previous performance.

Targets focus on student mastery. Breadth: Teachers must select a learning objective that applies to all students in

a class and a targeted objective focused on low-performing students. State guidance outlines three levels of student preparedness on which to base targets. Targets should cover all major standards and skills.

Rigor: Relies upon the judgment of the teacher and evaluator. Meaningful: In setting targeted objectives, teachers must consider those

students that enter the course inadequately prepared and may choose to focus on a few key standards or all standards.

Consistency: Teachers in subjects/grades covered by an approved assessment must adhere to state established content mastery standards. The state provides guidance on mastery using other assessments.

Timing: Targets must be set at initial conferences (by October), with a mid-course check-in to discuss progress (in January -February).

18

Rhode Island Targets

Rhode Island provides teachers and principals with a checklist of criteria to evaluate targets. The state also provides exemplar and non-exemplar SLOs. Measurable: Targets must be measurable and based on baseline/historical data

or previous classes' performance. Targets can be mastery or growth. Breadth: Targets must cover all students, but can include multiple tiered groups

for subpopulations of students. Targets should cover all major standards and skills.

Rigor: Targets must correspond with at least one year of learning and attainment suggest that students are on track in that subject/course.

Meaningful: Targets must include a rationale for the selection of the target based on student needs.

Consistency: Teachers across subjects/grades within a school should have the same targets. Targets should be vertically aligned with school and district priorities.

Timing: Targets must be set at initial conferences (by October) and then revised (in January) if there are major changes in class composition or other factors that would necessitate a change.

19

III. Reviewing Results and Scoring

A specific process for review of results and scoring of SLOs must be developed as part of the broader evaluation system.

Key considerations include: Evidence: Who gathers and evaluates the evidence? What amount of evidence

is sufficient? Assessment of Progress: How is a judgment about teacher progress on a

student learning objective made? By whom? By when? Teacher and Leader Input: What is the level of professional discretion in

making judgments? What is the process for teacher feedback and input? External Validation: Is there a process to ensure broader, external validation

of judgments? Overall Scoring: How are judgments and evidence incorporated into the larger

evaluation summative ratings process? Ongoing Development: How is this process linked to summative feedback for

professional learning?20

DCPS Reviewing Results and Scoring

DCPS principals judge evidence of student learning through SLOs outcomes and assign a rating aligned with the rubric established at the initial conference. Evidence: Teachers must provide evidence of student performance in meeting

targets at their year-end conferences with evaluators. Assessment of Progress: DCPS provides a general rubric with suggested criteria for

each rating level. Principals and teachers agree on a rubric at the beginning of the year. At the end of the year, principals validate student scores and assess student progress against this rubric.

Teacher and Leader Input: Teachers propose SLOs, but they must be approved by principals (at an initial conference).

External Validation: The DCPS central office reviews all SLOs for feasibility (but not quality).

Overall Scoring: SLOs are rated according on a 4-level rating scale that is specified at the initial conference.

Ongoing Development: Principals and teacher discuss student learning outcomes at the end of the year to identify a final rating and determine professional growth strategies. This supplements ongoing conversations with principals and master educators on observational feedback. 21

Indiana Reviewing Results and Scoring

Indiana evaluators judge evidence of student learning through SLOs outcomes and, following discussion with the teacher, assign a rating based on professional judgment. Evidence: Teachers must provide evidence of SLOs 48 hours prior to their summative

conference with evaluators. Evidence of class objectives should only include student scores on the assessment, while targeted objectives may also require additional evidence, such as classwork, student projects, etc.

Assessment of Progress: Evaluators use professional judgment to evaluate student learning in comparison to established targets and then determine a rating, based on four performance levels, for each objective.

Teacher and Leader Input: SLOs are set collaboratively among teachers and approved by evaluators.

External Validation: The state (for teachers covered by pre-approved assessments) or the evaluator approves the content, rigor, and quality of assessment as acceptable before approving each SLO.

Overall Scoring: All teachers receive SLO ratings on a 4-point scale. A weighted average is used to determine the final SLO score. (Class and targeted objectives count equally.)

Ongoing Development: Evaluators and teacher discuss student attainment of learning objectives, and the evaluator establishes ways to support the teacher. 22

Rhode Island Reviewing Results and Scoring

Rhode Island principals judge evidence of student learning through SLOs outcomes and assign a rating based on professional judgment. Evidence: Teachers must provide evidence of SLOs prior to their final

conferences with principals. Assessment of Progress: Evaluators use professional judgment to evaluate

student learning in comparison to established targets and then rate performance against all targets on a five-level scale.

Teacher and Leader Input: Teachers propose SLOs, but they must be approved by principals (at an initial conference).

External Validation: The principal must rate the content, rigor, and quality of assessment as acceptable before approving each SLO.

Overall Scoring: All teachers receive SLO ratings on a 5-point scale based on professional judgment of their performance across 2-4 SLOs.

Ongoing Development: Evaluators and teacher discuss student attainment of learning objectives, what strategies were effective and ineffective, and what the teacher learned about curriculum, assessments, and instruction through the SLO process. 23

Questions?

Robin Gelinas, EducationCounsel: robin.gelinas@educationcounsel.com

Margie Yeager, EducationCounsel: margery.yeager@educationcounsel.com

Questions?

24