Measures of Student Learning CFN 204 Summer 2013 CFN 204 · Diane Foley · Network Leader 1

Measures of Student Learning

CFN 204Summer 2013

CFN 204 · Diane Foley · Network Leader 1

CFN 204 · Diane Foley · Network Leader

Purpose & Context This work, particularly around MoSL, is very nuanced and

complex. There are places where some policy decisions are still being made.

Throughout the day we want you to capture your reflections and thinking to help you process and distinguish between what will become feedback to the DOE and what you need to move the work forward in your schools.

The goal for today is to understand, so that you are ready to lead the work in your schools.

The day is packed with content, but we will be making connections throughout the day and the institute so that you can see how to pull it all together for your schools.

2


DOE Resources and Training

3

The following resources will assist principals and the School Local Measures Committee throughout the decision-making process and implementation:

• Measures of Student Learning Guide: You can find more information on assessment and measurement options, as well as sample selections in the Guide, found at http://www.learndoe.org/dhr/recording-mosl-overview/.

• Interactive Selection Tool: The DOE will release a tool to select assessments and measurements in August.

• MOSL Specialists: Experts to help networks and schools implement in the fall.

Additional training opportunities: • MOSL Specialists will be providing additional support to networks,

who will in turn support schools, as the school year begins.

http://www.learndoe.org/dhr/recording-mosl-overview/


MoSL Options Template Required assessments are checked and shaded in gray.

State Measures decisions made by principal are color-coded in orange and Local Measures decisions made by the school committee are color-coded in blue (same as MoSL guides).

For State Measures, teachers who teach both ELA & math must have an ELA and math component for their State Measure, but not their Local Measure. The Local Measure can be any subject they teach. For schools that departmentalize, teachers are only responsible for the content they teach.

Pending policy decisions, for Local Measures, school committees may select up to 2 measures per group of similar teachers, i.e. same subject and grade. (2 measures = 1 assessment with 2 populations, or 2 assessments)

For example: 1 assessment, 2 populations: ELA state exam – school and school lowest

third 2 assessments: ELA state exam – school, ELA performance tasks –

individual

4


All teachers will receive:

Two different Measures of Student Learning (40%) State Measures Local Measures

Initial planning conference and summative end of year conference to include discussion of student outcomes (Will be covered on Aug. 8th)

Multiple measures provide a more valid, robust picture of teacher performance, providing teachers with multiple sources of feedback

MoSL: Measures of Student Learning (40%)

5


State and Local Measures

6

Local Measures

State Measures

Although principals make decisions about State Measures and the School Local Measures Committee makes decisions about Local Measures, these

two decisions go hand in hand.

There are many reasons why it is important for the two measures to be determined in collaboration:

1. Teachers will receive the best picture of student growth with multiple measures, each telling a slightly different story of student performance.

For example: Principal reinforces a focus on school-wide learning by selecting a school-wide measure on a State Assessment for a given group of teachers’ State Measure. School Local Measures Committee then decides to base Local Measure on individual teacher performance on a 3rd party assessment (where possible) to include a variety of assessment perspectives.

2. Per law, teachers may not have the same exact measure for both State and Local (e.g., 4th grade teacher cannot have an individual classroom growth measure on ELA/Math state tests count for 40% of overall rating. Her Local Measure would have to measure another population either the lowest 1/3 or group measures such as school or grade).

3. The more alike the State and Local Measures are to each other, the more likely a teacher will receive the same rating on each. This may have positive or negative consequences.


Long-term Vision

More assessment options each year

Schools re-select

options each year, learning

with experience

LEARN WITH EXPERIENCE

MORE OPTIONS

7


Terminology – Common Mix Ups

State assessments can be used for both state and local measures. When we are talking about state tests, it doesn’t always mean we are talking about state measures.

The state made decisions about requirements on both the state and local side. Principals have a role in both as well.

Local assessments – NYC performance assessments – are required for different grades on both the state and local side. Therefore, when we talk about local assessments it does not always mean we are talking about local measures.

8

State Measures (20%)

The MOSL category that includes State Assessments or, where there are no State Assessments, a list of allowable assessments that can be used (chosen by principal where there is choice).

Local Measures (20%)The MOSL category with options chosen by the School Local Measures Committee and submitted to the principal who may accept the recommendation or opt for the default measures.


Individual Class

ELA Lowest 1/3, Grade,

SchoolState ELA Grade 4/5

State Local


Terminology – Common Mix Ups

Even though there are only 3 main decisions that need to be made for each measure, decisions impact each other and all options are not available in all cases.

10

Assessment(s)Student growth is measured based on one or more of the following types of assessments: State Assessments, 3rd Party Assessments, or NYC Performance Assessments.

Target Population

Refers to the students included in the measure – this can include students school-wide, in a particular grade level, or only those students a teacher teaches. If the same assessment is selected for the State and Local Measures, the target population for the Local Measures will be the lowest third of students.

Growth MeasurementsThe method by which student growth is measured on a given assessment. For MOSL, options include teacher/principal goal-setting or growth models.

For each measure, the following three items need to be considered:


NYC Assessment Options

11

Information on assessments available for specific grades/subjects is available in the Measures of Student Learning Guides

(http://www.learndoe.org/dhr/recording-mosl-overview/).

The following parameters guided the selection of assessments available for each grade and subject:

• State rules regarding allowable assessments: SED released rules about what assessments are allowed in each grade and subject, including specific rules about 3rd Party and NYC Performance Assessments.

• Assessment quality: NYC assessment options were evaluated to ensure instructional value, validity and reliability.

• Security and comparability: Assessments must be available across all NYC schools and students cannot have access to them prior to administration. School-created assessments are not allowed for outcome assessments.*

• Existing practice in schools: Whenever possible, NYC assessment options build upon existing assessment practices in NYC schools.

*Where baseline selections need to be made, schools may use school-based/created assessments as baselines. Schools may NOT use school-based/created assessments for final outcomes.

http://www.learndoe.org/dhr/recording-mosl-overview/


Assessment Type*

Target Populatio

n Options*

*

Description Example (K-5)

State Assessment

s

IndividualGradeSchool

Measure the performance of students based on

state-created assessments.

3-8 ELA & Math4 & 8 Science

NYSESLATMS Regents: Integrated Algebra,

Living Environment, Earth Science

3rd Party Assessment

s

IndividualSchool

Are created by assessment experts.

Assessment format varies (multiple choice,

performance based, etc.)

Math: K-2 Discovery Math, 3-8 Scantron

ELA: 3rd grade Scantron, 3rd grade Degrees of Reading Power (DRP)

NYC Performanc

e Assessment

s (developed

w/ NYC Teachers)

Individual Authentic tasks (e.g., evidence-based essays), scored against a common rubric. Created by DOE, NYC teachers, curriculum

and assessment experts to be used as Measures of

Student Learning in teacher evaluation.

K-8 ELA (2 options: tasks & running records)

3rd grade Math4, 6 & 7 Science

6-8 Social StudiesIntegrated AlgebraLiving Environment

NYC Assessment Options

12

NYC

Only state-approved 3rd Party Assessments can be used. Not all NYC assessments are on this list. A final list will be approved on August 1. **NYC is not using all 3rd Party Assessments that were previously approved for use throughout the state.** NYCDOE is funding 3rd Party Assessments used for teacher evaluation.

3rd Party & NYC Performance Assessments:Include both pre- and post-tests.Teachers (per law) can score their own baselines/pre-tests, but CANNOT score their own post-test results.


Activity: Preview NYC Performance Task

Review the NYC Performance Task and aligned Rubric sample.

Use the template to guide your reflections and thinking by considering the following prompts: 3 key points you learned from the NYC Performance Task

and Rubric.

2 questions or implications of the assessment.

1 message to present to school/local measures committee.

13


Baseline Assessments

14

A baseline assessment is one given in the beginning of the year to assess student starting points.

NYC Performance Assessments and approved 3rd Party Assessments come with baseline (pre) assessments.

The principal must select a baseline assessment when none is included or where there is a State Assessment without historical data (e.g., 3rd grade).

There are only a few instances where baselines will need to be selected for Local Measures. Where this is the case, the principal, not the committee, selects the baseline.

Schools that select school-based/created baselines must use goal setting for that measure.

Assessment

Baseline Recommendation

Other Baseline Options

3rd Grade Math/ELA NYC Performance Assessments Scantron ELA & MathDegrees of Reading Power (DRP) ELASchool based

ESL Prior Year NYSESLAT LAB-R (where prior year NYSESLAT isn’t available)School based

4th Grade Science3rd Grade Math, NYC Science Performance Assessment

School Based

8th Grade Science7th Grade Math, NYC Living Environment Performance Assessment

School Based


Target Population Options

Where are they available? All 3 options – Individual, Grade, and School – are available for the state tests for testing grade teachers.

For non-testing grades teachers, the only options available for state tests are grade and school. The only population available for NYC Performance Assessments is individual. For 3rd party assessments, the only options are individual and school (this does not mean the assessment

must be administered to all students in the school, it only means that the teacher will be evaluated based on everyone in the school who took the assessment).

Nuances: In some cases these distinctions overlap. For example, for a 4th grade science cluster teacher, her

individual student population will also be grade and school if she teachers the entire 4 th grade.

Requirements: For required State and Local Measures, the target population will always be the individual. Teachers who do not give approved individual classroom-level assessments will use either a grade- or a

school-wide measure. If an assessment is repeated across measures, the target population must differ. If a school wants to use

the same target population with the assessment, the Local Measures must focus on the lowest third of performers. You may only select the lowest 1/3 if the larger population was already selected as a measure.

15

Target population options include:

Individual only those students that a teacher teaches (only applicable if a teacher administers an assessment to his/her own students).

Grade students in a given grade level.

School all students within the school.


Measurement: Growth or Goal Setting

Growth Model Goal SettingGives teachers credit for the degree to which students exceed growth as compared to similar students.

Benefits & Challenges: Does not introduce additional work in

schools

Better able to account for unexpected outcomes resulting from unfamiliar, new assessments

Growth model score results are not available until after assessments have been administered (i.e., the following spring/summer)

Does not actively facilitate brainstorming and discussions about individual student performance and goals

Must use citywide assessments for baselines to select growth model

Teacher’s rating is based on the percentage of students who meet their target but does not consider the degree to which students fall below or exceed their target.

Benefits & Challenges: Particularly valuable for teachers/schools with

unique student populations or high mobility

Allows teachers and principals to individually tailor student goals

Requires additional time/resources

Targets must be set early in the school year, possibly before much diagnostic info is gathered

Setting goals may be challenging if: Teachers are not familiar with the

comparability between assessments Assessments are new or changing

Must use goal setting if using school-based/created baselines

16


Measurement: Growth Model Calculated by the state for 4-8 ELA & math;

calculated by the city for all other assessments including 3rd grade ELA & math, science, NYSESLAT, regents, etc.

Puts students in buckets according to 13 characteristics. Similar students are compared to each other to determine SGP.

Some characteristics describe the student

and some describe the classroom.

Controls for similar characteristics as the Progress Report, but in different ways.

The Progress Report controls for classroom characteristics in their peer grouping and comparison. For growth scores, classroom characteristics are embedded in the student comparison “buckets”.

The Progress Report accounts for student characteristics by adding weights to student scores, For growth scores, student characteristics are accounted for by putting students in buckets to compare them only to similar students.

17

Category Defining Characteristic of “Similar Students”

Academic History:

• Prior year test score, same subject• Prior year test score, different subject• Retained in grade• New to school in year other than entry

year• Average prior achievement and range

around average prior score in student’s class/course (same subject)

Student with Disability (SWD)

• Student has an IEP • SWD spends less than 40% of time in

general education setting• Percent SWD in student’s class

English Language

Learner (ELL)

• Student is an ELL• NYSESLAT scores• Percent of ELLs in student’s class

Poverty• Student poverty indicator (yes/no)• Percent poverty in a student’s class


Measurement: Growth Model

18

Ms. Smith’s Class

Prior Score

Current Score

SGP

Student A 450 510 45

Student B 470 500 40

Student C 480 525 70

Student D 500 550 60

Student E 600 650 40

If we compare student A’s current score to other students who had the same prior score (450), we can measure her growth relative to other students. We describe her growth as a “student growth percentile” (SGP).

Student A’s SGP is the result of a statistical model and in this example is 45, meaning she performed better in the current year than 45% of similar students.

To measure teacher performance, we find the mean growth percentile (MGP) for his or her students. To find an educator’s mean growth percentile, take the average of SGPs in the classroom. In this case:

Step 1: 45+40+70+60+40=255Step 2. 255/5=51

Ms. Smith’s mean growth percentile (MGP) is 51, meaning on average her students performed better than 51% of similar students.


Activity: Growth Model Understanding Check

Turn and Talk to your colleagues and discuss the two scenarios:

A teacher says to you: “I appreciate that the new evaluation system aims to measure growth, but my kids face so many challenges and just don’t grow as fast as other kids. I feel like I’m going to be disadvantaged by this new system.” In 2 minutes or less, explain how using a growth model will address her concern.

A teacher says to you: “I mean, we get these numbers called Student Growth Percentiles (SGPs), and we have no idea what they refer to. If Jose’s SGP is 65, does that mean he got 65% right on the test?” In one minute or less, explain the misconception this teacher has about SGPs, and what this number actually refers to.

19


MGPs and Statistical Confidence

87

Confidence Range

Upper

Limit

Lower

Limit

MGP

NYSED will provide a 95% confidence range, meaning we can be 95% confident that an educator’s “true” MGP lies within that range. Upper and lower limits of MGPs will also be provided.

An educator’s confidence range depends on a number of factors, including the number of student scores in their MGP and the variability of student performance in the classroom.

20


State Growth Model: Scoring and Results

NYC Teachers (n=10,544) Rest of State (n = 22,585)

Highly Effective 8% 6%Effective 76% 77%

Developing 10% 10%Ineffective 6% 6%

21

Most teachers scored Effective on the state growth score in 2011-12. The results for NYC were slightly higher than the rest of the state:

Note: SED has stated that it expects the future distribution of ratings to remain similar to what it was in 2011-12.


Goal-Setting Process1. Administer baseline

assessment

2. DOE sends predicted student targets

3. Teachers review DOE predicted targets

4. Principals approve or adjust targets

5. Administer end of year assessment

6. Teachers’ Ratings

Baseline assessment administered (not required for all assessments).

Report baseline assessment results.

DOE sends predictions for how individual students will perform.

Predictions are based on baseline performance, student achievement history, and student demographic characteristics.

Teachers may choose to adjust these targets based on additional information about their students.

Teachers submit student targets to principal.

Principal (or designee) report finalized student targets.

End-of-year assessment administered to students.

Teachers’ HEDI ratings are calculated with a conversion chart based on students’ performance on outcome assessments relative to their targets.

22


Goal-Setting Considerations

23

This is not the same as the goal-setting you may typically see in schools.

Goals are scored against a state conversion chart which makes the target-setting process difficult and non-intuitive.

Before considering goal-setting, make sure you understand the work and additional training this entails.


50% RuleAll teachers who have individual assessments (assessments for their students and subject) available must have State Measures that cover at least 50% of their students. Therefore, you may need to add additional measures to get to 50%.

Begin by accounting for courses that end in state exams that result in state growth scores (4-8 ELA & Math)

Next, add courses that end in state exams but do not lead to state growth scores. You must begin with the largest (highest enrollment) courses first.

Finally, add additional courses beginning with largest (highest enrollment) courses first.

50% of students covered is the minimum, where principals have choice, you may choose to cover a larger percentage.

24


Default Option

Strategic

Status Quo

School wide

Does not introduce additional

work

Accurate results

All Teachers


DEFAULT Default only affects the Local Measure and applies for the entire school. When you default, all local measure decisions and required assessments (ELA

NYC performance assessments) are eliminated and replaced with the default.

The local measure default is:

K-5 SED provided growth scores on 4-5 ELA & Math state tests Lowest 1/3 growth scores for teachers who have school-wide measures for

their State Measure

K-8 Schools SED provided growth scores on 4-8 ELA & Math state tests Lowest 1/3 growth scores for teachers who have school-wide measures for

their State Measure

K-3 & K-2 Schools An average of all measures on the State side.

Defaulting DOES NOT affect the State Measure. Principal’s are still required to make decisions highlighted in orange. All required assessments for the State Measure are still required.

26


DEFAULT - Nuances Pending upcoming policy decisions

If a committee does not make a decision for a particular grade and subject and it leaves it blank, the default option is automatically filled in and eliminates the required assessments.

For grades without required Local Measure options, the committee may select the default options.

If the committee selects the default options for a grade with required assessments, that means they are doing both the required assessments and the default.

27


Activity: Committee Decision Check

1. Review the committee’s choices for assessment and target population for errors.

Circle any selections that are not allowable within the guidelines.

Use your graphic organizer to note the error, why it is an error (reference the rule), and what suggestions you have for making changes.

2. Review the committee’s choices for assessment and target population for broader implications.

Draw a box around any selections that ARE allowable, but might have broader implications that are worth discussing before selections are finalized.

Use your second graphic organizer to note the selection and what broader implications this selection might have.

28


Activity: Problem Solving Scenarios – Part 1

Read and select 1-2 scenarios to discuss with your colleagues. Use the template to guide your conversation. For each scenario:

Describe the problem

Indicate the problem type (technical, cultural, contextual, indecision)

Brainstorm possible problem solving strategies

Identify 3 next steps for the school

29


Activity: State Measures DecisionsPractice using the MoSL Options Template to make your decisions for state measures for your teachers.

Use your Table of Organization: First, cross off any Pre-K teachers and teachers who do not

teach 40% of the time. Make a note that these teachers will be rated S or U.

Second, write the names of your teachers in the appropriate place in the left column of the MoSL Options Template.Exception: For teachers who do not have 50% or more

of their students in a particular grade and subject, make a note to include them on the last page of the template.

Check off the decisions you have made or are considering for your teachers.

Discuss your thinking and reasoning with your colleagues.

30


Activity: Connecting MoSL - Elevator Speech – Take 1

More than ever, the success of the school will be dependent on the clarity of school leaders vision and communication. In light of the new Principal Evaluation System, school leaders ability to clearly and concisely voice their “theory of action” will largely impact their Measures of Leadership Practice.

Over the course of the Summer Institute and throughout the year we will revisit the idea of the elevator speech as a way of working through how to best articulate the work we do.

Now, as a way to synthesize our thinking around MoSL and next steps for our school we will begin to draft our elevator speech regarding assessment in our school.

Use the elevator speech template to draft your thinking around the following prompts:

Goal & Action Steps: We are working on…by….

Data & Purpose: because…so that…..

Monitoring & Next Steps: The evidence we are looking for…. based on this data our next steps will be

31


Activity: Connecting MoSL - Considering our Community

Even though we may hold particular beliefs and a vision for assessment in our schools, depending on how your committee and teachers are processing this work, it may make sense to modify our decisions to honor/account for their perspective.

Turn and Talk to your colleagues and share:

How your teachers are reacting to or may react to this work

Does that align or not with where you are in this work

How might that impact your final decisions

32


Activity: Connecting MoSL - Elevator Speech - Take 2

Based on your considerations for your community, revise your elevator speech in “take 2” on the template.

How might this message impact your…

Meetings with your MoSL Committee

September faculty conferences

Initial planning conferences

Turn, Talk & Share with your colleagues.

33


Activity: Problem Solving Scenarios – Part 2

Use the Consultancy Protocol to discuss a challenge you are having at one of your schools.

Protocol overview: Presentation: Presenter provides overview of work and 1-3

questions (3 minutes)

Clarifying Questions: Colleagues ask presenter clarifying questions (3 minutes)

Probing Questions: Colleagues ask presenter probing questions (3 minutes)

Non-presenter Discussion: Colleagues discuss the presenters work and questions without the presenter (5 minutes)

Presenter Response and Open Discussion: Presenter responds and discusses work with colleagues (5 minutes)

Reflection/Debrief: Group reflects on the process (5 minutes)

34

Documents

Measures of Student Learning CFN 204 Summer 2013 CFN 204 · Diane Foley · Network Leader 1