Washington’s New Accountability Index

WERA Spring ConferenceMarch 2009

Pete Bylsma, EdD, MPAConsultant to the State Board of Education

http://www.sbe.wa.gov/spa.htmbylsmapj@comcast.net

1

Overview of Presentation• Rationale for creating a new system• Guiding principles• Process used to develop the proposed system• Proposed indicators and outcomes• Ratings and tier assignments• School and district results in 2007 and 2008• Handling special cases• Recognition categories and criteria• AYP vs. new system• Identifying “Priority” schools and districts• Remaining Issues

2

Why a state accountability system?• Legislative mandate for the SBE to “adopt objective,

systematic criteria” for successful schools and districts and for those in need of assistance

• Accountability is part a comprehensive state education reform system, which includes standards & assessments and assumes adequate funding

• Federal accountability system (AYP) is viewed as unfair, too complicated, and punitive

• State provides most of the education funding• Supports legislative efforts to revise funding system

3

Guiding Principles1. Be fair, reasonable, and consistent

2. Be transparent and simple to understand

3. Be valid and accurate

4. Use existing data

5. Rely on multiple measures

6. Include state assessment results from all grades and subjects

7. Apply to as many schools & districts as possible

8. Provide multiple ways to show success and earn recognition

9. Rely mainly on criterion-referenced measures

10.Use familiar concepts

11.Use concepts of AYP when appropriate

12.Be flexible enough to accommodate future changes4

Process to Date• Initial proposal reviewed by an advisory panel• Convened diverse set of advisors to provide input on

technical details for a revised proposal• Applied proposed rules to data to ensure validity, made

adjustments to increase precision• Ongoing feedback received from OSPI and others• Presentations and feedback at SBE meetings and ESDs

– More than 60 districts have received data for their schools

• Further refinements taking place

5

Proposed Outcomes/Indicators

Five OutcomesResults from 4 assessments (reading, writing, math, science)

aggregated together from all gradesExtended graduation rate for all students

Four Indicators1. Achievement by non-low income students (% meeting

standard/ext. grad rate)

2. Achievement by low income students (eligible for FRL)

3. Achievement vs. Peers (Learning Index and ext. grad rate controlling for ELL, low-income, special ed., gifted, mobility)

4. Improvement (change in Learning Index from previous year)

Creates a 5x4 matrix with 20 outcomes6

Outcome/Indicator Matrix

Outcomes

Indicator Reading Writing Math Science Ext. G.R. Avg.

Non-low inc. achievement

Low inc. ach.

Ach. vs. peers

Improvement

Average Index *

7

* Simple average of all rated cells

Outcome/Indicator Benchmarks & Ratings

8

Indicator Reading Writing Math Science Ext. grad rate

Achievement of- Non-low inc.- Low income (% met standard)

% MET STANDARD RATING90 – 100% 780 – 89.9% 670 – 79.9% 560 – 69.9% 450 – 59.9% 340 – 49.9% 2< 40% 1

RATERATING> 95 790 – 95% 685 – 89.9% 580 – 84.9% 475 – 79.9% 370 – 75% 2< 70% 1

- Achievement vs. Peers(Learning Index)

DIFFERENCE IN LEARNING INDEX RATING> .20 7.151 to .20 6.051 to .15 5-.05 to .05 4-.051 to -.15 3-.151 to -.20 2 < -.20 1

DIFFERENCEIN RATE RATING> 12 76.1 to 12 63.1 to 6 5-3 to 3 4-3.1 to -6 3-6.1 to -12 2< -12 1

Outcome/Indicator Benchmarks & Ratings

9

Indicator Reading Writing Math Science Ext. grad rate

- Improvement(Learning Index)

CHANGE IN LEARNING INDEX RATING> .15 7.101 to .15 6.051 to .10 5-.05 to .05 4-.051 to -.10 3-.101 to -.15 2< -.15 1

CHANGEIN RATE RATING> 6 74.1 to 6 62.1 to 4 5-2 to 2 4-2.1 to -4 3-4.1 to -6 2< -6 1

Achievement vs. “Peers”• Recognizes context affects outcomes (“statistical neighbors”)

• Makes “apples to apples” comparisons using multiple regressionto control for 5 student variables—percent ELL, low-income (FRL), special education, mobile, gifted

• Include expenditure level (adj.) in district regression

• Regressions weighted by number of students assessed

• Separate regressions for each type of school (e.g., elementary, middle, high, multiple grades)

• Non-regular schools not included in regression (not rated)

• Viewed as significant new contribution

• Others are starting to do this type of analysis

• Complexity presents a communications challenge10

Linear Regression

0.0 25.0 50.0 75.0 100.0

Pct low income

0.000

1.000

2.000

3.000

4.000

Mat

h L

earn

ing

Inde

x, 2

007

Math Learning Index, 2007 = 3.26 + -0.01 * PctLowIncR-Square = 0.70

A

B

Illustration of Ach. vs. Peers

11

7

1

4

Tier Index Range

Exemplary 5.50 – 7.00

Very Good 5.00 – 5.49

Good 4.00 – 4.99

Acceptable 2.50 – 3.99

Struggling 1.00 – 2.49

Priority (eligible for Innovation Zone)

12

Tier Names and RangesAssigned to a “tier” based on index score

School and District Results (2007)Tier Index

RangePct. of schools

Pct. of districts

Exemplary 5.50 – 7.00 4.0% .3%

Very Good 5.00 – 5.49 6.5% 3.1%

Good 4.00 – 4.99 29.4% 29.9%

Acceptable 2.50 – 3.99 48.7% 60.8%

Struggling 1.00 – 2.49 11.3% 5.8%

Priority (eligible for Innovation Zone)

TBD TBD

228 schools were in the Struggling tier with 74,000 students (1 in every 14 students); 98 were alternative schools or served other special populations.Over 2-year period, 7.5% of all schools averaged < 2.50 (Struggling tier)

17 districts were in the struggling tier in 2007 (average 1,000 students) 13

Key Changes Made• Slow down implementation schedule

• Avoid double-counting students by creating separate ratings for low income and non-low income students

• Move to a 7-point scale, add another tier

• Exclude results for ELL students in first 3 years or until reaching Level 3 on WLPT (whichever comes 1st)

• Think of other ways to hold alternative school accountable• Exclude improvement indicator for those with very high

performance• Simplify recognition system

14

Example Results(Middle School, 2008)

15

Indicator Reading Writing Math ScienceGrad Rate Average

Non-low inc. ach. 5 6 3 4 4.50Low-inc. ach. 3 4 1 1 2.25Ach. vs. peers 5 7 4 6 5.50Improvement 3 7 4 7 5.25Average 4.00 6.00 3.00 4.50 4.38

District Accountability

• Uses the same indicators, outcomes, benchmarks, ratings, recognition system as schools

• Combines all the grades together(no separate results for each grade band)

• N is 10 for entire district (very few left out)

• Control for funding level for “peer” analysis

16

0

5

1 0

1 5

Ele m e n tary M id d le /Jr H i H igh A lte rn ative

Num

ber o

f sch

ools Ex e m p lary

Ve ry Go o d

Go o d

A c c e p tab le

Stru gglin g

17

Showing Results for All Schools(2008)

0

2 ,5 0 0

5 ,0 0 0

7 ,5 0 0

1 0 ,0 0 0

Ele m e n tary M id d le /Jr H i H igh A lte rn ative

Num

ber o

f stu

dent

s Ex e m p lary

Ve ry Go o d

Go o d

A c c e p tab le

Stru gglin g

18

Showing Results for All Schools(2008)

19

Showing Index for All Schools

(2008)

1 .0 0 2 .0 0 3 .0 0 4 .0 0 5 .0 0 6 .0 0 7 .0 0

DCB

A lterna ti ve A

RQPONM

LKJI

HGFEDCB

Elem enta ry A

DCB

M iddle S chool A

IHGFEDCB

H ig h S chool A

20

Propose exempting ELL results in the first 3 years of enrollment or until advanced proficiency achieved on the WLPT (whichever comes first)

• Results currently count in AYP in 2nd year of enrollment

• Research found it usually takes at least 3 years to achieve “academic” proficiency in English

• OSPI requested this policy but was denied; WA could still use this policy when calculating the index

• ELLs would still take the test in their 2nd year, WLPT results would be made public to increase accountability

• Would not affect many students (most ELLs in tested grades have been in US for 3 years or have achieved intermediate proficiency); would have a small positive impact on index where there are many ELLs present

Special Case #1 – ELL Exemption

21

• Title III application submitted in January• Redefines “proficiency” for ELL cells in AYP based on

WLPT- Uses sliding scale: Lower scale score counts as proficient on the WASL for

each grade, subject, and WLPT Level. For example:

Grade 3 student in Level 2 of WLPT is considered proficient withWASL scale score of 359; if in Level 3, considered proficient with 388

- Cut scores are lower for math

• No scores proposed yet for writing or science• Concept similar to WASL-Basic for special education

(lower scale score considered proficient), rejected by feds• Has almost no effect on AYP results, but produces useful data

about student progress toward meeting standard

OSPI Proposal to US Education Dept.

22

• Designation at district discretion, some miscoding occurs (a special program may not be given a separate school code)

• Most are small: average 145 students, range from 1 to 2200+

290 schools (14%) with 42,000 students (4%)

• About half serve only secondary (usually 9-12), many serve K-12

• Many types exist- Correctional facilities (jails, prisons, detention centers)- Contracts for vulnerable groups (treatment centers, group homes)- Schools for specific types of students (ELLs, gifted, spec. ed.)- Growing number serve students learning via the Internet- Parent Partnerships- Some don’t use a normal school approach or a normal building

(e.g., college campus, night school)

Special Case #2 – Alternative Schools

“Alternative” Schools Index Results

23

• Given this diversity, no “peer” indicator is computed for non-regular schools

• Half do not have any index results due to N<10, many only have a few cells rated

• Average index is much lower (2.94 vs 3.95 for “regular”)

but some perform very well (magnet schools, gifted)33% in Struggling tier, 9% in Very Good or Exemplary tiers

• Usually made AYP (below the radar due to N< 30 cont. enrolled)

Two Options Being Considered

1: Use regular process, use in-depth analysis to determine if school is using best practices, showing progress, and their role/status/resource level within the district

2: Allow schools serving high-risk/special populations to use additional measures to determine their tier (e.g., credits earned, attendance, gains on pre-post tests)

24

Propose excluding improvement indicator when reaching very high achievement levels

• Improvement is difficult when achievement is very highCannot receive the highest rating (7) after the Learning Index

reaches 3.85/4.00 and when the graduation rate reaches 94%

• Would go into effect when achievement is in the top range two years in a row, making it impossible to receive the maximum rating in the second year

• No school or district has yet to met these criteria for the assessments; 11% met these criteria for graduation rate

• School/district could decide to include the results

Special Case #3Improvement by High Performers

Recognition System• Guiding principles

– Multiple ways to demonstrate success, earn recognition

– Criterion-based system

Theory of Change: People are motivated more by successthan guilt or blame; need clear, challenging, attainable goals; small wins build momentum for continued improvement

• Use same accountability matrix, receive recognition when meeting specific benchmarks

• Based on 2-year average

25

Recognition Recommendations1. Provide recognition to schools & districts for each of

the 20 cells when the 2-year average is at least 5.50 and when the index is averages at least 5.00 (21 cells)

2. Require some minimum conditions to occur• Must have ratings of 5-7 in both years for 20 “inner” cells

• Recognition for non-low income cells in reading and writing requiresa minimum 2-year average of the low income group of 4.00

• Must have at least 4 cells rated each year for recognition in Index cell

3. Coordinate recognition system with OSPI

4. Give recognition each fall via public announcement (joint SBE/OSPI press release), post on web

5. Consider giving special recognition in some cases26

Suggested Cells and Criteria

Outcomes

Indicator Reading Writing Math Science Ext. G.R. Avg.

Non-low inc. achievement

5.50*Low inc. ach.

Ach. vs. peers

Improvement

Average 5.00*

27

Recognize results in 20 cells + Index (21 total)

* Minimum 2-year average rating to earn recognition

EFFECTDistribution of Recognition in21 Cells

• Math, science, andlow income cells recognized least often

• Districts receive less recognition than schools

288 .0 %

1 3 .2 %

3 3 .4 %

2 3 .3 %

3 1 .0 %

1 2 .4 %

2 9 .7 %

3 0 .1 %

2 7 .4 %

2 6 .8 %

2 3 .2 %

1 3 .6 %

0 .1 %

0 .7 %

1 3 .1 %

9 .5 %

3 5 .4 %

5 .1 %

1 7 .8 %

2 5 .7 %

4 0 .7 %

0 % 2 0 % 4 0 % 6 0 % 8 0 % 1 0 0 %

Accounta bility Index

Ext. g ra d ra te im provem ent

Science im provem ent

M a th im provem ent

W riti ng im provem ent

R ea ding im provem ent

Ext. g ra dua ti on ra te a m ong peers

Science a m ong peers

M a th a m ong peers

W riti ng a m ong peers

R ea ding a m ong peers

Low -incom e ext. g ra d ra te

Low -incom e s cience a chievem ent

Low -incom e m a th a chievem ent

Low -incom e w riti ng a chievem ent

Low -incom e rea ding a chievem ent

N on-low incom e ext. g ra d ra te

N on-low incom e s cience a chievem ent

N on-low incom e m a th a chievem ent

N on-low incom e w riti ng a chievem ent

N on-low incom e rea ding a chievem ent

Complement Existing Recognition

Federal recognition (competitive)– Blue Ribbon Schools– Academic Achievement Award– Distinguished Schools Award– Title I Improvement Award

OSPI recognition (status TBD)– Schools of Distinction (top 5%)– Improvement Award (schools and districts)

29

Current Recognition

Outcomes

Indicator Reading Writing Math Science Ext. G.R. Avg.

Non-low inc. achievement

Low inc. ach.

Ach. vs. peers

Improvement

Average Index

30

Note: Improvement is measured in different ways.

Role of AYP• Widespread belief that AYP is overly complex, less

inclusive, too narrow, not transparent, and not valid for identifying success or most in need

• Recommend using disaggregated results to help determine which schools and districts need the most help (“Priority” designation)

• Will submit proposal to US Ed. Dept. to use the index to determine AYP and “steps” of improvement

31

Advantages of Proposed System• More valid – Uses performance of all students in more subjects, more differentiated

than a “Yes/No” system, does not count students multiple times, looks at improvement and how the outcomes compare with “peers”

• More inclusive/comprehensive – Uses smaller minimum N (10 students across entire school/district), includes results of all students (regardless of how long they have been enrolled), includes both writing and science (helps prevent a narrowing of the curriculum), uses Learning Index to measures performance across the range of assessment results (reduces the focus on bubble kids)

• Less volatile over time – Assessment results are combined across all grades (not individual grades) so N is larger and students do not change much from year to year

• More transparent – Does not include a margin of error, benchmarks are the same over time and among different subjects, fewer subgroups and rules, schools and districts evaluated using same criteria, same N for all groups and for both schools and districts

• Encourages high expectations/standards – “Compensatory” model gives no incentive to lower standards so all can be counted as “proficient”

• Requires deeper analysis before final decisions are made – Not solely a quantitative approach

32

Identifying Priority Schools

• Those in “Struggling” tier undergo deeper analysis to determine which need more help

• Many issues to examine:– Contextual issues– In-depth analysis of disaggregated WASL/WAAS– Federal AYP results– Other data

• Identification process similar to that used now by OSPI

33

Issues to be Resolved• Determine how index is used to made AYP

determinations. Ideas include the following:Minimum N=10, no margin of error used, no results by gradeOnly results from index (average of cells) are used over timePossible ways of not making AYP (warning year)

- In Struggling tier- In Acceptable tier, no increase in improvement average in 2 straight years- In Good/Very Good tier, index declines > .70 in 2 straight years- District with 20% of students in schools not making AYP

Entering a “Step”- Not making AYP two years in a row- Always requires “deeper data dive” before decision is made about moving

(may stay in warning year or current step, even if not making AYP)

Sanctions would differ, could be fewer steps for schoolsMove back a step if making AYP

34

Issues to be Resolved

• How to include other outcome measures andshow “reciprocal” accountability- Student and staff characteristics- District revenues and expenditures- Community and state support- More student outcomes

• How SBE and OSPI systems work together- Will develop joint proposal for US Education Dept.- Show results on Report Card- Determine how index results relate to recognition/state

assistance- What happens to those in a Step now

35

Issues to be Resolved

• Timing of implementationPhase I this fall for Recognition (?)

Announcing index results- Provides public more accurate picture of school and district

performance (AYP results provide a false picture)- Gives educators with useful data for improvement efforts- Introduces concepts before used for federal accountability- Provides SPI with additional data for assistance decisions

Phase II depends on federal response and resources to support those in Priority tier

36