Pla Methdology 3 19 2010

“Persistently Low Achieving” in Washington

Greg LobdellCo-founder and Director of ResearchCenter for Educational Effectiveness

greg@effectiveness.org

Today’s Outcomes• Understand the definition of

“Persistently Low Achieving” in Washington

• Understand the application of this definition in the context of WA

• Understand the Profile of these 47 buildings

• What’s Next…• Dialogue and Discussion

Project Background- The Why?

Definitions Persistently lowest-achieving: Schools

with three consecutive years of data in the lowest 5% in both reading and mathematics and secondary schools with a weighted average of graduation rates less than 60% over a three-year period.

4

Proficiency…

𝑅𝑒𝑎𝑑𝑖𝑛𝑔𝑃𝑟𝑜𝑓𝑖𝑐𝑖𝑒𝑛𝑐𝑦= 𝑁𝑢𝑚𝑏𝑒𝑟𝑂𝑓𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑠𝑀𝑒𝑒𝑡𝑖𝑛𝑔𝑅𝑒𝑎𝑑𝑖𝑛𝑔𝑆𝑡𝑎𝑛𝑑𝑎𝑟𝑑𝑇𝑜𝑡𝑎𝑙𝑁𝑢𝑚𝑏𝑒𝑟𝑂𝑓𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑠𝑇𝑒𝑠𝑡𝑒𝑑𝐼𝑛𝑅𝑒𝑎𝑑𝑖𝑛𝑔 NOTE: This is calculated ONLY when the Number of students is > 29.

Definitions Weighting is equal between reading and

mathematics Weighting is equal between elementary and

secondary schools Graduation rate weighted-average is based on

the number of students for each year Graduation rate is calculated as required in

Guidance on School Improvement Grants, January 21, 2010 consistent with C.F.R. § 200.19(b)

6

Definitions

• Title I eligible: Based on SY 2009-10 student data, a school is considered Title I eligible if:– Poverty percentage is 35% or more; or– The school’s poverty percentage is greater than or

equal to the district’s poverty average

• Lack of Progress: The school’s percent increase or decrease (slope of linear regression) over the most recent three-year period compared to the state slope

7

Progress…

42%45%

48%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

WASL2007 WASL2008 WASL2009

ABC Elementary

Slope == Progress

42%45%

48%

y = 0.03x + 0.39

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

WASL2007 WASL2008 WASL2009

ABC Elementary

3% Improvement

Per Year

Tier DefinitionsTier I School

Any Title I school in improvement, corrective action, or restructuring that- Is among the lowest-achieving five percent of Title I schools in improvement,

corrective action, or restructuring in the State or the five lowest-achieving such schools (whichever number of schools is greater); or

Is a high school that has had a graduation rate as defined in 34 C.F.R. § 200.19(b) that is below 60 percent over a number of years.

Tier II SchoolAny secondary school that is eligible for, but does not receive, Title I, Part A funds that- Is among the lowest achieving five percent of secondary schools or the five

lowest-achieving secondary schools in the State that are eligible for, but do not receive, Title I funds; or

Is a high school that has had a graduation rate as defined in 34 C.F.R. § 200.19(b) that is below 60 percent over a number of years;

Tier III SchoolAny Title I school in improvement, corrective action, or restructuring that is not a Tier I school.

10

Newly Eligible Tier I: Title I eligible elementary schools that are no

higher achieving than the highest-achieving school that is defined as a “persistently lowest-achieving school” in Tier I and that: Have not made AYP for two consecutive years.

Tier II: Title I eligible secondary schools that are (1) no higher achieving than the highest-achieving school that is defined as a “persistently lowest-achieving school” in Tier II or (2) high schools that have had a graduation rate below 60 percent over a number of years and that: Have not made AYP for two consecutive years. Is in Step 5 of Improvement with a declining

improvement trend11

Key References

• “SIG-G”: Guidance on School Improvement Grants under section 1003(g) of the ESEA of 1965. US Dept. of Education December 18, 2009 and updated January 20, 2010 http://www2.ed.gov/programs/sif/guidance20100120.doc

• Interim final requirements for School Improvement Grantshttp://www2.ed.gov/legislation/FedRegister/finrule/2010-1/012110a.pdf

12

Data Sources• 21 Independent data sources

– Demographic information by district: 2006, 2007, 2008, and 2009

– Demographic information by school: 2006, 2007, 2008, and 2009

– AYP & Title I information by school: 2006, 2007, 2008, and 2009

– Graduation rates by school 2006, 2007, 2008, and 2009

– 2009-10 Title I eligibility and status (Oct ‘09 data)

13

The “Alternative” Methodology…

Key Concepts

• ‘Persistently’ requires 3 years of data in Reading and Math

• Secondary: grades 7 – 12 (WAC 392-348-235)

• As SIG-G requires, proficiency calculated for all-students, continuously enrolled, following AYP Accountability Workbook– Minimum of N of 30 applied for all students

tested in a building by content area per year

16

Key Concepts

• Tier Size:– The number of schools in the consideration-

set (for Tiers I and II) is calculated based on Title and AYP status before “minimum N” rule is applied (as per SIG-G).

– Schools that had 2009 data but are closed in 2009-10 are removed from overall consideration-set before creating the sizes for each tier.

17

Defining Tiers: Tier I

18

Step 1: There are 2,066 schools in Washington State for which Adequate Yearly Progress is calculated

Tier I

Step 2: Of the 2,066 schools, there are a total of 920 Title I schools (removed 1146 schools that are not Title I)

Step 3: Of the 920 Title I schools, 464 schools are in improvement, corrective action, or restructuring (removed 456 schools that are not in improvement, corrective action, or restructuring)

Step 4: Given this data set, 5% of 464 is 23 schools

After N of 30 rule applied: 450 for Ranking

Defining Tiers: Tier II

19

Step 1: There are 2,066 schools in Washington State for which Adequate Yearly Progress is calculated

Tier II

Step 2: Of the 2,0066 schools, 1,017 serve secondary students (grades 7 through 10 ) (removed 1049 schools that serve no students in grade 7 through 10)

Step 3: Of the 1,017 schools, 499 are Title I eligible (removed 518 schools not eligible for Title I)

Step 4: Of the 499, 254 of these schools do not receive Title I funds (removed 245 that receive Title I)

Step 5: Given this data set, 5% of 254 is 12.7; 13 schools when rounded up.

After N of 30 applied: 190 Schools for Ranking

Key Concepts• Following SIG-G: Added Ranks Method

for 6 ranks– If there are N schools in consideration-set:

this results in a value for each school between 6 and6 x N

– If ranking 450 schools• If your school was top ranked in each year in

both reading and math your added-rank = 6• If your school was the bottom ranked in each

year in both reading and math your added-rank = 2700 (6 x 450)

20

Key Concepts

• FINAL rank-ordering (3-level sort):1. Schools in lowest-5% in BOTH reading and

math (at least once in each over 3 years)2. Total “added ranks”3. Progress

21

Added Ranks Method

22

Title-1-2010 St

ep

Read

ing-

09

Mat

h-09 2007

Reading Rank

2008 Reading

Rank

2009 Reading

Rank

2007 Math Rank

2008 Math Rank

2009 Math Rank

3-Yr Added Ranks

Yes 5 74.3% 22.8% 125 75 52 440 399 426 1517

Yes 5 74.5% 16.7% 347 403 50 393 437 444 2074Yes 5 43.8% 23.0% 413 399 424 435 420 425 2516Yes 2 43.1% 25.2% 431 432 428 415 402 413 2521Yes 4 42.2% 25.1% 441 445 431 423 423 414 2577

Yes 1 45.8% 17.6% 332 435 414 339 436 442 2398Yes 1 35.4% 21.5% 377 413 443 360 431 431 2455Yes 4 23.9% 20.7% 403 409 450 380 377 437 2456Yes 1 51.2% 21.4% 399 434 375 433 428 432 2501Yes 4 45.2% 21.4% 433 420 417 386 415 433 2504Yes 2 46.8% 20.8% 439 394 410 447 440 435 2565Yes 5 40.9% 24.2% 444 438 434 430 400 421 2567Yes 5 38.8% 26.1% 443 442 439 429 411 409 2573Yes 2 33.8% 22.7% 438 392 444 432 445 427 2578Yes 5 43.6% 29.1% 435 444 426 438 439 397 2579Yes 5 38.2% 21.2% 440 441 441 405 425 434 2586Yes 5 46.0% 19.8% 419 433 413 441 443 440 2589Yes 4 39.3% 16.3% 428 436 437 431 429 445 2606Yes 5 45.4% 20.4% 434 443 416 448 441 439 2621Yes 3 32.0% 16.7% 447 439 448 444 432 443 2653

Schools in lowest 5% in Both Reading and Math

Schools in lowest 5% only in 1 content area

Applying Sort-Order

23

2007 Reading

Rank

2008 Reading

Rank

2009 Reading

Rank

2007 Math Rank

2008 Math Rank

2009 Math Rank 3-

Yr A

dded

Ran

ks

3-yr Bott-5%

in BOTH

3Yr-Progress-Trend vs

State175 172 159 157 150 147 960167 142 133 164 180 175 961 No: -0.9%127 180 146 168 173 167 961 No: -2.5%183 168 164 163 149 139 966

185 188 188 182 179 183 1105 Yes No: -3.7%

Note two schools with “Added Rank” = 961. Progress defines who ranks above and below.

Summary Example

– School 4: in bottom 5% in both reading and math– School 3: Larger ‘added rank’ than 1 & 2– Schools 1 & 2: Tie in added ranks so next step is

“progress”

24

Tier 1: 450 schools in consideration set (ranks 1 to 450)School

2007 Reading

2008 Reading

2009 Reading

2007 Math

2008 Math

2009 Math

Added RanksProgress

1 403 386 418 436 437 428 2508 No: -1.0

2 405 413 403 417 433 437 2508 No: -2.8

3 416 445 441 420 421 425 2568 No: -0.5

4 444 448 419 449 449 449 2658 No: -4.5

Validation• US Department of Education

approval of methodology was the critical step!

• OSPI Student Information and Assessment division

• Integrated data-set validated back to individual components

• All “missing data” confirmed (or fixed)• Bottom-up and top-down creation of

tiers and the final lists

25

Who/Where/Why?

Profile of Tiers I and II

ESD101- Spokane ESD, 2, 4%

ESD105- Yakima Valley ESD, 13, 28%

ESD112- Vancouver ESD, 3, 6%

ESD113-Olympia/Coastal ESD,

4, 9%ESD114-

Bremerton/Olympic Peninsula ESD, 0, 0%

ESD121- Puget Sound ESD, 12, 26%

ESD123- Tri-cities ESD, 8, 17%

ESD171-Wenatchee/Okanogan

ESD, 2, 4%

ESD189- North Puget Sound ESD, 3, 6%

Geographical Distribution: Tiers I and II(Number of Schools and Percentage)

Alternative, 3, 6%

Institution, 0, 0%

Public School, 44, 94%

Tribal, 0, 0%

School Type: Tiers I and II(Number of Schools and Percentage)

Elem, 15, 32%

Middle, 21, 45%

High School, 6, 13%

Multi-Level, 5, 10%

School Level: Tiers I and II(Number of Schools and Percentage)

Step 1, 3, 6%

Step 2, 7, 15%

Step 3, 4, 9%

Step 4, 10, 21%

Step 5, 23, 49%

Not in improvement, 0, 0%

NCLB School Improvement Step: Tiers I and II(Number of Schools and Percentage)

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

American Indian Asian (incl. HI/Pac Isl.) African Amerian / Black Hispanic White

Tiers I and II: Ethnic Diversity(Percent of Enrollment)

Tier State

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

2006 2007 2008 2009

Tiers I and II: Poverty vs State(Percent of Enrollment)

Poverty State Poverty (OSPI)

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

2006 2007 2008 2009

Tiers I and II: ELL (Transitional Bilingual) vs State(Percent of Enrollment)

English Language Learners State ELL

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

2006 2007 2008 2009

Tiers I and II: Reading Proficiency vs. State(Percent Meeting Standard)

Reading Proficiency State Reading Proficiency

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

2006 2007 2008 2009

Tiers I and II: Math Proficiency vs. State(Percent Meeting Standard)

Math Proficiency State Math Proficiency

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

2006 2007 2008 2009

Tiers I and II: On-time Graduation Rate vs. State

On-time Graduation Rate State On-time Graduation Rate

Visually…

-12.0%

-10.0%

-8.0%

-6.0%

-4.0%

-2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

3-Ye

ar Im

prov

emen

t Rat

e

3-Year Combined Reading and Math: Percent of Students Meeting Standard

Elementary Schools: 3-Year Performance vs Improvement (1022 Schools)

WA Elems

ABC SD: Title I Elig.

ABC SD: Not Title I Elig.

Note: Schools with N<30 Excluded

Bottom 5%

Bottom 5%

Median

Median

Top 5%

Top 5%

-12.0%

-10.0%

-8.0%

-6.0%

-4.0%

-2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

3-Ye

ar Im

prov

emen

t Rat

e

3-Year Combined Reading and Math: Percent of Students Meeting Standard

Middle Schools: 3-Year Performance vs Improvement (324 Schools)

WA Middle Schools

ABC SD: Title I Elig.

ABC SD: Not Title I Elig.

Note: Schools with N<30 Excluded

Bottom 5%

Bottom 5%

Median

Median

Top 5%

Top 5%

Added Ranks Method

40

Title-1-2010 St

ep

Read

ing-

09

Mat

h-09 2007

Reading Rank

2008 Reading

Rank

2009 Reading

Rank

2007 Math Rank

2008 Math Rank

2009 Math Rank

3-Yr Added Ranks

Yes 5 74.3% 22.8% 125 75 52 440 399 426 1517

Yes 5 74.5% 16.7% 347 403 50 393 437 444 2074Yes 5 43.8% 23.0% 413 399 424 435 420 425 2516Yes 2 43.1% 25.2% 431 432 428 415 402 413 2521Yes 4 42.2% 25.1% 441 445 431 423 423 414 2577

Yes 1 45.8% 17.6% 332 435 414 339 436 442 2398Yes 1 35.4% 21.5% 377 413 443 360 431 431 2455Yes 4 23.9% 20.7% 403 409 450 380 377 437 2456Yes 1 51.2% 21.4% 399 434 375 433 428 432 2501Yes 4 45.2% 21.4% 433 420 417 386 415 433 2504Yes 2 46.8% 20.8% 439 394 410 447 440 435 2565Yes 5 40.9% 24.2% 444 438 434 430 400 421 2567Yes 5 38.8% 26.1% 443 442 439 429 411 409 2573Yes 2 33.8% 22.7% 438 392 444 432 445 427 2578Yes 5 43.6% 29.1% 435 444 426 438 439 397 2579Yes 5 38.2% 21.2% 440 441 441 405 425 434 2586Yes 5 46.0% 19.8% 419 433 413 441 443 440 2589Yes 4 39.3% 16.3% 428 436 437 431 429 445 2606Yes 5 45.4% 20.4% 434 443 416 448 441 439 2621Yes 3 32.0% 16.7% 447 439 448 444 432 443 2653

Consequence of this Methodology:• In very small schools, volatility (year to year) is

typically larger

• One year has a significant impact on “added rank”

-12.0%

-10.0%

-8.0%

-6.0%

-4.0%

-2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

3-Ye

ar Im

prov

emen

t Rat

e

3-Year Combined Reading and Math: Percent of Students Meeting Standard

High Schools: 3-Year Performance vs Improvement (311 Schools)

WA High Schools

ABC SD: Title I Elig.

ABC SD: Not Title I Elig.

Note: Schools with N<30 Excluded

Bottom 5%

Bottom 5%

Median

Median

Top 5%

Top 5%

Questions, Comments?greg@effectiveness.org

What About The WA Accountability Index?

• Reading, Writing, Math, & Science• Those outcomes are each measured using four

indicators: – achievement of students who are not from low-income families; – achievement of students from low-income families; – achievement of all students when compared to “peers” (those with

similar student characteristics, such as the percentage of students who have a disability, are learning English, are designated as gifted, come from low-income families, and are mobile); and

– improvement in the achievement of all students from the previous year.

• The average of the resulting 20 measures comprises the overall index

• SB6696 and Required Action…

Innovative Leadership

Getting Ahead of the Curve

Innovative Leadership

• Proactively working on internal “turnaround” projects– Segmentation of schools in a

performance management framework– Comprehensive School Review- the

“Deep Dive”– Aggressive research-based

interventions

Consider- If this was your district…

-12.0%

-10.0%

-8.0%

-6.0%

-4.0%

-2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

3-Ye

ar Im

prov

emen

t Rat

e

3-Year Combined Reading and Math: Percent of Students Meeting Standard

Elementary Schools: 3-Year Performance vs Improvement (1022 Schools)

WA Elems

ABC SD: Title I Elig.

ABC SD: Not Title I Elig.

Note: Schools with N<30 Excluded

Bottom 5%

Bottom 5%

Median

Median

Top 5%

Top 5%

What would you do as a leader?

• Policy to solution or is policy the problem?

• Beliefs drive behavior…or does behavior drive beliefs?

• Institutions: “collective values” or “collective interests”?More about this after dinner with Gene…

Questions, Comments?greg@effectiveness.org

Student Achievement

CEEData Foundation/Comprehensive School Profile

Com

preh

ensi

ve P

lann

ing

Org

aniz

atio

nal M

anag

emen

t

Fina

ncia

l Man

agem

ent

Mon

itorin

g of

Inte

rven

tion

Professional Developm

ent

Leadership

Teaching and Learning

Increased Student Achievement

References You Can UsePrimary• Beat The Odds (2006). Morrison Institute for Public Policy (2006). Why Some Schools With

Latino Children Beat the Odds…and Others Don’t. Tempe, AZ.: Morrison Institute for Public Policy, Arizona State University, jointly with Center for the Future of Arizona. (aka: “Beat The Odds (2006) ).

• Elmore, R. (2004). Knowing the Right Things to Do: School Improvement and Performance-Based Accountability. Washington, D.C.: National Governors Association- Center for Best Practices.

• Marzano, R. (2003). What Works in Schools: Translating Research Into Action. Alexandria, VA: ASCD.

• Fixen, D.L. et al. (2005). Implementation Research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Mental Health Institute, The National Implementation Research Network (FMHI Publication #231)

• School Turnarounds (2007). Public Impact (2007). School Turnarounds: A review of the cross-sector evidence on dramatic organizational improvement. Public Impact, Academic Development Institute- prepared for the Center on Innovation and Improvement. Retrieved from: http://www.centerii.org/ (aka: School Turnarounds (2007)).

• Shannon, G.S. & Bylsma, P. (2004). Characteristics of Improved School Districts: Themes from Research. Olympia, WA. Office of Superintendent of Public Instruction.

• Shannon, G.S. & Bylsma, P. (2003). Nine Characteristics of High Performing Schools. A research-based resource for school leadership teams to assist with the School Improvement Process. Office of Superintendent of Public Instruction. Olympia, WA.

• Sharratt, G. C., Mills, S., & Lobdell, G. (2008). Schools of distinction: What makes them distinct? Washington State Kappan, 2(1), 20-22.

Secondary• Center for Educational Effectiveness (CEE) (2005). Longitudinal Change in Staff Perceptions of

the 9 Characteristics of High Performing Schools in OSPI SIA Cohort-II and III Schools. Redmond, WA: Center for Educational Effectiveness.

• Elmore, R. (2000). Building a New Structure For School Leadership. Washington, D.C.: The Albert Shanker Institute.

• Elmore, R. (2002). Bridging the Gap Between Standards and Achievement. Washington, D.C.: The Albert Shanker Institute.

• Tschannen-Moran, (2004). Trust Matters, Leadership for Successful Schools. San Francisco, CA. Jossey-Bass.