1

Implementing a Three-Tiered State Evaluation Structure

Bob Putnam The May Institute

Karen ChildsUniversity of South Florida

2009 National PBIS Leadership Forum

Goals of Session

• Present the purpose of a state-wide evaluation

• Provide exemplars of the core components of three tier state-wide evaluations

• Provide an ongoing state-wide evaluation system

Purpose of Statewide SWPBS Evaluation

• Evaluation is the process of gathering information for decision-making.

• SWPBS decisions focus on levels of adoption, adaptation, and replication. – Efficiency – Effectiveness – Sustainability– Wide-spread adoptability

3

Funding Visibility PolicyPoliticalSupport

Training CoachingBehavioral Expertise

Evaluation

LEADERSHIP TEAM(Coordination)

Local School/District Implementation Demonstrations

Core Indicators• These indicators include considerations about the

– context in which implementation of SWPBS is to occur – inputs available to guide and assist with

implementation – fidelity with which core elements of SWPBS are put in

place – impact of those core elements on the social and

academic behavior of students.

• Assess the breadth of adoption, and sustainability of implementation.

5Algozzine, Horner, Sugai, Barrett, Eber, Kincaid & Lewis, 2009

Context

• What are/were the goals and objectives for SWPBS implementation?

• Who provided support for SWPBS implementation?

• Who received support during SWPBS implementation?

6

Context

Who received support

during SWPBS implementation?

Input

• What professional development was part of SWPBS implementation support?

• Who participated in the professional development?

• What was the perceived value of the professional development?

8

Participating Schools2004 Schools (21)2005 Schools (31)2006 Schools (50)

2000 Model Demonstration Schools (5)

2007 Schools (165)2008 Schools (95)

Who Is Receiving Training and Support?

Training/Technical Assistance

More than three-fourths of the counties in the state have at least one school participating in the North Carolina Positive Behavior Support Initiative.

Who Is Receiving Training and Support?

Tier 2 & Tier 3 Who Is Receiving Training and Support?

Tier 2 & Tier 3 Who Is Receiving Training and Support?

Tier 3 Who Is Receiving Training and Support?

Fidelity

• To what extent was SWPBS implemented as designed?

• To what extent was SWPBS implemented with fidelity?

14

School-Wide Evaluation Tool35 Coaches trained as SET assessors

15 Contractual SET assessors

• 97 SETs completed 2004• 154 SETs completed 2005• 157 SETs completed 2006• 104 schools have at least two SET scores• 80% Total score is considered Maintenance

Phase (IPI)• All regions met 80% criterion across schools• 69% increase after one year of implementation

To What Extent Have Practices Changed?

SET Scores by Region

42.52

25.00 29.0042.00

88.44 92.9086.60

95.58 91.04

0

20

40

60

80

100

Central Eastern Southern Special Western

Pre Post

To What Extent Have Practices Changed?

% at phase

11.32%

9.47%

23.16%

56.05%

Preparation Initiation Implementation Maintenance

To What Extent Have Practices Changed?

To What Extent Have Practices Changed?

www.pbisillinois.org

Tier 3To What Extent Have Practices Changed?

www.pbisillinois.org

Tier 3To What Extent Have Practices Changed?

www.pbisillinois.org

Impact

• To what extent is SWPBS associated with changes in student outcomes?

• To what extent is SWPBS associated with changes in other areas of schooling?

21

What extent is SWPBS associated with changes in student behavior?

www.pbisillinois.org

What extent is SWPBS associated with changes in student behavior?

www.pbisillinois.org

Tier 2 & 3

What extent is SWPBS associated with changes in student outcomes?

www.pbisillinois.org

Percent of Students at DIBELS Benchmark (Spring) and Major Discipline Referrals per 100 Students

To What Extent has Behavior and Academic Performance Changed?

MiBLSi Schools and Reading MEAP:Average Total Office Discipline Referrals per

100 Students per Day 2004-2005

To What Extent has Academic Performance Changed?

MEAP- 4MEAP- 4thth grade Reading Assessment grade Reading Assessment29 Elementary Schools In Michigan29 Elementary Schools In Michigan

Schoolwide: Over 55% of major discipline referrals from classroom

Schoolwide: Under 55% of major discipline referrals from classroom

Probability of scoring below 75% proficiency on 4th grade MEAP (Reading): .78

Probability of scoring above 75% proficiency on 4th grade MEAP (Reading): .75

To What Extent has Academic Performance Changed?

North CarolinaPositive Behavior Support Initiative

[A]chievement causes

[B]ehavior?

[B]ehavior causes

[A]chievement?

[Context causes [A]chievement

and [B]ehavior?

To What Extent has Academic Performance Changed?

Replication, Sustainability, and Improvement

• To what extent did SWPBS implementation improved capacity for the state/region/district to replicate, sustain, and improve behavior and other outcomes?

• To what extent did SWPBS implementation change educational/behavioral policy?

• To what extent did SWPBS implementation affect systemic educational practice?

29

What is the breadth of adoption, and sustainability

of implementation?

www.pbisillinois.org

What is the breadth of adoption, and sustainability of implementation?

465

030 42

6181

133

183

283

383

1434

64

118

178

263

368

559

0

100

200

300

400

500

600

700

FY 99 FY 00 FY 01 FY 02 FY 03 FY 04 FY 05 FY 06 FY07(P rojected)

Schools Trained Coaches Trainedwww.marylandpbis.org

Conclusions

• Increase of states reporting on the implementation of SWPBS

• Information on implementation and outcomes of Tier 1 interventions more available than Tier 2 and 3 interventions

• Research is needed on the types of and presentations of data included in state evaluations that will have best outcome on funding, visibility, political support and policy.

For More Information

Bob Putnam, Senior Vice President of School ConsultationMay InstitutePhone: (781) 437-1207)Email: bputnam@mayisnstitute.orgWebsite: http:/mayinstitute.org

Karen Elfner Childs, Research & Evaluation CoordinatorFlorida’s PBS ProjectPhone: (813) 974-7358Email: childs@fmhi.usf.eduWebsite: http://flpbs.fmhi.usf.edu