Scaling Up SW PBS Implementation:

Different Journeys to the Same Destination

Heather Peshak George, Ph.D.

APBS Conference, St. Louis, MO: March 26, 2010

2

Florida’s PBS Project

Our Mission• Increasing the capacity of Florida’s school districts to

address problem behavior using positive behavior support through a Response to Intervention framework.

What we do…• Provide training and technical assistance to districts across

the state in the development and implementation of positive behavior supports at the school-wide, targeted group, classroom and individual student levels

3

Exploration Phase

• Is there a need for change?• What current practices and initiatives exist

that are facilitators or barriers?• What is innovation and does it address

our problem?• How do we plan for implementation?• Is the team ready to begin installation of

innovation?

4

Installation Phase

• What individuals have authority to reallocate resources/facilitate implementation and connect with state improvement plan?

• Who will guide implementation?• What does implementation of the

innovation involve?

5

Initial Implementation Phase

• Who are initial implementers?• How do we begin implementation?• How do we monitor fidelity and outcomes?

6

Full Implementation Phase

• How do we build local training/coaching capacity?

• What resources do we need to add more schools?

• How do we sustain fidelity and positive outcomes over time?

7

Innovation and Sustainability Phase

• How do we braid innovation with new/existing initiatives?

• How do we become more efficient and effective?

• How do we share what we have learned?

9

Funding

• IDEA Part B Discretionary Grant

• Around 1997: $33,000/year

• Current: Almost $1.8 million/year

10

Visibility

Before:• Present at every state conference

• Special education• Expanded across disciplines

• ABA, Safe Schools, RtI, Reading, School Psychologists, Superintendent, Evaluation

Now:• Present when requested or “invited”

11

Click Here…

12

http://flpbs.fmhi.usf.edu/resources_newsletter.asp

Newsletter:

Read directly from the website or download to print.

Use to find useful tips, data, important dates, etc.

13

Political Support and Policy

• Expanded funding of project• State RtI Implementation Plan• Active member on the FL DOE’s State

Transformation Team for RtI• Title of Project to include RtI:B

• RtI• Bullying & Harassment Policy• Zero Tolerance

State Infrastructure• State Management Team

– Role: Provide leadership and facilitate policy-level changes to support implementation of effective educational practice

• State Transformation Team– Role: Analyze progress toward statewide efforts, recommend actions for

improvement, and support District Based Leadership Teams DBLT to build the capacity of districts to implement evidence-based practices and to establish integrated RtI academic and behavior systems in each school

• District Based Leadership Team– Role: Provide leadership, advisement, and training at the district level and

assist schools in their implementation efforts• School Based Leadership Team

– Role: Develop a school implementation plan. The school based team will become “trainers” and “coaches” for the school staff and will be responsible for school wide implementation

• Advisory Group– Role: Provide on-going stakeholder input– pp. 11-13 of the Statewide RtI Implementation Plan

District ResponsibilitiesBased on self-assessment results, and in conjunction

with the student progression plan and K-12 Comprehensive Reading Plan, develop an RtI implementation plan organized around building consensus, infrastructure, and implementation. Plans should also address:– How current resources will be used to implement

RtI and identify additional resources needed– How stakeholders will be educated– How stakeholders will be involvedp. 8 of the Statewide RtI Implementation Plan

Alignment with Existing InitiativesThe basic elements of RtI are required by

NCLB and IDEA, therefore, they are included in all broad-based initiatives for schools striving to meet (AYP), such as:– K-12 Reading Plan and Reading First– Florida’s Continuous Improvement Model (FCIM)– Florida’s Positive Behavior Supports (PBS)– PS/RtI Pilot Project– Bright Beginningsp. 1 of the Statewide RtI Implementation Plan

Tiered Model of School Supports & the Problem-Solving Process

ACADEMIC and BEHAVIOR SYSTEMS

Tier 3: Intensive, Individualized Interventions & Supports

The most intense (increased time, narrowed focus, reduced group size) instruction and intervention based upon individual student need provided in addition to and aligned with Tier 1 & 2 academic

and behavior instruction and supports.

Tier 2: Targeted, Supplemental Interventions & Supports.

More targeted instruction/intervention and supplemental support in addition to and aligned with the core academic and behavior curriculum.

Tier 1: Core, Universal Instruction & Supports.

General academic and behavior instruction and support provided to all students in all settings.

FL RtI State Transformation Team, Dec. 2009)

18

19

Training• School-Wide/Initial (Tier 1)• Booster/Retraining (Tier 1)• Classroom (Tiers 1 & 2) – online only• Targeted Group (Tier 2)• Individual Student (Tier 3) • Team Leader• Principal/Administrator• SWIS• Coaches’

Italics indicates both on-site & online modules available

20

Tier 1 Training Sequence

Day 1

Intro to RtI:B

Teaming

Developing Expectations & Rules

Developing a System for Teaching

Day 3

Evaluation

Classroom PBS

Facilitated work time

Day 2

Developing a Reward System

Effective Discipline Procedures•Definitions

•Forms

•Process

•Responses

Implementing PBS

# Florida Schools' Initial PBS Training

12

61

8781

53

77

119

219

0

50

100

150

200

250

2001-2002 2002-2003 2003-2004 2004-2005 2005-2006 2006-2007 2007-2008 2008-2009

Year

Tota

l Sch

ools

Tra

ined

22

Number of Schools Trained

(As of March, 15 2010)

6 Pre-K schools387 Elementary schools167 Middle schools82 High schools56 Alt/Center schools50 Other (e.g. K-8)

748 TOTAL SCHOOLS

Coaching & Behavioral Expertise

RtI:B Support StructureProject Contact

District Coordinator(District Leadership Team)

COACH

School-based Team Leader

PBS Team Members

25

Factors to Consider in Developing Comprehensive

Evaluation Systems1) Systems Preparation

– Readiness activities2) Service Provision

– Training and technical assistance3) Identification and Assessment of Behavior Problems

– Possible data sources4) Evaluation Process

– Timelines, data systems5) Evaluation Data (Across all three Tiers)

– Implementation Fidelity, Impact on Students, Attrition, Client Satisfaction

6) Products and Dissemination– Reports, materials, presentations, etc.

(modified from Childs, Kincaid & George, in press)

26

Florida’s Evaluation Model

TrainingOn-going technical assistance

FLPBS↓

Districts↓

Coaches↓

Schools

End-Year

ImpactOutcome data (ODR, ISS, OSS)FL Comprehensive Assessment TestBenchmarks of QualitySchool Demographic DataPBS WalkthroughDaily Progress ReportsBehavior Rating ScalesClimate Surveys

Implementation FidelityPBS Implementation Checklist (PIC)Benchmarks of Quality (BoQ)Benchmarks for Advanced Tiers (BAT)School Demographic DataSchool-wide Implementation FactorsTier 3 plan fidelity checklistBEP Fidelity checklist

Project ImpactAttrition Survey/Attrition RatesDistrict Action Plans

Client SatisfactionSchool-Wide Implementation FactorsDistrict Coordinator’s SurveyTraining Evaluations

Annual Reports Revisions to

training and technical assistance process

National, State, district, school dissemination activities

Website On-line training

modules

Systems Preparation

Service Provision

Evaluation Process

Evaluation Data

Products and Dissemination

MidYear

I

MidYearII

Identification/Assessment

Discipline Records ESE Referrals Surveys Walkthroughs PIC Classroom

Assessment Tool Student rank/rating Teacher requests Lack of response BAT Behavior Rating

Scale Daily Progress

Report Charts

• District Action Plan

• District Readiness

Checklist• SchoolReadinessChecklist• New School Profile (includes ODR, ISS, OSS)

27

(1) Systems Preparation• Tier 1

• District Readiness Checklist• District Action Plan

– District baseline and goals• School Readiness Checklist

• Baseline data• Tier 2

• School Readiness• Implementation of Tier 1• School Infrastructure

• Tier 3– District Action Plan

• Systems change• Evaluation of products and processes• Establish vision and goals

28

29

(2) Service Provision

Training and ongoing technical assistance

• Training

• Tier 1- District and multi-district on-site

• Tier 2 –District, multi-district, web-based

• Tier 3- Post assessment, goal setting, systems/process established

(3) Identification & Assessment

• Tier 1– Discipline records, attendance, ESE referrals, baseline BoQ, action

plans, climate surveys, coaches surveys, walkthrough (mini SET), PBS Implementation Checklist (PIC)

• Classroom– Discipline records, teacher requests, student rankings/ratings, ESE

referrals, observations, Classroom Assessment Tool• Tier 2

– Discipline records, teacher requests, student rankings/ratings (SSBD, TRF, etc…), lack of response to Tier 1, Daily Progress Reports, PBS Implementation Checklist (PIC), Benchmarks for Advanced Tiers (BAT)

• Tier 3– Above items, lack of response to Tier 2, Behavior Rating Scale,

observation data, intervention fidelity checklist, PBS Implementation Checklist (PIC), Benchmarks for Advanced Tiers (BAT)

31

Referrals per Student

0

10

20

Nu

mb

er

of R

efe

rra

ls p

er

Stu

de

nt

Students

Office Discipline Referrals

Student Initials Grade/Period I or E (Step 2) _________ ________ _________________ ________ _________________ ________ ________

Academic Personally Taught Personally Given Externalizing Concerns Expectations SW Reward 1. __________________ ____ Yes ____ Yes ____ Yes

2. __________________ ____ Yes ____ Yes ____ Yes

3. __________________ ____ Yes ____ Yes ____ Yes

• Rank top 3 externalizing and top 3 internalizing students

• Check “YES” if personally taught expectations to the student

• Check “YES” if personally given a SW-PBS reward to student

Teacher Nomination

Tier 2 Progress Monitoring

Behavior Rating ScaleBehavior Date

Hitting 8 or more6-7 times4-5 times2-3 times0-1 times

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

Profanity 16 or more times12-15 times

8-11 times4-7 times0-3 times

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

Requesting Attention/Assistance

55% or more40-55%25-40%10-25%0-10%

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

54321

(4) Evaluation Process• Timelines for State Evaluation

– Baseline (due date varies)– Mid Year I – due 10/31

• School Profile• PBS Implementation Checklist (PIC) (Tiers 1-3)

– Mid Year II – due 2/28• PBS Implementation Checklist (PIC) (Tiers 1-3)

– End Year – due 6/15• Benchmarks of Quality (BoQ) (Tier 1)• Benchmarks for Advanced Tiers (BAT) (Tiers 2-3)• Outcome Data Summary • School-wide Implementation Factors (SWIF)

• Web-based Data Entry and Reporting• PBSES• Statewide Student Database – Academic/Behavior

36

(5) Evaluation Data

a) Implementation Fidelity– PIC (All Tiers)– BoQ (Tier 1)– BAT (Tiers 2-3)– SWIF (All Tiers)– Walkthrough (Tier 1)– Tier 2 & 3 intervention

specific fidelity measuresc) Attrition

– Attrition Survey (All)

b) Impact on Students– Outcome data (ODR, ISS,

OSS) – Academic achievement– School Demographic

Data (e.g. ethnicity)– Attendance– DPR charting– Behavior Rating Scale

d) Client Satisfaction– SWIF– Climate surveys– Social validation

37

(a) Implementation Fidelity

1. Are schools trained in SWPBS implementing with fidelity? Across years? Across school types?– BoQ, BAT, School Demographic Data

2. What factors are related to implementing with fidelity? – SWIF survey, BoQ, BAT

38

Tier 1 Critical Element Implementation Level chart

39

PBS Implementation Level chart

40

High Implementing Florida PBS Schools(Scoring 70 or Above on BoQ)

57 5660

7774

0

10

20

30

40

50

60

70

80

90

100

School Year

Pe

rce

nt

of

Sc

ho

ols

Sc

ori

ng

70

+

2004-2005 2005-2006 2006-2007 2007-2008 2008-2009

Average Score

66

Average Score

66

Average Score

72

Average Score

77

Average Score

76

BoQ Totals by School Type Across Years

Average BoQ Total Score by School Type

69

64

70 6971

66 66

7674 73

63

8178

74 75

8080

74

67

80

0

10

20

30

40

50

60

70

80

90

Elementary Middle High Alt/Center

School Type

Ave

rag

e B

oQ

To

tal

Sco

re

2004-2005 2005-2006 2006-2007 2007-2008 2008-2009

38

Sc

ho

ols

77

Sc

ho

ols

10

4S

ch

oo

ls

27

Sc

ho

ols

54

Sc

ho

ols

55

Sc

ho

ols

6S

ch

oo

ls

14

Sc

ho

ols

17

Sc

ho

ols

15

Sc

ho

ols

26

Sc

ho

ols

28

Sc

ho

ols

14

9S

ch

oo

ls

22

1S

ch

oo

ls

59

Sc

ho

ols

93

Sc

ho

ols

17

Sc

ho

ols

25

Sc

ho

ols

32

Sc

ho

ols

37

Sc

ho

ols

Brief Walk-through

Benchmarks for Advanced TiersScore Summary for BAT Sections

0

10

20

30

40

50

60

70

80

90

100

Tie

r 1

T2&

3:C

omm

it

T2&

3:S

tu I

dent

T2&

3:M

onit

or

T2:

Sup

port

T2:

Mai

n Im

pl

T2:

Mai

n M

onit

or

T2:

2nd

Impl

T2:

2nd

Mon

itor

T2:

3rd

Impl

T2:

3rd

Mon

itor

T2:

4th

Impl

T2:

4th

Mon

itor

T3:

Sup

port

T3:

Ass

ess

T3:

Mon

itor

Tota

lBAT Sections

Per

cen

t P

ossi

ble

Sco

re

44

(b) Impact on Student Behavior

1. Do schools implementing SWPBS decrease ODRs, days of ISS, and days of OSS?– ODRs, ISS, OSS

2. Do schools implementing SWPBS realize an increase in academic achievement?– FCAT scores

3. Is there a difference in outcomes across school types?– ODRs, ISS, OSS, FCAT scores, school demographic data

4. Do schools implementing with high fidelity have greater outcomes implementers with low fidelity?– BoQ, ODRs, ISS, OSS

5. Do teams that work well together have greater outcomes than those that don’t work as well together?– Team Process Evaluation, ODRs, ISS, OSS

Percentage Change in Discipline OutcomeBetween Baseline and Year 1

-24%

-18%

-8%

-30%

-25%

-20%

-15%

-10%

-5%

0%

ODR ISS OSS

Per

cen

tag

e C

han

ge

110

Sch

oo

ls

89S

cho

ols

108

Sch

oo

ls

higher implementing schools reported 37% fewer ODRs per 100 students

162

114115

94

105100

107

85

115

73

0

20

40

60

80

100

120

140

160

180

Low (BoQ <70) High (BoQ >=70)

Implementation Level

Ave

rag

e O

DR

s p

er

100

Stu

de

nts

2004-2005 2005-2006 2006-2007 2007-2008 2008-2009

35

Sc

ho

ols

77

Sc

ho

ols

83

Sc

ho

ols

39

Sc

ho

ols

76

Sc

ho

ols

10

7S

ch

oo

ls

67

Sc

ho

ols

16

2S

ch

oo

ls

ODR Rates by Implementation Level Across School Years

10

2S

ch

oo

ls

25

6S

ch

oo

ls

ISS Rates by Implementation Level Across School Years

52

29

45

28

43

32

54

25

39

26

0

10

20

30

40

50

60

Low (BoQ <70) High (BoQ >=70)

Implementation Level

Ave

rag

e D

ays

ISS

per

100

Stu

den

ts

2004-2005 2005-2006 2006-2007 2007-2008 2008-2009

35S

cho

ols

77S

cho

ols

83S

cho

ols

39S

cho

ols

76S

cho

ols

107

Sch

oo

ls

67S

cho

ols

162

Sch

oo

ls

67S

cho

ols

162

Sch

oo

ls

overall average difference of 40% across the five school years

73

43

61

34

56

37

51

28

50

25

0

10

20

30

40

50

60

70

80

Low (BoQ <70) High (BoQ >=70)

Implementation Level

Av

era

ge

Da

ys

OS

S p

er

10

0 S

tud

en

ts 2004-2005 2005-2006 2006-2007 2007-2008 2008-2009

35

Sc

ho

ols

77

Sc

ho

ols

83

Sc

ho

ols

39

Sc

ho

ols

76

Sc

ho

ols

10

7S

ch

oo

ls

67

Sc

ho

ols

16

2S

ch

oo

ls

OSS Rates by Implementation LevelAcross School Years

67

Sc

ho

ols

16

2S

ch

oo

ls

overall average difference of 43% across the five school years

49

(c) Attrition

1. Why do schools discontinue implementation of SWPBS?– Attrition Survey

50

(d) Consumer Satisfaction

1. Are our consumers satisfied with the training, technical assistance, products and support received?– SWIF survey – District Coordinators survey– Training evaluation– Climate surveys

School-Wide Implementation Factors (SWIF)

Higher Implementing Lower Implementing

(70+ on BoQ) (-70 on BoQ)90% + respondents from high implementing schools identified these factors as Helpful:

80%+ respondents from low implementing schools identified these factors as Helpful:

Factors MOST

Helpful to Implementation of

SWPBS

Expectations and rules clearly definedAdministrator committed to PBS, willing to teach and model PBS, willing to reward studentsRepresentative and committed PBS TeamReward system worksPBS Coach’s guidance with processStudents responses to rewards and activities

Expectations and rules clearly definedAdministrator willing to reward studentsRepresentative PBS Team

25%+ respondents from high implementing schools identified these factors as Problematic:

50%+ respondents from low implementing schools identified these factors as Problematic:

FactorsMOST

Problematic to Implementation of

SWPBS

Adequate fundingTeam recognizes faculty participationStaff stability from year to yearStudent stability from year to year

Staff time for PBSStaff belief about effectiveness of PBSStaff philosophyStaff consistency in teachingStaff consistency in discipline procedures

53

(6) Products and Dissemination

• Annual Reports• Revisions to Training• Revisions to Technical Assistance process• Dissemination activities:

– National, state, district, school levels• Revisions to Website• On-line training modules

Model SchoolsFlorida's Model Schools

1739 56

91133

95

184221

280

410

050

100

150200250300

350400450

2004-2005 2005-2006 2006-2007 2007-2008 2008-2009

Number of Model Schools Number of Active Schools

32% of eligible schools achieved model school status in 2008-2009. Of the 133 schools = 27 Gold, 72 Silver, 34 Bronze

55

What’s New

• Train-the-Trainer Evaluation Process• Expansion of Online Training Modules• State-wide Behavioral Database• Possible State RtI Center• High Schools• Further Research• ???

56

Current Research

• PBS Implementation Checklist (PIC)

• Benchmarks of Quality (Revised)

• Benchmarks for Advanced Tiers (BAT) • Walkthrough

• Tier 3

• Model Demonstrations at Tier 2/3

57

Challenge

58

Contact

Heather Peshak George, Ph.D.• Phone: (813) 974-6440• Fax: (813) 974-6115• Email: flpbs@fmhi.usf.edu• State Website: http://flpbs.fmhi.usf.edu• National Website: www.pbis.org• Association on PBS: www.apbs.org