Scaling Up Evidence-based Practices Dean Fixsen, Karen Blase, Rob Horner, and George Sugai

Scaling UpScaling UpEvidence-based Evidence-based Practices Practices

Dean Fixsen, Karen Blase,Rob Horner, and George Sugai

Current ChallengesCurrent ChallengesEffective practices exist for literacy, social behavior and safety

Traditional TA approaches (dissemination and compliance) are insufficient

Methods are available to reliably produce benefits to students statewide

© Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008


CurrentlyCurrently

EBPs now are boutique operations

◦Now have convincing demonstrations that EBPs can work in the real world

◦This is encouraging, but EBPs are not used on a sustainable scale sufficient to solve social problems


FutureFutureStart with the end in mindWhat will it take to:

◦make statewide use of education innovations

◦that produce increasingly effective outcomes

◦for the next 50 years?

ProblemProblem Most States already have

attempted to implement a variety of these science-based practices

Some attempts achieve their purposes, but others fall short



Key Issues Key Issues Four big issues have emerged:

◦ Select what to scale up

◦ Determine how to implement new education practices so they actually produce the intended benefits to students (effective)

◦ Define how to scale up effective practices so they are available to all students who could benefit from them (effective, accessible)

◦ Define how to align system structures and functions to fully support scale up efforts as part of “education as usual” (effective, accessible, sustainable)

When is a “practice” ready for TA When is a “practice” ready for TA InvestmentInvestmentDefining a “practice”

◦ A “practice” is a procedure, or set of procedures, designed for use in a specific context, by individuals with certain skills/features, to produce specific outcomes for specific individuals.

Operationally defined proceduresTarget population/ ContextImplementer CharacteristicsDefined outcomes

The scope/size of a practice The scope/size of a practice mattersmattersPractices can be small, medium or

large in scope◦Time delay◦Functional behavioral assessment◦First Step to Success

Scale up practices only when they are bundled to a level where they have impact on a core social outcome.


When is a “practice” ready for TA When is a “practice” ready for TA InvestmentInvestment

Wisdom from: Karin Guldbrandsson H. Gotham Dean Fixsen and Karen Blase George Sugai Bruce Joyce and Beverly Showers

Eight Features of a “practice” that make it “ready for implementation” investment


Substantive impact on highly valued outcome.

Practice is comprehensive enough that it addresses a highly valued outcome (not just a piece)

There is a perceived need for the outcome Effect is dramatic (both in relative and absolute

terms)

The practice is defined with precision The practice can be described with operational

precision and substance. What is the “it”

Compatibility with existing practices


Evidence-based Experimental documentation of

relationship between practiced and impact published in peer-reviewed journal. (Effect Size of at least .50)

Emerging Promising Evidence-based

Efficiency Minimal additional resources to perform

new practice Minimal additional resources to transition

to new practice

When is a “practice” ready for TA When is a “practice” ready for TA InvestmentInvestmentSocial Validity

The people required to behave differently are comfortable with the new behavior(s) they are being asked to perform

Documentation of effectiveness trials

Adaptability The practice can be adapted to “fit” different

contexts The challenge of being precise and manualized

versus being flexible and adaptive (be clear about core).

Generalizability The practice is effective across the array of

contexts normally encountered.

ValuedOutcomes

Practice Implementation

Identifying& Modifying

Practices

Efficiency

EffectivenessPriority

Continuous Regeneration

Continuous

Measurement

Data-BasedProb.

Solving

Capacity

Building

What “packaging features” What “packaging features” are necessary for successful are necessary for successful adoption?adoption?Materials for exploration and awareness

What is the need, what is the practice, what is the documented link between the practice and valued outcome?

Materials for Installation and Initial Implementation

Materials for teaching people to master the new practice

Protocol for implementation

What “package features” What “package features” are necessary for are necessary for successful adoption?successful adoption?Evaluation tools

Is practice being implemented with fidelity

Is practice producing valued outcome

Are training segments producing implementation of core elements of “practice”


Research Measures

Annual Self-Assessment Measures

Progress Monitoring Measures

Universal Tier of SWPBS

-School-wide Evaluation Tool (SET)

-Self-Assessment Survey (SAS)-Benchmarks of Quality (BoQ)

-Team Implementation Checklist (TIC)

Secondary and Tertiary Tiers of SWPBS

-Individual Student School-wide Evaluation Tool (I-SSET)

-Benchmarks of Advanced Tiers (BAT)

(To be developed)

Overall Summary of Implementation

-Implementation Phases Inventory (IPI)-Phases of Implementation (POI)


Activity Activity Task Analysis Who When

1. Establish Commitment

Administrator Top 3 goal 80% of faculty Three year timeline

a.

b.

c.

d.

e.

2. Establish TeamRepresentativeAdministratorEffective team operating proceduresAudit of teams/initiatives

a.

b.

c.

d.

e.

Evaluation Guided Action Planning


Key IssuesKey IssuesInterventions that are and remain

effective through several generations of teachers, principals, superintendents, and State and national leaders

Implementation supports that are and remain effective through several generations of trainers, coaches, evaluators, administrators, and State leaders

Teaching–Family Teaching–Family ReplicationsReplications


0

100

200

300

400

500

600

700

800

900

≤1972 1974 1976 1978 1980 1982

0

50

100

150

200

250

300

CumulativeCouples

CumulativeHomes

Fixsen, Blase, Timbers, & Wolf (2001)


Follow Through ProgramsFollow Through Programs

Figure 1: This figure shows the average effects of nine Follow Through models on measures of basic skills (word knowledge, spelling, language, and math computation), cognitive-conceptual skills (reading comprehension, math concepts, and math problem solving) and self-concept. This figure is adapted from Engelmann, S. and Carnine, D. (1982), Theory of Instruction:

Principles and applications. New York: Irvington Press.

School AdoptingSchool-wide Positive Behavior Support


Insufficient MethodsInsufficient MethodsExcellent experimental evidence for what does not work

Diffusion/dissemination of information by itself does not lead to successful implementation (research literature, mailings, promulgation of practice guidelines)

Training alone, no matter how well done, does not lead to successful implementation


Insufficient MethodsInsufficient MethodsExcellent experimental evidence for what does not work

Implementation by laws/ compliance by itself does not work

Implementation by “following the money” by itself does not work

Implementation without changing supporting roles and functions does not work Paul Nutt (2002). Why Decisions Fail


Sobering Sobering ObservationObservation

"All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get." R. Spencer Darling

Business Expert

Rise in Incidence of Autism

Reduction in Incidence of Mental Retardation

and Learning Disabilities

The Oregon Department of Education has released graduation rates for all public high schools.

Nearly one-third of all high school students don't receive a diploma after four years of study.

by Betsy Hammond,

The Oregonian Monday June 29, 2009,

Sobering ObservationsSobering ObservationsInnovative practices do not fare well in old organizational structures and systems

Organizational and system changes are essential to successful implementation◦Expect it◦Plan for it


Build on local strengths, context and accomplishments

Implement 1-3 evidence-based practices at a sustainable scale of social importance.

Document value of implementation to achieve valued student outcomes


Benefits to focusing on Benefits to focusing on Scaling ImplementationScaling Implementation

Benefits to focusing on Benefits to focusing on Scaling ImplementationScaling Implementation

Develop capacity of your State to implement any evidence-based practice with fidelity and good outcomes

De-silo implementation of evidence-based practices, innovations, legislative mandates in the State


Scale UpScale UpTo scale up interventions we must first scale up implementation capacity

Building implementation capacity is essential to maximizing the use of EBPs and other innovations

Large scale, real time change© Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

Academic Systems Behavioral Systems

1-5% 1-5%

5-10% 5-10%

Intensive, Individual Interventions•Individual Students•Assessment-based•High Intensity•Of longer duration

Intensive, Individual Interventions•Individual Students•Assessment-based•Intense, durable procedures

Targeted Group Interventions•Some students (at-risk)•High efficiency•Rapid response

Targeted Group Interventions•Some students (at-risk)•High efficiency•Rapid response

80-90% 80-90%Universal Interventions•All students•Preventive, proactive

Universal Interventions•All settings, all students•Preventive, proactive

Multi-tier Model

Attention, Effort,

Precision


Capacity DevelopmentCapacity Development

State Management Team

State Transformation Team

Regional Implementation Teams

State Management

Group

State Transformation

Team

Regional Implementation

Team

N = 50 – 200 Schools


Team



Team



Team


IMPLEMENTATION CAPACITY FOR

SCALING UP EBPs

SIS

EP

Su

pp

ort

& 2

FT

E

State Management

Group

First Regional Implementation

TeamN = 9 Staff

Po

licy En

abled

P

ractice (PE

P)

Pra

ctic

e In

form

ed

Po

licy

(P

IP)

Sys

tem

C

han

ge

SIS

EP

Su

pp

ort

& 2

FT

E

ContextContext

Many initiatives to improve educationInitiatives too often conflicting and/or

competing

Early Intervention

Literacy

Math

Wraparound

Positive Behavior SupportFamily Support

Response to Intervention

Applied Behavior Analysis


Using RTI to Align Using RTI to Align SystemsSystems

Literacy

Wraparound

Math

Family Support

Behavior Support

AL

IGN

ME

NT

Early Intervention

Res

po

nse

to

In

terv

enti

on

/Pre

ven

tio

n

Student Outcomes

Primary Prevention

Universal Screening

Multi-tiered

Support

Early Interventio

n

Progress Monitoring

Systems to support

practices


Implementation TeamImplementation Team

A group that knows the innovation very well (formal and practice knowledge)

A group that knows how to implement that innovation with fidelity and good effect

A group that accumulates data & experiential knowledge -- more effective and efficient over time (information economics, K. Arrow)


Implementation TeamImplementation Team

Implementation Team

Prepare Communities

Prepare schools faculty, staff

Work with Researchers

Assure Implementation

Prepare Districts Assure Student Benefits


Capacity DevelopmentCapacity Development

Start with too many overqualified people

“Generation 1” RIT members become: ◦Generation 2 expanded RIT members

◦Generation 3 State Transformation Team leaders

◦Generation 4 State and federal officials

State Management

Group

State Transformation

Team2 FTE & Others


Team



Team



Team



Team


IMPLEMENTATION CAPACITY FOR

SCALING UP EBPs

[Phase 2]

SIS

EP

Su

pp

ort

Intensive Development

Saturation

State Capacity Development

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12

Years

Reg

iona

l Im

pl. T

eam

s RITs

STTs

Intensive Development

Saturation44/51

Coaching DefinedCoaching DefinedCoaching is the active and iterative

delivery of: ◦ (a) prompts that increase successful

behavior, and ◦ (b) corrections that decrease unsuccessful

behavior.

◦ Coaching is done by someone with credibility and experience with the target skill(s)

◦ Coaching is done on-site, in real time ◦ Coaching is done after initial training◦ Coaching is done repeatedly (e.g. monthly)◦ Coaching intensity is adjusted to need

Outcomes of CoachingOutcomes of CoachingFluency with trained skillsAdaptation of trained concepts/skills

to local contexts and challenges And new challenges that arise

Rapid redirection from miss-applications

Increased fidelity of overall implementation

Improved sustainability Most often due to ability to increase

coaching intensity at critical points in time.

Training Outcomes Related to Training Components

Training Outcomes

Training Components

Knowledge of Content

Skill Implementation

ClassroomApplication

Presentation/ Lecture

PlusDemonstration

Plus Practice

Plus Coaching/ Admin SupportData Feedback

10% 5% 0%

30% 20% 0%

60% 60% 5%

95% 95% 95%

Joyce & Showers, 2002

Coaching Coaching within SWPBS within SWPBS ImplementationImplementation

Context:◦9600 schools implementing SWPBS

nationallyDefining the Role

Internal vs External

Selecting CoachesTraining and support for coachesAssessing Impact

Rehearsal and CoachingRehearsal and CoachingAfter initial training, a majority of

participants (211 of 213) demonstrated poor implementation. Decision-makers should pair training prior to implementation with on-going rehearsal and performance feedback (coaching)◦Test et al 2008


0

1

2

3

4

5

6

7

Sep Oct Nov Dec Jan Feb Mar Apr May

Avg. Refe

rrals

per

Day

05-06 06-07

Example of the Impact of Coaching on Student Outcomes:Example of the Impact of Coaching on Student Outcomes:Average Major Discipline Referrals per Day per MonthAverage Major Discipline Referrals per Day per Month

Coach Coach returns from returns from

leaveleave

Leadership Team

FundingVisibility Political

Support

Training Coaching Evaluation

Local Demonstration Schools

Active Coordination

BehavioralExpertise

““Even if you’re on the right track, you’ll get Even if you’re on the right track, you’ll get run over if you just sit there” run over if you just sit there”

- Will Rogers - Will Rogers

Documents

Scaling Up Evidence-based Practices Dean Fixsen, Karen Blase, Rob Horner, and George Sugai