Evidence-Informed Planning: Benefiting from Evidence- Based Interventions Defending Childhood January 26, 2011 Melissa K. Van Dyke, LCSW Associate Director

Evidence-Informed Planning:

Benefiting from Evidence-Based Interventions

Defending Childhood

January 26, 2011

Melissa K. Van Dyke, LCSWAssociate Director

National Implementation Research Network

University of North Carolina, Chapel Hill

Copyright © Dean L. Fixsen and Karen A. Blase, 2010

Two Sides of the Same Coin

To successfully implement and sustain evidence-based and evidence-informed interventions, we need to know:

The WHAT - What is the intervention (e.g. Al’s Pals, FFT, PCIT, Second Step)

AND

The HOW - Effective implementation and sustainability frameworks (e.g. strategies to change and maintain behavior of adults)


The Challenge

“It is one thing to say with the prophet Amos, ‘Let justice roll down like mighty waters’ …

William Sloane Coffin

Social activist and clergyman

… and quite another to work out the irrigation system.”


Science “to” Service

SCIENCE SERVICEGAP

Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions.

Why Focus on Implementation?“Children and families cannot benefit from interventions they do not experience.”

IMPLEMENTATION


Goals for Today’s Session

The “What”Review general information about evidence-based practices

The “How”Present ‘stage-related’ work necessary for successful service and system changePresent the Implementation Drivers that result in competence and sustainabilityExplore “improvement cycles” and how to use them at a number of levels

The “Who”Discuss the roles and responsibilities of implementation team and program purveyors


Which intervention is right for you?

What are the needs of your population?

What interventions are available to address those needs?

What is the strength of the evidence of those interventions?

Which interventions are a good fit for our community?

Do we have what is required to fully and effectively implement these interventions?

EBP: 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4.

High Medium Low

Need

Fit

Resources Availability

Evidence

Readiness for Replication

Capacity to Implement

Total Score:

Need in Agency, SettingSocially Significant IssuesParent & Community Perceptions of NeedData indicating Need

Need

Fit

Fit with current - •Initiatives• State and Local Priorities• Organizational structures• Community Values

ResourceAvailability

Resource AvailabilityITStaffingTrainingData SystemsCoaching & SupervisionAdministrative & system supports needed

EvidenceOutcomes – Is it worth it?Fidelity dataCost – effectiveness data Number of studiesPopulation similaritiesDiverse cultural groupsEfficacy or Effectiveness

Evidence

Assessing Evidence-Based Programs

and Practices

Intervention Readiness for ReplicationQualified purveyor Expert or TA availableMature sites to observe# of replicationsHow well is it operationalized?Are Imp Drivers operationalized?

Intervention Readiness

for Replication

Capacity to ImplementStaff meet minimum qualificationsAble to sustain Imp Drivers • Financially• Structurally• Buy-in process operationalized• Practitioners• Families• Agency


© National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland


Becoming an Informed Consumer

NREPPhttp://www.nrepp.samhsa.gov/ - Descriptive information - Outcomes

- Quality of Research - Study Population

- Readiness for Dissemination - Costs

- Replications

Questions to ask model developers or model purveyors:http://www.nrepp.samhsa.gov/pdfs/Questions_To_Ask_Developers.pdf

http://www.nrepp.samhsa.gov/

http://www.nrepp.samhsa.gov/pdfs/Questions_To_Ask_Developers.pdf

http://www.nrepp.samhsa.gov/pdfs/Questions_To_Ask_Developers.pdf


NREPP Program Review Sample

For example –Second Step

Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)

Implementation Materials 4.0

Training and Support 4.0

Quality Assurance 3.5

Overall Rating 3.8


Science-to-Service Gap

Implementation Gap

What is adopted is not used with fidelity and good outcomes

What is used with fidelity is not sustained for a useful period of time

What is used with fidelity is not used on a scale sufficient to impact social problems


Implementation

Review and synthesis of the implementation research and evaluation literature (1970 – 2004)

Multi-disciplinary

Multi-sector

Multi-national


Insufficient MethodsImplementation by laws/ compliance by itself does not work

Implementation by “following the money” by itself does not work

Implementation without changing supporting roles and functions does not work

Diffusion/dissemination of information by itself does not lead to successful implementation

Training alone, no matter how well done, does not lead to successful implementation

Fixsen, Naoom, Blase, Friedman, Wallace, 2005


Implementation Pre-Requisites

Start with Data related to NeedLook for “best evidence” to Address the Need

An Evidence-Based Practice or ProgramAn Evidence-Informed InitiativeSystems Change and Its Elements

Clearly operationalize the program and/or practice features or the systems change elementsOperationalize

Part of Speech: verb Definition: to define a concept or variable so that it can be measured or expressed quantitativelyWebster's New Millennium™ Dictionary of English, Preview Edition (v 0.9.7) Copyright © 2003-2008 Lexico Publishing Group, LLC


What Works

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION

INT

ER

VE

NT

ION Actual Benefits

(Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)

Inconsistent; Not Sustainable; Poor outcomes

Poor outcomes; Sometimes harmful



What Works

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION

INT

ER

VE

NT

ION Actual Benefits

(Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)

Inconsistent; Not Sustainable; Poor outcomes



from Mark Lipsey’s 2009 Meta-analytic overview of the primary factors that characterize effective juvenile offender interventions – “. . . in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.”


Implementation Frameworks

Practice, program and systems change through…

Multi-dimensional, fully integrated use of

Implementation Drivers

Implementation Stages

Implementation Teams

Improvement Cycles


Implementation Drivers

Common features of successful supports to help make full and effective uses of a wide variety of innovations

Staff Competency

Organizational Supports

Leadership

© Fixsen & Blase, 2008

Performance Assessment

(fidelity measurement)

Coaching

Training

Selection

Systems Intervention

Facilitative Administration

Decision Support Data System

Integrated & Compensatory

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Improved outcomes for children and

families Implem

entation

Drivers


Coaching

Training

Selection

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers

Graphics by Steve Goodman,2009

Implementation Lens

Implem

entation

DriversPerformance Assessment



families



Purposes:

Measure fidelity

Ensure implementation

Reinforce staff and build on strengths

Feedback to agency on functioning of

Recruitment and Selection Practices

Training Programs (pre and in-service)

Supervision and Coaching Systems

Interpretation of Outcome Data


Coaching

Purposes:

Ensures fidelity

Ensures implementation

Develops clinical and practice judgment

Provides feedback to selection and training processes

Grounded in “Best Practices”


Training and Coaching

OUTCOMES% of Participants who Demonstrate Knowledge, Demonstrate New Skills in a Training Setting,

and Use new Skills in the Classroom

TRAINING

COMPONENTS

Knowledge Skill

Demonstration

Use in the

Classroom

Theory and Discussion

10% 5% 0%

..+Demonstration in Training 30% 20% 0%

…+ Practice & Feedback in

Training

60% 60% 5%

…+ Coaching in Classroom

95% 95% 95%

Joyce and Showers, 2002


Training

Purposes:

“Buy-in”

Knowledge acquisition

Skill Development


Selection

Purposes:

Select for the “unteachables”

Screen for pre-requisites

Set expectations

Allow for mutual selection

Improve likelihood of retention after “investment”

Improve likelihood that training, coaching, and supervision will result in implementation


Coaching

Training

Selection

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers


Implementation Lens

Implem

entation




families

Copyright © Dean L. Fixsen and Karen A. Blase, 2010© Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

Organizational Change

"All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get."

R. Spencer DarlingBusiness Expert


Coaching

Training

Selection



Decision Support Data SystemC

ompe

tenc

y D

river

s

Com

pete

ncy

Driv

ers

Organization D

rivers

Organization D

rivers


Implem

entation




families



Purposes:To make a difference for children and families

Provide information to assess effectiveness of evidence-based practices

Analyze the relationship of fidelity to outcomes

To guide further program development

Engage in continuous quality improvement

Interaction with Core Implementation Components

Celebrate success

Be accountable to consumers and funders



Purposes:

Facilitates installation and implementation of the Drivers

Aligns policies and procedures

Takes the lead on Systems Interventions

Looks for ways to make work of practitioners and supervisors easier!!



Purposes:

Identify barriers and facilitators for the new way of work

Create an externally and internally “hospitable” environment for the new way of work

Contribute to cumulative learning in multi-site projects.


Coaching

Training

Selection




Adaptive

Technical


Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers



Implem

entation




families


Integrated and CompensatoryImplementation Drivers

Integrated

Consistency in philosophy, goals, knowledge and skills across these processes (S/T/C/SE/DSDS/FA/SI)

Compensatory

At the practitioner level

At the program level


Coaching

Training

Selection




Adaptive

Technical


Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers



families

Major Implementation Initiatives occur in stages:

Exploration (Sustainability)

Installation (Sustainability)

Initial Implementation (Sustainability)

Full Implementation (Sustainability & Effectiveness)

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

Implementation Takes Time

2 - 4 Years

Stages of Implementation


EXPLORATION


Com

pete

ncy D

river

s Organization Drivers

Leadership


Exploration

Goals:Examine degree to which the Evidence Based Practice, best practice, systems change meets the needs in the settings identified

Determine whether moving ahead with the initiative and implementation is desirable and feasible

Create readiness for change at many levels

“Pay now or pay later.”


Sustainability

Goals:Financial:

Ensure funding streams for desired change and necessary infrastructure

Programmatic:

Ensure high fidelity and positive outcomes through infrastructure improvement and maintenance

Plan for turnover

“The only thing harder than getting there is staying there.”



EXPLORATION

INST

ALL

ATI

ON


Com

pete

ncy D

river


Leadership


Installation

Goal:

To make the structural and instrumental changes necessary to initiate services

“If you build it, they will come”. . . but you actually have to built it!



EXPLORATION

INSTALLATIO

N

INITIAL

IMPLEMENTATION


Com

pete

ncy D

river


Leadership


Initial Implementation

Goals:

Survive the awkward stage!Learn from mistakes

Continue “buy-in” efforts

Manage expectations

“Anything worth doing…is worth doing poorly.”


2 - 4

Years


EXPLORATION

INSTALLATIO

N

INITIA

L

IMPLEMENTATIO

N

FULL

IMPLEMENTATION


Com

pete

ncy D

river


Leadership


Full ImplementationGoals:

Maintaining and improving skills and activities throughout the systemComponents integrated, fully functioningSkillful practices by front line staff, supervisors, administratorsChanges in policy that are reflected in practice at all levelsReady to be evaluated for expected outcomes

“The only thing worse than failing and not knowing why you failed, is succeeding and not knowing why you succeeded.”

~ Jane Timmons-Mitchell



Major Implementation Initiatives occur in stages:

Exploration (Sustainability)

Installation (Sustainability)

Initial Implementation (Sustainability)

Full Implementation (Sustainability & Effectiveness)



Exploration

“Many implementation efforts fail because someone underestimated the scope or importance of preparation. Indeed, the organizational hills are full of managers who believe that an innovation’s technical superiority and strategic importance will guarantee acceptance.”

Leonard-Barton & Kraus,

Harvard Business Review, 1985


Exploration: In Depth

What happens during Exploration?

Form “exploration workgroup”

Analyze data related to “needs”

Identify options and assess feasibility

Reassess, revise, prioritize, re-scope

Formalize structures










Form an “Exploration Workgroup”

Formation of an exploration workgroup

Focal point for the exploration work

Empowered to make decisions and/or to make recommendations

Representative of the “stakeholders”

Develop collaboration / co-ownership in the community










Analyze Data Related to “Needs”

Assessment of current outcomes

Dimensions – Root cause analysis (5 Whys)

Prevalence of the problem(s) – How frequent and pervasive?

Persistent nature of the problem – Have we been struggling for a long time?

Social significance – If this changed, would it make a significant different for students?

Leverage point – If these few indicators changed then other outcomes would be likely to be “pulled along.”







Reassess, revise, prioritize re-scope



Identify Options &Assess Feasibility

Needs

Fit

Resource availability

Evidence

Readiness for replication or degree to which it is operationalized

Capacity

Initiative : 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4.

High Medium Low

Need

Fit

Resources Availability

Evidence

Readiness for Replication


Total Score:

Need in State, District, SchoolsSocially Significant IssuesParent & Community Perceptions of NeedData indicating Need

Need

Fit

Fit with current - •Initiatives•State, District, School Priorities• Organizational structures• Community Values

ResourceAvailability

Resources StaffingTrainingData SystemsCoaching & SupervisionAdministrative & system supports neededTime

Evidence – is there any?Outcomes – Is it worth it?Fidelity or process dataCost – effectiveness data Number of studiesPopulation similaritiesDiverse cultural groupsEfficacy or Effectiveness

Evidence

Assessing Fit and Feasibility of Initiatives

ReadinessQualified purveyor Expert TA availableMature sites to observe# of replicationsHow well is it operationalized?Are Imp Drivers operationalized?

Intervention Readiness for Replication

CapacityStaff meet minimum qualificationsAble to sustain Imp Drivers • Financially• StructurallyBuy-in process operationalized• Practitioners• Families• Agency and Departments


© National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland










Does this change initiative still address the most critical needs?Does it fit our current political and social context?Do we have the necessary resources and support?Do we have the capacity and access to necessary expertise to proceed?Have we bitten off more than we can chew?Is this our leverage point?What has emerged during Exploration that impacts our decisions?

Reassess. . .










Formalize Structures

Formalize structures and processes – develop implementation teams based on…Design elements (e.g. components of the initiative)

LegalPractitioner levelAgency levelDistrict/State/Tribal workStakeholders

Identify linkages among the ‘structures’





Review data related to “needs”

Identify options and feasibility

Reassess, revise, re-scope



Resistance to Change

There is no such thing – only inadequate preparation

It is not “their” problem, it is ours.


Creating Readiness for Change

Individual readiness for change

Transtheoretical Model or Stages of Change

Precontemplation

Contemplation

Preparation

Action

Maintenance

Prochaska and DiClemente


Stages of Change

Stage of Change for Pre-Action Individuals:

Precontemplation – 40%

Contemplation – 40%

Preparation – 20%

“If only 20% of employees in organizations are prepared to take action. . . .”

Janice M. Prochaska, James O. Prochaska, and Deborah A. Levesque (2001)


2 - 4

Years


EXPLORATION

INSTALLATIO

N

INITIA

L

IMPLEMENTATIO

N

FULL

IMPLEMENTATION


Com

pete

ncy D

river


Leadership


“Who”

Purveyors

Intermediary Organizations

Technical Assistance Centers

Implementation Teams

Organized, Implementation Support

Provider Agency Supports

Management (leadership, policy)

Administration (HR, structure)

Supervision (nature, content)

Practitioner Competence

State and Tribal Leadership

Regional Authority Supports

Dev

elo

per

s T

ech

nic

al A

ssis

tan

ceIm

ple

men

tati

on

Tea

m

Simultaneous, Multi-Level Interventions

Organized, Implementation Support

Provider Agency Supports

Management (leadership, policy)

Administration (HR, structure)

Supervision (nature, content)

Practitioner Competence

State and Tribal Leadership

Regional Authority Supports

Dev

elo

per

s T

ech

nic

al A

ssis

tan

ceIm

ple

men

tati

on

Tea

m

Simultaneous, Multi-Level Interventions“We tend to focus on snapshots of

isolated parts of the system and

wonder why our deepest problems

never seem to get solved.

(Senge, 1990)

Systems trump programs!…Patrick McCarthy, Annie E.

Casey

"All organizations are designed, intentionally or unwittingly, to achieve precisely the results they get.” …R. Spencer Darling

System Change


Changing on Purpose

New practices do not fare well in existing organizational structures and systems

• Effective practices are changed to fit the system, as opposed to existing systems changing to support effective evidence-based practices.


EXISTING SYSTEM

Effective approaches are Changed to Fit the

System

Or Operate in the Shadows

(Ghost System)

Effective System Change

EXISTING SYSTEM IS

CHANGED TO SUPPORT

THE EFFECTIVENESS OF

THE APPROACH

(Host System)

EFFECTIVE APROACH


Changing on Purpose

People, organizations, and systems. . .• Cannot change everything at once (too big;

too complex; too many of them and too few of us)

• Cannot stop and re-tool (have to create the new in the midst of the existing)

• Cannot know what to do at every step (we will know it when we get there)

• Many outcomes are not predictable (who knew!?)

Trial & Learning


Shewhart (1924); Deming & Juran (1948); Six-Sigma (1990)

Plan – Decide what to do

Do – Do it (be sure)

Study – Look at the results

Act – Make adjustments

Cycle – Do over and over again until the intended benefits are realized

PDSA Cycles: Trial & Learning


Improvement Cycle Uses

Rapid Cycle Teams

Problem-solving

Practice Improvement

Usability Testing

Practice-Policy Feedback Loops

Transformation Zones


Implications

Clearly understand/define the “What”Stage-matched activities guide the processBuild processes/systems to continuously improve “drivers”Local and/or state systems will need time to implement effectivelySupport the development of organized, skilled implementation support to build organization and system capacity to implement well


For More InformationFor More Information

Melissa Van Dyke [email protected]

http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/

Documents

Evidence-Informed Planning: Benefiting from Evidence- Based Interventions Defending Childhood January 26, 2011 Melissa K. Van Dyke, LCSW Associate Director