Selecting and Implementing Evidence-Based Practices Webinar

Webinar Wednesdays are brought to you by the Northeast ATTC

Jim Aiello, MA, MEd

December 14, 2011

A nationwide, multidisciplinary resource for professionals in the addictions treatment and recovery services field, the ATTC Network serves to:

What are the ATTCs?

– Raise awareness of evidence-based and promising treatment and recovery practices,

– Build skills to prepare the workforce to deliver state-of-the-art addictions treatment and recovery services, and

– Change practice by incorporating these new skills into everyday use for the purpose of improving addictions treatment and recovery outcomes

History

• Established in 1993 by the Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Treatment (CSAT).

• The ATTC Network is comprised of 14 Regional Centers and a National Office which serve the 50 U.S. states, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, and the Pacific Islands.

www.attcnetwork.org

Click on “Find a Regional Center” to locate the ATTC that serves your region

A definition of “technology transfer”

A multidimensional process that intentionally promotes

the use of an innovation. Technology transfer begins

during the development of an innovation, continues

through its dissemination, and extends into its early

implementation. This process requires multiple

stakeholders and resources, and involves activities

related to the translation and adoption of an innovation.

Technology transfer is designed to accelerate the

diffusion of an innovation.

A little perspective….

In the process of using evidence-based practices, we are here

Workshop Objectives:

• Outline a comprehensive strategy for selecting and implementing a practice

• Discuss barriers and challenges to implementing a practice

• Review the need for organizational “self-assessment” to determine readiness for change

• Examine a framework for evaluating evidence-based practices

• Describe a model for technology transfer

• Describe the necessary steps for successful implementation of an evidence-based practice

• Discuss steps necessary to insure fidelity

POLL #1

An EBP Definition

A program or practice that has been

demonstrated through scientific

studies to be effective in improving

outcomes for a specific population.

Blase, K. and Fixsen, D. (2008). Evidence Based Programs to Support Caregivers:

Real World Implementation Challenges and Solutions. National Implementation

Research Network.

POLL #2

Why Should You Care?

• Research directly or indirectly affects the practice of prevention and treatment on a daily basis.

• Research is ongoing and evolving…what “works” today may be replaced by more effective treatments later.

• Payers/funders are increasingly concerned with outcomes

• Payers/funders are requesting (and some places, demanding) the use of EBPs

AND, most importantly…

• Because they are tools that can be effective in helping clients to get better!

In many ways, addiction services have been greatly

influenced by:

•Faith: Turning it over to a Higher Power and working the Steps

•Belief: “It worked for me”

•Anecdotal Evidence: “Seems to work for most people”

•Influence: “Everybody does it this way”

•Tradition: “We’ve always done it this way”

•Mandate: “We have to do it this way”

Why not science?

Just because a practice or

innovation has an “evidence base”

doesn’t necessarily mean it’s right

for your region or agency.

One day Alice came to a fork in the road and saw a Cheshire cat in a tree.

“Which road to I take?” she asked.

His response was a question: “Where do you want to go?”

“I don’t know,” Alice answered.

“Then,” said the cat, “it doesn’t matter.”

Lewis CarrollAlice in Wonderland

Develop a sense of where you need to go

before you begin the process of

implementing an evidence-based practice.

Poll #3

Guidelines for Building an Effective Change Process

Guidelines – The Foundation

1. Base objectives on client or agency need (support with data when possible).

2. Select objectives that management will fully support.

3. Assess for a match with agency mission and values.

4. Make objectives specific, achievable and measurable.

5. Appoint a change leader.

Like the foundation for a

house, the first 5

guidelines form the basis

for moving ahead with

any change initiative.

Guidelines – The Framework

6. Assemble a team to plan the improvement (be sure to include Opinion Leaders).

7. Develop a detailed plan and start small.

8. Thoroughly prepare supervisors and staff.

9. Monitor fidelity via direct observation.

These four steps are the

framework for change.

Like the walls and roof of a

house, they comprise what

most people see when

viewing a change.

Too often the change process ends here, but the

finishing work that makes the house something

you want to buy or live in has not been done.

Guidelines – The Finishing Work

10. Track change data to monitor progress.

11. Make adjustments as needed based on staff/client feedback.

12. Keep all staff informed about progress.

13. When objectives are achieved, celebrate and spread what works.

14. Develop a sustainability plan to assure maintenance of improvement.

Now the house is

complete.

Modifications and

improvements have

been made and it is

ready for move-in.

Barriers or challenges can include:

• Administrative philosophy

• Organizational policy

• System structure

• Agency staff (resistance, insufficient communication)

• Board of Directors

• Client population

Other barriers to adoption include:

• The innovation itself may be difficult to implement • Specialized training and ongoing supervision may be

required to fully implement the innovation

• Limited organizational resources• Unavailability of physicians or nursing staff for medications

• The technology may not be there (computers, EHR systems, etc)

• physical space constraints

• Financing issues• The approach may not be reimbursed by third parties

• Start up costs (free vs. proprietary materials)

Is your region or agency ready to

begin the process of implementing

an evidence-based practice?

The task of assessing the

readiness of your region or

agency is part of the preparation

stage of EBP implementation.

Questions to Consider

• What is your organization’s level of readiness?

• What aspects of your organization point to readiness?

• What aspects of your organization would require additional support or resources?

How can we conceptualize and

rank (and thus evaluate)

evidence-based practices?

What Makes a Practice Evidence-based?

Source: Drake et al (2001). Implementing evidence based practices in routine mental health service settings. Psychiatric Services, 52, 179-182.

Gold StandardMultiple randomized clinical trials

Second TierConsensus reviews of available science

Third TierExpert opinion based on clinical observation

Two specific frameworks for evaluating a practice or innovation

• NREPP: SAMHSA’s National Registry of Evidence-Based Programs & Practices (196)

– http://nrepp.samhsa.gov/

– Useful for assessing whether or not an intervention is an EBP

• IOWA Practice Improvement Collaborative (PIC) Criteria

– http://www.uiowa.edu/~iowapic/files/Criteria%20for%20Evaluating%20EBPs.pdf

– Useful for assessing whether or not an intervention is the right one for your region or agency

POLL #4

National Registry of Evidence-based Programs and Practices (NREPP)

• NREPP is a searchable online registry of mental health and substance abuse interventions that have been reviewed and rated by independent reviewers.

• The purpose of this registry is to assist the

public in identifying approaches to preventing and treating mental and/or substance use disorders that have been scientifically tested and that can be readily disseminated to the field.

http://www.nrepp.samhsa.gov/

NREPP

• NREPP can be a first step to promoting informed decision making.

• The information in NREPP intervention summaries are provided to help you begin to determine whether a particular intervention may meet your needs.

NREPP: SAMHSA’s National Registry of Evidence-Based Programs & Practices

• Provides descriptive information

• Evaluates the supporting science in terms of the quality of research

• Provides an indicator or measure of the intervention’s readiness for dissemination (and thus readiness for adoption)

NREPP: SAMHSA’s National Registry of Evidence-Based Programs & Practices

• Descriptive information includes:– Areas of interest (e.g. AOD or MH)– Demonstrated outcomes– Population descriptors (ages, sex, races, etc.)– Settings (OP, home, community, urban vs. rural, etc.)– Implementation history– Funding sources– Adaptations– Adverse effects– Costs

NREPP: SAMHSA’s National Registry of Evidence-Based Programs & Practices

• Quality of research measures- reviewers are doctoral level and have a strong background in methods of evaluation. Ratings are from 0-4 in each of the following categories:

– Reliability of measures– Validity of measures– Intervention fidelity– Missing data and attrition– Potential confounding variables– Appropriateness of analysis

NREPP: SAMHSA’s National Registry of Evidence-Based Programs & Practices

• Readiness for dissemination measures- reviewers are consumers of service, service providers, or experts in the field of implementation. Ratings are 0-4 in each of the following categories:

– Availability of implementation materials

– Availability of training and support resources

– Availability of quality assurance procedures

Let’s take a look

NREPP: SAMHSA’s National Registry of Evidence-Based Programs & Practices

– http://nrepp.samhsa.gov/

NREPP is concerned with evaluating practices and

innovations and presenting them in a way that allows

providers to review, evaluate and determine whether the

practice is appropriate for their region or agency. It does

not tell you, however, which practice you should use for

your specific region or agency.

IOWA PIC Guidelines

Practice Improvement Collaborative (PIC) –

a way to assess an EBP and its

appropriateness for your agency

http://www.uiowa.edu/~iowapic/files/Criteria%20for%20Evaluating%20EBPs.pdf

IOWA PIC Criteria

• The practice has at least one randomized clinical trial that has shown this practice to be effective.

• The practice has demonstrated effectiveness in several replicated research studies using different samples, at least one of which is comparable to the treatment population of our region or agency.

IOWA PIC Criteria (cont’d)

• The practice either targets behaviors or shows good effect on behaviors that are generally accepted outcomes.

• The practice can logistically be applied in our region or agency.

• The practice is feasible for our region or agency: It can be used in different formats (e.g. group), is attractive to third party payers, is low cost, and training is available.

IOWA PIC Criteria (cont’d)

• The practice is manualized or sufficiently operationalized for staff use. Its key components are clearly laid out.

• The practice is well-accepted by providers and clients.

• The practice is based on a clear and well-articulated theory.

IOWA PIC Criteria (cont’d)

• The practice has associated methods of ensuring fidelity.

• The practice can be evaluated.

• The practice shows good retention rates for clients.

• The practice addresses cultural diversity and different populations.

• The practice can be used by staff with a wide diversity of backgrounds and training.

A few thoughts about fidelity

Nothing is more noble, nothing more venerable than fidelity.

- Marcus Tullius Cicero

• Training Fidelity Strategies ensure consistent information and skill development

• Intervention Delivery Strategies ensure an EBP is delivered by staff as close to the original model as possible

• Intervention Receipt Strategies ensure the intervention is having the intended effect

Fidelity questions• Is the formal program manual being used (if one is

available)?

• Did you develop your own written procedures and practices for implementation?

• Did you provide staff training required by this EBP?

• Is the amount of supervision consistent with the EBP in terms of time, frequency, duration, etc.?

• Are the supervision techniques utilized consistent with the requirements of the EBP (e.g. one-on-one, observation, record review, etc.)?

• Did the implementation vary from the model?

• If so, was the variation supported by a planned rationale?

• Does your organizational culture continue to support the EBP?

• Does your agency have the necessary supplies to effectively continue the EBP?

• Does your agency have sufficient staff trained so normal attrition doesn’t threaten sustainability?

Adaptations

Successful adaptations:•Are made in consultation with the program developer•Are consistent with the theoretical foundations of the program•Are documented/referenced when conducting/reporting evaluation data

• If possible, make adaptations after the original model has been implemented with fidelity (“first do it right, then do it differently”)

• Conduct an evaluation of the adaptations after they have been implemented

(Boberly, 2005)

Hallmarks of “Watered Down” Programs

• Agency issues/agendas taking priority over and evidenced-based practice or program’s mission

• Implementers picking and choosing from model/curriculum components at will

• Program staff who cannot give a clear and detailed description of the program model

• Modifications not being clearly identified within the implementation process

Ideas for Keeping or Getting “Back on Track”

• Don’t sweep concerns under the rug; identify areas that pose challenges, lack of clarity

• Take stock of strength and weaknesses related to program fidelity and adaptation

• Start documenting what you’ve done (take a “no looking back” approach and start documenting what your doing as of today)

• Make the new or renewed approach to fidelity and adaptation a team effort

• Access support and resources from program partners and developers

The Change BookA Blueprint for Technology Transfer

www.attcnetwork.org/explore/priorityareas/.../tools/changebook.asp

“It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change.”

- Charles Darwin

Thank you for your time and attention!

Visit www.ireta.org/webinars

to read about and register for upcoming webinars from the Northeast ATTC.