Evidence Based Practice – An Overview

Webinar for Reclaiming Futures October 23, 2008

Randolph Muck, M.Ed.CSAT/SAMHSAContact Info: randy.muck@samhsa.hhs.gov240-276-1576

Evidence Based Practice

What is it?

Why do it?

Points to consider

Evidence Based Practice

The term evidence-based practice (EBP) refers to preferential use of mental and behavioral health interventions for which systematic empirical research has provided evidence of statistically significant effectiveness as treatments for specific problems. Alternate terms with the same meaning are evidence-based treatment (EBT) and empirically-supported treatment (EST).

Evidence Based Practice

Tested with good outcomes

Manual exists so it can be replicated/trained

A training program exists

Supervision leading to certification

Ongoing monitoring

Outcomes measurement

Ways of Viewing EBP

EBP is a process. EBP is a way of doing practice that integrates the best evidence with clinical expertise and consumer values. (EBP as a verb.) (Sackett et al., 2000)

PractitionerExpertise

BestEvidence

Client Values & Preferences

EBP

Ways of Viewing EBP

EBP is a product. An evidence-based practice is any practice that has been established as effective through scientific research according to some set of explicit criteria. (EBP as a noun.) (Drake, 2001)

– EB Interventions. (A-CRA, MET/CBT5)– EB Skill sets. (CBT, Behavioral Parent Training)

Definition of Implementation

“…Specified set of activities designed to put into practice an activity or program of known dimensions…such that independent observers can detect its presence and strength.”

(Fixsen et al, 2004, p. 5)

Definition of Fidelity

Strategies used to monitor the faithful delivery of a manual-guided behavioral intervention

Important dimensions include – adherence (i.e., extent to which intervention

procedures were delivered as prescribed in the treatment manual)

– competence (i.e., qualitative measure of the skillfulness in which intervention procedures are delivered)

Different Types of Manuals

Session Driven

Procedure Driven

Principle Driven

Randomized Clinical Trials (RCT) are to Evidence Based Practice (EBP) like Self-reports are to Diagnosis

They are only as good as the questions asked (and then only if done in a reliable/valid way)

They are an efficient and logical place to start But they can be limited or biased and need to be

combined with other information Just because the person does not know something

(or the RCT has not be done), does not mean it is not so

Synthesizing them with other information usually makes them better

The field is increasingly facing demands from payers, policymakers, and the public at large for “evidence-based practices (EBP)” which can reliably produce practical and cost-effective interventions, therapies and medications that will

– reduce risks for initiating drug use among those not yet using, – reduce substance use and its negative consequences among those who are

abusing or dependent, and– reduce the likelihood of relapse for those who are recovering

NIDA Blue Ribbon Panel on Health Services Research (see www.nida.nih.gov )

Context

So what does it mean to move the field towards Evidence Based Practice (EBP)?

Introducing reliable and valid assessment that can be used – At the individual level to immediately guide clinical judgments

about diagnosis/severity, placement, treatment planning, and the response to treatment

– At the program level to drive program evaluation, needs assessment, and long term program planning

Introducing explicit intervention protocols that are– Targeted at specific problems/subgroups and outcomes– Having explicit quality assurance procedures to cause adherence

at the individual level and implementation at the program level

Having the ability to evaluate performance and outcomes – For the same program over time, – Relative to other interventions

The Current Renaissance of Adolescent Treatment Research

Feature 1930-1997 1997-2005

Tx Studies* 16 Over 200

Random/Quasi 9 44

Tx Manuals* 0 30+

QA/Adherence Rare Common

Std Assessment* Rare Common

Participation Rates Under 50% Over 80%

Follow-up Rates 40-50% 85-95%

Methods Descriptive/Simple More Advanced

Economic Some Cost Cost, CEA, BCA

* Published and publicly available

Juvenile Justice involved youth increasing presence in the treatment system

Support for funding relies on ability to demonstrate effectiveness Treatment needs of the youth that we see and the need to

incorporate appropriate and effective interventions for these needs

Continuing Care is as or more important than the treatment delivered

Issues to Consider

-

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

Juve

nile

Jus

tice

Sch

ool

Sel

f/F

amil

y

Oth

erC

omm

unit

y

Oth

er S

A T

xA

genc

y

Oth

er H

ealt

hC

are

Em

ploy

ee/E

AP

0%

20%

40%

60%

80%

100%

120%

140%

1993

2003

Change

Change in Referral Sources: 1993-2003

Source: Treatment Episode Data Set (TEDS) 1993-2003.

JJ referrals have doubled, are 53% of 2003 admissions and

driving growth

Other sources of Referral have grown, but less than expected

41%

37%

12%

37%

114%

115%

5%

61% growth

53% Have Unfavorable Discharges

Source: Data received through August 4, 2004 from 23 States (CA, CO, GA, HI, IA, IL, KS, MA, MD, ME, MI, MN, MO, MT, NE, NJ, OH, OK, RI, SC, TX, UT, WY) as reported in Office of Applied Studies (OAS; 2005). Treatment Episode Data Set (TEDS): 2002. Discharges from Substance Abuse Treatment Services, DASIS Series: S-25, DHHS Publication No. (SMA) 04-3967, Rockville, MD: Substance Abuse and Mental Health Services Administration. Retrieved from http://wwwdasis.samhsa.gov/teds02/2002_teds_rpt_d.pdf .

0% 20% 40% 60% 80% 100%

Outpatient(37,048 discharges)

IOP(10,292 discharges)

Detox(3,185 discharges)

STR(5,152 discharges)

LTR(5,476 discharges)

Total(61,153 discharges)

Completed Transferred ASA/ Drop out AD/Terminated

Despite being widely recommended, only 10% step down after intensive treatment

Median Length of Stay is only 50 days

Source: Data received through August 4, 2004 from 23 States (CA, CO, GA, HI, IA, IL, KS, MA, MD, ME, MI, MN, MO, MT, NE, NJ, OH, OK, RI, SC, TX, UT, WY) as reported in Office of Applied Studies (OAS; 2005). Treatment Episode Data Set (TEDS): 2002. Discharges from Substance Abuse Treatment Services, DASIS Series: S-25, DHHS Publication No. (SMA) 04-3967, Rockville, MD: Substance Abuse and Mental Health Services Administration. Retrieved from http://wwwdasis.samhsa.gov/teds02/2002_teds_rpt_d.pdf .

0 30 60 90

Outpatient(37,048 discharges)

IOP(10,292 discharges)

Detox(3,185 discharges)

STR(5,152 discharges)

LTR(5,476 discharges)

Total(61,153 discharges)

Lev

el o

f C

are

Median Length of Stay

50 days

49 days

46 days

59 days

21 days

3 days

Less than 25% stay the

90 days or longer time

recommended by NIDA

Researchers

Past 90 day HIV Risk Behaviors

Source: CSAT AT Outcome Data Set (n=9,276 adolescents)

84%

38%

32%

26%

21%

3%

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Sexually active

Sex Under the Influence of AOD

Multiple Sex partners

Any Unprotected Sex

Victimized Physically, Sexually, orEmotionally

Any Needle use

Recovery Environment

Source: CSAT AT Outcome Data Set (n=9,276 adolescents)

57%

49%

28%

74%

65%

14%

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Social Peers Getting Drunk Weekly+

School/Work Peers Getting Drunk Weekly+

Others at Home Getting Drunk Weekly+

Social Peers Using Drugs

School/Work Peers Using Drugs

Others at Home Using Drugs

Co-Occurring Psychiatric Problems

Source: CSAT AT Outcome Data Set (n=9,276 adolescents)

79%

54%

45%

37%

26%

17%

59%

47%

31%

25%

16%

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Any Co-occurring Psychiatric

Conduct Disorder

Attention Deficit/Hyperactivity Disorder

Major Depressive Disorder

Traumatic Stress Disorder

General Anxiety Disorder

Ever Physical, Sexual or Emotional Victimization

High severity victimization (GVS>3)

Ever Homeless or Runaway

Any homicidal/suicidal thoughts past year

Any Self Mutilation

Past Year Violence & Crime

*Dealing, manufacturing, prostitution, gambling (does not include simple possession or use)

Source: CSAT AT Outcome Data Set (n=9,276 adolescents)

82%

69%

66%

51%

49%

45%

84%

68%

39%

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Any violence or illegal activity

Physical Violence

Any Illegal Activity

Any Property Crimes

Other Drug Related Crimes*

Any Interpersonal/ Violent Crime

Lifetime Juvenile Justice Involvement

Current Juvenile Justice involvement

1+/90 days In Controlled Environment

Multiple Problems* are the Norm

Source: CSAT AT Common GAIN Data set

NoneOne

Two

Three

Four

Five to Twelve

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Most acknowledge 1+ problems

Few present with just one problem

(the focus of traditional research)

In fact, over half present

acknowledging 5+ major problems

* (Alcohol, cannabis, or other drug disorder, depression, anxiety, trauma, suicide, ADHD, CD, victimization, violence/ illegal activity)

No. of Problems* by Severity of Victimization

Source: CSAT AT Common GAIN Data set (odds for High over odds for Low)

* (Alcohol, cannabis, or other drug disorder, depression, anxiety, trauma, suicide, ADHD,

CD, victimization, violence/ illegal activity)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Low (31%) Moderate (17%) High (51%)

Five or More

Four

Three

Two

One

None

Those with high lifetime levels of

victimization have 117 times higher

odds of having 5+ major problems*

GAIN General Victimization Scale Score (Row %)

Most Lack of Standardized Assessment for…

Substance use disorders (e.g., abuse, dependence, withdrawal), readiness for change, relapse potential and recovery environment

Common mental health disorders (e.g., conduct, attention deficit-hyperactivity, depression, anxiety, trauma, self-mutilation and suicidality)

Crime and violence (e.g., inter-personal violence, drug related crime, property crime, violent crime)

HIV risk behaviors (needle use, sexual risk, victimization)

Child maltreatment (physical, sexual, emotional)

Summary of Problems in the Treatment System

The public systems is changing size, referral source, and focus – often in different directions by state

Major problems are not reliably assessed (if at all) Less than 50% stay 50 days (~7 weeks) Less the 25% stay the 3 months recommended by

NIDA researchers Less than half have positive discharges After intensive treatment, less than 10% step down

to outpatient care While JJ involvement is common, little is known

about the rate of initiation after detention

EBPs and Treatment for Youth in the Juvenile Justice

System

Some Programs Have Negative or No Effects on recidivism

“Scared Straight” and similar shock incarceration program

Boot camps mixed – had bad to no effect

Routine practice – had no or little (d=.07 or 6% reduction in recidivism)

Similar effects for minority and white (not enough data to comment on males vs. females)

The common belief that treating anti-social juveniles in groups would lead to more “iatrogenic” effects appears to be false on average (i.e., relapse, violence, recidivism for groups is no worse then individual or family therapy)

Source: Adapted from Lipsey, 1997, 2005

Meta Analysis of the Effectiveness of Programs for Juvenile Offenders

N of

Offender Sample Studies

Preadjudication (prevention) 178

Probation 216

Institutionalized 90

Aftercare 25

Total 509

Source: Adapted from Lipsey, 1997, 2005

Most Programs are actually a mix of components

Average of 5.6 components distinguishable in program descriptions from research reports

Intensive supervisionPrison visitRestitutionCommunity serviceWilderness/Boot campTutoringIndividual counselingGroup counselingFamily counselingParent counselingRecreation/sportsInterpersonal skills

Anger managementMentoringCognitive behavioralBehavior modificationEmployment trainingVocational counselingLife skillsProvider trainingCaseworkDrug/alcohol therapyMultimodal/individualMediation

Source: Adapted from Lipsey, 1997, 2005

Most programs have small effectsbut those effects are not negligible

The median effect size (.09) represents a reduction of the recidivism rate from .50 to .46

Above that median, most of the programs reduce recidivism by 10% or more

One-fourth of the studies show recidivism reductions of 30% or more

The “nothing works” claim that rehabilitative programs for juvenile offenders are ineffective is false

Source: Adapted from Lipsey, 1997, 2005

Major Predictors of Bigger Effects

1. Chose a strong intervention protocol based on prior evidence

2. Used quality assurance to ensure protocol adherence and project implementation

3. Used proactive case supervision of individual

4. Used triage to focus on the highest severity subgroup

Impact of the numbers of Favorable features on Recidivism (509 JJ studies)

Source: Adapted from Lipsey, 1997, 2005

Usual Practice has little

or no effect

Program types with average or better effects on recidivism

AVERAGE OR BETTER BETTER/BEST

Preadjudication

Drug/alcohol therapy Interpersonal skills training

Parent training Employment/job training

Tutoring Group counseling

Probation

Drug/alcohol therapy Cognitive-behavioral therapy

Family counseling Interpersonal skills training

Mentoring Parent training

Tutoring

Institutionalized

Family counseling Behavior management

Cognitive-behavioral therapy Group counseling

Employment/job training Individual counseling

Interpersonal skills trainingSource: Adapted from Lipsey, 1997, 2005

Cognitive Behavioral Therapy (CBT) Interventions that Typically do Better than Practice in Reducing Recidivism (29% vs. 40%)

Aggression Replacement Training Reasoning & Rehabilitation Moral Reconation Therapy Thinking for a Change Interpersonal Social Problem Solving Multisystemic Therapy Functional Family Therapy Multidimensional Family Therapy Adolescent Community Reinforcement Approach MET/CBT combinations and Other manualized CBT

Source: Adapted from Lipsey et al 2001, Waldron et al, 2001, Dennis et al, 2004

NOTE: There is generally little or no differences in mean effect size between these brand names

Implementation is Essential (Reduction in Recidivism from .50 Control Group Rate)

The effect of a well implemented weak program is

as big as a strong program implemented poorly

The best is to have a strong

program implemented

well

Thus one should optimally pick the strongest intervention that one can

implement wellSource: Adapted from Lipsey, 1997, 2005

Moderate to large differences in Cost-Effectiveness by Condition

Source: Dennis et al., 2004

$0

$4

$8

$12

$16

$20

Cos

t per

day

of

abst

inen

ce o

ver

12 m

onth

s

$0

$4,000

$8,000

$12,000

$16,000

$20,000

Cos

t per

per

son

in r

ecov

ery

at m

onth

12

CPDA* $4.91 $6.15 $15.13 $9.00 $6.62 $10.38

CPPR** $3,958 $7,377 $15,116 $6,611 $4,460 $11,775

MET/ CBT5MET/

CBT12FSN MET/ CBT5 ACRA MDFT

* p<.05 effect size f=0.48** p<.05, effect size f=0.72

Trial 1 Trial 2

* p<.05 effect size f=0.22 ** p<.05, effect size f=0.78

MET/CBT5 and 12 did better

than FSN

ACRA did better than MET/CBT5, and both did better than MDFT

Choosing an EBP

Best evidencePractitioner experienceYouth/Family values and preferencesReadiness for change (buy-in at all levels

of agency, but particularly the top)Cost/ResourcesAbility to implement well

What are the pitfalls of EBP?

EBP generally causes some staff turnover EBP often shines a light on staff or work place problems

that would otherwise be ignored EBP often impact a wide range of existing procedures and

policies – requiring modification and provoking resistance EBP (and most organizational changes) will fail without

good senior staff leadership EBP typically require going for more funds from grant or

other funders On-going needs assessment will create demand for more

change and more EBP

A Fearless Appraisal… We are entering a renaissance of new knowledge in this area, but are only

reaching 1 of 10 in need

Several interventions work, but 2/3 of the adolescents are still having problems 12 months later

Effectiveness is related to severity, intervention strength, implementation/adherence, and how assertive we are in providing treatment

As other therapies have caught up technologically, there is no longer the clear advantage of family therapy found in early literature reviews

While there have been concerns about the potential iatrogenic effects of group therapy, the rates do not appear to be appreciably different from individual or family therapy if it is done well (important since group tx typically costs less)

Effectiveness was not consistently associated with the amount of therapy over a short period of time (6-12 weeks) but was related to longer term continuing care

Common Strategies you can do NOW Standardize assessment and identify most common problems Pool knowledge about what staff have done in the past, whether it

worked, and what the barriers were Identify system barriers (e.g., criteria to local access case management,

mental health) that could be avoided if thought of in advance Identify existing materials that could help and make sure they are

readily available on site Identify promising strategies for working with the adolescent, parents,

or other providers Develop a 1-2 page checklist of things to do when this problem comes up Identify a more detailed protocol and trainer to address the problem,

then go for a grant to support implementation

Evidenced Based Practice - Summary

Achieving reliable outcomes requires reliable measurement, protocol delivery and on-going performance monitoring.

The GAIN is one measure that is being widely used by CSAT grantees and others trying to address gaps in current knowledge and move the field towards evidenced based practice.

Standardized and more specific assessment helps to draw out treatment planning implications of readiness for change, recovery environment, relapse potential, psychopathology, crime/violence, and HIV risks.

Adolescents entering more intensive levels of care typically have higher severity.

Multiple problems and child maltreatment are the norm and are closely related to each other.

There is a growing number of standardized assessment tools, treatment protocols and other resources available to support evidenced based practices.