Upload
ucla-ctsi
View
120
Download
0
Embed Size (px)
Citation preview
Measuring Sustainment of Multiple EBPs Fiscally Mandated in Children’s Mental
Health Services: Knowledge Exchange on Evidence-Based Practice Sustainment
(4KEEPS) Study Lauren Brookman-Frazee
Southern California Regional Dissemination, Implementation and Improvement Science Webinar Series
June 1, 2016
Objectives
1. Provide an overview of system driven EBP implementations and community context and the 4KEEPS Study.
2. Describe study methods and initial findings: • Characterizing sustainment outcomes
• Characterizing potential inner context determinants of sustainment outcomes
Public Mental Health Care System (Local, State)
Organizations contracted to provide MH Care
Therapists providing direct services to consumers
Exploration Phase
Preparation Phase
Implementation Phase
Sustainment Phase
Phases and Levels of EBP Implementation
Adapted from Aarons, Hurlburt & Horwitz, 2011
Public Mental Health Care System (Local, State)
Organizations contracted to provide MH Care
Therapists providing direct services to consumers
Exploration Phase
Preparation Phase
Implementation Phase
Sustainment Phase
Sustained Delivery of Multiple EBPs in System-Driven Implementation Efforts
Adapted from Aarons, Hurlburt & Horwitz, 2011
Statewide Reform: California’s Mental Health Services Act
Prevention and Early Intervention
(PEI)
Los Angeles County
• Nation’s largest county mental health department • 75 directly operated sites • 288 contracted agencies
(120 with child MH services)
• 8 Service Provision Areas range from rural to inner city communities
• Serves an ethnically diverse, disadvantaged population
Hispanic/Latino
Asian/Pacific Islander
African American
Applying Aarons et al (2011) Model to LACDMH PEI Timeline
Applying Aarons et al (2011) Model to LACDMH PEI Timeline
Characteristics of Practices Practice Age Range
(years) Target Problem(s)
Triple P 2-12 conduct
CPP 0-6 trauma; attachment
MAP 0-21 anxiety; trauma; depression; conduct
TF-CBT 3-18 trauma
CBITS 11-15 trauma
Seeking Safety 13-18 trauma; substance use
Applying Aarons et al (2011) Model to LACDMH PEI Timeline
Applying Aarons et al (2011) Model to LACDMH PEI Timeline
4KEEPS Study Aims
• A neutral, observational study investigating the sustainment of original 6 practices with implementation support.
• Aim 1: Characterize sustainment outcomes • EBP Concordant Care • Volume/penetration of each practice over time
• Aim 2: Use mixed methods to characterize inner context factors and early implementation conditions that potentially predict EBP sustainment.
• Aim 3: Identify inner context and early implementation conditions that determine sustainment outcomes
Funded by NIMH Grant # R01 MH100134 MPI Anna Lau and Lauren Brookman-Frazee
Multiple Sources of Data
LACDMH Administrative
Claims Data
LACDMH Site Visit
Documents
Online Surveys of Program Leaders and Therapists
Semi-Structured Interviews
Session Recordings
Sustainment outcomes and potential inner context determinants of outcomes
Today
2014 2015 2016 2017 2018
4KEEPS Study
launch
1/1/2014
Online survey 1
launch
3/1/2015
In-depth sample
launch
5/1/2015
Online survey 1
supplement
2/1/2016
In-depth sample wrap-up
Spring 2017
Online survey 2 launch
Spring 2017
4KEEPS Study Timeline
4KEEPS Events Timeline
Sustainment outcomes
• # & % agencies continuing to be reimbursed for the practice
• # & % Therapists continuing to claim to the practice
• # & %Unique clients served by the practice
• # $ % Units of services being provided within each practice
Practice Volume/Penetration
• Degree to which a therapist’s practice resembles the essential strategies one would expect within an evidence-based protocol for a given problem focus.
EBP Concordant Care
Practice Volume/Penetration
94
agencies
8,514 providers
87,100 children
2,331,000 psychotherapy claims
3,014,353 total claims
2009 2015
Cumulative # of Claims Per Practice • MAP = 905,395
• TF-CBT = 662,184
• Seeking Safety = 515,208
• Triple P = 140,147
• CPP = 105,231
• CBITS = 2835
Volume Over Time
Initial increases in raw volume for most practices in the first quarters with peak and leveling off after the initial few years Highlight the rapid impact of a fiscal policy change restricting reimbursement to specific practices and training in these practices
Gross Penetration Over Time
During the initial ramp-up period, some practices ramped up more quickly than others. Reordering and stabilization after 2 years MAP, TF-CBT, SS > Triple P, CPP, CBITS
Characteristics of Practices
Higher Penetration (MAP, TF-CBT, CPP) • Apply to a broad age range of
clients
• Can be delivered in multiple settings
• Addresses common presenting problem -trauma
• MAP also covers a range of presenting problems
• Train-the-trainer capacity (MAP, SS)
• Minimal training required for billing (SS)
Lower Penetration (Triple P, CPP, CBITS) • Apply to a narrower age
range of clients (CPP, CBITS)
• Restrictions on settings/format • Group (CBITS, Triple P) • School setting (CBITS) • Caregiver directed (CPP,
Triple P)
• Additional requirements to deliver such as MOAs between programs and schools (CBITS)
Sustainment outcomes
• # & % agencies continuing to be reimbursed for the practice
• # & % Therapists continuing to claim to the practice
• # & %Unique clients served by the practice
• # $ % Units of services being provided within each practice
Practice Volume/Penetration
• Degree to which a therapist’s practice resembles the essential strategies one would expect within an evidence-based protocol for a given problem focus.
EBP Concordant Care
Measuring Sustainment: EBP Concordant Care
Addresses concerns about the feasibility and
appropriateness of using traditional fidelity instruments to
asses ongoing delivery of multiple EBPs
EBP Concordant Care Assessment (ECCA) Development Process Reviewed existing practice inventories.
Adapted/selected/generated items (strategies).
Collected data from practice experts about how essential/interfering strategies are to their practice.
Delphi rating system used to determine item selection for therapist-report ECCA (alpha version).
Collected data from large sample of therapists using ECCA (alpha version).
Examine the properties of the therapist-report ECCA (alpha version).
Validate therapist-report ECCA with observational coding system.
Finalize therapist-report ECCA based on #6 and 7.
Current ECCA
Purpose Assesses the extent to which a therapist delivered individual psychotherapeutic strategies considered essential for a given EBP Target
Online Versions
Session (therapist report, observer rated) and Bi-monthly (therapist report)
Items 38 psychotherapeutic content and techniques EBP Targets #Content #Techniques
Conduct 18 9 Trauma 10 5 Anxiety 7 7 Depression 9 6
ECCA Data Collection To Date
• 710 therapists from 54 agencies completed a Bi-Monthly ECCA
Full Sample
• 71 therapists submitted 459 Session ECCAs with corresponding audio recordings of sessions as well as Bi-Monthly versions for each client
Validation Sub-Sample
ECCA Preliminary Findings
High internal consistency for each EBP Target Composites for Anxiety, Conduct, Depression and Trauma (alphas ranged from .86 to .95; M=.90).
Discriminant validity of the scales supported by significant differences in EBP Target composites by EBP delivered When therapists delivered an EBP that targeted Conduct or Trauma,
their ratings on the content items from the corresponding composite were significantly higher than for other EBP targets
Construct Validity supported by Item Response Theory Analyses indicating that items with high item difficulty appeared to include strategies that reflect high integrity EBP concordant care (Ignoring/Differential Reinforcement of Other Behaviors > Praise)
ECCA Preliminary Findings Continued Concordance between Session and Bi-monthly
version supporting by significant and large correlations between the average Session ECCA with the Bi-Monthly version on the EBP Target composites (range= .62 to .71; M=.68).
Concordance between therapist report and direct observation in process Early data based on a small subset of therapist reports
with corresponding observer ratings indicate greater concordance for strategies that are part of the practice type being delivered.
Next Steps
1) Finish validation of therapist-report ECCA with observational coding system 1) Projected sample of 1080 sessions with therapist report
and observer ratings
2) Examine inner context factors associated with ECCA composite scores.
3) Refine ECCA instrumentation based on concordance analysis and end user feedback on utility.
Potential Inner Context Determinants of Sustainment
• Changes in state regulatory requirements
• Changes in implementation strategies used
• Adoption, de-adoption of EBPs over time System Level
• Implementation support
• Organizational climate
• Early implementation condition
Organizational Level
• Therapist attitudes: perceptions of effectiveness and fit
• Therapist clinical adaptations to practices Therapist Level
Therapist and Practice Characteristics as Facilitators of Multiple EBP Implementation
Practices
Therapists
• The PEI Context and fiscal mandate for EBP reform presents a unique context • EBP vs. Practice as Usual • EBP1 vs. EBP2 vs. EBP3…
• Allows for parsing the variance in attitudes associated with • Therapists and therapist
characteristics • Practices and practice
characteristics
Demographics M (SD) or %
Age 37.00 (9.28)
Gender (female) 88%
Hispanic 43%
Race
White 50% Asian/ Pacific Islander 11% African American 7% American Indian/Alaska Native 1% Multiracial 8% Other 23%
Deliver Services in >1 Language
Spanish 47%
Other Language 10%
Survey: Therapist Characteristics (n=790)
Training Background M (SD) or %
Years Practiced as Therapist 7.21 (6.16)
Years Worked at Current Agency 5.12 (4.56)
Current # Clients 14.65 (10.48)
MH Discipline
MFT 55%
Social Work 30%
Clinical Psychology 11%
Counseling 2%
School Psychology <1%
Other 2%
Measuring Therapist Attitudes Practice-Specific Attitudes General Attitudes towards EBP
Perceived Characteristics of Intervention Scale, Cook et al., 2014
• Relative Advantage • Complexity • Compatibility • Potential for Reinvention • Total
Evidence-Based Practice Attitudes Scale, Aarons, 2004
• Openness • Divergence
Attitudes Differ by Practice
F b
Intercept (grand mean) 9667.23*** 3.29***
TF-CBT 183.10 *** .45***
CPP 29.73*** .28***
Triple P 11.77** .16**
MAP 5.28* .08*
SS 11.47** -.12**
CBITS 117.16*** -.84***
*p<.05; **p<.01; ***p<.001
Practice Characteristics
Practice PCIS
Alphas
Narrow Age
Range
Consultation
Required
Prescribed
Session
Content/ Order
CBITS (N = 65) .96
CPP (N = 140) .94
MAP (N = 527) .92
SS (N = 491) .93
TF-CBT (N = 582) .93
Triple P (N = 184) .94
Therapist Attitudes Differ by Practice Characteristics
1
2
3
4
5
Prescribed Session Content/Order
Consultation Required Narrow Age Range
PC
IS IT
EM M
EAN
PRACTICE CHARACTERISTIC
Yes
No
* * *
* = p < .001
Very Great Extent
Not at All
• Use sequential QUANqual design to understand:
• The types of adaptations that therapists make in community mental health settings.
• Which therapist characteristics predict types of adaptations (fidelity consistent vs. inconsistent).
• The reasons therapists make adaptations.
Therapist Adaptations
Fidelity Consistent
Modify presentation
Integrate supplemental
content
Lengthen/extend Pacing
Fidelity Inconsistent
Remove/skip components
Shorten/condense pacing
Adjust order of sessions/components
Characterizing Adaptations
1
2
3
4
5
Mea
n E
xten
sive
nes
s
Fidelity Consistent
CBITS CPP SS TFCBT TP
1
2
3
4
5
Fidelity Inconsistent
Types of Adaptations
• Therapists reported more fidelity consistent adaptations (EMM= 9.30; SE = 1.79) than fidelity inconsistent (EMM= 5.88; SE = 1.79) adaptations • F(1, 1,075.59) = 1,332.00, p <
.001
• Therapist-reported attitudes towards a practice: • Did not predict fidelity
consistent adaptations B = -.02, t = -1.47, p = .14
• Did predict fidelity inconsistent adaptations
• B = -.07, t = -5.67, p < .001
0
1
2
3
4
5
6
7
8
9
10
Fidelity Consistent Fidelity Inconsistent
Mea
n C
om
po
site
CBITS CPP SS TFCBT TP
Reasons for Adaptations
Culture
Crises
Adjust Order
Extend Pacing
Modify Presentation
Omit
Developmental Level I think the length of treatment is part of that too,
especially with the Hispanic families. Like you can’t just
jump right in; you need to spend a little more time
building rapport.
Some of the examples that they have there I think, you
know, might not necessarily fit like this particular culture
so I’ll find something that they could relate to a little bit
better, a situation they can relate to a little bit better.
I did work with one client that was developmentally
delayed. So for some of the cognitive behavioral piece it
was just really difficult to understand…I did my best to
teach it, but I didn’t feel like I focused a lot on it because
dude wasn’t really understanding it.
Oh, we’ll still go back to some of the relaxation
components, or we’ll kind of focus on what we can do to
manage our anxiety about it and whatnot.
You know, with people that are developmentally delayed or whether there needs to be more repetition, I’ll repeat it more.
I think it’s just different working with a child that young.
So we do a lot of art. So instead of like a trauma
narrative, we’ve been doing a lot of art on the
chalkboard.
You do have those clients that maybe might become homeless, or maybe parents are having a really hard time, maybe with other kids, older kids. So you might have a lot of crises that might surface throughout treatment, and that might postpone certain parts of it
Summary of Preliminary Findings
The scope and size of the PEI transformation provides a critical opportunity to examine multiple EBP implementation in a large and diverse natural laboratory
Sustainment varies by practice and over time
Inner context factors are critical to examine within EBP implementation efforts
• Therapist attitudes vary across practices.
• Therapists report more extensive fidelity consistent adaptations than fidelity inconsistent adaptations
Funding Source: NIMH R01MH100134