Organized Delivery System DRAFT Evaluation Plan Darren Urada, Ph.D., Valerie Antonini, MPH, Cheryl Teruya, Ph.D., Elise Tran, Kate Lovinger, M.S., Howard

Organized Delivery System DRAFT Evaluation Plan

Darren Urada, Ph.D., Valerie Antonini, MPH, Cheryl Teruya, Ph.D., Elise Tran, Kate Lovinger, M.S., Howard Padwa, Ph.D., Diego Ramirez, June

Lim, Ph.D., Doug Anglin, Ph.D., Richard Rawson, Ph.D.

UCLA Integrated Substance Abuse Programs

CBHDA SAPT+ Meeting

March 26, 2015

Acknowledgments

Thanks to Lynn Brecht, Desiree Crevecoeur, Christine Grella ,Yih-Ing Hser, Michael Prendergast, Beth Rutkowski, and Liz Evans for their suggestions and advice.

Suggestions & advice appreciated!

DRAFT

Evaluation Goals• Evaluate the Organized Delivery System

(DMC, SAPT) in terms of:– Access to care– Quality of care– Coordination of care

• Within SUD continuum of care • With recovery support services• With mental health and primary care services

– Costs (might be led by DHCS)

• Help inform implementation via feedback.

Realistic Data Goals

• Use existing data where possible.

• Align measures with existing or expected future data requirements.

• Where necessary, supplement with new data collection while attempting to minimize the burden on stakeholders.

Design

• Randomized controlled trials are ideal but impractical here.

• Pre-Post Comparisons

• County comparisons (Opt-in vs. Opt-out)

• Qualitative data

State Timeline (DRAFT)

Simplest Scenario:No Overlap Between Phases

Possible Scenario:Overlapping Phases

Likely Scenario:Overlapping Phases and Start Dates

BaselineBaseline Yr 1 follow-upYr 1 follow-up

Phase 1

County A (Phase 1)

2015 2016


County B (Phase 1)


County C (Phase 2)


County D (Phase 2)

Phase 2

Solution:Examine Counties by Start Date, Not Phases


Solution:Examine Counties by Start Date, Not Phases

County A (Phase 1)

Start Date Start Date + 1 Year

County B (Phase 1)

County C (Phase 2)

County D (Phase 2)

BaselineBaseline

BaselineBaseline

BaselineBaseline

Yr 1 follow-upYr 1 follow-up



Note: Creates up to 371 time points, with multiple measures at each one.

T-1 (now)T-1 (now) T0 (start/baseline)T0 (start/baseline) T1 (end of yr 1)T1 (end of yr 1) T2,3,4 ?T2,3,4 ? TendTend

Provider Survey

Waiting list q EBPs; staffing Integrated Practice Assessment Tool (IPAT) and single-item version

Patient Survey

Perceptions of care Quality of life (SF-12)

Provider Survey

Waiting list q EBPs; staffing IPAT

Key Informant Interviews (County Focused)

Plans for implementation

Perceptions of system (other providers)?


Recommendations for future phases; insights, challenges, lessons learned

Patient Survey


Smaller Provider Survey (if needed)

Smaller Patient Survey (if needed)

Provider Survey

Waiting list q IPAT

Patient Survey



Data to be

Collectedby UCLA

T-1T-1 T0T0 T1T1 T2,3, 4?T2,3, 4? TendTend

Data

CalOMS OSHPD Medi-Cal DATAR Licensed beds NSDUH Chem hospitals, freepsych KL Capacity/Access Project

Data

CalOMS OSHPD Medi-Cal DATAR Licensed beds NSDUH Chem hospitals, freepsych Kate Special Project

Data


Data


Data


County/DHCS Audit; EQRs

Use of EBPs Staffing MOUs Beneficiary number

ASAM

(ASI)

ASAM

(ASI)





ASAM

(ASI)

Existing Data to be Analyzed by UCLA

Overview of Measures• Access - Has access to treatment increased in

counties that have opted in to the waiver?

• Quality - Has quality of care improved in counties that have opted in to the waiver?

• Integration & Coordination of Care - Is SUD tx being coordinated within the continuum of care? With recovery support services? With primary care and mental health services?

• Cost (might be led by DHCS) - Is the waiver cost effective?

Access

Potential Measures of AccessHas access to treatment increased?

• Capacity– # admission by type of service (CalOMS-Tx)– # of admissions (Medi-Cal claims)– Licensed beds (includes privately funded beds, inpatient detox)– Residential capacity (DATAR)– Newly certified and de-certified sites (SMART6i data)– Capacity by zip code/city and modality

• Penetration Rates– % beneficiaries receiving services (CalOMS-Tx, Medi-Cal)

Access (continued)• MAT use

– Prescriptions filled for MATs (Medi-Cal claims)– NTPs, outpatient-medicated (CalOMS-Tx)– # of physicians able to prescribe Bup by ZIP/city

• Time to treatment/between treatment– Transition times between levels (CalOMS-Tx)– Typical wait time by modality (Provider survey, ASAM,

CalOMS-Tx)

• Telehealth– billing for teleheath increased? (Medi-Cal claims)– offer telehealth? (Provider survey)

Access (continued)• Existence of a functioning/up-to-date beneficiary

access number, provider directory for patients– Web searches, calls.– Ability to get an appointment?

• Other medical services– MH use, ER use, hospital inpatient days (Medi-Cal)– Chemical dependency recovery hospitals and freestanding

psychiatric hospitals (OSHPD).

• Misc. from Provider Survey– After-hours care/urgent care– Interpretation services.

Quality

Potential Measures of Quality

Has quality of care improved?

• Appropriate placement (ASAM, CalOMS-Tx): – Comparison of ASAM scores and actual placement– Use of continuing ASAM assessments, appropriate

transitions– % of referrals with successful treatment engagement

(e.g., stepping down)

Quality (continued)

• # of admissions to chemical dependency recovery hospitals and free standing psychiatric hospitals (OSHPD)

• # of ER and psychiatric emergency visits (Medi-Cal, CSI-MH)

• # of hospital inpatient days (Medi-Cal, CSI-MH)• % of patients who initiated and engaged in AOD

treatment (if receiving services from primary care; Healthcare Effectiveness Data & Information Set [HEDIS])

• Data indicator reports• # of grievances (Grievance reports)• Workforce (County or DHCS monitoring?):

– Staff turnover– Staff to client ratios– Staff /client population ethnic/racial diversity– Staff use of EBPs (fidelity)– Training (e.g., cultural competency)– Staff certification documented

Quality (continued)

• Outcomes• AOD use (CalOMS-Tx, patient survey)• Social support (CalOMS-Tx, patient survey)• Living arrangements/housing (CalOMS-Tx,

patient survey)• Employment (CalOMS-Tx, patient survey)• Quality of Life / Functioning (CalOMS-Tx,

patient survey)• Use of other services (e.g., ER, mental health)

(CSI-MH, Medi-Cal, OSHPD)

Quality (continued)

• Client perceptions of care (Mental Health Statistics Improvement Plan [MHSIP] Consumer Survey)– General satisfaction– Perception of access– Perception of quality and appropriateness– Perception of participation in treatment planning– Perception of outcome of services– Perception of functioning– Perception of social connectedness

Quality (continued)

(to be adapted)

• Provider perceptions of service delivery and implementation of the DMC ODS (Provider survey)– Staffing (e.g., counselor-client ratio) and turnover– Use of EBPs– Cultural competency training– Patient centered care (e.g., shared decision

making, provision of information about treatment options, family involvement)

– Implementation challenges, what’s working well– Training/technical assistance needs– Suggestions for improving implementation

Quality (continued)

• Stakeholders’ (e.g., county level administrators, consumer advocates) perceptions of the quality of services being delivered and implementation of the DMC ODS (Key informant interviews)– Biggest challenges (anticipated/current)– Strategies to address challenges– What’s working well– Communication– Training and technical assistance needs– Recommendations for improvement– Lessons learned

Quality (continued)

Integration / Coordination

Potential Measures of Integration and Coordination of Care

Is SUD treatment being coordinated within the continuum of care? With recovery support services?

With primary care and mental health services?

TWO Components:1.WITHIN SYSTEM

• SUD Continuum of Care

• Recovery Support Services

2.ACROSS SYSTEMS • SUD + MH• SUD +PH

Integration and Coordination

Summary of Data Collection Activities

1. Survey – County Administrators

2. Qualitative Interviews – County Administrators

3. Survey – Provider Executives

4. Patient Survey

5. Considering Qualitative interviews – Providers Purposive sample from providers with varying levels of care

integration

6. Secondary analysis Quantify referrals to and from primary care and mental health Quantify referrals across the SUD continuum

Component 1 – Within System

• County level (Administrator survey, interview)– Guidelines for SUD providers regarding

establishing MOUs/formal partnerships with other levels of care within the continuum

– Procedures for MOU enforcement / monitoring/reporting among providers?

**Reference County Implementation plan and initial survey response to guide interview

Component 1 – Within System• Provider Level (survey)

– Levels/Types of Treatment services offered

– Treatment Partnerships/MOUs • bi-directional agreements and utilization

– Recovery Support Service Partnerships• What/how do they incorporate RSS in their treatment program• Patient navigator services?

– Communication• Pt. Information Transfer (EHR and data sharing protocols)• Interpersonal communication (pt care plan exchanges)

– Organizational Dynamics• Leadership buy in/mission statements• Operational reliability to facilitate the continuum for all patients

• Patient survey

Component 2 – Across Systems

• County level (Administrator survey and interview)– Communication between departments (scheduled

meetings/frequency, etc)– Dept level MOUs (SUD and health plans, and mental health)– Guidelines for SUD providers regarding establishing

MOUs/formal partnerships with MH and physical health– Procedures for MOU enforcement / monitoring/reporting

among providers

**Reference County Implementation plan and initial survey response to guide interview


• Provider Level (survey)– SUD + MH– SUD + PH

• Assess provider perception of current level of integration/collaboration and goal

• Implement IPAT survey – Integrated Practice Assessment Tool – Developed by SAMHSA-HRSA – Decision tree leading to a determined Level of Integration/

Collaboration Care (Level 1-6)

• Patient survey

Component 2 – Across SystemsSAMHSA-HRSA Six Levels of Integration/Coordination


• After determining IPAT level of Integration…– More in-depth for levels 1-4 addressing the following

domains:• Systematic Screening for add’l services (including PH, MH,

recovery, and patient navigation services)• MOUs/referral protocols• Array of expertise/services available onsite (co-located or

integrated)• Proportion of patients that are being coordinated across systems• Physical proximity of partnered service providers• Separation time between referral and scheduled visit• Available patient navigators• Engagement of patients and family in care plan

Questions

• Collecting ASAM data

• Getting CalOMS-Tx summaries back to providers

• Are there other things counties plan to measure?

Health Plan Use of ASAM Criteria

• Magellan:http://dhhs.ne.gov/behavioral_health/Documents/ASAMChecklist.doc

• Cigna:http://apps.cignabehavioral.com/web/basicsite/media/consumer/educationAndResourceCenter/articles/comparisonSummary.pdf

• Beacon:http://www.beaconhealthstrategies.com/private/pdfs/MA%20LOC%202014%20Final.pdf

• Managed Health Network:https://www.mhn.com/static/pdfs/MHNLOC.pdf

Note: Sources were compiled by Los Angeles County Evaluation System (LACES) evaluation staff at UCLA ISAP.

Use of ASAM Criteria by Washington State

• Washington Administrative Code Chapter 388-877: Behavioral Health Services Administrative Requirementshttp://app.leg.wa.gov/WAC/default.aspx?cite=388-877&full=true

• Washington Administrative Code Chapter 388-877B: Chemical Dependency Services http://app.leg.wa.gov/WAC/default.aspx?cite=388-877B&full=true

See you next time!

Questions? Comments?

Darren Urada, Ph.D.

[email protected]