PCCYFS 2012 Annual Spring Conference 02/2/12 Fidelity Management: Increasing Provider Negotiating Strength in the Environment of Evidence-based Programs

PCCYFS 2012 Annual Spring Conference02/2/12

Fidelity Management: Fidelity Management: Increasing Provider Increasing Provider

Negotiating Strength in the Negotiating Strength in the Environment of Evidence-Environment of Evidence-

based Programs based Programs

Presented By:Wesley Spectrum Services

Doug Muetzel, Pamela Meadowcroft & Pamela Weaver

For more information, please contact Doug Muetzel, CEO, 412-831-9390, [email protected]

1

PCCYFS 2012 Annual Spring

Conference

2

Agenda

• Introductions and “setting the stage” for CEOs in the EBP environment

• Scenario Building: Experiences with EBP• Case Study: WSIH – survived and

thrived• Workshop case: participant selection• Summary lessons learned

Wesley Spectrum Services (c) 2


Conference

3

Learning Objectives

• Benefits and limitations of evidence-based models

• Alternative to packaged models (comparable level of accountability for less cost)

• How to use in your organization / program

• How to use “Fidelity Management” to create funding case



Conference

4

1. Setting the stage

• Introductions• CEO perspective

– it starts with mission– must be sustainable– “mantra” to live by

• Blue Print Evidenced Based Models – create a dilemma for CEO’s



Conference

5 Wesley Spectrum Services (c) 5

Payor ROI Confidence

Refe

rrals

•a large“gap”emerged… payors had minimal data and lacked confidence in their ROI• EBP a reasonable solution that filled the gap• the realities have created a real dilemma for CEOs


Conference


VolumeQ

uality

• inherent in “fee for service” model is the requirement of maximizing volume to sustain the business• mission advancement and sustainability driven by quality• CEO charged with sustainability and mission advancement


Conference

7Wesley Spectrum Services (c) 7

Costs

Qu

ality

• does quality ever cost too much?• quality is related to costDigress

Unit costs and reducing cost for sustainability


Conference


Quantity (census)

Qu

ality

• do we really know what payors will pay for?• balancing quality and quantity


Conference

9

Stakeholder perspective helps us understand complexity of social services and our marketing context:

Allegheny County• Marc Cherna• Marcia Sturdivant• Pat Valentine• Case managers

Providers• CEO• Program Directors• Direct care workers

Commonwealth• Legislators• DPW Secretary• Deputy Secretary

Managed Care (CCBH)• CEO/management• Case managers

Sellers of Evidenced Based Programs

• Researchers• Consultants

Context: Marketing



Conference

10

EBP Service Contracts

In the future ….

•Standardized contracts to require agreed upon evidence-based performance measures and outcome data•Adherence to the principles of the Standardized Program Evaluation Protocol (SPEP)

– Dosage should be consistent with what research indicates is effective

– Do providers have internal protocols for monitoring adherence to written policies?

•Fiscal policy supporting evidence-based programming

[based on Pennsylvania’s Juvenile Justice System Enhancement Strategy (JJSES) 12/12/11)



Conference

11

2. Scenario Building


What has been your experience with Evidence-based practices, programs, or models?


Conference

12

Evidence based practice environment

• Context: increased accountability in resource strained times; lack of consistent outcomes; program clutter; “fee for service” volume

• Benefits: increased confidence in outcomes, guaranteed results and efficiency for specific populations, funder commitment

• Limitations: cost, limited external validity, fidelity and scalability



Conference

13

Programs / interventions can be placed along a “proof of effectiveness” continuumHow confident are we that a program will improve outcomes?


Not much

confidence

Best Practices

“We’ve done it and we like it” Promising Practices

“We really think this will work… but we need time to prove it”

Research-based“This program is based on sound theory informed by research”

Evidence-based“This program has been rigorously evaluated and shown to work”

Very much

confidence

Bumbarger, B. K. (2009). Promoting the Use of Evidence-based Prevention: Application in the Real World. Available online at https://breeze.psu.edu/p22215525.


Conference

14

Evidence-based Programming and InterventionsThe Evidence-based Prevention and Intervention Support Center (EPISCenter at Penn State University)


The EPISCenter supports the dissemination, quality implementation, sustainability, and impact assessment of proven-effective prevention and intervention programs, and conducts original research to advance the science and practice of evidence-based programs.

Evidence-based Programs Supported

[From Pennsylvania’s Juvenile Justice System Enhancement Strategy (JJSES) 12/12/11]


Conference

15

What is Value?What is Value?

15

Wesley Spectrum Services (c)


Conference

16

3. Case Study: WSIH survived and thrived



Conference

17

Wesley Spectrum In Home: History of Tracking Outcomes



Conference

18

But… Why Good Outcomes?

• Easier population? OR• Something we are DOING (our

interventions/program model)?• In other words: TRACKING

OUTCOMES IS NOT ENOUGH

18


Conference

19

We Know a Lot About What Works!

• Meta-analyses on thousands of studies• Many programs ARE using research-

based practices• They just have not MEASURED and

TRACKED their work!!!• Which lead WISH down a different path:

Fidelity Management

19


Conference

20

Alternative to proprietary EBP

Fidelity Management: Five Key Steps

1. Define program key ingredients that are research-supported

2. Develop Fidelity metrics for use during supervision

3. Develop/refine outcome measures and incorporate into tracking tools

4. Validate your model (relate Fidelity measures to Outcome measures)

5. Use in CQI



Conference

21

Step #1: Define the Program

• Logic Model • Key program components

– Specific population– Staff selection and training– What the staff does– How they are supervised– Expected outputs and outcomes– Suggested measures

• Example of Logic Model• Developing from existing performance

data



Conference

22

Verify Program Elements with Existing Research

• Literature review or “common elements” (Lipsey or Chorpita)

• Eliminate from tracking anything that doesn’t have existing research support

• Examples



Conference

23

Step #2 and #3: Develop and Track Model Fidelity and Outcomes

•Therapist and Supervisor Checklist (Intake, Monthly, Discharge) Scores:

– Who we are serving (population assessments)

– What are we doing (outputs related to key activities, intensity of services)

– How did we do (client outcomes)

•Consumer Satisfaction Survey Scores– Items relate to key program activities;

additional output measures



Conference

24

Step #4: Relate Fidelity and Outcomes

• High fidelity to the model leads to the best outcomes (ideal results and actual sample data)



Conference

25

Higher Model Fidelity Improved Child Well-Being: Strong Relationship between Model Fidelity and Outcomes



Conference

26

Step #5: Build in CQI: Model Fidelity Comparison of Two Sites



Conference

27

4. Workshop Case

• Why your programs can and should do Fidelity Management– Marketing of what your program DOES

THAT MATTERS (i.e., is supported by existing research)

– Increased likelihood that you’ll produce positive outcomes

– Efficiency (unit cost) and effectiveness (related to outcomes)



Conference

28

Workshop Case: participant program

• Try-out the five steps of Fidelity Management



Conference

29

More Reasons to Locally Develop EBP



Conference

30

Yet More Reasons...

Mark Lipsey, “Evidence-based Practice More than One Approach.” MST and FFT (two brand-names) show positive results, the dark boxes, but even “generic” interventions showed better results.


From http://cjjr.georgetown.edu/pdfs/ebp/ebppaper.pdf


Conference

31

Yet more reasons…cost-benefit



Conference

32

Cost Comparison MST vs. WSIH

• Based on Washington State Institute Public Policy work

• Main benefit of Youth Intervention programs is reduction in crime

• Compared Allegheny youth who completed either MST or WSIH treatment

32Wesley Spectrum Services (c)


Conference

3333Wesley Spectrum Services (c)

Program Benefit Completed cases

Average hrs/clien

t

Cost/unit service

Cost per completed

case

Return on investment

(per $1 invested)

MST ( ) $14,996 per

participant

22 or 49% of all

served (12 months 2010)

55.5 hrs/client

$138.96/hr

$8,531.474 0.76

Wesley Spectrum In Home

$14,996 per

participant

69 or 78% of all served (six months 2011)

48.16 hrs/client

$51.94/hr $1,936.78 6.74


Conference

34

Brand-name EBP vs. Local-developed EBP

Proprietary EBP•$millions for research and evaluation•Many decades research/development•Highly prescribed•High level accountability•High level marketability•Low adaptability•High effort•Scalability (and ongoing cost)•Community connections have to be built (and sometimes don’t happen)


Locally Developed EBP• Low-cost research and

evaluation in short-time• Moderate level program

requirements• Lower program cost• Very good level

accountability• Need to develop marketing

zeitgeist• Greater utility across

populations• Greater access for consumers• Embedded in CQI• Tools for incorporating new

practices• Staff commitment


Conference

35

Marketing

• Brand-name EBPs leg-up• How-to for locally developed (using

Workshop Case from participants)



Conference

36

Key Conclusions

• Evidence based models pose limitations that Fidelity Management does not

• Fidelity Management is replicable so other programs could do the same

• The process gives programs the supervision and monitoring tools for continuous improvement AND for making the case of value to stakeholders

• Gives CEOs a tool for aligning unit cost and outcomes -- sustainability



Conference

37

How’d We Do?

• Benefits and limitations of evidence-based models

• Alternative to packaged models (comparable level of accountability for less cost)

• How to use in your organization/program

• How to use “Fidelity Management” to create funding case



Conference

38

Don’t Forget…


Documents

PCCYFS 2012 Annual Spring Conference 02/2/12 Fidelity Management: Increasing Provider Negotiating Strength in the Environment of Evidence-based Programs