Pittsburgh Nonprofit Summit - Measuring Change - Supplemental Slides

Measuring ChangeMoving from Outputs to Outcomes to Impact

10:00am – 11:15amF ilit t Li K Ri h d Ki M ll F d tiFacilitator: Lisa Kuzma, Richard King Mellon Foundation

Speakers: Mike Bangser, MDRC / Junlei Li, Office of Child Development / Art Maxwell and Pam Meadowcroft, Meadowcroft

& Associates / Tim Weidemann, Rondout Consulting & Associates / Tim Weidemann, Rondout Consulting

Converting YOUR Program to a V lid EBP Fid li MValid EBP: Fidelity Management

Meadowcroft & Associates and Wesley Spectrum ServicesFor more information, please contact Pamela Meadowcroft, Ph.D. at [email protected] 412.683.7275

5/20/2011Meadowcroft&Associates and

Wesley Spectrum (c) 2

Evolution of what payers want AND ways id i dproviders improved programs

Old days: Then Came:OUTCOMES

Old days:PROCESS

N F t EBP+ Then EBP:Near Future: EBP+ continuous improvement (assess-plan-monitor-i )

Then EBP:process+outcomes



improve)

What is the Difference?Levels of confidence in the outcomes

Evidence-based practice (rigorously evaluated; most often proven via RCT)E id i f d ti / h b dEvidence-informed practice/research-based (existing research support)Best Practices (expert opinion)Best Practices (expert opinion)Promising practice (acceptable treatments, anecdotal) )InnovationsIntuition, “the way it’s always done”



, y y

WE KNOW A LOT ABOUT WHAT WORKS!!!!!

Meta-analyses on thousands of studiesMany programs ARE using research-based practicesThey just haven’t MEASURED and TRACKED their work!!!



Mark Lipsey, “Evidence-based Practice More than One Approach.” MST and FFT (two brand-names) show positive results the dark boxes but evenand FFT (two brand-names) show positive results, the dark boxes, but even

“generic” interventions showed better results.



From http://cjjr.georgetown.edu/pdfs/ebp/ebppaper.pdf

Wesley Spectrum In Home: History of Tracking Outcomes

91% 93% 92% 93%89%

100%

90% 90%

100%100%

Clients Discharged to Same or Less Restrictive Environments

82% 86%86% 89% 90% 90%

60%

70%

80%

90%

30%

40%

50%

60%

Allegheny

Westmoreland

0%

10%

20%

FY 2009 Qtr 1 FY 2009 Qtr 2 FY 2009 Qtr 3 FY 2009 Qtr 4 FY 2010 Qtr 1 FY 2010 Qtr 2


Wesley Spectrum (c) 75/20/2011 7

FY 2009 Qtr. 1 FY 2009 Qtr. 2 FY 2009 Qtr. 3 FY 2009 Qtr. 4 FY 2010 Qtr. 1 FY 2010 Qtr. 2

B t Wh G d O t ?But… Why Good Outcomes?

Easier population? ORSomething we are DOING (our

/ d l)interventions/program model)?In other words: TRACKING OUTCOMES IS NOT ENOUGHOUTCOMES IS NOT ENOUGH



Id l R ltIdeal Results

High “fidelity” to the model leads to High fidelity to the model leads to the best outcomes

45 00

30.00

35.00

40.00

45.00

15.00

20.00

25.00 Model Adherence Scores

Change in Outcomes Scores

0.00

5.00

10.00

Family 1 Family 2 Family 3 Family 4



Family 1 Family 2 Family 3 Family 4

Steps for Building a local EBP: Fidelity Management

fDefine the programDevelop and Track Model Fidelity ( )(outputs)Develop and Monitor OutcomesValidate the Locally-Developed Program Model (link outputs to

)outcomes)Build-in CQI



T l f A i M d l Fid litTools for Assuring Model Fidelity

Th i d S i Ch kli Therapist and Supervisor Checklists (Intake, Monthly, Discharge) include:

Who we are serving (population assessments)Who we are serving (population assessments)What are we doing (outputs related to key activities, intensity of services)How’d we do (client outcomes)How’d we do (client outcomes)

Consumer Satisfaction SurveysItems relate to key program activities; e s e a e o ey p og a ac es;additional output measures

Embed in CQI (participating in QII)



Model Adherence Scores for Wesley Spectrum In Home Model Elements: Two Sites Compared to Be Used for CQI4

3.754

3.75 3.75 3.753 6

4

3.5 3.5 3.5

3

3.53.25

3.5 3.6

2 5

3

3.5

1.5

2

2.5

Westmoreland n=4

0.5

1Allegheny n=10

0



Higher Model Fidelity Improved Child Well-Being: Strong Relationship g g pbetween Outputs and Outcomes

NCFAS-G Child Well-Being Change Scores

0.6

0.7

NCFAS-G Child Well-Being Change Scores

0 3

0.4

0.5

Lower Adherence Group

0.1

0.2

0.3Higher Adherence Group

0Overall Child Well-Being

Child's Behavior

School Performance

Child's Relationship with Parents

Motivation to Maintain Family



Overall Child Well-Being and Child's Behavior significant at p<.05 level

Wesley Spectrum In Home VS Multi-Systemic Therapy (name-brand EBP) with y py ( )Comparable Populations

Percent of Consumers Who Achieved Outcomes

0.92

0.96

0.8

0.84

0.88

MST

WSIH-All Cases

0.72

0.76

0.8

WSIH-Higher Adherence Cases

0.68Completed

Therapy / Planned Discharge

No Arrests / Planned Discharge

Child in Home Child in School



Child In School significant for MST vs. WSIH Higher (p<.05).

Brand-name EBP vs Local-developed EBP

Purchased EBP Model$millions for research and evaluation

Home Grown Model Building ProcessLow-cost research and evaluation in short timeand evaluation

Many decades research/developmentHighly prescribed

evaluation in short-timeModerate level program requirementsLower program cost

Low adaptabilityHigh effortOngoing high program C t (

Greater utility across populationsEmbedded in CQITools for incorporating Cost (e.g.,

recertification)Tools for incorporating new practicesStaff commitment



K C l iKey Conclusions

E id b d d l li it ti th tEvidence based models pose limitations that our model building process does not

Our model building process is replicable so other programs could do the same

The process gives programs supervision and monitoring tools for continuous improvement ANDmonitoring tools for continuous improvement AND for making the case of value to stakeholders



Documents

Pittsburgh Nonprofit Summit - Measuring Change - Supplemental Slides