Implementation for the Massesimplementationscience.web.unc.edu/files/2015/01/...practice 80 - 95% use in practice (coaching) Fidelity of use in practice 29% EBP outcomes if low fidelity

Dean Fixsen, Karen Blase,

Melissa Van Dyke, Allison Metz National Implementation Research Network

FPG Child Development Institute

University of North Carolina at Chapel Hill

Implementation for the Masses

UNC Implementation Science

Student Group

!  Past – Before 2005

!  Present – 2005-2015

!  Future – Starts today

Science to Service Gap Implementation Science

!  Implement = Use

!  Implementation Science = the study of factors that influence the full and effective use of innovations in practice !   The goal is not to answer factual questions

about what is, but rather to produce evidence for what is required (invent the future)

Science to Service Gap Implementation Defined

Eisenhower: National System of Interstate and Defense Highways (1956)

•  Goal of 21,185 miles (50%) reached in 1966

Kennedy: Decision to send astronauts to the moon and return them safely (1961)

•  Goal reached in 1969

Johnson: Great Society Programs (1964) •  Goals not reached yet in 2014

Implementation Science

Human services involve interaction-based sciences and services

•  Inherently more complex than atom-based sciences – e.g., atom-based ingredients don’t

change from one engineer to the next or change overnight or decide to quit


National Assessment of Education Progress (NAEP)

Churning Around a Mediocre Mean

Original and Revised Assessment Formats

Federal SPENDING on K-12 Education and NAEP READING Scores (Age 9)

It’s not (just) the money or top-down support!

Convergence in the new millennium 1.  Innovation science 2.  Implementation science 3.  Improvement science 4.  Complexity theory


Innovation science •  Rigor and relevance of data re

WHAT to do –  “Evidence-based” programs and

innovations


A key lesson from the Great Society is that the effects of social programs in practice hover near zero, a devastating discovery for social reformers – Rigorous data supporting effective

innovations are necessary but not sufficient


Rossi and Wright (1984)

Implementation Science (get started) and Improvement Science (get better)

•  HOW to assure the full, effective, and consistent use of innovations in practice

•  HOW to resolve issues of fit and function (innovation, delivery, organizations, systems) – never ends!


Complexity Theory •  Interactions among moving parts •  Emergent adaptation •  Plan for the unexpected •  Large scale change in real time


•  Summaries of literature in any domain including agriculture, business, child welfare, complexity theory, computer science, engineering, health, juvenile justice, manufacturing, medicine, mental health, nursing, social services, and others

•  Summaries of best practices by experts doing implementation work successfully

•  Synthesis of over 40 implementation and evaluation frameworks

E Pluribus Unum

Socially Significant Outcomes

Effective Innovations

Effective Implementation

Enabling Contexts

Convergence

!  Usable Innovations !  Implementation Stages !  Implementation Drivers !  Improvement Cycles !  Implementation Teams !  Enabling Change

Convergence: Active Implementation Frameworks

Haphazard Implementation Informed

Training Benefits 5 - 15% use in practice

80 - 95% use in practice (coaching)

Fidelity of use in practice

29% EBP outcomes if low fidelity use

81% EBP outcomes if use meets criteria

Implementation Team

18% fidelity with no/poor Drivers

83% fidelity if Drivers at criteria

Training + Coaching + Fidelity

22% staff retained 3+ yrs.

58% staff retained 3+ yrs.

Competency + Organization Drivers

17% sustain 6+ yrs.

84% sustain 6+ yrs.

Implementation Benefits

8 X More

3 X More

4 X More

3 X More

5 X More

Stacey (1996)

Zone of Complexity

•  Letting it happen – Recipients are accountable

•  Helping it happen – Recipients are accountable

•  Making it happen – Purposeful use of implementation practice and

science –  Implementation Teams are accountable Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004);

Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)


Approach Usage Success Months Intervention/Facilitation 8% 87% 14

Participation/Internal Team 18% 73% 16

Persuasion 37% 47% 21 Edict 37% 35% 15

Nutt (2001)

Implementation Science Approaches to implementation of innovations in 376 organizations (public, private, profit, non-profit)

Approach Usage Success Months Intervention/Facilitation 8% 87% 14

Participation/Internal Team 18% 73% 16

Persuasion 37% 47% 21 Edict 37% 35% 15

Nutt (2001)

Implementation Science Approaches to implementation of innovations in 376 organizations (public, private, profit, non-profit)

Tell and Sell

Lead and Support

!  Minimum of three people (four or more preferred) with expertise in: !  Operationalizing innovations !  Using implementation methods effectively !  Helping organizations change

!  Tolerate turnover; teams are sustainable even when the players come and go (Higgins, Weiner, & Young, 2012; Klest & Patras, 2011)

Implementation Teams Teams

© Fixsen & Blase, 2009

80%

Implementation Team

Prepare Organizations

Prepare Practitioners and Staff

Work with Researchers

Assure Implementation

Prepare Regions Assure Intended Benefits

Create Readiness

Patients and Stakeholders 20%

Prochaska, Prochaska, & Levesque (2001)


Organization Supports Management (leadership, policy)

Administration (HR, structure) Supervision (nature, content)

Practitioner/Staff Competence

State/Community Supports Regional Authority Supports

Impl

emen

tatio

n Te

am

Simultaneous, Multi-Level Interventions

Federal and National Supports


Implementation and Innovations !  Rapid cycle (PDSA) problem solving

!  Shewhart (1931); Deming (1986)

!  Usability testing !  Rubin (1994); Nielsen (2000)

Improvement Cycles Teams

Improvement Cycles

Nielsen (2000)

Teams

Usability Testing

•  A planned series of tests of an innovation or of implementation processes.

•  Use proactively to test the feasibility and impact of a new way of work prior to rolling out the innovation or implementation processes more broadly

Improvement Cycles

More is learned from 4 cycles with 5 participants each than from 1 pilot test with 20 participants

Teams

Expert Impl. Team NO Impl. Team

Effective

Effective use of Implementation Science & Practice

IMPLEMENTATION

INN

OVA

TIO

N

80%, 3 Yrs 14%, 17 Yrs

Balas & Boren, 2000

Green, 2008

Fixsen, Blase, Timbers, & Wolf, 2001

Saldana & Chamberlain, 2012

Letting it Happen Helping it Happen


"All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get.” …R. Spencer Darling

Human Services are not improving as fast as problems are changing.

The Nature of Systems

•  Innovative practices do not fare well in existing organizational structures and systems (crushed by established routines)

•  Organizational and system changes are essential to successful use of innovations – Expect it – Plan for it

The Nature of Systems

Existing System

Effective Innovations Are Changed to Fit The System

Existing System Is Changed To Support The Effectiveness Of

The Innovation

Effective Innovation

Enabling Contexts

•  Cannot change everything at once (too big; too complex; too many of them and too few of us)

•  Cannot stop and re-tool (have to create the new in the midst of the existing)

•  Cannot know what to do at every step (we will know it when we get there)

•  Many outcomes are not predictable (who knew!?)

Creating Enabling Contexts

Implementation Team

Policy Makers &

Administrators

Practitioners Innovations

Children/Families

“Ext

erna

l” S

yste

m C

hang

e Su

ppor

t

Prac

tice

Polic

y C

omm

unic

atio

n Lo

op

Policy Enabled Practice

♦Look for Faulty Assumptions & Errors; ♦Make Needed Changes; ♦Invite System to Respond


“The fault cannot lie in

the part responsible

for the repair.” Ashby (1956)

Implementation Team

Policy Makers &

Administrators


Children/Families

Syst

em

Cha

nge

Adaptive Challenges • Duplication •  Fragmentation • Hiring criteria •  Salaries • Credentialing •  Licensing •  Time/ scheduling • Union contracts • RFP methods •  Federal/ State laws

“Ext

erna

l” S

yste

m C

hang

e Su

ppor

t

Prac

tice

Polic

y C

omm

unic

atio

n Lo

op






Local Agency LIT

Executive Management

Team


Beneficiaries

“Ext

erna

l” S

yste

m C

hang

e Su

ppor

t

Prac

tice-

Polic

y C

omm

unic

atio

n Lo

op


Implementation Infrastructure




Municipality MIT

Regional Entity RIT

State and Regional Capacity Assessment

State Capacity Assessment: Fixsen, Duda, Blase and Horner, 2009

•  Stop wasting time and money on things that don’t work (and never have!) – Sugai’s Law: For every new initiative, stop

two current ones that have poor outcomes – De-scale; Avoid layering and fragmentation

•  Set aside 20% of funds for implementation •  Require quarterly reports of fidelity data

System Reinvention

A fundamental truth: •  People cannot benefit from

innovations they do not experience


Effective Innovations


Enabling Contexts

Socially Significant Outcomes

Formula for Success

!  Usable Innovations !  Implementation Stages !  Implementation Drivers !  Improvement Cycles !  Implementation Teams !  Enabling Change

Active Implementation Frameworks

National Assessment of Education Progress (NAEP)

Leverage Visible Improvements!

Original and Revised Assessment Formats

Educationally Significant Outcomes

Effective Interventions


Enabling Contexts

!  There is sufficient information to support a rapidly evolving science of implementation

!  On the verge of creating a virtuous circle where evidence-based implementation practice improves implementation science that improves implementation practice …

Science to Service Gap Implementation Science

GIC May 26-28, 2015 Dublin, Ireland

www.globalimplementation.org

Allison Metz [email protected]

Melissa Van Dyke [email protected]

Frank Porter Graham Child Development Institute University of North Carolina

Chapel Hill, NC

http://nirn.fpg.unc.edu/ www.scalingup.org


HTTP://NIRN.FPG.UNC.EDU

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Implementation Research: A Synthesis of the Literature

Get Connected!

www.scalingup.org

SISEP @SISEPcenter

For more on Implementation Science http://nirn.fpg.unc.edu


• Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology, 77(2), 270-280. doi: 10.1037/a0013223 • Ashby, W. R. (1956). An introduction to cybernetics. London: Chapman & Hall. • Balas, E. A., & Boren, S. A. (2000). Managing clinical knowledge for health care improvement. In J. Bemmel & A. T. McCray (Eds.), Yearbook of Medical Informatics 2000: Patient-Centered Systems. (pp. 65-70). Stuttgart, Germany: Schattauer Verlagsgesellschaft. • Deming, W. E. (1986). Out of the crisis. Cambridge, MA: MIT Press. • Elmore, R. (2002). Bridging the gap between standards and achievement: The imperative for professional development in education. Washington, DC: The Albert Shanker Institute. • Fixsen, D. L., Blase, K. A., Timbers, G. D., & Wolf, M. M. (2001). In search of program implementation: 792 replications of the Teaching-Family Model. In G. A. Bernfeld, D. P. Farrington & A. W. Leschied (Eds.), Offender rehabilitation in practice: Implementing and evaluating effective programs (pp. 149-166). London: Wiley. • Fixsen, D. L., Blase, K., Duda, M., Naoom, S., & Van Dyke, M. (2010). Implementation of evidence-based treatments for children and adolescents: Research findings and their implications for the future. In J. Weisz & A. Kazdin (Eds.), Implementation and dissemination: Extending treatments to new populations and new settings (2nd ed., pp. 435-450). New York: Guilford Press. • Green, L. W. (2008). Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Family Practice, 25, 20-24. • Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581-629. • Grigg, W. S., Daane, M. C., Jin, Y., & Campbell, J. R. (2003). The nation's report card: Reading 2002. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

• Higgins, M., Weiner, J., & Young, L. (2012). Implementation teams: A new lever for organizational change. Journal of Organizational Behavior. doi:10.1002/job.1773 • Joyce, B., & Showers, B. (2002). Student achievement through staff development (3rd ed.). Alexandria, VA: Association for Supervision and Curriculum Development. • Kutner, M., Greenberg, E., Jin, Y., Boyle, B., Hsu, Y., & Dunleavy, E. (2007). Literacy in everyday life: Results from the 2003 National Assessment of Adult Literacy (NCES 2007–480). Washington, DC: U.S. Department of Education, National Center for Education Statistics. • Linehan, M. M., Dimeff, L. A., Reynolds, S. K., Comtois, K., Welch, S. S., Heagerty, P., & Kivlahan, D. R. (2002). Dialectical Behavior Therapy Versus Comprehensive Validation Therapy Plus 12-Step for the Treatment of Opioid Dependent Women Meeting Criteria for Borderline Personality Disorder. Drug and Alcohol Dependence, 67(1). doi: 10.1016/S0376-8716(02)00011-X • Kivlahan, D. R. (2002). Dialectical Behavior Therapy Versus Comprehensive Validation Therapy Plus 12-Step for the Treatment of Opioid Dependent Women Meeting Criteria for Borderline Personality Disorder. Drug and Alcohol Dependence, 67(1). doi: 10.1016/S0376-8716(02)00011-X • Marzano, R., Waters, T., & McNulty, B. (2005). School leadership that works: From research to results. Alexandria, VA: Association for Supervision and Curriculum Development (ASCD). • Metz, A., Bartley, L., Ball, H., Wilson, D., Naoom, S., & Redmond, P. (2014). Active Implementation Frameworks (AIF) for successful service delivery: Catawba County child wellbeing project. Research on Social Work Practice, 1-8. doi: 10.1177/1049731514543667 • Morgan, G., & Ramirez, R. (1983). Action learning: A holographic metaphor for guiding social change. Human Relations, 37(1), 1-28. • Nielsen, J. (2000). Why you only need to test with 5 users. Retrieved April 22, 2007, from http://www.useit.com/alertbox/20000319.html.

• Nutt, P. (2002). Why decisions fail: Avoiding the blunders and traps that lead to debacles. San Francisco: Berrett-Koehler Publishers Inc. • Nutt, P. C. (2001). Contingency approaches applied to the implementation of strategic decisions. International Journal of Business, 6(1), 53-84. • Pressman, J. L., & Wildavsky, A. (1973). Implementation: How Great Expectations in Washington are Dashed in Oakland. Berkeley, CA: University of California Press. • Rhim, L. M., Kowal, J. M., Hassel, B. C., & Hassel, E. A. (2007). School turnarounds: A review of the cross-sector evidence on dramatic organizational improvement. Lincoln, IL: Public Impact, Academic Development Institute. • Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155-169. • Saetren, H. (2005). Facts and myths about research on public policy implementation: Out-of-fashion, allegedly dead, but still very much alive and relevant. The Policy Studies Journal, 33(4), 559-582. • Saldana, L., Chamberlain, P., Wang, W., & Brown, H. C. (2012). Predicting program start-up using the stages of implementation measure. Administration and Policy in Mental Health, 39(6), 419-425. doi: 10.1007/s10488-011-0363-y • Schofield, J. (2004). A model of learned implementation. Public Administration, 82(2), 283-308. • Stacey, R. D. (1996). Complexity and creativity in organizations. San Francisco: Berrett-Koehler Publishers. • Strouse, M. C., Carroll-Hernandez, T. A., Sherman, J. A., & Sheldon, J. B. (2004). Turning Over Turnover: The Evaluation of a Staff Scheduling System in a Community-Based Program for Adults with Developmental Disabilities. Journal of Organizational Behavior Management, 23(2), 45-63. • Sundell, K., Soydan, H., Tengvald, K., & Anttila, S. (2009). From opinion-based to evidence-based social work: The Swedish case. Research on Social Work Practice. doi: 10.1177/1049731509347887 • Vernez, G., Karam, R., Mariano, L. T., & DeMartini, C. (2006). Evaluating comprehensive school reform models at scale: Focus on implementation. Santa Monica, CA: RAND Corporation.

©Copyright Dean Fixsen and Karen Blase

This content is licensed under Creative Commons license CC BY-NC-ND, Attribution-NonCommercial-NoDerivs. You are free to share, copy, distribute and transmit the work under the following conditions: Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work); Noncommercial — You may not use this work for commercial purposes; No Derivative Works — You may not alter or transform this work. Any of the above conditions can be waived if you get permission from the copyright holder.

http://creativecommons.org/licenses/by-nc-nd/3.0

Documents

Implementation for the Massesimplementationscience.web.unc.edu/files/2015/01/...practice 80 - 95% use in practice (coaching) Fidelity of use in practice 29% EBP outcomes if low fidelity