An advanced training and conversation hosted by the ... · • Flexible components-based treatment –Clinicians can pick, choose, and tailor elements for each individual’s clients

An advanced training and conversation hosted by

the California Evidence-Based Clearinghouse and

members of Florida’s practice community

September 9, 2015 2015 Child Protection Summit

http://www.bing.com/images/search?q=university+of+central+florida&FORM=IARRTH&ufn=university+of+central+florida&stid=3539fded-bde5-56f7-e5f7-bf47ecbda4b1&cbn=EntityAnswer&cbi=0&FORM=IARRTH

DAVID FAIRBANKS, Ph.D. Deputy Secretary

Florida Department of Children and Families 1317 Winewood Boulevard, Bldg.

Tallahassee, FL 32399 850-410-5283

[email protected]

mailto:[email protected]

Why we all have a stake in this

topic

• Florida’s practice model changes – Are designed to improve safety, permanency and well-being

outcomes

– Are the focus of a five year rigorous evaluation design funded by Children’s Bureau to determine evidence base (Washoe Co., NV)

• Broadening use of EBPs is a requirement in Florida’s Title IV-E waiver

• Waiver evaluation conducted by FMHI must include assessment of progress with implementation of EBPs

• Upcoming federal Child & Family Services Review (CFSR)

Improving Outcomes

Training Part 1:

The Fundamentals of EBPs

that Florida Leaders Should

Consider

Training Part 2:

Strategic Leadership and

System Change

CHARLES WILSON, MSSW Senior Director

Chadwick Center for Children and Families and Sam and Rose Stein Endowed Chair in Child Protection,

Rady Children's Hospital- San Diego Director or co-director California Evidenced Based Clearinghouse for

Child Welfare, Chadwick Trauma Informed Systems Project, and California Screening, Assessment and Treatment Initiative.

Co-Director of the Centers of Developmental and Behavioral Sciences, Rady Children’s Hospital-San Diego

CEBC’s Definition of EBP

for Child Welfare

& Consistent with Family Client

Values

Best Clinical Experience

Best Research Evidence

EBP

[Institute of Medicine (IOM), 2001]

The National Registry of Effective

Programs and Practices

www.nrepp.samhsa.gov

http://www.nrepp.samhsa.gov/





The Blueprints for Youth Development www.blueprintsprograms.com

What Works (in education) www.ies.ed.gov/ncee/wwc

http://www.ies.ed.gov/ncee/wwc/

What works (Child Welfare) www.cebc4cw.org

Scientific Rating Scale [**Based on a Continuum**]

NR

Number of Programs by Rating

Intergenerational Trauma

Kimberly Renk, Ph.D., Associate Professor

Director, Clinical Psychology Program Director, Understanding Children and Families Laboratory

Department of Psychology University of Central Florida

4000 Central Florida Boulevard Psychology Building (99), Room 353

Orlando, FL 32816 Phone: 407-823-2218/Fax: 407-823-5862

http://psychology.cos.ucf.edu/people/renk-kimberly/ http://understandingchildren.cos.ucf.edu


http://psychology.cos.ucf.edu/people/renk-kimberly/



http://understandingchildren.cos.ucf.edu/

Trauma Can Happen for Both Parents and Children!!

• Intergenerational trauma may be present in about one quarter of cases where child maltreatment is present

• Domestic violence and child maltreatment co-occur

frequently (American Medical Association, 1995; American Psychological Association,

1996).

40% median co-occurrence rate (Appel & Holden, 1998).

Young children at elevated risk (Fantuzzo et al., 1997; Slep &

O’Leary, 2005).

Same risk factors (Bedi & Goaddard, 2007).


The Adverse Childhood Experiences (ACE) Study

acestudy.org

Abuse by Category Prevalence of positive responses

Psychological (by parent) 11%

Physical (by parent) 11%

Sexual (by anyone) 22%

Household Dysfunction by Category

Substance Abuse 26%

Mental Illness 19%

Mother Treated Violently 13%

Imprisoned Household Member 3%


Cumulative ACEs Increase the Risk of Negative Outcomes

0

2

4

6

8

10

12

14

0 1 2 3 4+

AttemptedsuicideInjected druguseAlcohol problem

Illicit drug use

Depressed

Chronicbronchitis50+ sexpartnersSTDs

Poor Self-ratedhealthCurrent Smoker

Cancer

Severe obesity


Three-Headed Monster?

Violence

Depression Substance Abuse


PTSD Can Follow Such Experiences for Both Parents and Children…

• 31% to 84.4% of women who experienced DV met PTSD criteria (Golding, 1999)

• 56% of children met criteria for PTSD after exposure to DV (Lehmann, 1997)

• PTSD in mothers affects their parenting behaviors (Ammerman et al., 2012; Schechter et al., 2010)

• Parenting behaviors can affect PTSD symptoms in children (Gewirtz et al., 2012)


Maternal

Depression

Parenting

Behavior

Parent-Child

Relationship Family/Marital

Functioning

Attachment-Emotion Regulation

Child

Development

Cummings (1994). Journal of Child Psychology and Psychiatry 35, 73-112.

Family Violence Substance Abuse


Symptom Correlates for Trauma

in Young Children (De Young, Kenardy, & Cobham, 2011; Finkelhor et al., 2005; Lieberman & Knorr, 2007;

Scheeringa & Zeanah, 2008; Scheeringa et al., 2003)

• Posttraumatic stress disorder

• Emotional difficulties, such as anxiety and depression

• Behavioral difficulties, such as ADHD and Oppositional Defiant Disorder

• Attachment difficulties with caregivers and interaction difficulties with family members.

• Difficulties achieving developmental tasks

• Difficulties coping and with frustration tolerance

• Comorbid socioecological conditions

(e.g., poverty) may be likely.


Making Good Choices About Which Treatment(s) Should Be Pursued…

• So, do you go with a parent-involved or parenting program?

The Circle of Security (Cooper, Hoffman, Marvin, & Powell, 1999)

Child-Parent Psychotherapy (Lieberman & Van Horn, 2008)

• Or, do you go with an individual treatment for either the

parent or the child or both?

Trauma-Focused Cognitive-Behavioral Therapy (Cohen,

Mannarino, & Deblinger, 2006).

Cognitive Processing Therapy (Resick & Schnicke, 1993


Making Good Choices About Which Treatment(s) Should Be Pursued…

• Certainly prevention efforts are warranted,

but treatments must address the sequelae

of these experiences in our youngest

children (McKinney et al., 2006; Renk et al., 2002; Renk et al., 2008).

• No single treatment approach will be

applicable to all children (Cohen, Berliner, & Mannarino,

2000) or all families.


Trends • Dramatic increase in number of solid research

studies on interventions relevant to child welfare

• Dramatic increase in use of EBPs over the past decade

• Funders are requiring the use of EBPs

• Let the buyer beware-Evidence Based Practice as a marketing term

Challenges

• Workforce issues

– Match of skill set, compatibility, culture and climate

• Training staff in multiple models

– How many do you need?

• Staff turnover and retraining

• Fidelity and drift over time

Controversies

• Practice Based Evidence

• Common elements

• Application to Different Cultural Groups

Early

Adopters

Early

Majority

Late

Majority

Traditionalists

Innovators

The Rogers Curve

How Things Traditionally Change in Child Welfare

https://www.youtube.com/watch?v=fW8amMCVAJQ



Levels of Implementation

• Paper

• Process

• Performance

Real organic organizational change at the cultural level

http://nirn.fmhi.usf.edu/resources/detail.cfm?resourceID=31

Stages of Change- Motivational Interviewing

• Pre-contemplation Stage

• Contemplation Stage

• Preparation Stage

• Action and Sustainment

EPIS: Stages of Implementation

Lessons Learned

• Importance of Assessment

• How much evidence is enough?

• Referral Pathway

• Power of a change agent

• Importance of sustained effect

Implementation Tools and Resources

Selection and Implementation Guide

What research informed programs are available in Florida?

Protect me Comfort me Delight in me Organize my feelings

Circle of Security Parent Attending to the Child’s Needs

I need you to

Support My Exploration

Welcome My Coming To You

I need you to

Watch over me Help me Enjoy with me • Delight in me

I need you to

I need you to

© Cooper, Hoffman, Marvin, & Powell (1999)


Circle of Security Parenting Program (Marvin, Cooper, Hoffman, & Powell, 2002)

1. Welcome to the Circle of Security: Introduction to program.

2. Exploring Our Children’s Needs All the Way Around the

Circle: Increase parents’ observation and inferential skills.

3. “Being With” on the Circle: Build a parent-child relationship

where children’s feelings can be shared and a secure

attachment can be built.

4. Being With Infants on the Circle: Teach attunement to

infants’ shifts in attention and emotion.

5. The Path to Security: Steps to promoting security.

6. Exploring Our Struggles: Be “bigger, stronger, wiser, and

kind”.

7. Rupture and Repair in Relationships: Learn that children

sometimes act out in an effort to manage their needs.

8. Summary and Celebration.


Child-Parent Psychotherapy (Lieberman & Van Horn, 2005)

• A relationship-based intervention grounded in psychoanalytic, attachment, and trauma theory.

– It includes social learning and CBT interventions for change.

• Core premise includes the idea that young children rely on their parents for protection and safety.

– Trauma can alter the perception that parents can provide these things.

• Therapists support the parent-child dyad with play, words, and interactions to express and respond to emotional needs, breaks the taboo of silence about the trauma, that modulates unmanageable traumatic stress, and that restores trust.

• Scientific Rating CEBC: 2


Child-Parent Psychotherapy (CPP)

Target population:

Children from birth through age five and their caregivers.

Treatment length and frequency:

60-90 minute weekly sessions for 52 weeks

Goals:

Strengthen the parent-child relationship, increase safety and attachment and decrease behavior problems and mental health symptoms.


Trauma-Focused Cognitive-Behavioral Therapy (Cohen, Mannarino, & Deblinger, 2006)

• Cognitive and behavioral techniques.

• Flexible components-based treatment – Clinicians can pick, choose, and tailor elements for

each individual’s clients symptoms and specific trauma.

• Effective for children 7- to 16-years of age (Webb et al., 2014).

• Also effective for children 3- to 6- years of age (Scheeringa et al., 2011).

• Diverse traumas: Bullying, natural disasters, sexual abuse, physical abuse, domestic violence, community violence, traumatic loss).



Preschool PTSD Treatment (PPT; Scheeringa, Amaya-Jackson, & Cohen, 2010)

• TF-CBT modified and downscaled for young children (ages 3- to 6-years).

• More behavioral techniques (e.g., SUDS, habituation) compared to TF-CBT (e.g., no inclusion of cognitive coping or restructuring).

• Use of binder to keep treatment materials (drawings, worksheets).

– Children experienced decrease in PTSD symptoms, depression, oppositionality, and separation anxiety after 12-week treatment protocol (Scheeringa et al., 2011).


Cognitive Processing Therapy (CPT; Resick & Schnicke, 1993)

• Cognitive and Behavioral techniques:

– Cognitive restructuring (e.g. Identifying “Stuck Points”, Socratic questioning, Use of worksheets).

– Exposure (e.g., Written accounts, Read to self daily, Read aloud in session).

• Goal of exposure is not habituation.

• Feel, process, re-story with more adaptive thoughts.

• Decrease negative emotions associated with trauma memories.

• CPT shown to reduce symptoms in many populations.



Stephanie C. Kennedy, MSW Doctoral candidate and Legacy Fellow

College of Social Work The Florida State University

296 Champions Way University Center, Building C Tallahassee, FL 32306-2570

850-644-4751 [email protected]

mailto:[email protected]

Right intervention, right population redux

• Evaluating the effectiveness of an

established EBP with a child welfare

population

• Parent Child Interaction Therapy (PCIT)

• Empirically supported, scientific rating of 1

– supported at the highest level

• BUT, does it “work” as delivered to

child welfare involved families?

Parent Child Interaction Therapy (PCIT) Original target population: Children aged 2-7

with externalizing behavior problems

Treatment length and frequency:

60-minute weekly sessions for 14-20 weeks.

Goals: Enhance the child-caregiver

relationship, improve child compliance, and

model positive parenting and non-physical

discipline.

Mechanics: Uses in vivo coaching to train

parents to positively engage their child in

play and respond appropriately to child’s

behavior.

Phases (7-10 sessions each): Child-Directed Interaction (CDI)

Goal: Enhance the parent-child relationship

PRIDE skills

Parent-Directed Interaction (PDI) Goal: Improve child compliance

Gaps in the existing support for PCIT

• Child-level outcomes are common

• Externalizing behaviors

• Mental health symptoms

• School behavior

• Thomas and Zimmer-Gembeck (2007) review

parent-level outcomes

• Parent reports of child’s behavior

• Parent self-reported stress

• Clinic report of observed parenting

• Studies conducted on a variety of

samples – not exclusively child welfare

The real question ...

• Does PCIT reduce future abuse when implemented among physically abusive families?

• How the current project grew

• Method

– Quasi-experimental or experimental studies

– Child welfare involved or “at risk” families

– Must have in home custody of at least one child

• Parent-level outcomes

– Child abuse potential

– Parenting stress

– Physical abuse recurrence

What did we find?

• Child Abuse Potential Inventory – Non-significant, although 95% confidence intervals

suggest clinically meaningful treatment effects

• Parenting Stress Index – Significant reductions for PCIT completers (v controls)

pre- to post-intervention, although the pooled treatment effects were small (g = .35)

• Physical abuse recurrence – Significant reductions for PCIT completers (v

controls), with a medium treatment effect (g=0.52)

– Long follow-up period

– Survival analysis and hazard risk ratios

– Informal and formal reports of abuse.

What do the results suggest?

• PCIT appears to be effective at reducing future abuse, even among parent-child dyads with a history of maltreatment/at-risk for maltreatment

• Disclaimers!

– Few studies examine parent-level outcomes in general

– Long-term physical abuse recurrence based on only two studies

• Alterations to PCIT for a child welfare population

– Children aged 2-12

– “Hands off” protocol

Opportunities for collaboration

• Consider your local colleges and universities as potential partners in building empirical evidence for your programs

• Faculty and doctoral students offer statistical and research methodology expertise

• You provide a deep understanding of your clients needs and the strengths and limitations of your current programs

Training Part 2: Strategic

Leadership & System Change

• BFP Pathway to Excellence

• Volusia County Super-community

• Questions and Discussion

Dr. Patricia Nellius

Chief Executive Officer Brevard Family Partnership

2301 Eau Gallie Blvd., Suite 104 Melbourne, FL 32935

Phone: 321-752-4650 ext. 3002 Cell: 321-626-0261 Fax: 321-752-3188 www.brevardfp.org

http://www.brevardfp.org/

BFP Pathway to Excellence Initiative

Launched in 2010

• To strategically assess and build EBP capacity in the local system of care.

• Contracted with EBA and Chadwick Center to assess our network of existing promising and EBP’s to assist in the creation of a Road Map to Excellence.

Findings

• BFP has a large and varied network of providers, ranging from large, multi-site agencies to individual providers.

• Some providers reported they already deliver well recognized EBP’s.

• BFP has a strong track record of implementing innovative changes and adopting new practices to respond to the needs of their community.

Findings, continued

• There are EBP’s in place in the community that do not contract with BFP.

• Providers are very open to EBP and willing to move in that direction with assistance from BFP.

BFP Road Map to Build EBP Capacity

Expand the EBP’s already in place in Brevard.

Build evidence for promising programs already in place.

Add new EBP’s in areas of needed capacity.

Build and strengthen the BFP infrastructure.

Define Assessment and Referral Pathways.

Create Implementation Pathways.

Implement Fidelity Monitoring.

Results • BFP offered 3 grants for programs identified in

the assessment as having an evidence base to expand research and build child welfare capacity.

• BFP contracted with Evidence Based Associates (EBAs) and Chadwick Center to provide TA and consultation to the grantees.

• Continue C.A.R.E.S. research efforts.

• BFP created referral pathways to prescribe criteria.

• BFP implemented an assessment protocol using the CFARS to collect client specific baseline and outcome data from providers.

Sustainable Results: Capacity Building: Evidence Based Practice

• Child-Parent Psychotherapy-Scientific Rating of 2 and Child Welfare Relevance Rating High. –grant awarded to increase EBP’s in our SOC.

• Trauma Focused Cognitive Behavioral Therapy-Scientific Rating of 1 and Child Welfare Relevance Rating High –grant awarded to increase EBP’s in our SOC.

• Implementation of North Carolina Family Assessment.

Results: Capacity Building: EBP

• Duluth Model for Batterer’s Intervention- Scientific Rating of 3 and Child Welfare Relevance Rating of Medium

• Nurturing Parenting Curriculum - Scientific Rating of 3 and Child Welfare Relevance Rating High

• Wraparound-Scientific Rating of 3 and Child Welfare Relevance Rating High

• Implementation of Wraparound Fidelity Measurement using the WOF.

Lessons Learned

• The limited capacity able to be served led to stringent referral criteria which created barriers to sustainability in some instances.

• BFP funded the EBP programs with fixed price contracts for a limited period as start up funding with intent to move to a unit rate in accordance with the SOC funding model.

– Some agencies were not able to sustain following unit rate conversion and BFP could not retain the program due to a lack of referrals.

Mark D. Jones Chief Executive Officer

Community Partnership for Children 135 Executive Circle

Daytona Beach, Fl 32114 main line 386-238-4900 direct line 386-254-3936 cell phone 386-547-8924

http://communitypartnershipforchildren.org/

https://webmail.myflorida.gov/exchweb/bin/redir.asp?URL=https://webmail.myflorida.gov/exchweb/bin/redir.asp?URL=http://communitypartnershipforchildren.org/

https://webmail.myflorida.gov/exchweb/bin/redir.asp?URL=https://webmail.myflorida.gov/exchweb/bin/redir.asp?URL=http://communitypartnershipforchildren.org/

Volusia County, Super-community Grant Recipient

• Chadwick Trauma-Informed Systems Dissemination & Implementation Project

• Funded by the Substance Abuse and Mental Health Services Administration (SAMHSA) as a “Category II Center” with the National Child Traumatic Stress Network

• One of five “Super-communities” who will serve as communities of excellence and lead transformation of child welfare agencies into trauma-informed systems

Why “Volusia County” Chose to Become a Super-community

• Need for Systematic approach to implement practices to support training that has been provided over the past several years

• Need for Strategic Change to address several issues, including intergenerational abuse and compassion fatigue

• Opportunity to work with the experts in the field to improve our system of care

What It Really Means to Be a Super-community

• Intentional shift in the culture of the child welfare

workforce, as well as our system of care

• Incident focused Trauma Lens

• Challenges and successes

Who is Part of a Super-community

Partner with Agencies and Systems that Interact with Children and Families is one of the essential elements of a Trauma-Informed System

• DCF and CPC

• Community Assessment

• Community Kick Off

• Steering Committee

• Community Trainings

• Quarterly Updates to Community Stakeholders

• To learn more about this process, I invite you to attend tomorrow’s workshop:

Taking Trauma-Informed Care to the Next Level: One Trauma-Informed “Super-community’s” Integration of the Florida Safety Decision Making Methodology (FSDMM)

What Does it Take to Become an EBP? Brevard C.A.R.E.S. EBP Credentialing Journey

• The Formation of C.A.R.E.S.

– Designed and piloted model in house at BFP in response to community priority to implement a front end prevention and diversion program.

– Once fully implemented with fidelity to the model, BFP collected post discharge recidivism data at 6, 12, 18 and 24 months.

– An independent data validation study was conducted before creating separate 501 c. 3.

How Long Does it Take? C.A.R.E.S. EBP Credentialing Timeline

Year 1

•2009

•AED (now FHI 360) launched a data validation study

Year 2

•2010

•AED began 2nd more rigorous research study

Year 3

•2011

•AED-BFP created a Manual Replication Toolkit

Year 4-6

•2012-2015

•Child Trends began 3rd randomized control group study

Year 7

•2015

•Research findings on 2010 study published in the Journal of Families and Society in April 2015

•Application to become EBP being filed in October 2015

Challenges Associated with EBP Capacity Building and Implementation

• Conducting research using randomized selection of clients.

• The cost of implementation, training and management; and the impact of staff turnover.

• Maintaining fidelity to the model with limited referrals.

• The amount of financial and human capital required to collect data, provide ongoing training, monitoring, and cost to purchase the program license.

• The cost of EBP programs and the prescriptive nature and limited number of clients able to be served.

Questions????

January 23-24 2016Institutes January 25-28, 2016 Diego Conference

Sheraton San Diego Hotel and Marina, San Diego, CA

The San Diego Conference offers a terrific agenda of evidence and research-based topics, as well as current issues and new ideas. The conference is presented by international experts and practitioners in the field. Our top notch faculty will impart strong science-based interventions with practical experience and real world solutions. This multi-disciplinary program offers courses in medicine, mental health, trauma treatment, child welfare, law, investigations, forensic interviewing, research, early childhood mental health, technology, child welfare, secondary traumatic stress & resilience, domestic violence, prevention, and global perspectives topics for professionals working with children and families involved in maltreatment.

Register online before November 20, 2015 for early bird registration

www.sandiegoconference.org

Presented by Chadwick Center for Children and Families, Rady Children’s Hospital-San Diego

Join us at our bayside venue! Sheraton San Diego Hotel & Marina Special rates available until January 4, 2016

Leading the Way in Preventing Child and Family Maltreatment

http://www.sandiegoconference.org/

http://www.sandiegoconference.org/

Selecting and Implementing Evidence-Based Practices:

A Guide for Child and Family Serving Systems

Cambria Walsh

Jennifer Rolls Reutz

Rhonda Williams

April 2015

Acknowledgements

We would like to acknowledge the contributions of several individuals to the development of this

guide. First, Lynne Marsenich, LCSW, assisted with the framing and execution of the guide, and

provided feedback throughout its drafting. We thank the CEBC Implementation Science Panel for

their input, particularly Lawrence Palinkas, PhD, Patricia Chamberlain, PhD, and Sonja Schoenwald,

PhD, who reviewed full drafts and provided comments. Charles Wilson, MSSW, CEBC Director, and

John Landsverk, PhD, CEBC Scientific Director, provided guidance throughout the development of the

guide. We also thank Andrea Hazen, PhD, from the Child and Adolescent Services Research Center for

her edits and review. Finally, we appreciate the efforts of internal Chadwick Center staff including

Jennifer Hossler, MSW; Lisa Conradi, PsyD; and Molly Robb who reviewed drafts and provided

feedback, and especially Jennifer Demaree for her efforts in editing and formatting the guide.

Suggested Citation:

Walsh, C., Rolls Reutz, J., & Williams, R. (2015). Selecting and implementing evidence-based practices: A guide

for child and family serving systems (2nd ed.). San Diego, CA: California Evidence-Based Clearinghouse for Child

Welfare.

Copyright 2015 by the California Evidence-Based Clearinghouse for Child Welfare, Rady Children’s Hospital,

San Diego. All Rights Reserved.

Document Available from:

California Evidence-Based Clearinghouse for Child Welfare

Rady Children’s Hospital, San Diego

3020 Children’s Way, MC 5131

San Diego, CA 92123

[email protected]

Also available on the web at www.cebc4cw.org

http://www.cebc4cw.org

Introduction ................................................................................................................................................ 1

Background on Evidence-Based Practice .................................................................................................... 2

CEBC definition of evidence-based practice ................................................................................... 2

Dissemination and training on EBPs ................................................................................................ 3

Implementation definitions and concepts ...................................................................................... 4

Why look at implementation? ........................................................................................................ 5

How to support implementation: Strategies and Frameworks ...................................................... 6

Implementation Strategies .................................................................................................. 7

Implementation Frameworks .............................................................................................. 7

Overview of the EPIS Framework ................................................................................................................ 8

EPIS and using the CEBC for implementation ................................................................................. 9

Detailed Description of EPIS Phases ......................................................................................................... 11

Exploration Phase .......................................................................................................................... 11

Preparation Phase ......................................................................................................................... 17

Implementation Phase .................................................................................................................. 19

Sustainment Phase ........................................................................................................................ 22

Putting it all Together ................................................................................................................................ 24

References ................................................................................................................................................. 25

About the Authors ..................................................................................................................................... 26

Appendices ................................................................................................................................................ 27

A. Glossary

B. Common Implementation Strategies that Have Been Used in CWS

C. Sample Questions to Address in Each EPIS Phase of Implementation

D. Key Implementation Steps by EPIS Phase

E. Exploration Phase: Resources and Tools

F. Preparation Phase: Resources and Tools

G. Implementation Phase: Resources and Tools

H. Sustainment Phase: Resources and Tools

Table of Contents

In 2004, the California Department of Social Services (CDSS) contracted with Chadwick Center for

Children & Families at Rady Children’s Hospital-San Diego to create the California Evidence-Based

Clearinghouse for Child Welfare (CEBC). The CEBC is one of the California Department of Social

Services’ targeted efforts to improve the lives of children and families served within the child welfare

system.

The mission of the CEBC is “To advance the effective implementation of evidence-based practices for

children and families involved with the child welfare system.” The current CEBC website

(www.cebc4cw.org) provides a searchable database of programs that can be utilized by professionals

that serve children and families involved with the child welfare system. As the registry grew and

feedback from users was obtained, it became clear that users needed tools and assistance to help

them use the vast amount of information available on the website. Therefore, the CEBC began to

develop and identify tools for not only selecting evidence-based practices (EBPs), but also for

implementing and sustaining the use of those practices in community settings.

This guide was created to be a companion to the CEBC registry of programs and is based on the

emerging body of research known as implementation science and on lessons learned through CEBC

technical assistance efforts with county child welfare systems. In contrast to the many excellent

academic and scholarly texts available on the topic of implementation, this guide was designed

specifically for child welfare administrators and social services providers to

provide information and examples of implementation relevant to those

working with children and families in the child welfare system. It provides

concrete information that child welfare systems across the nation can use

to evaluate what their system needs, examine what programs are currently

being used in their system, make decisions about which new programs to

add, and plan for implementation activities. Numerous resources

referenced throughout this guide are provided in the Appendices including

a glossary in Appendix A that provides definitions of key implementation

terms.

Selecting and Implementing EBPs 1 © Apr. 2015 CEBC, www.cebc4cw.org

Introduction


CEBC definition of evidence-based practice

There are many terms currently used to describe the

research base for child welfare related practices. One of

those terms is evidence-based practice (EBP), which may

have a specific meaning to some professionals but a

dramatically different meaning to others. At this time, a

universal definition does not exist. The CEBC adapted the

Institute of Medicine’s definition for EBP (IOM, 2001),

incorporating child welfare language:

Best Research Evidence

Best Clinical Experience

Consistent with Family/Client Values

This definition builds on a foundation of scientific research while honoring the clinical experience of

child welfare practitioners and being fully cognizant of the values of the families which are served.

The main focus of the CEBC website is research evidence defined by the CEBC as research study

outcomes that have been published in a peer-reviewed journal. The CEBC developed their Scientific

Rating Scale to evaluate practices based on the available research evidence. The CEBC Scientific

Rating Scale creates a continuum approach to looking at the concept of EBP. Thus, none of the

programs on the CEBC are defined as evidence-based or not evidence-based. Instead, they are all

evaluated using the 1 to 5 rating of the strength of the research evidence supporting a practice or

program. A scientific rating of 1

represents a practice with the

strongest (well-supported) research

evidence and a 5 represents a

concerning practice that appears to

pose substantial risk to children and

families. Some programs listed on the

CEBC website do not currently have enough peer-reviewed published research evidence to be rated

on the Scientific Rating Scale and are classified as NR (Not able to be Rated).

The CEBC definition of EBP also recognizes that a program that has strong research evidence, but is

not implemented with the necessary clinical skill or is inconsistent with the family/child’s values

Background on Evidence-Based Practice


would not fit the definition of evidence-

based. The CEBC provides extensive

information on each program to help

address these concerns through the

detailed questionnaire that each program

completes. The standard format allows for

comparison between programs on the CEBC

website.

Several sections are particularly relevant to

the clinical skills and child/family values

components of the CEBC’s EBP definition.

For example, information on necessary

clinical skills for each program can be found

in the “Minimum Provider Qualifications”

and “Training Information” sections in the

detailed description of the program. For

cultural issues, review the “Brief

Description,” “Target Population,” and

“Essential Components” sections for

information on the goals and application of

the program. The research summaries also

provide information on the racial/ethnic

composition of the studies and where they

took place (e.g., rural versus urban,

university versus community, etc.). There is also a Cultural Resource section on the CEBC website

which provides information about EBPs and the cultural background of the clients or communities

served by the organization. This section provides articles with further information on cultural

adaptation and whether or not an EBP can be used in populations that were not represented in the

research studies.

Dissemination and training on EBPs

The CEBC has been disseminating information about the level of research evidence for child welfare

related programs and interventions since 2006 with a goal of improving services and outcomes for

children and families through the increased use of EBPs. Much has changed since that time. The

registry of programs on the CEBC started with 2 topic areas and 17 programs in 2006 and the CEBC’s

initial efforts focused on dissemination regarding the concept of EBPs and how the CEBC defined EBPs

Thinking Beyond the Numbers

The numbering system of the Scientific Rating

Scale is meant to help CEBC users quickly see

differences in the evidence levels of programs.

However, there is danger in not thinking beyond

the numbers. For instance, there is wide

variability in programs that are rated a 3. Some

programs that are rated a 3 have one study with

a control group, while others have multiple RCTs,

but have not yet published outcomes collected

on the participants at least 6 months after the

intervention ended. Looking closer at the

“Relevant Published, Peer-Reviewed Research”

section of a program to find out more detail on

the research can help inform the decision making

process. Also, there are entire topic areas where

none of the programs were able to be rated.

However, some of these programs may have

evaluations, dissertations, or other research that

do not meet the threshold for the CEBC Scientific

Rating Scale. Again, these can provide some

assistance in differentiating between programs

and is one of several steps in the decision making

process.


in an effort to increase basic awareness and

understanding of the need for EBPs in child welfare.

Dozens of in-person trainings were provided at the

national, state, and local level to child welfare

administrators and staff. The CEBC has also offered an

extensive series of webinars and on-line trainings to

reach a broader audience through distance learning.

Now, there are over 325 programs in 42 topic areas in

2015. In addition, recent statements from the

Administration for Children and Families (ACF) (U.S.

Department of Health and Human Services *DHHS+,

2012), as well as the Pathways to Mental Health

Services: Core Practice Model Guide (California

Department of Social Services, California Department

of Health Care Services, & UC Davis Extension Center

for Human Services, 2013) indicate the expectation of

the use of EBPs in child welfare services. However,

concerns and misinformation continue to exist

regarding EBPs. The CEBC will continue to disseminate

information on EBPs through distance learning and

periodic in-person trainings in an effort to address

these issues.

Implementation definitions and concepts

As defined by the National Institutes of Health (NIH), “implementation is the use of strategies to

adopt and integrate evidence-based...interventions and change practice patterns within specific

settings” (U.S. DHHS, 2013, Research Terms section, para. 4). Implementation is commonly defined as

putting something into effect or action, but the NIH definition above recognizes that implementing a

program or initiative may be complicated, specifically mentioning “adopt,” “integrate,” and “change”

and incorporating the use of strategies. Essentially, implementation is the multistep process of

moving an EBP into routine practice through the use of concrete supports.

On the surface, implementation would seem like a straightforward endeavor - identify a program and

put it into practice. In reality, implementation can be complex, confusing, and challenging, and it is

important to move through the implementation process in a thoughtful way in order to meet with

success. Implementation science is the study of methods to promote the integration of research

A Changing Focus

As the term EBP became more common

and understood in child welfare systems

and the number of programs on the

CEBC grew, it became apparent that

systems needed additional assistance

determining which EBPs were

appropriate for their needs and how to

successfully implement them. Common

questions asked of the CEBC were “How

can my system be more evidence-

based?” and “Which program should we

adopt?” It was clear to the CEBC that

simply providing information about the

evidence level for programs was no

longer sufficient. The training focus of

the CEBC has moved towards supporting

the implementation of EBPs in child

welfare systems through more hands-on

technical assistance, leading to the

development of this guide.


findings and evidence into policy and practice. Research on implementation has progressed over the

past decade and a growing body of evidence exists on implementation science. However,

implementation science is still developing and research on implementation in complex settings, such

as child welfare, is still limited.

Why look at implementation?

While disseminating information about EBPs by groups such as the CEBC is important, it is clearly not

enough to ensure that EBPs are implemented universally or consistently. Studies have shown that it

takes almost two decades for a new medical treatment to become common practice (Chadwick

Center, 2004). Dissemination creates awareness of the EBP, but knowledge alone will not ensure that

effective practices are utilized. Child welfare systems have begun to recognize that improving services

designed to support the needs and well-being of children and families is influenced as much by the

process of implementing EBPs as by the specific practices selected for implementation (Aarons,

Hurlburt, & Horwitz, 2011).

As the use of EBPs has increased, it has become clear that a gap often exists between the outcomes of

the program seen in research studies and the outcomes of the program in a real world setting. While

this has led some to question the value of EBPs, it has also spurred additional attention to the process

of implementation to determine what is occurring. Here are three issues that can lead to the failure

to implement models successfully:

The EBP that has been adopted is not being used as it was designed. This

could be due to many causes, such as insufficient training for providers (e.g.,

condensing training, a lack of follow-up or coaching, not planning for staff

turnover), lack of buy-in from providers, use with an inappropriate target

population (e.g., program for school-age children applied to adolescents), a

lack of accountability or monitoring of fidelity, or local adaptations to the

program that affect key components of the intervention (e.g., individual

intervention used in a group setting).

The EBP is put in place with fidelity, but does not last long enough to see

any meaningful change. This often happens when a program is introduced

through a time-limited grant or other dedicated funding – when the funding

goes away, unless there has been careful planning, the program will go

away as well, or be altered in such a way to make it meaningless.

Fidelity

Sustainment


The EBP is put in place with fidelity, but on such a small scale that the true

impact cannot be seen. For example, a pilot study of an EBP in a given

county may be too small to see an impact on common child welfare

outcomes, such as re-abuse or permanency.

A concrete illustration of implementation issues with an EBP is hand washing in health care. This

practice has been commonly recognized for over a century as a key component in disease prevention.

Numerous dissemination efforts have been conducted (e.g., training for medical staff on proper

techniques, commonplace signs in restrooms and commercial kitchens, etc.) and the need for hand

washing has become common knowledge. However, hand washing is not

always done or done correctly. During cold and flu season, there are

increased reminders to wash one’s hands regularly and properly as a way to

fight the spread of illnesses. However, disease outbreaks still occur which

could have been prevented if hand washing was done properly. Even in

hospitals, proper hand washing continues to be targeted by quality

improvement initiatives, as evidence shows that proper hand washing

techniques are not always followed in these settings. It is unlikely that anyone would argue to stop

washing hands because society has not achieved complete success getting everyone to wash their

hands routinely and correctly to date – instead, efforts are made to improve hand washing techniques

through increased education efforts (e.g., public service announcements, print and social media, etc.)

and addressing any barriers that may exist (e.g., ensuring that sufficient sinks and supplies are

available and offering hand sanitizers in public areas).

How to support implementation: Strategies and frameworks

Support for the implementation of new practices takes two basic forms: implementation strategies

and implementation frameworks. While there is some overlap between the two, frameworks typically

have an underlying theory and are broader in scale, while strategies are typically more discrete in

nature and offer specific approaches to the implementation of a practice.

An implementation strategy is a systematic process to adopt and integrate evidence-based

innovations into usual care (Powell et al., 2011). Many different implementation strategies exist and

can be classified into three groups:

Scale


Discrete implementation strategies involve the use of a single strategy, such as staff

training on the new procedure, fiscal incentives to implement the practice, or a

computerized reminder system to support use of the practice.

Multifaceted implementation strategies combine two or more discrete strategies, such as

staff training and reminders used together.

Blended implementation strategies are more comprehensive and integrate several discrete

strategies to impact implementation at multiple levels. These are often packaged as

branded strategies, such as the Community Development Team (CDT) and the

Breakthrough Series Collaborative (BSC) strategies. These blended strategies have been

used in child welfare settings and are described in more detail in Appendix B.

An implementation framework is a model of factors likely to impact implementation and sustainment

of EBP. They are developed based on literature reviews and may focus on specific service contexts.

Several useful conceptual frameworks have been developed that have basic elements in common

(Aarons, Hurlburt, & Horwitz, 2011; Damschroder et al., 2009; and Fixsen et al., 2005):

Phases or Stages of Implementation where the implementation process is divided into

discrete phases, each of which has multiple component activities.

Identifying Needs and Assessing Current Practice begins the implementation process with

an examination of the needs and status of the implementing agency and/or community.

Strength of Evidence includes consideration of the evidence supporting a proposed

practice as a key component of decision making during adoption.

Multi-level Context and Intervention Fit considers how a proposed intervention will fit

within the adopting agency as well as within the environment in which it operates (client,

community, staff, etc.).

Implementation Outcomes where several outcomes are examined, including the process of

implementation, the outcomes of the intervention implemented, and the sustainability of

the intervention.

In general, an implementation strategy should be used along with an implementation framework and

have clear underlying logic regarding why the strategy should work as desired. The strategy should

address implementation on multiple levels. For example, a strategy may have components that

address child welfare system policies and practices, while also targeting social workers directly. In

order to be successful, strategies need to be acceptable to stakeholders, feasible in the service setting

and robust enough to be adapted and scaled as needed (Mittman, 2010).

Imp

lem

enta

tio

n S

trat

egie

s Im

ple

men

tati

on

Fra

mew

ork

s


The CEBC has adopted the Exploration, Preparation, Implementation, and Sustainment (EPIS)

framework (Aarons, Hurlburt, & Horwitz, 2011) for use in its technical assistance efforts. The EPIS

framework was developed by CEBC-affiliated implementation scientists at the Child and Adolescent

Services Research Center (CASRC) through funding from the National Institute of Mental Health

(NIMH) and is based on existing research on implementation. It was developed specifically for use in

child welfare and similar service sectors.

The EPIS framework has four phases - Exploration, Preparation, Implementation, and Sustainment –

and examines contextual factors at two primary levels: outer and inner. The outer context represents

larger, often external, factors that can either support or slow implementation, such as federal, state,

county or local policies, funding and mandates, and organizational relationships. The inner context

represents what is happening within a community or organization that is implementing an EBP, such

as staffing, policies and procedures, and organizational culture and climate. The four phases are

reviewed briefly here, with a more detailed examination in the next section of this Guide.

Exploration Phase – Potential implementers consider what EBPs might best solve a clinical or

service problem, while also considering opportunities or challenges in the outer and inner

contextual factors.

Preparation Phase – Implementers plan for integrating the EBP into the existing system,

including a realistic and comprehensive assessment of implementation challenges.

Implementation Phase - The adopted practice is implemented. This is where the rubber

meets the road and the implementers will find out if their work during the Preparation phase

addressed the major issues.

Sustainment Phase – The intervention is ingrained in the organization, including stable

funding and ongoing monitoring and/or quality assurance processes.

Overview of the Exploration, Preparation, Implementation, Sustainment (EPIS) Framework


EPIS and using the CEBC for implementation

As the CEBC became more established, information on implementation was added to its website, but

it was not clear whether or how the CEBC was being used for implementation purposes. The CEBC

conducted a self-evaluation in 2011 that involved a web survey of child welfare administrators and

staff in California, as well as a focus group with child welfare services providers. The evaluation

showed that, while users were familiar with the CEBC and many had visited it, the information was

not commonly being used for implementation purposes, such as selection of an appropriate program.

Feedback from the focus group, as well as queries and comments from website users, showed us that

users needed assistance to help apply the information on the CEBC registry to the implementation

process.

The CEBC began to provide technical assistance to support implementation efforts of interested

communities in 2012. The technical assistance was hands on and typically involved several face-to-

face meetings with county agency staff, providers, and stakeholders. Work started with the

Exploration Phase of implementation to determine what, if anything, needed to be changed. The EPIS

stages and information on how CEBC used this model to provide technical assistance are outlined in

the sections below, with links to concrete resources and tools supporting each stage in Appendices C

through H.

The four phases of EPIS are illustrated above; although the phases proceed in order, there is often

some overlap between activities in the different phases. Within each phase, multiple domains are

addressed, including organizational characteristics, funding, client advocacy, and interorganizational

networks. A comparison of sample domains across phases is provided in the table on the next page.

The CEBC provides information relevant to each phase of EPIS through the program registry materials

and the implementation resources available on the CEBC website and in this guide, such as

Appendix C, which provides sample questions to address during each phase of implementation and

Appendix D which is a list of the Key Implementation Steps by EPIS Phase.

Sample progression of EPIS phases


Domain

Exploration Phase

Preparation Phase

Implementation Phase

Sustainment Phase

Organizational Characteristics

Identify skills and focus of agency that will provide the chosen EBP and determine how this will impact program selection

Identify the challenges and benefits of existing infrastructure in the implementing agency. Develop plans for monitoring fidelity and assessing outcomes

Monitor staffing, including job satisfaction, turnover, culture, and climate to assess impact of new EBP and respond Review plans for fidelity monitoring and assessing outcomes

Address turnover and staffing changes in the supervision and quality assurance procedures to insure the EBP continues with fidelity

Funding Consider what funding is available and compatibility with potential EBPs

Secure funding for chosen EBP delivery and staff training

Ensure billing is being submitted and approved and reimbursement is occurring

Identify continued funding in cases where program was a pilot or a time-limited contract

Client Advocacy Identify and involve local and/or national advocates, including caregivers and families, in the implementation process

Engage the local or national advocates to support the EBP

Gather satisfaction information from clients and referral sources and local advisors and develop a feedback loop

Engage the local or national advocates to support sustainment of the practice

Interorganizational Networks

Engage professional stakeholders in implementation process to provide input

Form partnerships among multiple agencies who will be implementing the new practice

Look to networks to provide problem solving and support when challenges arise

Ensure networks are institutionalized

Summary of Sample Domains across EPIS Phases


Exploration Phase

The Exploration Phase is the initial phase in EPIS and involves awareness of a clinical or service issue

that needs to be addressed in a more effective way. For example, a child welfare system may want to

reduce re-entry to services to address a Child and Family Services Review goal. During this phase,

possible interventions to address the targeted area will be considered and an adoption decision will

be made based on the specific agency and/or community needs.

The first step involves setting up a small group of individuals to work on the project - the

Implementation Team. It is important to incorporate several different types of members on this

team (e.g., administrators, supervisors, front line workers, contractors, key stakeholder

representatives, etc.) to ensure that different perspectives are represented. The group determines

who will chair the meeting and how administrative support will be provided (i.e., who will take

minutes, oversee scheduling, etc.). The Implementation Team Membership Tracking Tool (Appendix

E1) is a sample form to track team membership, and The Critical Role of

Implementation Teams and their Evolution through EPIS (Appendix E2) is an

overview document. It is also important at this juncture to explore the

support, resources, and timeline for working on the exploration process.

This may be done as part of setting up the Implementation Team or with

that team once it is formed. The Exploration Worksheet (Appendix E3)

provides a sample format to track basic information essential to this

process; it can be modified to fit individual needs.

Once the group is established, the problem identification step occurs. The group identifies 3-5

potential areas that they feel need to be addressed. They describe each area in 2-3 sentences and

then begin to look at what data is available to describe the problem, such as population

demographics, indicators of need, referral and retention rates, as well as any data on existing services

(e.g., numbers served, outcomes, waiting times, etc.).

Detailed Description of EPIS Phases

Some of the key steps in the Exploration Phase include:

Form an Implementation Team

Identify the problem

Narrow the focus

Conduct a needs assessment

Identify potential solutions

Determine program fit

Create a written summary


Each problem is further clarified through a series of questions aimed to clarify more about the

problem. Example questions are provided in the Identifying and Clarifying the Problem handout

(Appendix E4). This step ends with the further refinement of the 3-5 potential problem areas, using

the information obtained during the review and adding in more detail about the problem and causes.

One tendency CEBC staff have noted in some communities is the temptation to jump to a solution and

suggest specific EBPs that might address the identified problem area before the underlying issues

causing the problem are thoroughly understood. To address this, it is important to ask detailed

questions about each identified area to determine the root cause. One way to do this is to use an

approach from the quality improvement field called “Ask Why 5 Times” combined with support from

existing data. An example can be seen below. To avoid coming to erroneous or simplistic solutions, it

is important for the Implementation Team to work together on the questions and for the answers to

be based on data, not supposition. In some cases, it may take several weeks to work through the

questions, as data needs to be pulled and examined before proceeding. For more information on data

sources that may be used during this process, see Data Sources to Consider (Appendix E5).

Ask Why 5 Times

Problem: The re-entry rate to Child Welfare is too high.

Initial suggested solution: County needs to add practices listed under reunification on the CEBC.

Question: Is that the correct response?

Ask Why 5 Times

Why do children re-enter care? Families are being re-reported 6-12 months after case closure.

(Extracted time frame from data reports.)

Why? Families typically have recurrent substance abuse issues. (Examined reasons cases were

re-opened.)

Why? Families had received substance abuse treatment services while in CWS but still relapsed.

(Examined services received in previous CWS case.)

Why? Many families who re-entered received few, if any, of the offered aftercare services.

(Examined service records and compare to non-returning cases.)

Why? Families have not engaged in aftercare services or linked with needed services in the

community. (Explored EBPs that have demonstrated improved capacity to engage families

following the completion of substance abuse treatment.)

The next step involves narrowing the focus of the effort to one problem area that will be the initial

focus of the group’s efforts. An initial area of focus may have become obvious during the problem


identification step, or it may be necessary to have the group vote on

which area to focus on first. Other areas can be added over time, but it

is helpful to have success in a single area and become confident in the

process before trying to address several areas simultaneously. This

step involves looking in detail at the current system and processes

related to the problem area. For example, in the scenario given in “Ask

Why 5 Times” box on the previous page, this would entail examining

the process for referral to and discharge from substance abuse services, as well as the decision-

making and risk assessment process for closing cases, and the current procedures for referring to and

engaging clients in aftercare services. Based on the identification of the problem area, additional

stakeholders may be identified to join the implementation group. For example, in this case, a

representative from substance abuse services should be asked to join the work group.

The next step is to conduct a thorough needs assessment

for the area of interest. This will likely involve a deeper

look at the data than was done during the problem

identification step and may involve new data collection in

the form of surveys or focus groups. Again, using the

scenario given in the “Ask Why 5 Times” example, an

agency would want to determine the number of families

in the system currently who may be in the target

population and conduct focus groups or key informant

interviews to find out more about why families do or do

not engage in aftercare services.

As part of the needs assessment, any existing services in

place to address the problem area should also be

examined. Using the current scenario, this would involve

looking at existing services, such as reunification

programs, substance abuse services, and aftercare

services. For each type of program, the services should be

examined in detail including what program model is being

used, if it is evidence based, how services are delivered, if

it is being delivered with fidelity, and what outcomes the

participants have experienced. Before deciding on

whether a new program is needed, it is important to know

what is currently available in the community. CEBC

experience has shown that agencies are often unaware of

Exploration Example: County A

County A wants to reduce their

placement disruption rate, which

is higher than the state average.

They conduct a needs assessment,

including chart reviews and phone

calls with foster and kinship care-

givers and caseworkers, to better

understand the reasons for place-

ment disruption. The data shows

that a large percentage of the

moves are initiated by the caregiv-

er and involve children with a

mental health diagnosis. The

phone calls identify that caregivers

feel unable to sufficiently address

the behavioral and mental health

needs of the children in their care,

leading to frustration and eventu-

ally a notice of removal. County A

decides to look at models that

help address youth mental health

problems while providing support

for caregivers.


exactly which models their contractors are using and that EBPs may be in place in the community

without the agency knowing. In addition, there may be locally developed programs that have good

track records of outcomes and community support. These should be examined in more detail to

determine if the evidence base can be developed for the local program.

Finally, the last part of the needs assessment step is to clearly define the issue and the outcomes

desired from practice changes. This typically involves a brief written summary of the problem area

that has been selected, the reasons why it was selected, and the concrete indicators of outcomes

after practice changes have been made. The summary should be reviewed and approved by the

group members to ensure that everyone is on the same page.

Once the problem area is clearly identified

and examined, the next step, identifying

potential solutions, can begin. It is

important to keep in mind that solutions do

not always involve the addition of a new

program. Instead, they may involve making

changes to internal processes to ensure that

existing services are delivered more

effectively.

If the problem can clearly be addressed by a discrete program, then there are 3 options to consider:

1) Expanding existing EBPs in place in the community that need more capacity (e.g., agency

currently funds several providers of The Incredible Years, but there is a lengthy wait list)

2) Add new EBPs (e.g., add providers in the community who are certified in Trauma-Focused

Cognitive-Behavioral Therapy)

3) Build the evidence for a locally developed program in place that seem promising (e.g., a home-

grown parent training program has been in place for many years and appears to have good

outcomes, but needs additional research to verify effectiveness).

If it appears that a new practice will need to be added to the system (e.g., need to add trauma-

focused mental health services to existing array), or that an existing practice needs to be changed

(e.g., change current locally developed home visiting program to one that has evidence of effectively

targeting neglect), then the CEBC registry of programs can be an effective resource. Begin the process

of searching for potential programs by developing a list of keywords that are relevant to the program

being sought. For example, when looking for a home visiting program that targets neglect, “home

visiting” and “neglect” would be appropriate terms to list. Additional terms to describe the target

population for the program, such as “young children,” or the type of staff who are currently delivering

Examples of Changes to Internal Processes

Assessment and referral processes may need to

be adjusted to ensure that families are being

referred to the most appropriate service availa-

ble in the area.

Addition of supervision and a monitoring system

for an existing practice to ensure that it is being

delivered with fidelity and consistency.


home visiting services (if these existing staff would be

utilized for the new program), such as “paraprofessional” or

“bachelors’ level” may also be useful. The Identifying

Potential Solutions form (Appendix E6) can be used to keep

track of the chosen keywords. Once a list of several

keywords has been developed, review the CEBC Topic Areas

to identify the topic areas that seem relevant and indicate

them on the form. An individual or subcommittee can then

review the programs in each topic area and examine the

“About This Program,” “Brief Description,” and “Minimum

Provider Qualifications” sections for each to identify

potential programs and exclude those that clearly do not fit

local needs. At this point, be fairly broad and include all

programs that appear to fit on the list. Fill out the basic

information on each potential program on the Identifying

Potential Solutions form.

The next step is determining the fit of each potential program using

the CEBC Selection Guide for EBPs in Child Welfare (Appendix E7),

which was derived from a review done by Greenhalgh, Robert,

Macfarlane, Bate, & Kyriakidou, 2004. Each program on the

Identifying Potential Solutions form should be reviewed using the

Selection Guide and Selection Guide Worksheet (Appendix E8) by a

small group or subcommittee; it can also be done independently by

several individuals with the results compared to achieve consensus.

The selection areas listed in the box to the left are described in

more detail in the Selection Guide, along with the section(s) of the

CEBC program registry that may have applicable information.

In addition, during the Exploration Phase, it is beneficial to have

direct contact with representatives from the potential programs,

preferably program developers or their dissemination and/or

training staff (see Working with Program Developers *Appendix E9+

for more information). These interactions will allow the Implementation Team to gain some insight

into the way that the program staff works, as well as how their work style fits with the team’s work

style. If the program is selected, the team will likely be working closely with these people and a good

fit will be helpful. The team can also use these initial contacts to identify any clear barriers to

Exploration Example: County B

While considering the fit of sever-

al potential programs, County B

learns that there is currently a 3-6

month wait list for training for

Program X, which is typically de-

livered at the developer’s offices

(in another state). For on-site

training, at least 15 staff must be

trained. The Implementation

Team considers this information,

along with training costs, when

examining program fit and deter-

mines that this is program is still

feasible, given their timelines and

budget constraints.

Selection Areas

Ease of use

External compatibility

Financial considerations/

relative advantage

Internal compatibility

Knowledge requirements

Match of skill set

Observability of benefits

Reinvention/adaptability

Risk

Training/support

Trialability


implementation. For example, if

the program’s training staff is

fully booked for the coming year

but the new program needs to

start within the next six months

in order to capitalize on existing

funding, other options will need

to be considered by the team.

After completing the Selection

Guide Worksheet for any

potential programs identified,

review the forms with the

Implementation Team and see if consensus can be reached on 1 or 2 programs that appear to be the

best fit. At this point, it is often helpful to present information on this narrow list of programs to the

larger Implementation Team and to existing groups of child welfare stakeholders to get feedback on a

broader perception of fit. In some cases, community forums may be used to allow for public

discussion and input. This step can be time-consuming and is most appropriate for large-scale

changes, such as major alterations to the case work model. Either manner of obtaining feedback

allows for the incorporation of additional points of view and also helps identify potential champions

for the planned change, as well as barriers to change that may exist.

If these services are going to be contracted out, this is the time to make a critical decision as to

whether a specific practice will be designated in the contract or whether the applying agencies might

have some input and a role in the exploration process. The Considerations when Contracting For

Services document can be found in Appendix E10.

The final step in the Exploration Phase is to create a written summary of the

work completed to date. This does not need to be a lengthy or extremely

formal document, but should summarize the process that was completed as

well as any feedback that was obtained on the potential programs. Include

basic information such as Implementation Team or other workgroup rosters,

meeting agendas and minutes (if available), and completed versions of the

CEBC tools that were used. The document should make a clear

recommendation on what intervention the workgroup decided on and provide

support for the choice. Finally, the document should lay the basic ground work for the next steps that

need to occur – the Preparation Phase. The Template for Exploration Summary Report with suggested

sections to help create this report can be found in Appendix E11.

Exploration Phase Tip

One way to help gather information on the specific items in

the CEBC Selection Guide for EBPs in Child Welfare is to

identify other agencies using the programs being

considered. This information can be obtained from the

programs’ developers or through formal or informal

networks, such as child welfare directors associations,

listservs, or technical assistance centers. Often, a discussion

with someone who has implemented the program can give

very useful insight into the fit, the implementation process,

as well as the pros and cons of the individual program.


Preparation Phase

During this phase, the system/organization has made the decision to adopt a specific EBP and is doing

the planning and preparation work necessary to effectively implement it. The adoption decision is

often seen as a one-time event while, in practice, it begins the preparation for implementation. Some

of the key considerations in this phase are similar to those in the Exploration Phase but now move

from identification to problem-solving. The Preparation Phase will often involve working closely with

the program developer, who may have developed materials relevant to preparation; this information

can be found in the Implementation section of the CEBC website or from the developer directly.

A key initial step in the Preparation Phase is ensuring leadership buy-in and confirming that relevant

agency leadership are championing and supporting the adoption of the chosen EBP. If these

individuals were not part of the Implementation Team during the Exploration Phase, it may be

necessary for the team to arrange to meet with each leader, review the exploration process and

results, and address any concerns that they may have. Without the clear support of leadership, it will

be very difficult for implementation of a new practice to occur. It may be helpful to assess leadership,

supervisor, and front line worker support for implementation through the use of existing assessment

tools; information on these tools is available on the CEBC website in the Implementation section

under “Tools and Resources.”

In addition to getting leadership on board, it will be necessary to develop

an implementation support system and build the infrastructure to

support implementation of the new practice. This ranges from

designating or hiring a change agent or coordinator to take day-to-day

responsibility for implementation to establishing a clear relationship with

the program developer (see Contracting with Program Developers

*Appendix F1+) and/or their implementation and training providers, to

determining how outcomes and fidelity will be examined (refer to Data & Outcomes *Appendix F2+

and Assessing Fidelity *Appendix F3+). It is also important to evaluate what resources will be required

and what resources may already exist and can be transitioned as the new program is implemented

(see Resources for Implementation *Appendix F4+). Many agencies have a training department that

will need to be involved in planning for introductory and ongoing training, regardless of whether or

Some of the key steps in the Preparation Phase include:

Ensure leadership buy-in

Develop an implementation support

system

Work with stakeholders

Ensure that the chosen EBP fits with

consumer concerns

Identify viable funding streams

Develop timetables


not the trainings will be delivered by local staff. Some agencies may

have quality assurance or similar units that will oversee monitoring of

the program. If not, the Implementation Team will need to determine

how these functions will occur and develop clear timelines and

partnerships for each. If necessary, external contracts may be set up

to support implementation. For example, a county implementing a

new home visiting program countywide may decide to use a Request for Proposal (RFP) for an

external evaluator to ensure that the evaluation is impartial.

Working with stakeholders is a crucial part of the Preparation Phase. Stakeholders will vary

depending on the area being targeted, but will likely include community agencies and partners,

consumer groups, and advocacy organizations. Hopefully, some of the stakeholders were involved in

the Exploration Phase; however, if they were not, engaging these groups and individuals early in the

Preparation Phase will be crucial to success. Stakeholders will play a role in providing support for the

program, including referral and client engagement. For example, they can help address early on how

the new practice will fit into the existing service and practice system. In addition, these stakeholder

groups can help ensure that the chosen EBP fits with consumer concerns. Concerns might include

the applicability of the selected practices for the needs and culture of clients in the targeted

populations and the potential for stigma.

Initial work to identify viable funding mechanisms should have been done during the Exploration

Phase, and this work can be solidified during this phase. Example activities include revising billing

manuals, forms, and training to accommodate

the new program and updating the electronic

billing and/or health record systems to

document and bill for the services

appropriately. Since funding is essential, clearly

written communication should be obtained

from the funder stating that the funding stream

will cover the new services. Refer to

Determining the Funding Stream (Appendix F5)

and the Funding Stream Inventory Worksheet

(Appendix F6) for information on identifying and

evaluating viability of funding streams.

Finally, develop timetables for key processes in the Preparation and Implementation Phases to keep

the effort on track. This task will likely be done by the change coordinator or similar individual in

charge of the day-to-day implementation process and in conjunction with the Implementation Team

Preparation Example: County A

County A is implementing Multidimensional

Treatment Foster Care (MTFC) and works

closely with them to identify recruitment and

advertising strategies that highlight the

unique features and role of MTFC foster par-

ents. In addition, MTFC helps County A de-

velop foster care reimbursement rate struc-

tures that are likely to yield a number of fos-

ter parent applicants.


and stakeholders. Timelines will need to address

the realities that exist in the system, such as

avoiding peak service times during the year (e.g.,

start of school year, holidays) or conflicting with

already scheduled requirements (e.g., week of

external review by funding agency, budget

development time period). If services are being

contracted out, consider the time that will be

required for the procurement process, as this can

often take several months. In addition, timetables

for staff training and certification will need to align

with service and productivity requirements of the

service system or organization; if existing staff are

being used, it will need to be determined how to

cover their caseload while they attend training. The

timeline for obtaining and distributing training

materials and manuals, as well as documentation

and billing forms, will need to be established. Finally, planning for outcomes and fidelity monitoring

should be included in the timeline. Program developers or implementation staff may have draft

timelines already established that can be adapted for local use. The following documents may be

useful to review when developing timelines: Referral System (Appendix F7), Staffing Plan (Appendix

F8), and Training & Coaching Considerations (Appendix F9).

Implementation Phase

The system/organization is in the process of implementing (e.g., training, knowledge and skill transfer,

service delivery, capacity building) a specific EBP at this time. The active Implementation Phase is

when the “real” work of implementation has begun – often this is the first time that change is visible

to the community.

Preparation Example: County B

County B has decided to implement The

Incredible Years. While the program

has no rules regarding staff education

and experience, it is recommended that

at least one of the group leaders have a

Master’s degree or above and

knowledge of child development. In

addition, staff who have had experience

leading groups are more likely to be

successful than those who have not.

The Implementation Team uses this in-

formation when developing job descrip-

tions and identifying whether/how in-

ternal staff would be appropriate for

the new positions.

Some of the key steps in the Implementation Phase include:

Verify buy-in

Ensure priority

Complete training

Prepare materials

Confirm referral processes

Monitor fidelity to the EBP

Collect and evaluate outcomes

Explore scale-up in the service system(s)


At the start of the Implementation Phase, it will be important

to verify buy-in to confirm that stakeholders have bought into

the change that the organization is implementing and that they

are fully in support of the change. This will involve addressing

any remaining concerns and ensuring that all stakeholders are

informed, on the same page, and clearly supportive of the

change to the EBP in order for the implementation to be

successful. The role of the Implementation Team has changed as it moved into this phase, with more

focus on quick response to challenges encountered; for more information on Implementation Teams

during this phase, refer to The Critical Role of Implementation Teams and their Evolution through EPIS

(Appendix E2). Monitoring and Feedback Systems (Appendix G1) and Reviewing the Billing/Financial

Process (Appendix G2) contain information on the types of issues the Implementation Team should be

addressing during this phase.

In addition, it will be necessary to ensure priority of the change to the new EBP and that other issues

do not take precedence over the implementation of this EBP. Querying involved partners to

determine if any unseen or pressing issues have arisen and then addressing them in the

Implementation Team will be necessary. For example, changes in agency leadership will require

outreach to the new leaders to ensure they understand and support the implementation process. It

can be helpful to discuss with the Implementation Team how common barriers to implementation

would be addressed, such as changes in leadership, funding, and policies, so that needed responses

can be more proactive and timely.

During the Preparation Phase, training timelines were developed which will be followed during the

Implementation Phase. This will include procuring training

venues and materials, conducting the actual trainings, and

providing opportunities for new trainees to practice what

is being learned. It will be essential to complete training

and begin service delivery shortly after training to ensure

that skills are not lost before they can be applied. In

addition, it is necessary to prepare materials for service

delivery (e.g., manuals, checklists, fidelity rating forms) and

distribute them according to the established timelines.

Shortly before service delivery of the EBP commences, it

will be important to confirm referral processes are firmly

in place so that service provider trainees can begin seeing

clients while attending to the need for system, site, or

Implementation Example:

County A

County A’s program supervisor

has weekly telephone consulta-

tion with MTFC regarding treat-

ment plans, progress and prob-

lems. Consultation includes re-

view of videotapes from the

weekly foster parent and clinical

team meetings. This is an integral

part of both MTFC and County A’s

effort to ensure fidelity to the

model.


practice adaptations. Training for referral sources may be necessary and team

members should plan to check in with the referral sources after services begin to get

feedback on the referral process. If referrals are not being received, it may be

necessary to re-deploy service providers to do outreach and determine why clients

are not being referred to the EBP. The Supporting Initial Implementation – Go Live

Checklist (Appendix G3) lists similar activities that should occur when services begin.

From the start of services, it will be essential to monitor fidelity to the EBP and to collect and

evaluate outcomes (refer to Monitoring and Feedback Systems *Appendix G1+ and Examining

Outcomes *Appendix G4+). Doing these from the start ensures that they are considered part of

standard practice and allows the implementation team to identify and address any challenges quickly.

The outcomes and fidelity monitoring plans developed during the Preparation Phase should be put

into place and evaluated on a regular basis, with changes made as needed by the change coordinator

and Implementation Team. Early focus will be on model fidelity (e.g., self-report checklists,

observation) and process outcomes (e.g., number of clients referred, number of sessions completed),

as well as the collection of baseline indicators (e.g., strengths, challenges, symptoms). There should

be a plan to examine initial client outcomes at an appropriate time as determined by the program

developer and implementation team.

Finally, at the start of the Implementation Phase, all required

contracts to support the EBP implementation and use should

be in place. Timelines were established during the

Preparation Phase and should be verified at the start of this

phase to identify any delays or problems. As the service roll

out begins, examine and explore scale-up in the service

system(s) and how the funding and resources are being used

to support the EBP. This will help to identify issues early on

(e.g., multiple referrals have been made to a provider but the

corresponding bills have not been submitted within the

established timeframe), and inform future service expansion

as additional capacity, or adjustments to resource

distribution, may be needed as the new program rolls out.

For example, additional services may be needed in a

particular region, resulting in amendments to the client

targets in the corresponding contract and, thus, the addition

of staff.

Implementation Example:

County B

County B is implementing The

Incredible Years. The Imple-

mentation Team had concerns

about parent retention in the

first set of parenting groups

and decided to purchase some

telephone consultation with

the developers to discuss sug-

gested strategies to improve

participation. After the call,

the Team selected two strate-

gies to try for the next set of

groups (transit vouchers and

reminder calls) and see if they

had any impact on retention.


Sustainment Phase

The system or organization has implemented a specific EBP and is now using it as part of its regular

service or treatment model with an appropriate proportion of clients. At this stage, the EBP becomes

everyday service as usual in the agency or community. This will typically occur at least six to twelve

months (or longer) after the Implementation Phase occurred. The Sustainment Phase involves the

continued use of an EBP with fidelity after initial implementation is complete. The Sustainment Phase

can also include expansion of an implemented practice and further scaling-up or spread of the model

within a service system beyond an initial implementation effort.

An important step for the Sustainment Phase is to identify and procure ongoing stable funding and

support for systems, organizations, and staff delivering an EBP (see Sustainable Funding *Appendix

H1+). When a new EBP is established using one-time or short-term funding, plans for sustainment are

often required from the start but may be cursory or vague. It will be important to determine whether

services can be funded through existing ongoing mechanisms (e.g., mental health services dollars,

Medicaid). In addition, support for ongoing training needs, including ongoing and booster training for

existing staff and training of new staff due to expansion or staff turnover, will need to be identified

along with support for other necessary resources, such as program materials and meeting space. For

more detailed information see Ongoing Training and Coaching Needs (Appendix H2).

Ensuring the program continues with fidelity is a

primary component of the Sustainment Phase - fidelity

monitoring, coaching, and support for staff delivering

the intervention need to be continued to ensure that

the program is being delivered as designed and with

integrity. Fidelity can be maintained by building these

efforts into standard quality of care and training

practices and incorporating them into Policy and

Procedure manuals. In addition, outcomes will need to

be examined on a regular basis to ensure that the

intervention is having the desired effects and to ensure

that it is not having unintended effects. These can

similarly be built into standard practice, such as

Sustainment Example: County A

County A has been implementing

MTFC for three years. They establish

a line item in the annual budget to

provide for ongoing booster training

and for training of new staff and fos-

ter parents. The amount is deter-

mined based on staff counts and

turnover rates. In addition, they ap-

ply to become an MTFC-certified pro-

gram, to support ongoing assurance

of model fidelity and outcomes.

Some of the key steps in the Sustainment Phase include:

Funding and support

Ongoing training needs

Ongoing fidelity monitoring

Outcomes

Making refinements

Reviewing referral process


monthly or quarterly reports from contractors,

to reduce burden and encourage sustainability.

For more information, see Maintaining Fidelity

(Appendix H3).

Once the EBP has been up and running for

several months, it may be necessary to make

refinements to the delivery process or to the

intervention itself to fit the needs of the

community or population being served. Any

refinements must be discussed with the

developer, as they will have the best idea of

which components of the program are essential

and cannot be altered and which may be open

to adaptation to meet local needs. Similarly,

the referral process should be reviewed on a

regular basis to improve the ease of referral and ensure that any feedback loops remain in place. As

systems change over time and providers and processes change in the broader system, the referral

process may need to be adjusted accordingly. These are just a few considerations and identifying

those unique to the local system and organization, as well as to the chosen EBP, will help to complete

this phase.

Sustainment Example: County B

County B is implementing The Incredible

Years; services are being delivered by sever-

al contracted agencies. The Incredible

Years’ fidelity tool is routinely used in super-

vision within each agency, with results sub-

mitted to the funding agency quarterly.

Each agency has at least one staff person

who can provide booster training on an as

needed basis. Agencies work together to

provide annual and new staff training on a

countywide basis. Group leaders across the

agencies meet on a quarterly basis.


The CEBC has learned four key lessons over the past few years as its staff has worked with counties

around implementation of EBPs:

1) The importance of careful planning before deciding what, if anything, to adopt. Taking the

time to work through the Exploration Phase, and clearly identifying the problem is crucial

to the success and sustainability of the implementation effort.

2) The importance of data-driven decisions. It may take some time to look at the data, but in

the long run it will result in a much stronger likelihood of choosing a practice that is the

best solution to the identified problem.

3) The importance of the referral system. A thorough job of creating referral systems and

educating those making referrals about the practice must be completed before it starts

being delivered and the implementation team must be willing to refine the referral system

as needed when the practice is implemented.

4) Training is necessary, but not sufficient. If staff members are trained without putting any

further support (e.g., coaching and monitoring) in place, it is unlikely that the practice will

be implemented with fidelity and thus will not bring the expected results.

In conclusion, it is important to note that at this time the field of implementation science is still in the

early stages of development. This guide captures what the CEBC currently knows about implementing

EBPs in child welfare, however, the information will evolve as more is learned. The CEBC will continue

to provide additional information on implementation on the CEBC website which will supplement this

guide. Please visit the website at www.cebc4cw.org to see what is new and join the CEBC email alert

list to be kept up to date as new information is added to the website.

Putting it All Together



References

Aarons, G. A., Hurlburt, M., & McCue Horwitz, S. (2011). Advancing a conceptual model of evidence-

based practice implementation in public service sectors. Administration and Policy in Mental

Health and Mental Health Services Research, 38, 4-23.

California Department of Social Services, California Department of Health Care Services, & UC Davis

Extension Center for Human Services. (2013). Pathways to mental health services: Core practice

model guide. Retrieved from http://www.dhcs.ca.gov/Documents/

KACorePracticeModelGuideFINAL3-1-13.pdf

Chadwick Center for Children and Families. (2004). Closing the quality chasm in child abuse treatment:

Identifying and disseminating best practices. Retrieved from http://www.chadwickcenter.org/

Documents/Kaufman%20Report/ChildHosp-NCTAbrochure.pdf

Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009).

Fostering implementation of health services research findings into practice: a consolidated

framework for advancing implementation science. Implementation Science, 4, 50. doi:

10.1186/1748-5908-4-50. Retrieved from http://www.implementationscience.com/

content/4/1/50

Fixsen, D. L., Naoom, S. F., Blase, K. A., & Friedman, R. M. (2005). Implementation research: A

synthesis of the literature. Retrieved from http://cfs.cbcs.usf.edu/_docs/publications/

NIRN_Monograph_Full.pdf

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in

service organizations: Systematic review and recommendations. Milbank Quarterly, 82(4), 581-

629.

Institute of Medicine (US), Committee on Quality of Health Care in America. (2001). Crossing the

quality chasm: A new health system for the 21st century. Retrieved from https://www.iom.edu/

~/media/Files/Report%20Files/2001/Crossing-the-Quality-Chasm/Quality%20Chasm%202001%

20%20report%20brief.pdf

Powell, B. J., Proctor, E. K., & Glass, J. E. (2011, October). A systematic review of implementation

strategies in mental health service settings. Presentation at Seattle Implementation Research

Conference, Seattle, Washington. Retrieved from http://www.seattleimplementation.org/wp-

content/uploads/2011/11/Powell-SIRC-2011-Presentation-Final.pdf

U. S. Department of Health and Human Services (DHHS), Administration for Children Families,

Administration on Children, Youth and Families, Children’s Bureau. (2012). Information

memorandum (Log No: ACYF-CB-IM-12-04). Retrieved from http://www.acf.hhs.gov/programs/

cb/laws_policies/policy/im/2012/im1204.pdf

U. S. Department of Health and Human Services (DHHS), National Institutes of Health. (2013). Funding

Opportunity Announcement (Number PAR-13-054). Retrieved from http://grants.nih.gov/grants/

guide/pa-files/PAR-13-054.html


http://www.dhcs.ca.gov/Documents/KACorePracticeModelGuideFINAL3-1-13.pdf

http://www.dhcs.ca.gov/Documents/KACorePracticeModelGuideFINAL3-1-13.pdf

http://www.chadwickcenter.org/Documents/Kaufman%20Report/ChildHosp-NCTAbrochure.pdf

http://www.chadwickcenter.org/Documents/Kaufman%20Report/ChildHosp-NCTAbrochure.pdf

http://www.implementationscience.com/content/4/1/50

http://www.implementationscience.com/content/4/1/50

http://cfs.cbcs.usf.edu/_docs/publications/NIRN_Monograph_Full.pdf

http://cfs.cbcs.usf.edu/_docs/publications/NIRN_Monograph_Full.pdf

https://www.iom.edu/~/media/Files/Report%20Files/2001/Crossing-the-Quality-Chasm/Quality%20Chasm%202001%20%20report%20brief.pdf



http://www.seattleimplementation.org/wp-content/uploads/2011/11/Powell-SIRC-2011-Presentation-Final.pdf

http://www.seattleimplementation.org/wp-content/uploads/2011/11/Powell-SIRC-2011-Presentation-Final.pdf

http://www.acf.hhs.gov/programs/cb/laws_policies/policy/im/2012/im1204.pdf

http://www.acf.hhs.gov/programs/cb/laws_policies/policy/im/2012/im1204.pdf

http://grants.nih.gov/grants/guide/pa-files/PAR-13-054.html

http://grants.nih.gov/grants/guide/pa-files/PAR-13-054.html

About the Authors

Cambria Rose Walsh, LCSW, has worked for Chadwick Center for Children and Families at Rady

Children's Hospital–San Diego since 2001. She is currently the Project Manager of the CEBC,

overseeing the day-to-day operations of the project. Cambria has experience in working with

counties and agencies on the implementation of SafeCare® and Trauma-Focused Cognitive-

Behavioral Therapy (TF-CBT). She has provided training and consultation on the Exploration and

Preparation Phases of the EPIS model to counties in California as well as to California’s Office of

Child Abuse Prevention (OCAP).

Jennifer Rolls Reutz, MPH, is the Research coordinator for the CEBC, as well as an evaluator for

several grants at the Chadwick Center for Children and Families at Rady Children’s Hospital-San

Diego. She was previously a Research Coordinator with the Child and Adolescent Services Research

Center (CASRC), overseeing several National Institutes of Mental Health (NIMH) funded grants,

including Caring for Children in Child Welfare (CCCW) and the Implementation Methods Research

Group (IMRG) Advanced Center, along with numerous foundation, state, and local projects. Her

research interests include policies, practices, and time use in public service sectors, such as child

welfare and mental health, and the use and implementation of evidence-based practices in real-

world settings. She has provided training and consultation on time use and implementation issues

to several counties, as well as to California’s Office of Child Abuse Prevention (OCAP).

Rhonda Williams, MA, is currently the Research Associate for the CEBC. She was previously an

assistant research administrator for the Child and Adolescent Services Research Center (CASRC),

and also a research assistant on several NIMH-funded grants, including Caring for Children in Child

Welfare (CCCW) and the Patterns of Care (POC) grant while working for CASRC. Rhonda has

experience teaching social and communicative skills to children with Autism Spectrum Disorders.

While working with the CEBC, she has provided consultation and trainings on the utilization of the

Exploration and Preparation phase of the EPIS model to California’s Office of Child Abuse Prevention

(OCAP).

Chadwick Center for Children and Families, Rady Children’s Hospital, San Diego

The Chadwick Center is a child advocacy center with facilities located throughout San Diego County. It is one

of the largest centers of its kind and is staffed with more than 75 professionals and para-professionals in the

field of medicine, social work, psychology, child development, nursing, and education technology. The center’s

mission is to promote the health and well-being of abused and traumatized children and their families.

The California Department of Social Services (CDSS) provides leadership in targeted efforts to improve the

lives of children and families served within the child welfare system. As part of their improvement strategies,

CDSS selected the Chadwick Center for Children and Families - Rady Children's Hospital-San Diego, in

cooperation with the Child and Adolescent Services Research Center (CASRC), to create the California Evidence

-Based Clearinghouse for Child Welfare (CEBC).


A. Glossary

B. Common Implementation Strategies that Have Been Used in CWS

C. Sample Questions to Address in each EPIS phase of Implementation

D. Key Implementation Steps by EPIS Phase

E. Exploration Phase: Resources and Tools

Implementation Team Membership Tracking Tool ............................................................................... E1

The Critical Role of Implementation Teams and their Evolution through EPIS ...................................... E2

Exploration Worksheet ......................................................................................................................... E3

Identifying and Clarifying the Problem ................................................................................................. E4

Data Sources to Consider ...................................................................................................................... E5

Identifying Potential Solutions .............................................................................................................. E6

CEBC Selection Guide for EBPs in Child Welfare .................................................................................... E7

Selection Guide Worksheet ................................................................................................................... E8

Working with Program Developers ....................................................................................................... E9

Considerations when Contracting For Services ................................................................................... E10

Template for Exploration Summary Report......................................................................................... E11

F. Preparation Phase: Resources and Tools

Contracting with Program Developers .................................................................................................. F1

Data & Outcomes .................................................................................................................................. F2

Assessing Fidelity .................................................................................................................................. F3

Resources for Implementation .............................................................................................................. F4

Determining the Funding Stream .......................................................................................................... F5

Funding Stream Inventory Worksheet ................................................................................................... F6

Referral System ..................................................................................................................................... F7

Staffing Plan .......................................................................................................................................... F8

Training & Coaching Considerations ..................................................................................................... F9

G. Implementation Phase: Resources and Tools

Monitoring and Feedback Systems ...................................................................................................... G1

Reviewing the Billing/Financial Process ............................................................................................... G2

Supporting Initial Implementation- Go Live Week Checklist ................................................................ G3

Examining Outcomes ........................................................................................................................... G4

H. Sustainment Phase: Resources and Tools

Sustainable Funding ............................................................................................................................ H1

Ongoing Training and Coaching Needs ............................................................................................... H2

Maintaining Fidelity ............................................................................................................................ H3

Appendices


Appendix A: Glossary

Adaptation refers to making slight changes, planned in conjunction with the developer, to a practice

while maintaining fidelity to the core elements of the intervention in order to improve fit with client,

organization, and/or system characteristics. Conversely, it is often the case that service systems and

organizations need to adapt to the delivery standards of an evidence-based practice in order to

support implementation and sustainment with fidelity.

Adoption is the decision to make full use of an innovation.

Dissemination is the targeted distribution of information and intervention materials to a specific

public health or clinical practice audience. The intent is to spread knowledge and the associated

evidence-based interventions.

Implementation is the use of strategies to introduce or change evidence-based interventions within

specific settings.

Inner Context refers to the interplay for intraorganizational characteristics with individual adopters

and others in the organization that can support or detract from effective evidence-based practice

implementation. Strong leadership that supports the importance of evidenced-based practices in the

organization can help to promote more positive staff attitudes toward adopting evidence-based

practices.

Intraorganizational Characteristics refers to the leadership, culture, and climate of an organization.

It also includes the policies and practices that are sanctioned and supported by organization

management. Such characteristics can be important in creating a fertile environment for the

implementation and sustainment of evidence-based practices.

Observability is being able to see the process and outcomes or interim results/measures for a given

evidence-based practice.

Outer Context is also known as the sociopolitical context and refers to the larger system-wide

political, legislative, and funding environment of a service system. This can be construed at the

federal (i.e., country), state, county, or city level, depending on the nature of the service system. The

United States has certain legislative mandates that influence policies related to required services and

funding to support those services.

Scale-up refers to a staged approach to implementation in which the implementation process begins

with a portion of the organization or service system and eventually moves to complete

implementation in the whole organization or larger community.

Trialability is the extent to which an evidence-based practice lends itself to being tried out and tested

in a way that gives a determination of its applicability before deciding on a full-scale implementation.

Selecting and Implementing EBPs Appendix A: Page 1 of 1 © Apr. 2015 CEBC, www.cebc4cw.org

Appendix B: Common Implementation Strategies that Have Been Used in CWS

Several implementation strategies have been developed in health care, mental health, and social

service settings and are commonly referred to in the literature. Information on these is provided as a

general reference, as these strategies may be discussed in documents or presentations. At this time,

the science behind these approaches is, for the most part, in its infancy. Inclusion here is for

information purposes and does not imply any endorsement by the CEBC.

It is important to note that most of these implementation approaches were not developed specifically

for child welfare settings. The basic principles and premises may generalize across service settings and

there is a great deal of variability in how and where approaches have been tested.

ARC (Availability, Responsiveness, Continuity)

Developed by the Children’s Mental Health Services Research Center at the University of Tennessee.

Knoxville (http://cmhsrc.utk.edu/arc/).

ARC is an organizational intervention focused on improving the context for service delivery with a

particular focus on culture and climate of organizations, programs, and teams. The ARC intervention

includes three types of activities to improve effectiveness. First, it enacts organizational tools and

practices (e.g., feedback or teamwork) that are used to identify and address service barriers (e.g.,

bureaucratic red tape or ineffective treatment models). Second, it embeds principles of organizational

effectiveness (e.g., results orientation or participation-based events) to guide improvement efforts

within the organization. Finally, ARC modifies cognitive models (e.g., fear of failure or escalating

commitment) that hamper improvement efforts within organizations.

Breakthrough Series

Developed by the Institute for Healthcare Improvement (IHI, www.ihi.org).

The Breakthrough Series is designed to help organizations make breakthrough improvements in

quality while reducing costs. The driving vision behind the Breakthrough Series is this: sound science

exists on the basis of which the costs and outcomes of current health care practices can be greatly

improved, but much of this science lies fallow and unused in daily work. There is a gap between what

we know and what we do. The Breakthrough Series creates a structure in which interested

organizations can easily learn from each other and from recognized experts in topic areas where they

want to make improvements.

Selecting and Implementing EBPs Appendix B: Page 1 of 3 © Apr. 2015 CEBC, www.cebc4cw.org

http://cmhsrc.utk.edu/arc/

http://www.ihi.org/

CDT (Community Development Team)

Developed by the California Institute of Mental Health (CIMH) *now known as the California Institute

for Behavioral Health Solutions (CIBHS), http://www.cibhs.org/sites/main/files/file-attachments/

cdt_report_0.pdf+

The Community Development Team (CDT) Model is a multilevel training and technical assistance

strategy that has grown out of effort to promote innovation in services and operations of mental

health programs. The model is designed to promote model-adherent adoption of program and/or

operational innovations by publicly operated/administered agencies and nonprofit community-based

organizations. The CDT structure is designed so that participants are able to:

Develop a realistic and concrete implementation plan

Learn and apply clinical or technical information about a specific innovation

Overcome implementation barriers

Evaluate program performance outcomes

Develop capacity to sustain model-adherent programs

A CDT is composed of a group of counties or agencies that are committed to implementing a new

practice in common. Training and technical assistance are provided, in partnership with model

developers, through a series of multiagency meetings and augmented by individualized agency

specific assistance as needed.

Dynamic Adaptation Process (DAP)

Developed by Gregory A. Aarons, PhD (available via email at [email protected])

DAP is designed to allow for evidence-based practice (EBP) adaptation and system and organizational

adaptations in a planned and considered, rather than ad hoc, way. The DAP involves identifying core

elements and adaptable characteristics of an EBP, then supporting implementation with specific

training on allowable adaptations to the model, fidelity monitoring and support, and identifying the

need for and solutions to system and organizational adaptations.


http://www.cibhs.org/sites/main/files/file-attachments/cdt_report_0.pdf

http://www.cibhs.org/sites/main/files/file-attachments/cdt_report_0.pdf

QUERI (Quality Enhancement Research Initiative)

Developed by the Veterans Affairs’ Health Services Research and Development Service (http://

www.queri.research.va.gov/)

QUERI works to improve the quality of healthcare for veterans by implementing research findings into

routine clinical practice. QUERI is a detailed process for testing and implementation of quality

improvement and EBP in the Veterans Affairs (VA) system:

Identify high risk/high prevalence diseases or problems

Identify best practices

Define existing practice patterns and outcomes across the VA and current variation from best

practices

Identify/implement interventions to promote best practices

Document that best practices improve outcomes

RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance)

Developed by Russ Glasgow, Shawn Boles, and Tom Vogt (www.re-aim.hnfe.vt.edu/index.html).

RE-AIM is a conceptual model to help identify key factors to implementation. It is a systematic way for

evaluating public health interventions that assesses 5 dimensions: Reach, Efficacy/Effectiveness,

Adoption, Implementation, and Maintenance. These dimensions occur at multiple levels (e.g.,

individual, clinic, organization, community) and interact to determine the public health or population

based impact of a program or policy.

Reach is the absolute number, proportion, and representativeness of individuals who

participate in a given program.

Efficacy/Effectiveness is the impact of an intervention on important outcomes. This includes

potential negative effects, quality of life, and costs.

Adoption is the absolute number, proportion, and representativeness of settings and staff who

are willing to offer a program.

Implementation, at the setting level, refers to how closely staff members follow the program

that the developers provide. This includes consistency of delivery as intended and the time

and cost of the program.

Maintenance is the extent to which a program or policy becomes part of the routine

organizational practices and policies. Within the RE-AIM framework, maintenance also applies

at the individual provider level.


http://www.queri.research.va.gov/

http://www.queri.research.va.gov/

http://www.re-aim.hnfe.vt.edu/index.html

Appendix C: Sample Questions to address in each EPIS phase of Implementation

Exploration Phase:


How will the new intervention be delivered – directly by

child welfare agency staff, by an external agency through

contract, etc.?

What are the pre-existing knowledge/skills within the

agency providing the evidence-based practice (EBP)? For

example, what is the education level of the staff at the

agency, and will this impact the program decision? Some

EBPs require certain types of staff (e.g., master’s level,

nurses, etc.).

What is the readiness for change of the agency(s) that

will be implementing the EBP? Agencies may want to

consider the agency’s culture, climate, or readiness for

change.

What is the current culture (i.e., beliefs and shared

expectations) of the agency in relation to EBPs,

innovations, and change?

What is the current climate of the organization (i.e., shared perceptions of the psychological

impact of the work environment on the provider) and does this support implementation of a new

EBP? If not, how will this be addressed?

Example: The agency has recently undergone a major change to the casework practice model,

resulting in increased staff turnover. It may be necessary to assess whether implementing

another new program is appropriate at this time.

Are there past successes or failures in implementing new practices? What lessons can be

learned from those past experiences that will help in this new implementation process?

Example: A new home visitation program was implemented 5 years ago using grant funding.

Although the program appeared to have had the desired outcomes, after the grant ended, the

program was discontinued.

Note: If the intervention will be

contracted out, then consider a

few options: 1) Determine the

underlying problem that needs

to be addressed and choose a

single EBP that will be required

or 2) Determine the underlying

problem that needs to be

addressed and choose more

than one potential EBP then ask

the applying agencies to answer

some of the following questions

to support the single EBP that

they choose.

Selecting and Implementing EBPs Appendix C: Page 1 of 6 © Apr. 2015 CEBC, www.cebc4cw.org

Individual Adopter Characteristics

Are there early adopters/innovators in the organization who are likely to act as champions for a

selected EBP? How can these individuals be involved in the implementation process to ensure

success?

What are the attitudes and perceptions of the need to adopt an EBP by the individuals who will

be providing the EBP?

Leadership

How does leadership within the agency perceive the need for EBPs? Does the support for this

program to be implemented exist in all levels of leadership (e.g., executive, program, supervisory,

and peer leaders)? If not, how will the leadership support be developed?

Funding

What are the current budget restrictions and how might those impact exploring new programs?

How will the program be funded (e.g., service grants, research grants, foundation grants, health

insurance, county or state funding streams, etc.) and how stable is that funding source?

What types of services can be funded under existing funding streams?

If the funding is through a time-limited source, such as a grant, how will future funding be

secured to allow for sustainability of the program?

Is there the need to partner with other agencies to blend funding for services?

Client Advocacy

What role does the mental health council, parent partner organizations, or other family advocacy

groups play in the community and how will these stakeholders be engaged in the

implementation process?

What are the current service demands from clients and how are these impacting individual

providers?


What professional stakeholders need to be involved in this process?

How are the networks set-up? Are there existing interorganizational network structures of

meetings in place that can be utilized?

What resources exist for implementation support? What can be accessed currently or will be able

to be accessed in the future?

Examples: Other counties that have implemented in the past, the developers of the chosen

EBP, technical assistance centers, universities, etc.


Does the agency which would provide the EBP have experience implementing EBPs and does it

interact with other agencies that provide EBPs? (This type of network increases likelihood of

adopting an EBP.)

Does the child welfare system contract with other departments that have successfully

implemented EBPs? Are there lessons that can be learned from these successes? (Again, these

types of connections assist in success in adopting EBPs.)

Sociopolitical Context

Are there mandates at the state or federal level that may impact funding and/or the focus of new

programs in the short and long term?

Are there legal requirements in place for the implementing agency that may impact the decision-

making process? For example, are there any lawsuit settlements or consent decrees that require

specific changes or actions by the agency?

What outcome requirements and benchmarks exist for the child welfare programs and how

might this impact choice of practice for the community?

What type of monitoring currently exists or will need to exist to allow outcomes to be collected?

For example, is there existing state or federal data reporting requirements to examine outcomes?

Preparation Phase:


How well does the EBP match the mission, values, and service provider tasks of the agency?

Are there specialized roles within the organization that lend themselves to the adoption of the

selected EBP?

What challenges/benefits does the size and infrastructure of the agency have on the

implementation of the selected EBP?

What is the size of the staff? Is it adequate to successfully implement the program?

Does the chosen EBP require specific expertise or knowledge of the program that is being

implemented?

How will referrals be supported and/or enhanced so that service providers will have clients to

work with?

How will outcomes be assessed to confirm that the EBP is meeting the needs of the community

and is effective in addressing the underlying problem identified during exploration?

How is fidelity being assessed as the EBP is implemented and on a continuous basis to avoid

future drift?



What role are the early adopter/champions playing in preparation to ensure buy-in from

providers?

How will feedback from providers be obtained as the practice is implemented?

Funding

Are there existing contracts that will be shifted to require EBPs? How will support be provided to

the agencies to ensure that there are successful applicants?

Leadership

How will the agency leadership support the adoption of the EBP?

Client Advocacy

Are there local and/or national advocates who are supporting the EBP? How might they support

the agency’s efforts?

How will client feedback/satisfaction be measured?


Does it make sense to fund more than one agency so that the agencies can partner to share the

costs of training and/or technical assistance?

Sociopolitical

Is there current legislation that is influencing funding for specific services that are related to the

chosen EBP?

Implementation Phase:


Are there formalized policies in the agency for supporting the use of EBPs? Is this part of the

values and mission of the agency?

Are there clear priorities and goals related to EBP?

Based on the agency’s culture (i.e., the implicit norms and assumptions of a work group that

guide behaviors), will the chosen EBP be accepted and implemented with fidelity?

How does the EBP fit with other administrative and practice needs (e.g., record-keeping,

productivity requirements, etc.)? What will need to be modified or changed as the EBP is

implemented?


Is fidelity being monitored based on the plans developed in the Preparation Phase, are

adjustments to the plan necessary now that implementation is underway?

Are outcomes being assessed based on the plans developed in the Preparation Phase?

Are there information technology (IT) or management information system (MIS) resources to

provide a data system for monitoring process indicators, client outcomes, and fidelity?


What is the feedback from providers about the training, coaching and implementation and how is

it being addressed?

Intervention Developers

How involved will the intervention developers be in the implementation (i.e., are there specific

trainers and consultants who will play a role as the program rolls out)?

Leadership

How will agency leadership promote a positive implementation climate and support the staff in

implementing the EBP? Encouragement and reinforcement will be key.

Funding

How will funding be obtained for necessary resources (e.g., computers, materials, etc.)?

Client Advocacy

How is the plan for obtaining client feedback working and what is being done with that data?


How can agencies learn more about successes and challenges that other agencies have

encountered when implementing an EBP in order to help learn how to address implementation

challenges that arise?

How can interorganizational networks assist in facilitating appropriate referrals, subcontracting

arrangements, and training opportunities/knowledge about EBPs?

Sustainment Phase:


Is there strong buy-in by agency leadership to help create a climate conducive to continued use

of the EBP?

Are there policies in place to sustain EBPs within the agency?


Is the EBP used consistently by the staff who have implemented it?

Is there social network support for the use of the EBP?


Are there specific selection criteria for new staff when turnover occurs?

Do the staff selection criteria include knowledge, skills, abilities, attitudes and other

characteristics important to effectively learning and delivering the selected EBP?

Fidelity monitoring and Support

Is fidelity being monitored internally by agency staff or tracked by the contracting agency?

Is training support reliant on external expertise or have internal trainers been appointed? If

internal, how has this been institutionalized to support sustainment?

Leadership

How will leadership continue to support the staff who have implemented the practice especially

as funding changes or new staff is brought on?

As turnover occurs, how will they identify new staff and support training for this new staff?

How will leadership support be managed during future agency or program leadership changes?

Funding

Will there be additional funding to sustain the EBP? If not, how will agencies be supported to find

and secure additional funding to sustain the EBP?

How will funding be obtained to compensate for staff turnover and the need for additional

training?


Is there a mechanism for continued involvement of multiple stakeholders?

Is there a mechanism for ongoing troubleshooting and problem solving across the various

partners?

What ongoing technical support is in place across the network?

What role can external networks play in ensuring continued support for the EBP during periods of

leadership or political transition or financial challenges?

Sociopolitical

Are leaders in the community supportive in creating policies that will sustain the EBP?


Appendix D: Key Implementation Steps by EPIS Phase

Exploration:

Form an Implementation Team

Identify the Problem

Narrow the Focus

Conduct a Needs Assessment

Identify Potential Solutions

Determine Program Fit

Create a Written Summary

Preparation:

Ensure Leadership Buy-In

Develop an Implementation Support System

Work with Stakeholders

Ensure that the Chosen EBP fits with Consumer Concerns

Identify Viable Funding Streams

Develop Timetables

Implementation:

Verify Buy-in

Ensure Priority

Complete Training

Prepare Materials

Confirm Referral Processes

Monitor Fidelity to the EBP

Collect and Evaluate Outcomes

Explore Scale-up in the Service System(s)

Sustainment:

Funding and Support

Ongoing Training Needs

Ongoing Fidelity Monitoring

Outcomes

Making Refinements

Reviewing Referral Process

Selecting and Implementing EBPs Appendix D: Page 1 of 1 © Apr. 2015 CEBC, www.cebc4cw.org

Appendix E: Exploration Phase: Resources and Tools

Implementation Team Membership Tracking Tool .................................................................................. E1

The Critical Role of Implementation Teams and their Evolution through EPIS ........................................ E2

Exploration Worksheet ............................................................................................................................. E3

Identifying and Clarifying the Problem .................................................................................................... E4

Data Sources to Consider ......................................................................................................................... E5

Identifying Potential Solutions ................................................................................................................. E6

CEBC Selection Guide for EBPs in Child Welfare....................................................................................... E7

Selection Guide Worksheet ..................................................................................................................... E8

Working with Program Developers .......................................................................................................... E9

Considerations when Contracting For Services ...................................................................................... E10

Template for Exploration Summary Report ........................................................................................... E11

Selecting and Implementing EBPs Appendix E: Table of Contents © Apr. 2015 CEBC, www.cebc4cw.org

Selecting and Implementing EBPs Appendix E1: Page 1 of 3 © Apr. 2015 CEBC, www.cebc4cw.org

Implementation Team Membership Tracking Tool

Note: A PDF version that can be typed into and saved is available on the CEBC website (www.cebc4cw.org). From the home page, click on the Select and Implement Programs button, then the Tools & Resources button, and then Technical Assistance Materials from the list.

A. Senior Child Welfare Administrator Overseeing Project Name: ________________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

B. Day-to-day Child Welfare Liaison for Project (Key Contact)

Name: ________________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

C. Other Key Child Welfare staff (relevant to topic areas being examined)

Name: ________________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

Name: ________________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________


Name: ________________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

Name: ________________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

Name: ________________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

D. Stakeholders (consumers, advocacy groups, etc. working in this topic area)

Name: ________________________________________________________________________

Agency/Role: ___________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

Name: ________________________________________________________________________

Agency/Role: ___________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________


Name: ________________________________________________________________________

Agency/Role: ___________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

E. External Providers/Contractors (primary contact for each main provider) Name: ________________________________________________________________________

Agency: _______________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

Name: ________________________________________________________________________

Agency: _______________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

Name: ________________________________________________________________________

Agency: _______________________________________________________________________

Title: _________________________________________________________________________

Email: ________________________________________________________________________

Telephone: ____________________________________________________________________

The Critical Role of Implementation Teams and their Evolution through EPIS

Purposeful, active, and effective implementation work is done by implementation teams.

Implementation team members have special expertise regarding programs, implementation science,

improvement strategies, and organizational capacity building. They are accountable for making it

happen: for assuring that effective interventions and effective implementation methods are in use to

produce the intended outcomes. The roles and responsibilities of implementation team members

need to evolve through the phases of EPIS.

In the Exploration Phase, the Implementation Team must include those who have authority to make

decisions about which practice to choose to implement. In addition, membership should include

those with content expertise in the area being explored (for instance, if the area being explored is

Mental Health, then it would be important to have representation of those in child welfare who

oversee program/policy management of mental health contracts, as well as leaders in mental health

at the table). Relevant stakeholders (e.g., providers, advocates, family members, etc.) should be

included in the Exploration process. Finally, members who are not decision makers, such as front line

staff, provide information on the practicalities of implementing a new practice. Examples might be

infrastructure, workforce, and caseload requirements that will impact the implementing organization.

Once a specific practice has been identified and the agency has moved into the Preparation Phase,

membership in the Implementation Team may need to be adjusted. During this phase, work shifts to

the development of timelines and contracts; much of the work will focus on identifying and

addressing potential stumbling blocks. A change agent or coordinator should be identified; this

person’s role may include day-to-day leadership of the Implementation Team. Leaders in the

agencies implementing the new practice, as well as those responsible for training and quality

assurance, will have a key role in preparing for active implementation. Depending on the practice, a

representative from the program being implemented might also be added to the team, or included on

an as-needed basis. During this phase, the group should meet on a regular basis to keep the process

moving forward.

The Implementation Phase begins as the providers begin to deliver the new program and, at this

point, the Implementation Team will require members who have both authority and responsibility to

provide quick resolution to problems encountered in the implementation process. Leadership roles

should be clear and agreed to by all. At least one member, typically the change agent, should be

identified who will have responsibility for monitoring and communicating difficulties and successes in

a timely fashion in order to insure quality implementation. During this phase, the Implementation


Team may need to be available to meet via telephone on short notice, or hold virtual discussions and

resolve issues via email.

During the Sustainment Phase, the Implementation Team should be expanded to include members

who represent additional agencies that will be implementing the practice, as well as representatives

from new funding sources who have been identified, such as local foundations. Members should also

include quality assurance and training staff from the implementing agencies, to ensure that fidelity

and outcomes continue to be a focus. During this phase, the group may meet less frequently than in

previous phases – the new program has become ingrained in the service system and challenges and

changes are less common.



Exploration Worksheet

This is a sample format to track basic information essential to the Exploration Phase; it can be

modified to fit individual needs. It may be used with the Implementation Team during initial meetings

or prior to the creation of the Implementation Team to help guide its formation.


Discussion of who will be involved (county level, partners in the county, region/geographic area,

etc.)

Will this be a countywide or regionalized effort?

Who do you see being involved in this? Who are the stakeholders from within the county and

who would partner with the county that would be involved?

Do you foresee challenges in engaging community-based organizations (CBOs) or other

partners in this process?

Brief explanation of your agency and leadership’s motivation for starting the Exploration Phase

What do you hope to accomplish throughout this process?


Briefly discuss and review the process for achieving these goals:

Information on any major linkages with other public service sectors (e.g., Probation, Mental Health,

etc.)

What joint projects, requests for proposals (RFPs), etc., are in place or projected to be in place

that may impact this effort?

Information on existing evidence-based practices (EBPs) and ones that have been identified as

possibilities for implementing in the county

Are you aware of any EBPs already in use within your county? If yes, please list them.

How would you find additional information about what is currently in use?

Have there been previous attempts to implement EBPs in the past?


Are there any specific EBPs for implementation that you have in mind?

Recent or upcoming changes to the system

Are there any upcoming changes to the system that might impact the assessment process or

implementation of a new practice?

Review of timeline and personnel

What is your timeline for this effort?

Who will be leading this process? Are they at a level where they will be able to generate

staff/stakeholder buy-in?


Identifying and Clarifying the Problem


What is the primary problem? (2-3 sentences)

What data do you have to help understand the problem and its causes?

What do you think the main factors are that drive it?

What are you currently doing to address the problem?

Using the existing data, review the following areas:

Target Population

1) Who is affected?

2) What are the ages of the children impacted?


3) Is it disproportionally impacting certain racial or ethnic groups or other specific demographic

groups (special needs, medically fragile, etc.)?

Time Frame

4) When in the child welfare process is it happening (before child welfare involvement, at referral,

reunification, etc.)?

Location

5) Is there a geographic area that is most affected by the problem? (clustered in one region of the

county, spread throughout, etc.)?

6) What type of delivery setting is most conducive to the area identified above (in home, school

based, clinic based, etc.)?

Data Sources to Consider

A key step in the Exploration Phase is to clearly identify the problem areas of interest. Examining

existing data sources, and collecting new data as needed, is crucial to ensure that the problem is well

understood and correctly addressed. Agencies often have access to large quantities of data, but may

not be sure how to utilize it effectively. Below are some suggested data sources.

1) Reports available from CWS-CMS through the California Child Welfare Indicators Project

(CCWIP) *University of California at Berkeley (UCB) and the California Department of Social

Services (CDSS), http://cssr.berkeley.edu/ucb_childwelfare/ReportDefault.aspx+ -

This is a good site to start with, as descriptive data on areas of interest can be examined, and

very little training is required.

The reports cover many different topic areas.

Reports can typically be generated by age, gender, and racial/ethnic groups.

Comparisons can be made between counties and the state, as well as across time periods.

2) Existing annual reports - Board of Supervisors reports, Board of Directors reports, etc.

3) Data from Community Self-Assessments (CSAs) and System Improvement Plans (SIPs)

4) Data from Structured Decision Making (SDM) or other decision support systems

5) Reports to funders - Foundations and other funding groups typically require periodic reports on

process and outcomes which may contain relevant information.

6) Other databases to which data is being provided - Efforts to Outcomes, First 5 databases, etc.

7) Data on referrals to outside providers (e.g., mental health, substance abuse, etc.) - if these are

tracked

8) Research projects - Has the agency participated in any research projects, or authorized any

research with their clients, that may have applicable datasets?

9) Chart reviews - These can be done on hard copy or electronic charts by examining a subset of

charts, randomly selected from a specific population (e.g., children, ages 0-3, in out-of-home

care over 30 days, etc.).

10) Focus groups - These can be conducted on the target population to further refine the problem

area. Also, it is important to determine if any relevant focus groups have been done, either by

the agency or involving agency clients.


http://cssr.berkeley.edu/ucb_childwelfare/ReportDefault.aspx

11) Surveys - Are there existing surveys that may have information of interest? Existing surveys

such as the Youth Risk Behavior Survey (YRBS) and the California Health Interview Survey (CHIS)

may be sources of relevant county-level data. The Census surveys also provide relevant

information and can be publicly accessed. Many agencies obtain satisfaction surveys from

clients on a periodic basis that may contain useful information. Finally, it may be necessary to

conduct a brief survey to obtain specific information.


Identifying Potential Solutions

A. List key words that describe the program you are looking for:

B. List the relevant CEBC (www.cebc4cw.org) topic areas to examine (from the home page, click on the View Programs button and

then Topic Areas on the sidebar on the left):

C. List of Potential Programs – Using the table on the next page, fill in the basic information from the CEBC as a starting point for

discussion. List one program per row. Feel free to make copies of the table if more than two programs are being considered. Note: A PDF version that can be typed into and saved is available on the CEBC website. From the home page, click on the Select and Implement Programs button, then the Tools & Resources button, and then Technical Assistance Materials from the list.

Selecting

an

d Im

plem

entin

g EB

Ps

A

pp

en

dix E6

: Page

1 o

f 2

©

Ap

r. 20

15

CEB

C, w

ww

.cebc4

cw.o

rg

Program Name Brief Description Goals Target Population Rating

Notes:

Notes:

Selecting

an

d Im

plem

entin

g EB

Ps

A

pp

en

dix E6

: Page

2 o

f 2

©

Ap

r. 20

15

CEB

C, w

ww

.cebc4

cw.o

rg

CEBC Selection Guide for EBPs in Child Welfare

The California Evidence-Based Clearinghouse for Child Welfare (CEBC) was designed to provide clear

and concise information about practices that are commonly used in child welfare. Each practice is

rated for both the level of scientific evidence as well as its relevance to child welfare. Selecting a

practice that is a good fit with one's organization goes beyond choosing a practice that is

scientifically rated a "1" on the website. This guide is designed to assist child welfare professionals

in selecting which practices to implement in their agency.

The information in this guide is based on the work of Trisha Greenhalgh and her colleagues, who

conducted a systematic literature review that addressed the question: "How can we spread and

sustain innovations in health service delivery and organization?" Using the key findings of this

review, we have designed a guide to help make critical decisions about selecting a practice for

implementation.

Please note that there is an Implementation Information section in the CEBC Program Description

for each program that is rated a 1, 2, or 3 on the CEBC Scientific Rating Scale. The section includes

information on Pre-Implementation Materials, Formal Support for Implementation, Fidelity

Measures, and Implementation Guides. These are additional items that may be helpful to consider

when comparing different programs for selection.

This Guide is meant to assist in leading discussion about the selections process. It is suggested that

it be used in conjunction with the Selection Guide Worksheet (Appendix E8) to note information

from the discussion on each of the practices being considered.

Ease of Use

Key Questions:

How complex is the program?

How easily will the key players be able to understand the practice?

Will the complexity make it more difficult to describe the practice to stakeholder and key

internal and external partners?

Can the program be broken down into smaller, more manageable parts for implementation?

CEBC Program Description Sections to Review:

Essential Components

Recommended Parameters (Duration and Intensity)

Identified Resources Necessary to Implement Program


Education and Training Resources

Real World Example:

An agency explored multiple parent training programs and decided to choose the one that would be

easiest for the staff to understand. The one they chose had the option to train the staff on the

program in segments, learning new skills a few at a time, as opposed to training the staff on the

whole program and learning all the new skills at the same time.

External Compatibility

Key Questions:

How compatible is the practice with the beliefs and values of the local community and

clients?

Is the practice compatible with the referral sources currently in place in the community – will

they feel comfortable referring clients to it?


Target Population


Recommended Parameters

Delivery Setting

Languages

CWS Relevance

Relevant Research (look at the types of populations involved in the research – how similar

are they to the desired target population?)

Real World Example:

A community has been struggling with finding services that will engage parents in treatment for

their substance abuse issues. After holding a stakeholders meeting and discussing possible

treatment programs, it is agreed that adding Motivational Interviewing (MI) into the existing

substance abuse programs would help to increase parents' engagement in substance abuse

services.

Financial Considerations/Relative Advantage

Key Questions:

What financial resources to fund the practice exist, both in the short and long term?

What is the cost for training and consultation?


Does the practice have a clear advantage for the organization, in terms of efficiency or cost-

efficiency, compared to what is currently being done?




Identified Resources


Real World Example:

After implementing Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT), a short-term

therapeutic intervention, the organization was able to better serve its clients by decreasing the time

clients waited to receive services.

Internal Compatibility

Key Questions:

How does the practice fit with the agency/workforce norms, values, and beliefs?

Will it require a radical change in thinking or process?

How much change will be required of the existing workforce – training, new processes, new

forms, etc.?


Target Population



Delivery Setting

CWS Relevance

Relevant Research (look at the types of populations involved in the research – how similar

are they to the desired target population?)

Real World Example:

Therapists at an agency have been trained in psychodynamic techniques and have a long history of

using psychodynamic approaches with clients. The agency director is considering implementing

Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT). Based on the clinical background of the

therapists, this could be a difficult transition and would have to be explored in depth to determine if

the therapists would be willing to make a radical shift in their treatment approach.


Knowledge Requirements

Key Questions:

How much training is required to implement the program?

Can the skills required to deliver the program be applied in other contexts in which the staff

works (e.g., can home visiting skills be applied to routine case work practice)?



Minimum Provider Qualifications


Real World Example:

An agency with only two full-time staff and multiple interns is considering having the staff trained in

Triple P, but decides that it would be too difficult to have that knowledge transferred to the next

group of interns once the current interns leave.

Match of Skill Set

Key Questions:

What education level or pre-existing skill set is required for staff?

How does this fit with the existing workforce in the community?

Are staff with the appropriate skill set/education level available to recruit?



Minimum Provider Qualifications

Real World Example:

An agency's existing workforce is composed primarily of social workers. The agency is interested in

implementing a home visiting program. Despite the high level of research evidence for the Nurse

Family Partnership (NFP) program, a decision is made to not select NFP since a nursing degree is a

minimum qualification for providers. This would not be a good match of skill set.

Observability of Benefits

Key Questions:

Are the outcomes of the program, either short or long term, easily observable?

How soon can results be seen (e.g., how long is the program)?


How will program results and outcomes be measured and does this measurement fit with

the existing data collection or outcomes process?


Relevant Research (look at the outcomes that were examined and how they were measured)

Recommended Parameters: Duration and Intensity

Real World Example:

An agency looks at Parent-Child Interaction Therapy (PCIT) and presents the findings of the

research, as well as videos of a session, to the staff. The staff is excited by the idea of implementing

a practice that shows an appreciable benefit so quickly.

Reinvention/Adaptability

Key Questions:

Can the practice be adapted, refined, or modified to meet local needs?

Will this adaptation influence the fidelity and outcomes?


Target Population

Brief Description (does it list any existing adaptations?)

Relevant Research (has research been done on any adapted or modified versions?)

Contact Information (Need to confirm with developers what adaptations, if any, are possible)

Real World Example:

An agency is interested in implementing Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT),

but does not have the ability to conduct 90-minute therapy sessions. After consulting with the

program developers, they are able to modify the sessions to fit their standard 50-minute therapy

session by extending the number of total sessions that clients attend.

Risk

Key Questions:

How risky is the adoption of the program?

Is there a large cost upfront (e.g., training, supplies, licenses, etc.)?

How big a change would this be to individual worker practice, as well as that of the

organization as a whole (i.e., a radical change may feel more risky)?


What are the potential costs and benefits of implementing the practice? How strong is the

research support for the program?




Identified Resources

Relevant Research

Scientific Rating

Real World Example:

A small agency is considering whether to implement Parent-Child Interaction Therapy (PCIT). After

considering the costs of training their staff and remodeling their offices to allow for this practice,

they decide that there is too high a risk of spending a lot on training and construction and not

getting enough referrals to justify this cost. The agency decides to use a parent training program

that requires fewer resources.

Training/Support

Key Questions:

How much training and consultation is required before the program can be delivered?

How will current staff be trained (e.g., impact on caseloads, time off for training, etc.)?

Is the training for the practice currently available, or if there is a waitlist, will it be available

in the timeframe necessary for it to be implemented?

Does ongoing consultation get transferred from the trainers to the agency level to continue

support once the initial training is finished, or will there be a need to contract for ongoing

training?

What is the process for new staff to get trained when turnover occurs?



Implementation Information

Contact Information (Need to confirm with developers the specific regarding training and

support costs)

Real World Example #1:

A small agency in a rural area reviews the training costs and availability and uses this information


to select a practice where training is provided on-site. It is determined that this is more efficient

than sending their workers to training off-site. This on-site trainer will also provide phone

consultation as a follow-up to the training, which will allow staff to get further support.

Real World Example #2:

An agency director is interested in implementing an evidence-based parent training practice. After

contacting the program about training, it is discovered that there is a long wait-list for being trained

in this practice. The agency director then explores the training requirements for a second evidence -

based parent training program and after contacting the program about training, discovers that

there is not a wait for being trained in this practice. This leads to the agency implementing the

second practice, because they are able to get the needed support and training in a timely manner.

Trialability

Key Questions:

Does the practice lend itself to being tried out on a small scale before a full implementation

takes place?

Is it possible to attend a training session, review program manuals, or visit another agency

implementing the program prior to making a decision?



Contact Information

Real World Example:

After talking to the program developer, the agency has one therapist trained in Motivational

Interviewing (MI). Once the therapist is comfortable and it is clear that MI is benefiting the clients,

then additional therapists in the agency are trained.

Reference: Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systemic review and recommendations. The Millbank Quarterly, 82(4), 581-629.


Selection Guide Worksheet

Use this worksheet to document information discussed while reviewing the CEBC Selection Guide for EBPs in Child Welfare [Appendix E7].

Area Program Name: Program Name: Program Name:

Ease of Use

External Compatibility

Financial Consideration

Selecting

an

d Im

plem

entin

g EB

Ps

A

pp

en

dix E8

: Page

1 o

f 4

©

Ap

r. 20

15

CEB

C, w

ww

.cebc4

cw.o

rg


Internal Compatibility

Knowledge Required

Match of Skill Set

Selecting

an

d Im

plem

entin

g EB

Ps

A

pp

en

dix E8

: Page

2 o

f 4

©

Ap

r. 20

15

CEB

C, w

ww

.cebc4

cw.o

rg


Observability of Benefits

Reinvention/ Adaptability

Risk

Selecting

an

d Im

plem

entin

g EB

Ps

A

pp

en

dix E8

: Page

3 o

f 4

©

Ap

r. 20

15

CEB

C, w

ww

.cebc4

cw.o

rg


Training/Support

Trialability

Selecting

an

d Im

plem

entin

g EB

Ps

A

pp

en

dix E8

: Page

4 o

f 4

©

Ap

r. 20

15

CEB

C, w

ww

.cebc4

cw.o

rg

Working with Program Developers

It is crucial to engage in discussions with program developers in advance of the selection and

throughout the process of adoption of a specific program or practice.

During the Exploration Phase, meetings with the developers (likely via telephone or web) will help

determine whether the program is a good fit for the agency’s needs. Issues such as total cost,

availability of and lead time for scheduling training and consultation, availability of fidelity measures,

and identification of the primary developer contact should be addressed during these meetings to

clarify if the program will meet local needs.

For some programs, the developer is the only one that provides training and consultation, and it is

necessary to work with them directly. Some developers use a train-the-trainer model, in which a

small number of staff from the implementing agency are trained by the developer and they, in turn,

train the rest of the staff in the local agency, while other developers establish a more loosely

organized pool of qualified trainers or consultants across the country who are certified or approved to

provide training on their behalf. Finally, other programs have established a separate organization

with multiple trainers and consultants whose sole focus is on the implementation of the practice. It is

important to clarify who will be providing the training and who will be supporting overall

implementation of the practice.

Once a practice has been chosen and the Preparation Phase begins, agreements will need to be

developed with the person or group who will be providing training and consultation (refer to the

Contracting with Program Developers *Appendix F1+ resource for additional information).

Below are some questions to consider during both the Exploration and Preparation Phases:

Who will be training/consulting and what exactly will they provide?

1. Will this person/people be a good fit for the agency’s needs?

2. How available will they be?

3. How easy are they to work with?

4. Where will the training take place? (Is it only held in a certain place or does the trainer travel

to the recipients?)

What are the program and personnel costs?

1. Ask the developer to specify implementation costs. Be sure to cover all of the relevant issues.

Cost for staff training including travel

Program materials


Additional equipment such as video recording devices or lap top computers

Technical assistance

2. Costs related to infrastructure should also be considered. Examples include:

The education, training, and experience required for staff to deliver the program

Computer program or internet access

Caseload standards that may be dramatically different from the usual standard of care

Documentation requirements that may necessitate either a new system or, in some cases,

double work for staff

How will program fidelity be ensured?

1. Delivering the program with fidelity is critical to achieving desired outcomes; therefore a

thorough understanding of what constitutes fidelity for the program prior to implementing is

advised:

Ask for specification of the core components.

Ask about the developer’s experience implementing with diverse populations including

accommodations/adaptations to insure client participation.

Clarify the mode of service delivery. For example, if the program is delivered in a group, is

there an adaption for delivering to an individual or a family?

Clarify the caseload standard. Many developers are willing to make accommodations if

discussed in advance.

Ask for specification regarding fidelity criteria including review of any tools used to

measure fidelity.

Planning for sustainment

1. Engage in a candid discussion regarding proprietary issues. Many EBPs are “owned” by the

developer and cannot be used unless there is an ongoing relationship with the developer.

There may be additional costs to maintaining the program including access to resources.

2. Ask about train-the-trainer options or other strategies that may help sustain the program in

the organization. Be clear about costs and criteria.

3. Discuss how future staff can get training and if there is availability of booster training or

support beyond the initial implementation period.


Considerations when Contracting for Services

Request for Proposal (RFP)/Contract for a Specific Program (County completed Exploration Phase

and identified specific program)

RFP Should Contain:

Background on program and selection process of the program

Detailed description of goals and target population of the program

Information on the selection process for the identified program

What the RFP will and will not fund

Training for the program

Service delivery

Materials

Staffing

Minimum provider qualifications for chosen program

How much start-up time to have staff hired/transitioned or training, etc., will be allotted?

Training Logistics

Is the training being provided by the contractor or is this something that the agency will

need to arrange?

If the contractor is providing, what is timeframe for training to take place and where?

Resources

Clearly spell out the necessary resources for implementing the program – AV equipment,

manuals, etc. Ask respondents to indicate which of these resources will need to be

obtained (for instance, if the program is a mental health therapy program, the agency may

already have existing therapy supplies that are available)

Fidelity monitoring

Specifically determined by the contract or up to responding agency to propose

Referrals

How will referrals be obtained?

Outcomes

Need to develop measureable outcomes based on goals of program


Process outcomes (e.g., number of clients to be served) vs. client outcomes (e.g., amount

of improvement on parenting measure)

Use of standardized tools

Sustainability

Expectation for service being continued beyond the terms of the contract

How will staff turnover be addressed, or request respondents to provide information on

how turnover will be addressed.

RFP Response Should Contain

Background

Agency experience using any similar programs

Agency experience with implementing EBPs

Staffing

Will agency hire all new staff or be retraining existing staff? How will this impact start-up

time?

If keeping current staff, review resumés to ensure they meet minimum qualifications.

What will be the timing of transition to new services if currently delivering a different

service?

If hiring new staff, what will be qualifications, job description, etc.? What is timeline of

hiring?

Training Logistics

(If not being arranged or funded by contractor) Initial training – how will this be delivered

(on-site, at program developer’s site, etc.) and funded?

Ongoing training when turnover occurs and new staff is hired – how will this be delivered


Resources

Clearly respond to specific resources that will be needed – AV equipment, manuals, etc. –

and how will this be funded (through this contract specify budget, other funding)?

Referrals

Provide information on referral pathway.

Fidelity monitoring

If not specified by the RFP, propose how this will be maintained.


Outcomes

How will these be collected?

Sustainability

Funding, staffing, service delivery beyond contract time, etc.

RFP/Contract where potential contractor selects program (County began Exploration Phase,

identified the problem, but did not narrow down the programs)

RFP Should Contain:


Clearly state the identified problem and target population (demographics including age,

gender, cultural backgrounds when appropriate, etc.)

May suggest potential programs which should be thoughtfully chosen through the

exploration process

Provide requirements for chosen programs (level of research, rating on CEBC or other

clearinghouses, rationale for choosing)

What the RFP will and will not fund

Training for the program

Service delivery

Materials

Outcomes

What are concrete outcomes that are expected?

RFP Response Should Contain:


Clear rationale for proposed program and why it will impact the identified problem.

Identify CEBC, NREPP, or similar rating of program – using criteria to show rating if

program not rated on one of the sites.

Training

Information on how training and program materials (manuals, videos, etc.) will be funded,

obtained, etc.

Initial training – how will this be delivered (on-site, at program developer’s site, etc.) and

funded?


Ongoing training when turnover occurs and new staff is hired – how will this be delivered


Staffing

Minimum provider qualifications for chosen program and corresponding job description

How much start-up time to get staff hired or transitioned, training, etc., will be allotted?

Information on how active implementation will occur including who will lead the effort and

what support exists for the chosen practice and how that support will be obtained.

Plans for outcomes assessment

Plans for fidelity monitoring

Plans for sustainability and addressing turnover issues

Resources

Clearly indicate the specific resources that will be needed – AV equipment, manuals, etc.,

and how these will be funded (through this contract specify budget, other funding)?

Referrals

How will referrals be obtained? Provide information on referral pathway.

Fidelity monitoring

How this will be maintained?

Outcomes

How will these be collected?

Sustainability

Funding, staffing, service delivery beyond contract time, etc.


Template for Exploration Summary Report

Assessment and Planning Initiative in XXXX County: A Road Map for Implementation of Evidence-Based Practices

Executive Summary

Overview of Existing Practices

A. Process/Methods

How was data obtained?

Who was surveyed and/or interviewed?

B. Existing practices – list separately for each practice examined

Description

Target population

Goals and Objectives

List of providers

Provider responsibilities (may be listed in contract)

Summary of experience with each practice (e.g., number of clients served, known issues with

practice, evaluation and/or satisfaction data)

Building The Infrastructure

A. Building Assessment and Referral Pathways

B. Building Implementation Pathways

Organizational Readiness

Change Coordination

Training/Consultation Model

Role of Supervision

C. Accountability and Implementation/Fidelity Monitoring


Integration of Evidence-Based/Evidence-Informed Practice in Service Delivery Continuum

A. Path 1: Expand Upon What Currently Exists in XXX County

B. Path 2: Determine the Evidence Base for Current Services

C. Path 3: Build the Evidence-Based Continuum

List potential programs by service domain (e.g., mental health, parent education, etc.) and

provide basic information on each, along with CEBC or similar rating

Closing Summary

Practical Action Steps

A. Outline what needs to occur next – basic introduction to the Preparation Phase


Appendix F: Preparation Phase: Resources and Tools

Contracting with Program Developers ..................................................................................................... F1

Data & Outcomes ..................................................................................................................................... F2

Assessing Fidelity ..................................................................................................................................... F3

Resources for Implementation ................................................................................................................. F4

Determining the Funding Stream ............................................................................................................. F5

Funding Stream Inventory Worksheet ..................................................................................................... F6

Referral System ........................................................................................................................................ F7

Staffing Plan ............................................................................................................................................. F8

Training & Coaching Considerations ........................................................................................................ F9

Selecting and Implementing EBPs Appendix F: Table of Contents © Apr. 2015 CEBC, www.cebc4cw.org

Contracting with Program Developers

Depending on the selected program, the amount of contact with the program developer varies

greatly. Some programs require a close working relationship during implementation, while others

simply require the purchase of materials and/or training. The requirements should have been

clarified during the Exploration Phase as part of the selection process. During Preparation, the

relationship should be formalized so that there are no unexpected issues as the implementation

moves forward.

Determine exactly what role, if any, the program developer will play in the site’s

implementation. In some cases, the developer may provide a referral to a trainer or

consultant and may not be directly involved. Clarify who this will be and what their role will

be. The developer may have documents describing the implementation process and roles, or

it may need to be discussed via email and phone.

○ In person visits?

○ On-site training required?

○ Coaching - on site or remotely?

○ Assessing fidelity and/or outcomes?

Develop a timeline in conjunction with the program developer or trainer for when the training,

coaching, etc., will take place.

Will a formal contract, Memorandum of Understanding (MOU), or similar document be

required by the developer and/or the site?

○ If Yes, what is the timeline and process to get the document approved and signed?

Does the document need to be signed before any work can begin?

○ If No, it may still be useful to spell out the relationship in writing so that all parties are

clear on the anticipated process.

Are there costs associated with the agreement with the developer?

○ Travel costs for meetings and trainings, conference call costs, coaching, etc.?

How will identified costs be paid?

○ Need for other paperwork, agreements for payment, etc.?

What happens if additional training or coaching is needed?

○ Option to establish a train the trainer program?

○ Costs and opportunities for additional training?

Selecting and Implementing EBPs Appendix F1: Page 1 of 1 © Apr. 2015 CEBC, www.cebc4cw.org

Data & Outcomes

During the Preparation Phase, the goals and objectives for the new program should be clearly

established and incorporated into any service contracts. In addition, the process for collecting and

reporting outcome measures should be determined. All of this should be discussed with the

developer, who may have requirements or recommendations.

Basic Definitions

Both goals and objectives are what is intended to happen when a change or a new program is

implemented:

Goals are the general aim or purpose of the program and are typically quite broad (e.g.,

reduce re-entry to foster care).

Objectives are measurable statements that describe the desired outcomes (e.g., reduce re-

entry to foster care by 25% over the next three years).

Outcomes are what are actually achieved when the data from the change or program is examined.

Determining Objectives

First, clarify the objectives of the program and then decide what can be measured to determine

whether the objectives are being met:

What are the objectives of the program? These should be concrete and measureable such as

“increase attendance and grade point average,” not “help students succeed in school.”

○ Develop process objectives (e.g., number of clients seen, number of clinicians trained, etc.)

and client objectives (e.g., improvement on behavioral measures, placement changes,

etc.).

○ Identify system-level objectives (e.g., decrease percentage of clients experiencing a

placement change by 5% in the next year) and individual-level objectives (e.g., improve

Parenting Stress Index scores between beginning and end of services).

How will each objective be measured?

○ Develop a concise metric for how each one should be examined; for example:

Attendance = (# of school days in time period - # absences)/ # of school days in time

period.


Writing SMART Objectives

One common way to write objectives is to use the SMART framework: each objective should be

Specific, Measureable, Achievable, Realistic/Relevant, and Time bound. Sample questions for each

component and an example are given below.

How to Measure Outcomes

Existing/Required Measures

Does the developer have outcome measures that are recommended or required?

○ Will the developer be collecting data on outcomes? How often and in what format?

Specific Measurable Achievable Realistic/Relevant

Time-phased

What exactly are we going to do, with or for whom? Example terms: Complete

Decrease Deliver

Develop

Improve Increase

Obtain Raise

Recruit Refer

Train

Is it measurable and can we measure it? Examples: Number

Average Percentage

(proportion) Change over

time

Can we get it done within the proposed timeframe? Can we address barriers or challenges that may arise?

Are there necessary staff and resources to attain the objective? Are goals achievable (reasonably high, not impossible)? Have other programs had similar results? Does the objective fit with overall goals?

When will we accomplish this objective? Examples: By the

beginning/ end of ___ quarter or ____ year

Example: In 2015, 80% of clients enrolled in Triple P parenting classes will complete at least 80% of the sessions.

Specific Measurable Achievable Realistic/Relevant

Time-phased

Clients enrolled in Triple P parenting classes

80% of clients 80% of sessions

Yes Yes/Yes In 2015


○ How will these measures be collected, who will compile the data?

Identifying Measures

If the developer does not have established measures to examine outcomes, the measures will

need to be identified:

○ Start by looking at the research on the program – how did these studies assess outcomes?

○ Talk to other sites implementing the program to see how they have assessed outcomes.

○ Look at measures developed for similar programs – could they be used as is or revised?

Initial Measurements

What is the baseline performance? This may have been done as part of the Exploration Phase

needs assessment and summarized in the Identifying and Clarifying the Problem form

(Appendix E4), but if not, determine using data from prior to program initiation.

○ It is helpful to examine data over several years to see if normal fluctuations are present.

For example, child welfare referrals tend to go up in the fall when schools re-open.

Use baseline performance to determine what type of change is reasonable to expect.

○ Again, using several years of historical data can be helpful to determine this.

○ Work with the program developer to get an idea of what type of result might be

reasonable, given the population and baseline data.

Collecting Data

Once the specific measures and indicators that will be used to measure outcomes have been

determined, a method to collect this data will need to be established.

When will be data be collected? At a minimum, data should be collected at entry and exit

from services, but for longer programs, it may be useful to assess during services as well to see

what progress is occurring.

How will data be collected? Different methods will need to be collected for client reported

data versus record abstraction data.

○ Consider work load issues:

Paper forms can be given to clients to complete when they check in for services, but

will need to be scored by hand or scanned.

Computerized forms that self-score can be useful, but require access to computer

equipment and availability of staff to assist with administration as needed.


How will the collected data be stored?

○ Is there space in current electronic record systems to enter entire measures or scores?

○ If not, will a separate system (e.g., Microsoft Excel spreadsheet, Microsoft Access

database) need to be developed?

How will the provider staff be trained on administering and collecting the outcomes data?

Reporting Data

A standard format for reporting outcomes data will need to be developed. Many systems use a

monthly or quarterly report from contractors as a way to monitor service provision. Outcomes data,

both process and client, should be an integral part of such a report.

How will outcomes data be reported by the service provider?

○ Monthly report, client level database, etc.

○ How frequently will outcomes be examined?

Other Considerations

Is there a start-up period before full outcomes can be expected? For example, a therapist may

need to see a few clients using a new treatment model before they are fully comfortable with

the process and delivering optimal services.

○ Discuss with developer to determine what is typical

What will the process be if outcomes objectives are not being met?

○ This should be specified in any contracts.

Fidelity should be addressed during any examination of outcomes – Refer to the Assessing

Fidelity tool (Appendix F3) for more information.

Other outcomes to review:

○ Satisfaction with services – family, child, etc.

Does the agency/county have an existing tool that can be used?

Does the developer have a recommended satisfaction tool?

○ Engagement – dropout rates, etc.

How does it compare with similar programs?

If dropout is high, consider doing follow-up with those who drop out to determine

reasons and address as necessary


Assessing Fidelity

It is vital for any new program to assess model fidelity from the start of services. This ensures that the

program is being delivered as intended, so that outcomes can be accurately assessed and tied to

service delivery. In addition, it can be used to identify training needs.

Each program rated on the CEBC has supplied information on the availability of fidelity tools. Please

refer to the Implementation Information section of the program outline for this information.

Questions to consider when measuring fidelity include:

Are there established fidelity tools for the program?

○ What is the format?

Self-report by provider, observation by supervisor, review of taped materials by

developer, etc.

○ How will these be integrated into services?

○ What should the fidelity expectations be?

Work with developer or available research papers to establish, and incorporate in

contracts and/or job descriptions as applicable.

○ How will fidelity information be reported?

Frequency

Individual and/or program level

If established tools do not exist, consider developing basic measures, either at the agency level

or in conjunction with the developer.

○ Review existing fidelity tools for similar types of services to get an idea of what to examine

○ Develop form to examine duration and frequency of services and essential components of

services

What should happen in each session or meeting?


Resources for Implementation

During the Preparation Phase, the resources needed to implement the practice will need to be

identified. Check with the developer to determine whether such a list exists. If not, make a

comprehensive list based on a thorough review of the program materials. Review the list with the

Implementation Team to see if any items are missing.

Below are some common examples of resources that may be needed for a program. This is not an

exhaustive list, but it is a starting place to begin thinking about what resources might be needed.

Keep in mind if an existing service is being transformed, there may be some existing resources that

can be utilized.

Space

○ What type of space is needed? Group meeting spaces, individual therapy offices, child

care facilities, etc.

○ Can existing resources be utilized to reduce costs and make the services feel more

community based? Libraries, government buildings, faith-based groups, etc.

○ Will any construction or alteration of space be required? For example, building mirrored

rooms for observation.

○ What is the process to reserve space – contracts, MOUs, etc.? Incorporate into timeline.

○ Are multiple spaces needed at the same time for concurrent parent/child sessions?

○ Are food service areas needed?

○ Is there access to public transportation and/or adequate parking?

AV equipment – computers, projectors, microphones, etc.

○ Is any of the required equipment available? May be able to repurpose existing equipment.

○ What is the lead time to obtain new equipment? Approval process, procurement process,

etc.? Incorporate into timeline.

Printing

○ Can staff manuals be printed locally or do they need to be purchased from the developer?

○ What other printed materials are needed? What is the lead time and process to order

them? Incorporate into timeline.

Brochures for outreach

Handouts for sessions

Referral forms


Documentation paperwork

Billing materials

Others?

Refreshments

○ Is provision of food or beverage part of the program (e.g., family meal at start of session,

snacks during after school program, etc.)?

○ Will the funding source allow for purchase of refreshments?

○ If not, are there any other ways to cover it? Community donations/sponsorship,

foundation grant, government feeding programs, faith-based groups, etc.

○ Is there a facility for storage of refreshments between sessions?


Determining the Funding Stream

During the Exploration Phase, the funding system for the new program should have been determined.

If not, it needs to be done early in the Preparation Phase so that the funding process can be clarified

in time for program start-up.

How will services be funded? For each potential funding stream, identify opportunities and potential

problems. Use the Funding Streams Inventory Worksheet (Appendix F6) to help organize the

information. Also refer to the Considerations for Documentation and Billing section (next page) which

may impact the choice of funding stream, and certainly must be addressed as part of the Preparation

Phase.

1. MediCal/MediCaid:

Funded through fee-for-service, managed care, or both?

Have the billing codes been established? If not, what is the process to do so?

Has the documentation process for services been determined? Develop new forms or

educational materials as needed.

Do new contracts need to be established, or existing ones expanded, to include the new

services? Determine process and timeline for completion.

2. Child Welfare funds

What Child Welfare funds will be used and are there restrictions or requirements in place?

Are the funds time limited? If so, need to consider how to continue services after the

funding ends.



3. State or Local Government funds

What government funds will be used and are there restrictions or requirements in place?

What is the approval process for use of the funds? Board of Supervisors, County Counsel,

etc.?

Are the funds time limited? If so, consider how to continue services after the funding

ends.



4. Grant from Foundation or other organization


What is the application process and timeline for receiving funds?

Are the funds time limited? If so, consider how to continue services after the funding

ends.



What report on services and outcomes will the organization require? How often will it be

due?

5. Other source?

Considerations for Documentation and Billing Process

It is important to evaluate the funding stream’s impact on the documentation and billing process and

make appropriate changes within the organization delivering the service to ensure that this process is

followed.

Will documentation and billing be electronic, paper, etc.?

Can existing systems be modified to accommodate new program? Lead time and cost to do

this?

Plan for training appropriate staff on documentation and billing (if appropriate) process

Is the training online, in-person, through manuals, etc.?

What is the timeline for preparation and delivery of training and materials?

What is the documentation timeline? How soon after service must documentation be

completed?

What is the billing timeline? How soon after service must bills be submitted?


Funding Stream Inventory Worksheet

Sample table:

Funding Source and Description Opportunities Potential Problems

Comments/Follow-Up Steps

Example: Community-Based Child Abuse Prevention (CBCAP) Established by Title II of the Child Abuse Prevention and Treatment Act Amendments of 1996. The purpose of the CBCAP program is: (1) To support community-based efforts to develop, operate, expand, enhance, and coordinate initiatives, programs, and activities to prevent child abuse and neglect and to support the coordination of resources and activities to better strengthen and support families to reduce the likelihood of child abuse and neglect; and (2) To foster understanding, appreciation and knowledge of diverse populations in order to effectively prevent and treat child abuse and neglect.

Use for primary and secondary prevention

Allocation based on population from the state to the counties (min. 20,000 people)

It can ONLY be used for primary and secondary prevention

Identify Contact person for CBCAP funding and involve them in conversation as to feasibility of converting current services funded by CBCAP to the potential Evidence-Based Practice(s) (EBP).

Example: Targeted Case Management (TCM) The TCM program reimburses participating counties for the federal share of costs (typically 50%) for case management services provided to Medi-Cal beneficiaries in specific target populations

It can be used to fund direct services.

Need more information and need a contact person.

Identify contact person for TCM that can provide additional information on whether the chosen EBP can be funded with TCM.

On the next page is a blank page that can be copied so that more than two potential funding sources can be listed. Note: A PDF version that can be typed into and saved is available on the CEBC website (www.cebc4cw.org). From the home page, click on the Select and Implement Programs button, then the Tools & Resources button, and then Technical Assistance Materials from the list.

Selecting

an

d Im

plem

entin

g EB

Ps

A

pp

en

dix F6

: Page

1 o

f 2

©

Ap

r. 20

15

CEB

C, w

ww

.cebc4

cw.o

rg

http://friendsnrc.org/direct-download-menuitem/doc_download/4-title-ii-of-capta



Funding Streams Inventory Worksheet

Funding Source and Description Opportunities Potential Problems

Comments/Follow-Up Steps

Selecting

an

d Im

plem

entin

g EB

Ps

A

pp

en

dix F6

: Page

2 o

f 2

©

Ap

r. 20

15

CEB

C, w

ww

.cebc4

cw.o

rg

Referral System

During the Preparation Phase, it will be determined how the existing system will be altered to ensure

adequate referrals to the new or expanded program.

What is the current referral system?

1. Self-referral by service recipient:

a. Update client brochures to ensure that the new program is included in any materials given to

clients.

b. Update websites and any resources materials with current program information.

c. Conduct outreach to workers and local organizations to ensure awareness of the new service.

d. Update or create new presentations, brochures, etc.

2. Direct referral from workers:

a. Provide training for workers about the new program and how it contributes to the service

array.

b. Provide information on how clients should be transitioned from existing services to the new

program, if applicable.

c. Provide referral materials – brochures, flyers, etc. – for use with clients when making referral.

d. Develop a referral pathway that clearly indicates when a referral to the program is appropriate

and should be made.

e. Invite workers to any kick-off/grand opening/open house event for the new program.

f. Send reminder email message to staff the day before the new program begins to remind them

of the changes, and resend shortly after opening.

3. Referral form submitted to centralized unit/worker who makes the referral:

a. Provide training for referral staff about the new program and how it contributes to the service

array.

b. Provide referral materials – brochures, flyers, etc. – for use with clients when making referral.

c. Invite workers to any kick-off/grand opening/open house event for the new program.

d. Send reminder email message to staff approximately 1 week before the new program begins

to remind them of the changes, and resend shortly after opening.

4. Other method?


After the program begins, track referrals closely to ensure that sufficient numbers are being received.

If referral numbers are low, additional outreach may need to be conducted:

Have program staff make phone calls or visits to worker or referral staff to increase awareness

and put a face on the new program.

Send thank you messages (phone or email) for referrals received to date.

Provide additional outreach in the community – community fairs, religious groups, etc.


Staffing Plan

An important part of preparing for implementation is to review the staffing requirements for the

practice. Confirm the staffing requirements with the program developer – be sure to clarify

requirements for all levels of staff. Basic information on staffing requirements is also listed in the

CEBC program outline in the Minimum Provider Qualifications section.

Below are questions to consider as the plan for staffing the program is being created:

What type of staff is required for the program? Master’s level, licensed, interns,

paraprofessional, etc.?

Are there any specific staffing needs to be considered? On call, 24-hour coverage, evening

coverage, weekend coverage, etc.?

Are the staff already in place in the agency that will deliver the program, or will they need to

be hired?

○ If transitioning existing staff, need to determine how this can be done - will it be necessary

reclassify staff into new positions, or do existing job descriptions work?

○ If transitioning existing staff, how different is new program from their current work? May

need to do additional work to ensure transition goes smoothly and workers are on board

with, as well as part of the changes.

○ If staff is to be hired, job descriptions will need to be created – check with the program

developer to see if any samples are available that can be adapted.

Are there any union issues to consider regarding moving or hiring staff?

Timeline for staff – need to get staff in place in time to complete training before services

begin. Allow plenty of time for hiring new staff and plan for what they will do before starting

training - do they need to be oriented to the agency, the community, etc.?


Training & Coaching Considerations

There are several considerations that must be made regarding training and coaching for a new

program:

What training is required for staff to implement the program? Refer to the Education and

Training Resources section of each CEBC program page for specifics on required training.

○ Duration, cost, location, etc.

Develop a training timeline in conjunction with the developer to ensure that it is clear on what

needs to be done and how long it will take. The developer may already have this available.

Clarify training requirements for supervisors and staff. At a minimum, supervisors should

complete the standard staff training; ideally, they will receive some additional training.

Does staff need to be certified in order to deliver the program?

○ How is the certification maintained?

Are booster trainings available or required? What is the recommended timing?

Is there ongoing coaching of staff? What is the process – consultation calls, videotape review,

etc.?

○ Who provides the coaching – developer, consultant, local staff, etc.?

○ What is the duration and intensity? Weekly one-hour calls for 12 weeks, monthly one-

hour calls for a year, etc.?

○ How will this support continue locally once the trainer/consultant is no longer involved

(will supervisors provide this in individual, group supervision, etc.)?

How will training be paid for? Does the funding source allow for an initial funding budget?

Will it cover ongoing trainings?

Can training costs be defrayed in any way? Discuss with developer there is an opportunity to

participate in a joint training with another agency, allow other individuals to attend the on-site

training, etc.

Does the developer have information/suggestions about how other agencies have covered

training costs?

What is the plan for training new staff when turnover occurs? If staff is already in place, look

at turnover history, but expect turnover as staff adjusts to new program.


Appendix G: Implementation Phase: Resources and Tools

Monitoring and Feedback Systems ......................................................................................................... G1

Reviewing the Billing/Financial Process .................................................................................................. G2

Supporting Initial Implementation- Go Live Week Checklist ................................................................... G3

Examining Outcomes............................................................................................................................... G4

Selecting and Implementing EBPs Appendix G: Table of Contents © Apr. 2015 CEBC, www.cebc4cw.org

Monitoring and Feedback Systems

Monitoring Service Delivery

Monitor service delivery on a weekly basis. One member of the team (typically the change

agent but can be another member as needed) should have responsibility for gathering and

organizing service delivery information. The necessary information depends upon the practice

being implemented. The following are offered as examples:

○ Are clients being served within the target population (e.g., court dependents placed at

home)?

○ Are clients being seen with the frequency necessitated by the practice?

○ Are documentation requirements being fulfilled?

If difficulties are identified, corrections should be sought and put into place to ensure quality

implementation

Monitoring Fidelity

Monitor fidelity on a weekly basis (See Assessing Fidelity - Appendix F3). One member of the

team should have responsibility for monitoring fidelity in collaboration with the practice

developer. Use fidelity data to make corrections and communicate expectations. If fidelity is

not being maintained, possible solutions include:

○ Determining if technical assistance or additional professional development is needed.

○ Working with the developer to see if an accommodation is feasible.

Developing Feedback Systems

Establish a feedback system to quickly identify and remedy problems that may be hindering

quality implementation. The change agent and the implementation team provide the bridge

between the practice/program developer and trainers and the practitioners working with

children and families.

Feedback from organizational supervisors/managers:

○ At least monthly calls or meetings with organizational supervisors/managers (i.e., those

with administrative responsibility for practitioners implementing the new practice).

Questions to ask:

Are consultants or trainers responsive to practitioner concerns?

Are consultants/trainers sensitive to the culture of clients being served and do they

provide assistance in making the practice relevant for diverse cultural populations?

Are there practitioners who may require more training to be successful?

Selecting and Implementing EBPs Appendix G1: Page 1 of 2 © Apr. 2015 CEBC, www.cebc4cw.org

Are there sufficient referrals?

Has there been any turnover – in practitioners and or organizational supervisors/

managers?

What is the feedback they are getting from direct service staff about the

implementation?

Feedback from developer/training organization/individual:

○ At least monthly phone calls with the developer or training organization to insure that all

practitioners are receiving consultation and feedback on their performance. Question to

ask:

Are all practitioners attending consultation?

If the practice utilizes a dashboard or other database, are all practitioners entering

information?

Are all practitioners building caseload at the expected pace?

Do practitioners have enough time to learn the new practice?

Are there organizational barriers that need to be addressed?

Is implementation on track at this point in time?

Are there early successes which should be highlighted?

How are the organizational supervisors doing?

Common Problems Uncovered from Monitoring and Feedback

High client dropout rates

○ Need to review reasons for dropout- is it because of transportation, timing, etc., or due to

something inherent to the practice?

○ Strategize how to address reasons, involve direct staff as well as leadership in this process

People not attending consultation calls

○ Have managers or trainers attend unit meetings to review implementation.

○ Follow up with individual providers about attendance issues.

○ Set consistent consequences for not attending calls.

Caseloads not being built/referrals not coming in

○ Review any breakdowns in the referral pathway.

○ Revisit whether referral pathway needs to be modified


Reviewing the Billing/Financial Process

During the Preparation Phase, the funding for the new program was identified and necessary

materials and procedures developed (refer to Determining the Funding Stream *Appendix F5+).

However, it is not until bills are submitted that it will be clear whether the system is functioning

properly.

Within the first few weeks after the start of a new program, the billing/financing process should be

reviewed to ensure that the process is going smoothly and that revenue will flow in a timely fashion:

Is documentation and billing completed according to the established time frames?

Is documentation and billing completed correctly (e.g., all required items and signatures filled

in, etc.)?

If the answer to both of these questions is No, then work with staff to determine why (e.g., provide

additional training, revise procedures as needed, etc.)

As billing statements are submitted to the funder, continue tracking the process to ensure prompt

reimbursement. If claims are being denied, work with the funder to determine the cause (e.g.,

insufficient documentation, inappropriate codes, etc.):

Provide a feedback loop with any necessary corrections to the staff developing the billing

material

Ensure that everyone is kept up to date on the current documentation and billing

requirements

○ Regular emailed or printed updates, version dates on all materials, etc.


Supporting initial implementation – Go Live Checklist

Things to consider before the first week and to monitor as implementation continues:

The week before Go Live:

1. Have all practitioners received the full dose of training?

a. Yes – Proceed.

b. No – Will they be able to start seeing clients immediately or have to wait to complete

training? This will depend upon criteria established by the program developer or the

implementing organization. Determine how they will be able to receive the additional

training.

2. Have equipment, space, materials been secured?

a. Yes – Proceed.

b. No – Determine what is needed, who is responsible for making it happen and resolve prior

to seeing first client(s).

3. Have coaching and technical assistance been arranged?

a. Yes – Proceed.

b. No – Determine what is needed (e.g., consultation calls, IT support, etc.) and arrange for it

as soon as possible.

4. Are monitoring and fidelity systems in place? Refer to the Monitoring and Feedback Systems

resource (Appendix G1) for more information.

a. Yes – Proceed.

b. No – Determine who is responsible for this component of implementation and ensure that

systems are in place prior to seeing the first client(s).

The Day Before Go Live:

1. Send a positive, inspirational email or other communication to all service providers about

tomorrow’s start. Include:

a. Last minute tips and reminders

b. Contact information in case of any problems

2. Send reminder email or other communication to all referral staff with any final instructions


The Day of Go Live:

1. Are there sufficient referrals so that practitioners are able to begin serving clients

immediately?

a. Yes – Proceed.

b. No – Problem solve and find a resolution as quickly as possible. Review current referral

pathway to look for breakdowns and make adjustments as necessary.

2. Follow-up with all service providers in group email or other communication asking for them to

respond with any problems, suggestions, or feedback.

One week after Go Live:

1. Follow-up with all service providers

a. Do they all have clients?

b. Any issues with service delivery, documentation, or billing?

2. Ensure fidelity and outcomes monitoring are occurring.

a. Have forms been submitted?

b. Are fidelity tools and/or observations being completed?

Monthly after Go Live:

1. See the Monitoring and Feedback Systems resource (Appendix G1) for ongoing monitoring and

feedback information.


Examining Outcomes

During the Preparation Phase, the goals and outcomes for the new program should have been clearly

established and incorporated into any service contracts. In addition, the process for collecting and

reporting outcomes measures should have been determined. Refer to the Data & Outcomes

(Appendix F2) resource in the Preparation section for more information.

During the Implementation Phase, the key focus will be on ensuring that the measures are being

collected on a timely and regular basis and also examining the outcomes to determine if the program

is having the desired effects.

Within the first month of services, begin preliminary analyses of the baseline data on clients. A large

focus will be on the quality of the data to ensure that it is being collected correctly. Questions to ask

include:

Are some clients missing entire measures? Why are they missing and how can this be

addressed to reduce missing data in the future?

Are certain items on the measures missing?

How soon after entering services were the intake measures collected?

Do the clients appear to be appropriate for the program? Correct age group, sufficient

impairment level, etc.

Within the first few months, begin examining outcomes from the program at both the process and

client level. Baseline levels should have been established and the focus now is looking to see what

changes, if any, are occurring.

If client level changes are not occurring, review the process outcomes and fidelity assessments and

also confer with the developer to determine what adjustments need to be made to the service

delivery process to improve results.

It may be helpful to display the outcomes results in chart or graph format, using Microsoft Excel or

similar programs, to show change over time. For example, a chart showing the number of clients

served each month by program site can help determine whether new clients are being referred at

appropriate rates and what the existing capacity in the program is. Some programs may choose to

establish a dashboard of several charts or graphs that are updated periodically as a way to quickly see

change over time.


Appendix H: Sustainment Phase: Resources and Tools

Sustainable Funding ................................................................................................................................ H1

Ongoing Training and Coaching Needs .................................................................................................. H2

Maintaining Fidelity ................................................................................................................................ H3

Selecting and Implementing EBPs Appendix H: Table of Contents © Apr. 2015 CEBC, www.cebc4cw.org

Sustainable Funding

Ideally, a plan for sustainable funding was considered during the Exploration Phase and finalized

during the Preparation and Implementation Phases. The Determining the Funding Stream (Appendix

F5) and Reviewing the Billing/Financial Process (Appendix G2) resources addressed the basic issues

related to funding and should be reviewed at this time, or completed, if they have not been, with an

eye towards sustainment.

The key funding issues for the Sustainment Phase are as follows:

1. Funding Stream

a. Are the funds currently being used to provide the program time limited?

i. If Yes, revisit the Funding Stream Inventory Worksheet (Appendix F6) form with a focus

on how to deliver services after the current funding ends.

2. Documentation and Billing

a. Is the current billing system sustainable? Incorporated into and funded as part of larger

data collection system, or a stand-alone system that will need maintenance over time?

i. If stand-alone, what is the plan to fund future maintenance, including training?

b. Are a sufficient percentage of bills being approved to meet actual costs? Examine the

claims denial rate and the denial reasons to determine if the approval rate and thus

income can be increased.

c. Is the payment rate sufficient to meet the program’s needs in the long term? It may be

necessary to periodically re-examine the appropriateness of the current reimbursement

rate for services and whether it covers all service delivered.

Selecting and Implementing EBPs Appendix H1: Page 1 of 1 © Apr. 2015 CEBC, www.cebc4cw.org

Ongoing Training and Coaching Needs

Ideally, a plan for sustainable training was considered during the Exploration Phase and finalized

during the Preparation and Implementation Phases. The Training & Coaching Considerations resource

(Appendix F9) addressed the basic issues related to training and should be reviewed at this time, or

completed if it has not been, with an eye towards sustainment.

The key training issues for the Sustainment Phase are as follows:

1. What are the current annual training costs? Include training for new staff, booster training for

existing staff, and coaching.

a. Is this cost covered by the current funding stream, or incorporated in the reimbursement

rate?

i. If No, are there any options to cover training costs separately?

2. Is there a certification process for the program?

a. Do the individuals or the organizations already have certification or are they planning to

become certified in the future?

b. What is the associated cost?

c. How often is certification renewed and what are the renewal requirements?

3. Opportunities to partner with other agencies on training? If several agencies in the region are

implementing the same program, there may be an opportunity to pool resources for training.


Maintaining Fidelity

At this time, fidelity monitoring systems should already be in place and fidelity should be reviewed on

a regular basis. Please review the Assessing Fidelity (Appendix F3) and Monitoring and Feedback

Systems (Appendix G1) resources for more information on how to establish these. Before entering

the Sustainment Phase, the program should be consistently meeting an acceptable level of fidelity, as

determined through consultation with the developer.

For the Sustainment Phase, it will be important to internalize the fidelity monitoring function within

the provider agency to the extent possible. This may involve moving responsibility for monitoring

fidelity from the Implementation Team to the agency implementing the service. Assessing and

reporting fidelity should still be incorporated as a contract requirement.

Key steps in this process include:

1. Determine how fidelity will be assessed in the long term – who has responsibility for each step

and how will it be reported and reviewed?

2. Identify the source of funding for any costs related to continuing to monitor fidelity, such as

the costs (if any) for purchasing the fidelity assessment tool(s) and reviewing fidelity issues

with the developer (as needed).

3. Determine an acceptable range of fidelity scores on the assessment tool (in conjunction with

the developer) if this was not done during the Preparation or Implementation Phases.

4. Develop a plan to increase fidelity if it drops below a certain level:

a. Acquire technical assistance or additional professional development from the developer.

b. Work with the developer to see if an accommodation to address deficiencies is feasible.


Reducing Risk:Families in

Wraparound Intervention

Reprinted from a research article published in Families in Society at the Alliance for Strong Families and Communities ©2015

Strengthening social work and those it serves through research, practice, and theory that propels lasting

positive change with families and their communities.

Brevard C.A.R.E.S. wraparound intervention reduces the occurrence of verifiable child maltreatment by applying core principles of wraparound with Family Team Conferencing to voluntarily enrolled families at risk for child maltreatment.

The intervention goes well beyond the traditional approach toward at-risk children and families, which is often characterized by systematic case management of multiple services in hopes of finding the “right” service array.

In contrast, the Brevard C.A.R.E.S. intervention actively engages the family in identifying and owning its strengths, natural supports, and vision of family success, while facilitating the family and its team to move toward that vision.

2Families in Society: The Journal of Contemporary Social Services©2015 Alliance for Strong Families and CommunitiesISSN: Print 1044-3894; Electronic 1945-1350

2015, 96(2), 91–98DOI: 10.1606/1044-3894.2015.96.18

Reducing Risk: Families in Wraparound InterventionAndrew J. Schneider-Muñoz, Rose Ann M. Renteria, Jesse Gelwicks, & Matthew E. Fasano

This study used a relative risk (RR) regression method to explore the extent to which children of caregivers in the Brevard C.A.R.E.S. (Coordination,

Advocacy, Resources, Education and Support) wraparound intervention experienced a reduction of maltreatment 6 months post completion. Brevard

C.A.R.E.S. is a wraparound intervention designed to reduce and prevent child maltreatment in identified at-risk families of children 0–17 years old.

The study of 308 children included 131 whose caregivers completed the intervention and a comparison group of 177 similarly situated children.

The study found that children whose caregivers completed the intervention experienced less verified maltreatment than children in the comparison

group. These findings have implications to enhance systems of care in the community that seek family interventions to support child maltreatment

prevention.

IMPLICATIONS FOR PRACTICE

• This intervention actively engages a family in identifying and owning its

strengths, natural supports, and vision of family success, while facilitat-

ing a Family Conferencing Team to move toward that vision.

The classic approach to child welfare has predominantly involved removing a child from a troubled home and neighborhood to the foster care system, which can cause long-term trauma to

the child (Doyle, 2008; Lee, Bright, Svoboda, Fakunmoju, & Barth, 2011; Walker & Bruns, 2007). This clinical and legal protocol, rising to the level of social policy, has produced diminishing returns and resulted in large numbers of children who have aged out of welfare programs. The cost to society grows at exponential rates as this vul-nerable population disproportionately suffers chronic mental illness, structural poverty, and crime (Heckman, 2006).

This study focuses on an intervention in Brevard County, Florida, that uses a wraparound approach early during family stress and/or crisis to prevent a child’s entry into the child welfare and dependency care systems. Wraparound is a definable planning process that re-sults in a unique set of community services and natural supports that are individualized for families and children and strive for positive outcomes (Bruns, Suter, Burchard, Leverentz-Brady, & Force, 2004; Bruns, Walker, & The National Wraparound Initiative Advisory Group, 2008).

System of Care Shifts

Brevard Family Partnership replaced the state’s traditional child wel-fare operation with a community-based system of care, leading to the development of the study’s intervention: a wraparound model that is strength-based and family-driven. The intervention is implement-ed while keeping at-risk children in their homes and communities, striving to prevent their system entry. Brevard’s wraparound model shifted service delivery from a deficit-laden focus—separating the child from the family—to a focus on building family strengths. Thus the unit of analysis in this study is the child within the family situ-ated in the community, rather than the child in relationship to the protective services system. Figure 1 contrasts the wraparound model and traditional child welfare services and approaches.

BackgroundStarting in 1998, the Florida legislature mandated the privatization of child welfare and related services to a community-based care model. Brevard County stakeholders responded with the develop-ment of a system of care anchored in a wraparound intervention, Brevard C.A.R.E.S. (Coordination, Advocacy, Resources, Educa-

tion and Support). Florida’s Title IV-E Waiver provided funding for such interventions, yielding a multimillion-dollar reduction in out-of-home care costs as communities like Brevard reinvested these savings in aggressive front-end prevention and diversion efforts, ex-panding child welfare capacity and fostering agency improvements (Armstrong et al., 2009).

Brevard C.A.R.E.S. provides prevention services, including the wraparound intervention, in Brevard County on Florida’s central east coast. With one of the country’s highest home foreclosure rates and the downsizing of the military and aerospace industries, Brevard’s approximately 550,000 residents (U.S. Census Bureau, n.d.) experi-ence high unemployment and serious economic challenges affecting both the traditionally poor and families slipping out of middle class.

Patricia Nellius-Guthrie of Brevard Family Partnership designed, piloted, and implemented the Brevard C.A.R.E.S. wraparound mod-el intervention in April 2005 after a local coalition of stakeholders prioritized aggressive front-end prevention and diversion to im-prove child safety and reduce the number of children entering the dependency system. The intervention’s wraparound approach was intended to reach families before the stressors they experience meet the threshold of abuse and neglect.

The intervention was formally defined by the National Wrap-around Initiative (NWI). Bruns et al. (2008) with the NWI developed standards for conducting high-quality and high-fidelity wraparound supporting measures of success. The intervention aims to assist fami-lies to identify and develop a system of formal and informal supports in the community and within family structures, and it maintains that even the most challenged parents and/or primary caregivers have the potential to make necessary changes.

In the intervention, families partner with and are guided by a cadre of service provider professionals credentialed in various disciplines, including mental health, substance abuse, and behavioral analysis. Weekly meetings link the family to formal and informal supports, in-cluding teachers, coaches, clergy, neighbors, friends, sponsors, service providers, and extended family, to stabilize immediate problems and build on family strengths to rear their children successfully.

Wraparound and Child Welfare OutcomesStudies examining the effectiveness of wraparound models have re-ported a wide range of outcomes. Some found little to no connection between family and youth outcomes and the provision of wraparound services as compared with “treatment as usual” (Bickman, Smith, Lambert, & Andrade, 2003). Others suggest a moderate link between family participation in wraparound and desired outcomes in emo-tional, behavioral, academic, or other areas (Clark, Lee, Prange, & McDonald, 1996). Still other research argues that strong associations exist between quality implementation of wraparound processes or ser-

Schneider-Muñoz, Renteria, Gelwicks, & Fasano | Reducing Risk: Families in Wraparound Intervention

3

vices and subsequent family success (Pullman et al., 2006). Bruns and Suter (2009) suggested that outcome discrepancies could be related to infidelity in the implementation of wraparound. Such observed discrepancies may be explained by methodological variation within the studies analyzed, disparities between chosen outcome indicators, and impact relative to practice setting and community context (Hyde, Burchard, & Woodworth, 1996; Myaard, Crawford, Jackson, & Alessi, 2000; Pullman et al., 2006). In terms of child welfare outcomes, recent studies have focused on varying outcome indicators such as rates of placement change, reduction of risk behavior, and increases in school attendance, all differences that make cross-study comparisons difficult (Clark et al., 1996; Mears, Yaffe, & Harris, 2009; Pullman et al., 2006). Specific to child welfare and wraparound, studies have found that, compared with youth in traditional child welfare programs, youth in wraparound programs experience significantly fewer placement changes, fewer days as runaways, fewer days incarcerated (for the rele-vant subset), and more older youths in a permanency plan at follow-up (Clark et al., 1996). Using a matched comparison research design, re-searchers found that after 18 months, 27 of the 33 youth (82%) who re-ceived wraparound moved to less restrictive environments, compared with 12 of the 32 comparison group youth (38%); family members pro-vided care for 11 of the 33 youth in the wraparound group compared with only six in the comparison group. Other positive outcomes for the wraparound cohort included increased school attendance, fewer school disciplinary actions, and higher grade-point averages (Bruns, Rast, Peterson, Walker, & Bosworth, 2006; Rast, Bruns, Brown, Peter-son, & Mears, 2007).

Other research using the Child and Adolescent Functional Assess-ment Scale, such as Mears et al. (2009), has found that youth in the wraparound group showed significantly greater improvement than youth receiving traditional child welfare services. Youth in the wrap-around group also showed significantly greater movement toward less restrictive residential placements as assessed by the Restrictiveness of

Living Environment Scale. And, while more wraparound youth expe-rienced a placement change, this was due to more youth in the wrap-around group moving to less restrictive placements during the study period.

To date, there remains a paucity of studies. Nevertheless, the above studies suggest that wraparound models are moving in the right di-rection and that key child welfare outcomes can be achieved. They fit a theoretical model of change, which suggests that system change with family strengthening as a guiding approach can lead to signifi-cant successes for families and their children. It is important, how-ever, to note the limitations of the current research on wraparound models (including interventions) and child welfare outcomes. Re-search studies are needed with the capacity to (a) focus research on front-end wraparound diversion programs, (b) examine child-level outcomes for specific family types, and (c) support knowledge de-velopment in wraparound evidence research and the child welfare research base.

The Current StudyOur study uses a relative risk (RR) regression method to explore the extent to which children with family members in the Brevard C.A.R.E.S. intervention can experience a reduction of risk of child maltreatment recidivism 6 months after receiving the intervention. The RR regression method reveals how risk is decreased or increased from an initial level, allowing risk to be measured readily and clear-ly. In application, an RR of 0.5 shows that the initial risk has been halved. By contrast, the odds ratio (OR) regression method relates to an event happening by simply stating the number of those who experience the event divided by the total number of people at risk of having that event (Davies, Crombie, & Tavakoli, 1998).

The RR regression method best answered our research question, and its conclusions are more applicable than OR regression. While the general form of RR and logistic regression are similar, RR regres-

Figure 1. Comparison of wraparound and traditional child welfare models.Area Wraparound model Traditional child welfare services model

Consensus, coordination, and collaboration—across systems

Care Coordinators integrate all aspects of care, bringing all parties to the table to devise a single customized, outcome-based plan.

Families with cross-system involvement encounter multiple plans with which families must comply.

Case planning Individualized and customized family-team planning process use key wraparound principles and the Family Team Conferencing approach.

Caseworkers develop plans based on system mandates and make formal determinations of type of child maltreatments to identify services.

Service array Flexible support and continuum of services purchased on an as-needed basis with family acuity driving service delivery.

Prescriptive service delivery—traditional and categorical in nature, which is less customized for families.

Utilization management Services are centrally authorized by Care Coordinators who have real-time access to services and community resources as alternatives to paid services.

Services are purchased and secured on a first-come, first-served basis; community resources and alternatives are utilized less frequently.

Funding Payment structure is based on a unit rate with swift flexible fund authorization and management in order to maximize use of alternative funding streams and community resources.

Funding sources are limited for use only by eligible populations and require multilevel approval processes.

Outcomes Individualized family-centered outcomes that begin with the initial assessment, care plan, authorization, and monitoring of services until the family graduates or completes program.

Typically provided through a state-automated information system where predetermined algorithms produce aggregated performance data (e.g., maltreatment and re-abuse rates).

Note. This chart was developed by the staff of Brevard C.A.R.E.S. after receiving training in the National Wraparound Initiative’s guiding principles (Bruns et al., 2008).

FAMILIES IN SOCIETY | Volume 96, No. 2

4

sion has underlying fundamental differences with respect to estima-tion. In particular, Agresti noted that in using small cell proportions within RR regression, “it is not possible to construct ‘exact’ confi-dence intervals for association measures that are not functions of the odds ratio” (Agresti, 1992, p. 135).

Our study is a secondary data analysis that matches state admin-istrative data on child welfare to Brevard C.A.R.E.S. administrative data on the program participants. Specifically, the study tracked whether or not a child experienced verified child maltreatment with-in 6 months after family members completed the program. The study compares children whose families completed the Brevard C.A.R.E.S. intervention with similarly situated nonparticipants.

Our primary research question is: When compared with children of similarly situated non-Brevard C.A.R.E.S. participants, are the children of the families that complete the C.A.R.E.S. intervention more or less likely to be maltreated 6 months after exiting the pro-gram? We hypothesized that children linked to families that com-plete the intervention would have reduced incidents of maltreatment, defined as experiencing verified child maltreatment as monitored by the Florida Safe Families Network, 6 months after program comple-tion of the intervention.

The comparison group comprised families that were referred to Brevard C.A.R.E.S. but did not participate in the intervention. Par-ticipation and enrollment in Brevard C.A.R.E.S. were voluntary, and families were under no obligation to participate. However, the refer-ral process provided information about the family history and rea-son for referral.

For all referrals, including families that did not participate in the intervention, Brevard C.A.R.E.S. staff conducted a risk assessment. Each family was assigned one of three intake levels based on the cir-cumstances of the referral, the level of need, and complexity. Staff assessed family history, prior child welfare involvement, past inter-ventions, status of final findings of maltreatment indicators, and the age and developmental needs of the children.

Intake Level III family referrals were the highest priority for the intervention, often with more than 10 prior maltreatment reports with the state and the highest acuity. Intake Level II referrals focus on families with 5 to 10 prior abuse reports with the state. Intake Level I referrals include families with no history or low history of prior abuse reports.

The risk assessment was designed to prioritize and guide the in-tervention, not as a research tool; however, its thoroughness made it the best choice for the primary independent variable. The Brevard C.A.R.E.S. database also included referral source, reason for referral, number and average age of children, household structure, location in county, and duration of participation. These variables were used to ensure that, while this was a convenient sample and not randomized, the two populations were similar enough to be comparable.

The study was based on secondary analysis and was completed af-ter the participants had completed the intervention, which precluded any human contact; therefore it was deemed exempt from an insti-tutional review. However, to ensure participants anonymity, Brevard C.A.R.E.S. staff removed all identifying personal data before the da-tabase was shared with the researchers. All revisions and additions to the database requested by the researchers were made by the staff.

Methods

Intervention DetailsAs described in detail below, the intervention examined in the cur-

rent study allowed families to receive the wraparound model that strives for high fidelity to key guiding principles (Walker & Bruns, 2007). Approximately 75% of families served are referred from the Florida child welfare system, while the remaining families are re-ferred through the community, including families that self-refer. The intervention targets various families, including “primary preven-tion families” in the general community who self-refer to decrease likelihood of maltreatment ever occurring; “secondary prevention families” with one or multiple risk factors (e.g., poverty, substance abuse); “tertiary prevention families” with current verified maltreat-ment and for whom the intervention is an alternative to traditional child welfare system involvement; and families transitioning from the child welfare system that choose wraparound for their aftercare support. The top five reasons for referrals were parenting support, ungovernable youth, domestic violence, mental health, and sub-stance abuse.

Any family living in the service area of Brevard County who serves as a primary caregiver of at least one child under age 18 is eligible to voluntarily enroll in the intervention, which lasts an average of 120 days. Each family is assigned a facilitator and a family support partner to help them develop an individualized plan for their own goals. A coach/supervisor consults and ensures wraparound fidelity. The initial family meeting includes an informal assessment leading to the family’s expression of how “Life will be better when…” as the basis going forward.

At the heart of the intervention is Family Team Conferencing, with an individualized and unified family team that enlarges the cir-cle of support around the family beginning with a wraparound plan outlining the family’s needs, challenges and barriers, action steps to-ward resolution, person(s) responsible, time frame, and outcome(s). Care Coordinators are wraparound practitioners and resource ex-perts able to authorize services on a unit basis via a utilization man-agement system. Family acuity dictates duration and frequency of services. Creative planning and unconditional support allow for “whatever it takes” to help the family. Incremental successes and a graduation ceremony are celebrated. A transition plan sustains fami-lies at exit.

The intervention’s core principles include family voice and choice, wherein family and youth/child perspectives are intentionally elicited and prioritized; planning is grounded in family members’ perspec-tives; and the team provides options that reflect family values and preferences. The intervention is also team-based; the wraparound team consists of individuals agreed upon by the family and com-mitted to them via informal, formal, and community support and service relationships. It features natural supports, actively pursuing full participation from the family members’ own networks of inter-personal and community relationships. A wraparound plan reflects activities and interventions that draw on sources of natural support.

Another approach of the intervention is collaboration: Team members work cooperatively and share responsibility for develop-ing, implementing, monitoring, and evaluating the family’s plan. The plan reflects a blending of team members’ perspectives, mandates, and resources. It guides and coordinates each team member’s work toward meeting the team’s goals. The intervention is community-based, ensuring service and support strategies take place in the most inclusive, responsive, accessible, and least restrictive settings pos-sible, and that safely promote child and family integration into home and community life.

The intervention is individualized: To achieve the goals outlined in the wraparound plan, the team develops and implements a custom-


5

ized set of strategies, supports, and services. It is strengths-based, with process and wraparound plans that identify, build, and enhance the capabilities, knowledge, skills, and assets of the child and family, their community, and other team members. Persistence is an impor-tant value in the intervention: Despite challenges, the team works toward the wraparound plan goals until everyone agrees that the for-mal intervention (wraparound model) is no longer required.

Further, the intervention is outcome-based, with the team link-ing goals and strategies to measurable success indicators, monitoring progress in terms of these indicators, and revising the plan accord-ingly. Finally, it is culturally competent: The wraparound team dem-onstrates respect for and builds on the values, preferences, beliefs, traditions, norms, and culture of the child, youth, family, and their community. The practitioner demonstrates an understanding of his or her own worldviews and those of the family while avoiding ste-reotyping and misapplication of scientific knowledge (Bruns et al., 2008).

Comparison Group Has No InterventionBecause program participation and enrollment in the Brevard C.A.R.E.S. intervention was voluntary, the comparison group did not participate in the intervention. All of the outcome data exam-ined in the study remain associated with families that were program eligible to participate and receive the intervention. In other words, all of the children included in the study were linked to families that met the Brevard C.A.R.E.S. program eligibility. With this in mind, our study compares children of similarly situated nonintervention participants with children in families who completed the interven-tion, while focusing on one outcome indicator: occurrence of verifi-able child maltreatment within 6 months after participants exited the intervention.

Study Design

Data Sources and VariableThe data for this study came from the C.A.R.E.S. de-identified par-ticipant database for all children referred to and participating in the program after July 1, 2009, and whose cases were closed by January 1, 2010. The data were analyzed only after the 6-month outcome data were available. Data were cleaned to identify any incomplete or in-accurate data, ensuring that administrative data on a child had ap-propriate and accurate data for every variable. The database included demographic information on the children and families including the type of reported abuse and services received. The study’s variable was monitored—whether or not children had experienced verified maltreatment (within 6 months)—with data from the Florida Safe Families Network. All cases were coded as verified maltreatment or no verified maltreatment.

No verified maltreatment included cases where no maltreatment was reported as well as where maltreatment was reported but not ver-ified. No verified maltreatment was also used as a binary dependent variable in the regression analysis.

SampleThe sample size was 308, including 131 children with at least one pri-mary caregiver who completed the C.A.R.E.S. intervention during the study period. The data were collected in July 2011. We compared the participants with 177 children with at least one primary care-giver who did not participate in the intervention. The sample came from a pool of 1,118 children whose parents who had opportunity

to receive and/or complete the intervention during the time period. This sample was a convenient one, selected from all eligible parents who could have participated due to a self-referral, a child protective investigator referral, a referral received from a community resource, or a 211 telephone call (for persons experiencing a crisis or in need of assistance).

Analyses

The intervention is designed to reduce the risk of child maltreatment. The analytical model chosen follows this goal, utilizing a general lin-ear model (GLM) to estimate risk ratios of the two groups. The re-gression model estimates risk ratios (Wacholder, 1986) and can be referred to as log-binomial (Blizzard & Hosmer, 2006). Results from this model estimate the probability of an occurrence of an event, such as determining whether the intervention reduces the risk of maltreatment for children with at least one primary caregiver who completes the intervention.

The risk ratio was estimated using the data analysis software Stata 11. Based on the nature of the intervention and the research ques-tion, the RR regression model allows for examination of the ratio of the probability of verified maltreatment occurring with children whose family member completes the intervention versus those who do not. Relative risk analyses (prevalence ratios) are a natural and familiar summary of association between a binary outcome—in this case, whether or not there is verified child maltreatment—and an ex-posure or intervention.

The dependent variable is the outcome of verified maltreatment, while the primary independent variable is the intake level. The latter was chosen to ensure that children in the two groups faced similar levels of risk for future maltreatment and to mitigate some of the po-tential differences between the groups. The intake level information was integrated into the study’s database at three different tiers to deter-mine priority, with Intake Level I posing the lowest risk of future mal-treatment and Intake Level III the highest risk. The families’ history of prior reports, level of need, severity of maltreatment, and notoriety determined the intake level.

Relative risk (RR) is a GLM with a log link and variance function [V(μ) = μ(1 − μ)] (McCullagh & Nelder, 1989). Unlike the standard logistic regression model, the relative risk model requires constraints to ensure that fitted probabilities remain in the interval [0,1]. While binary data is often estimated by logistic regression, the study used relative risk regressions because they provide a more useful sum-mary of association between a binary outcome and an exposure or intervention (Carter, Lipsitz, & Tilley, 2005; Cummings, 2009).

With binary data, the GLM model used is as follows: log link: log(Y) = constant + β*X + error. Relative risk (RR) is a ratio of the probability (P) of the event occurring in the exposed group versus a nonexposed group and is computed as follows: RR = P (no-maltreat-ment C.A.R.E.S.) / P (no-maltreatment non-C.A.R.E.S.).

The desired outcome was not to be maltreated, rather than to be maltreated. Therefore, the dependent variable was coded as 0 in cases where a verified maltreatment was reported and 1 when no verified maltreatment was reported. The data collection window allowed for examination of the 6-month outcome (no-maltreatment vs. mal-treatment) for both groups. Most cases fell into the no verified mal-treatment category and were coded 1.


6

Results

Comparative DataThe children were linked to at least one caregiver who either received the intervention (Group 1) or did not receive the intervention (Group 2).

Maltreatment reported within 6 months. Of the 308 children in the sample, 30 had verified maltreatment within 6 months after com-pletion of the Brevard C.A.R.E.S. intervention (10%). That means that 90% of the children, whether their families were in Group 1 or Group 2, had no verified maltreatment within 6 months (See Table 1).

Maltreatment reported within 6 months by group. When the sample was divided into children with a primary caregiver who com-pleted the intervention (Group 1) and children with a primary care-giver who did not complete the intervention (Group 2), 131 children were in Group 1, while 177 were designated to Group 2. Although the frequency of verified maltreatment was low for both groups, it was more commonly reported among Group 2, who accounted for 21 of the 30 cases, or 70% of the verified child maltreatment. The focus of the positive outcome is that 93% of Group 1 (completed the interven-tion) had no verified maltreatment reported, compared with 88% for Group 2 (see Table 2).

Maltreatment reported within 6 months by intake levels. When the sample was analyzed by intake levels, 286 of the 308 children, or 93%, were designated with Levels I or II (low to moderate risk) at intake. Of these 286 children in Levels I and II, a total of 262, or 92%, experienced no verified maltreatment, while 16 of the 22 children in Level III (highest risk), or 73%, reported no verified maltreatment. Although there are fewer children in Level III that had verified mal-treatment reported, the percentage is higher by 19% due to the small sample size (see Table 3).

Results from the GLM regression. After adjusting for intake level, using the GLM regression analysis, children with at least one pri-mary caregiver who finished the intervention (Group 1) were more likely not to have reports of later maltreatment, compared with chil-dren with a caregiver who did not finish the intervention (Group 2): risk ratio 1.10, 95% confidence interval 1.04 to 1.17. In these data, children linked to Group 1 had a greater risk of no reported maltreat-ment: 10% greater (95% confidence interval, 4% greater risk to 17% greater risk). This finding was statistically significant beyond p < .05 and p < .01 (see Table 4).

Results of this study add to the evidence that wraparound pro-grams such as the Brevard C.A.R.E.S. intervention can positively and significantly impact children. Put simply, comparing 100 youth whose primary caregiver completed the C.A.R.E.S. intervention to 100 youth with caregivers who are referred but do not complete the intervention, on average 10 more C.A.R.E.S. intervention “com-pleters” will have no reported maltreatment. While not definitive, these initial findings indicate that the program is successful, espe-cially as an early intervention program.

Discussion

First and foremost, this study links program effect to verified mal-treatment as the most powerful outcome for such an intervention, a measure that has not been fully applied in previous studies of wrap-around. The RR model used for the study reports the probability of nonmaltreatment as a result consistent with comprehensive and inten-sive early intervention designed to produce the absence of risk in the family at the community level.

Focusing on youth-level outcomes, Bruns and Suter (2009) had monitored and analyzed seven controlled, peer-reviewed studies ex-amining the provision of wraparound services. Unlike those seven, our study uses maltreatment as the outcome variable. Few studies have examined the secondary data that states routinely collect to manage maltreatment risk. Our study is unique in this regard, con-tributing to the wraparound field by adding a new dimension for analysis.

Brevard C.A.R.E.S. wraparound intervention reduced the occur-rence of verifiable child maltreatment 6 months postcompletion by applying core principles of wraparound with Family Team Confer-encing to voluntarily enrolled families at risk for child maltreatment. The intervention goes well beyond the traditional approach toward at-risk children and families, which is often characterized by system-atic case management of multiple services in hopes of finding the “right” service array. In contrast, the Brevard C.A.R.E.S. intervention actively engages the family in identifying and owning its strengths, natural supports, and vision of family success, while facilitating the family and its team to move toward that vision.

The results indicate that the intervention can reduce the prob-ability of later child maltreatment. To the extent that the Brevard C.A.R.E.S. wraparound intervention can promote and support fam-ily stability and strengthen families resulting in child safety, it offers a promising approach to primary prevention and diversion in child welfare.

LimitationsThe major limitation of the study is that the comparison group was a convenient sample; the study used existing secondary data that did not influence the selection process. As a result, it is possible that par-ticipants had greater motivation to make positive changes in their lives than nonparticipants. However, the study team attempted to mitigate selection bias by measuring maltreatment risk at intake with a three-level system. Our new study, currently underway, uses random assignment research procedures to further support rigorous sampling methodologies. Additionally, a minor limitation is that the study tracked only legally verified maltreatment because it was less biased than reported maltreatment. Other maltreatment may have occurred but not been verified.

Implications for Practice and Future Research

This study showed an association with strong effect on family func-tion in proximate terms. Families with no maltreatment for 6 months can be viewed as no longer in crisis.

The professionals delivering the intervention place strong empha-sis on high fidelity to a family-centered technique. Now that the pro-gram has demonstrated effectiveness at 6 months, more research is needed to better understand the proper dosage, how supports should be organized, and how professionals may need to provide continued support to sustain the family’s skills and progress. New areas of in-quiry raised by this study’s results include more in-depth exploration of how the program works, use of pre- and postassessments to track skills gained through the wraparound intervention, and examina-tion of how these skills affect participants’ behavior over time.

For the size of our national investment in high-fidelity wrap-around, little remains known about effective dosage, the clustering effect for supports in different kinds of families, or the trajectory of intensity resulting from the order in which the supports are de-livered. In applying wraparound intervention to early prevention,


7

ReferencesAgresti, A. (1992). A survey of exact inference for contingency tables. Statistical Science

7(1), 131–153.Armstrong, M. I., Vargo, A. C., Jordan, N., Sharrock, P., Sowell, C., Yampolskaya, S.,

& Kip, S. (2009). Evaluation brief on the status, activities and findings related to Florida’s IV-E waiver demonstration project: Two years post-implementation (Florida’s IV-E Waiver Demonstration Project—Evaluation Brief Series, 250–2). Tampa, FL: University of South Florida.

Bickman, L., Smith, C., Lambert, E. W., & Andrade, A. (2003). Evaluation of a congressionally mandated wraparound demonstration. Journal of Child and Family Studies, 12(2), 135–156.

Blizzard, J., & Hosmer, D. W. (2006). Parameter estimation and goodness-of-fit in log binomial regression. Biomedical Journal, 48(1), 5–22.

Bruns, E. J., Rast, J., Peterson, C., Walker, J., & Bosworth, J. (2006). Spreadsheets, service providers, and the statehouse: Using data and the wraparound process to reform systems for children and families. American Journal of Community Psychology, 38, 201–212.

Bruns, E. J, & Suter, J. S. (2009). Effectiveness of the wraparound process for children with emotional and behavioral disorders: A meta-analysis. Child and Family Psychology Review, 12, 336–351.

Bruns, E. J., Suter, J. C., Burchard, J. D., Leverentz-Brady, K., & Force, M. (2004). Assessing fidelity to a community-based treatment for youth: The Wraparound Fidelity Index. Journal of Emotional and Behavioral Disorders, 12, 69–79.

Bruns, E. J., Walker, J. S., & The National Wraparound Initiative Advisory Group. (2008). Ten principles of the wraparound process. In E. J. Bruns & J. S. Walker (Eds.), The resource guide to wraparound (Ch. 2.1, 1–10). Portland, OR: National Wraparound Initiative, Research and Training Center for Family Support and Children’s Mental Health.

Carter, R. E., Lipsitz, S. R., & Tilley, B. C. (2005). Quasi-likelihood estimation for relative risk regression models. Biostatistics, 6, 39–44.

Clark, H., Lee, B., Prange, M., & McDonald, B. (1996). Children lost within the foster care system: Can wraparound service strategies improve placement outcomes? Journal of Child and Family Studies, 5(1), 39–54.

Cummings, P. (2009). The relative merits of risk ratios and odds ratios. Archives of Pediatric & Adolescent Medicine, 163(5), 438–445.

Davies, H. T. O., Crombie, I. K., & Tavakoli, M. (1998). When can odds rations mislead? BMJ, 316, 989. Retrieved from http://www.bmj.com/content/316/7136/989

Doyle, J. (2008). Child protection and adult crime: Using investigator assignment to estimate causal effects of foster care. Journal of Political Economy, 116(4), 746–770.

Heckman, J. J. (2006). Skill formation and the economics of investing in disadvantaged children. Science, 312(5782), 1900–1902.

Hyde, K., Burchard, J., & Woodworth, K. (1996). Wrapping services in an urban setting. Journal of Child and Family Studies, 5(1), 67–82.

Lee, B., Bright, C., Svoboda, D., Fakunmoju, S., & Barth, R. (2011). Outcomes of group care for youth: A review of comparative studies. Research on Social Work Practice, 21(2), 177–189.

McCullagh, P., & Nelder, J. A. (1989). Generalized linear models (2nd ed.). Boca Raton, FL: Chapman & Hall/CRC.

Mears, S. L., Yaffe, J., & Harris, N. J. (2009). Evaluation of wraparound services for severely emotionally disturbed youths. Research on Social Work Practice, 19, 678–685.

Myaard, M., Crawford, C., Jackson, M., & Alessi, G. (2000). Applying behavior analysis within the wraparound process: A multiple baseline study. Journal of Emotional and Behavioral Disorders, 8(4), 216–229.

Pullman, M., Kerbs, J., Koroloff, N., Veach-White, E., Gaylor, R., & Sieler, D. (2006). Juvenile offenders with mental health needs: Reducing recidivism using wraparound. Crime & Delinquency, 52(3), 375–397.

Rast, J., Bruns, E. J., Brown, E. C., Peterson, C. R., & Mears, S. L. (2007). Impact of the wraparound process in a child welfare system: Results of a matched comparison study. Manuscript submitted for publication.

Brevard actually reconfigured the system of care to provide more formal and informal community supports in the home. This ap-proach proved surprisingly effective at strengthening families and reducing the risk of child maltreatment.

Little previous research on the wraparound approach compares the effectiveness of voluntary versus mandated participation. Future studies should explore better fitting such a model for patterns associ-ated with trust, as well as variables that report strengths sensitive to cultural constructions.

For future research, we would hypothesize that significant positive results can be anticipated when families receive the supports they need, as determined by each family. These supports serve to strengthen and connect relationships in the community to meet each family’s unique needs, empowering them with social and emotional skills, and edu-cational and financial resources to counter their challenges in times of crisis.

Table 1. Verified Maltreatment Reported Within 6 Months (Postcompletion of Brevard C.A.R.E.S. Intervention)

Verified maltreatment

Frequency (N = 308) %

Yes 30 10

No 278 90

Table 2. Verified Maltreatment Reported Within 6 Months by Group With Frequency and Percent


Group 1: C.A.R.E.S. (n = 131)

Group 2: Non-C.A.R.E.S.

(n = 177)

Yes 9 (7%) 21 (12%)

No 122 (93%) 156 (88%)

Table 3. Verified Maltreatment Reported Within 6 Months by Family Intake Level of Risk With Frequency and Percent


Intake Levels I & II

(n = 286)

Intake Level III

(n = 22)Total

(N = 308)

Yes 24 (8%) 6 (27%) 30 (10%)

No 262 (92%) 16 (73%) 278 (90%)

Table 4. Results From the General Linear Model (GLM) Regression: No Verified Abuse by Group (C.A.R.E.S.) Adjusting for Intake Level

No verified maltreatment Risk ratio SE z P > |z|

95% CI

Lower Upper

C.A.R.E.S. 1.103387 .0352 3.08 0.002 1.04 1.17

Level I + II 1.337156 .1759 2.21 0.027 1.03 1.73

Brevard Family Partnership2301 W. Eau Gallie Blvd., Suite 104Melbourne, FL 32935-3120

Brevard C.A.R.E.S.4085 South US 1Rockledge, FL 32955-5307ncfie.org/portfolio/brevard-c-a-r-e-s.html

Bring C.A.R.E.S. to Your Community!

Brevard’s family-centered, strength-based program is now available to agencies and organizations throughout the U.S. that want to reduce the number of children in their formal child welfare systems through an aggressive, front-end child abuse prevention model.

With the help of our team of experienced professionals, the C.A.R.E.S. model can be customized to the unique needs of the children and families in your community.

Implementing C.A.R.E.S. can help to:

- Reduce the number of children in a community’s formal child welfare system,

- Keep children safe and in their homes,

- Engage the community in providing the support families need when stressors could lead to child abuse, and

- Significantly reduce the costs of providing services (in-home vs. out-of-home care).

Interested parties may contact us to obtain more information about how we can customize our array of available services to support you and your community in protecting children, strengthening families and changing lives.

Celebrating 96 years in 2015, Families in Society is America’s most enduring social work journal. It advances a knowledge-into-practice approach to family and community research through innovative scholarship on the issues related to the capabilities of individuals and families, including consideration of the biopsychosocial, economic, and cultural factors that affect well-being.

Through the signature series, “Committing to Our Families,” the Alliance and its esteemed research and advocacy partners provide an in-depth focus on these significant issues that are facing America’s families.

©2015 Alliance for Strong Families and Communities.


8

U.S. Census Bureau. (n.d.). State & county quickfacts. Retrieved from http://quickfacts.census.gov/qfd/states/12/12009.html

Wacholder, S. (1986). Binomial regression in GLIM: Estimating risk ratios and risk differences. American Journal of Epidemiology, 123(1), 174–184.

Walker, J., & Bruns, E. J. (2007, March). Wraparound—Key information, evidence, and endorsements. Retrieved from http://www.nwi.pdx.edu/PDF/WraparoundEvidenceRecognition.pdf

Andrew J. Schneider-Muñoz, CYC-P, EdD, EdM, chief advancement officer, National Center for Innovation and Excellence. Rose Ann M. Renteria, PhD, MPA, BA, coordi-nator of evaluation and research, PHILLIPS Programs for Children and Families. Jesse Gelwicks, MA, BA, health care project manager, TechWorkers. Matthew E. Fasano, BS, doctoral candidate, La Salle University. Correspondence: [email protected]; Brevard Family Partnership, 2301 W. Eau Gallie Blvd., Suite 104, Melbourne, FL

32935.

Authors’ note. The authors wish to acknowledge consultation from Karen VanderVen and Benjamin Webman on methods and strategies. We thank Patricia Nellius-Guthrie, Valerie Holmes, and Tracy Little for the history of the development of systems of care and the C.A.R.E.S. site.

Manuscript received: May, 21, 2014Revised: February 13, 2015Accepted: February 16, 2015Disposition editor: Sondra J. Fogel