2
Factors Contributing to Successful Implementation of EBPs
The success of the intervention strategies themselves and
The success of the implementation
processes
Too often, evaluations focus more on changes in client outcomes without due consideration of fidelity to the intervention model or the effectiveness of the implementation process itself.
3
Implementation Defined
Implementation is a specified set of activities designed to put into practice an activity or program of known dimensions.
The National Implementation Research Network at the University of Southern Florida conducted an exhaustive review of research on implementation and is the primary source for this presentation.
4
A Conceptual Framework for Understanding Implementation of Evidence-Based Practices
The essential implementation outcomes are:
Changes in adult professional behavior
Changes in organizational structures and cultures
Changes in relationships to consumers, stakeholders, and system partners
5
Implementation Framework
6
Stages of Implementation
Exploration & Adoption
Program Installation
Initial Implementation
Full Operation
Innovation
Sustainability
7
Stages of Implementation
Exploration & Adoption
Identify the need for an intervention considering existing conditions
Assess the fit between the intervention and program and community needs
Prepare organization, staff, and resources for mobilizing information and support.This was essentially accomplished in the work of the Work Groups.
8
Program Installation
Program installation begins after an adoption has occurred and involves:
Ensuring availability of funding
Human resource strategies (recruitment, hiring, training, etc.)
Policy development to support change (e.g. referral mechanisms, reporting frameworks and outcome expectations)
Facility requirements
9
Initial Implementation
Initial implementation begins once conditions identified in the installation phase have been sufficiently completed.
Initial implementation must NOT be confused with full operation, as during initial implementation, many factors contributing to fidelity to the model may not be fully or even partially in place.
10
Barriers to Full Operation
Organizational Barriers
Personnel rules
Social stressors
Union stewards
Anxious administrators
Political pressures
Interpersonal rivalries
Staff turnover
11
Barriers to Full Operation
Human Barriers
Fear of change
Inertia
Investment in the status quo
Inherently difficult work of doing something differently
12
Full Operation
Full Operation ONLY occurs when:
New learning becomes integrated into practitioner, organizational and community practices, policies, and procedures.
Full case loads
Full staffing
Fidelity to the model being implemented
Only once fidelity measures are above criterion levels most of the time, can the effectiveness of an evidence-based practice be evaluated. This is a very important point.
13
Dangers of Premature Evaluation
Outcome evaluations should not be attempted until well after quality and participation have been maximized and documented in a process evaluation. Although outcome data can determine the effectiveness of a program, process data determine if a program exists at all.
Gilliam, Ripple, Zigler & Leiter (2000).
“
”
14
Innovation or Drift
When evidence-based practice meets new local conditions. Two results can occur:
Innovation: When desirable changes in the standard model are identified.
Drift: Undesirable changes in the standard model.
It is critical to implement the model with fidelity before considering innovation and to make changes only after due consideration of client benefit.
15
Innovation or Drift Illustrated
16
Sustainability
Throughout the two to four-year implementation process, the forces that led to and supported reform will change:
Staff and funding changes New social problems emerge Partnerships and political alliances change Champions and advocates move on to other
causes
Throughout the implementation process, it is essential that leadership maintain focus on sustaining the core components of reform.
17
Factors Contributing to Effective Implementation of EBP
Objective decision-making strategies that involved staff, good information about the reform and organizational leadership support during the exploration stage;
Evidence of a ‘learning culture’ within the organization implementing the reform;
A system in place for monitoring implementation;
Access to technical assistance throughout implementation;
The perceived ability of the organization to manage risks;
Belief in the validity of the reform; and most of all….
Adherence to the core components of BOTH the EBP and effective implementation (discussion follows).
18
Core Intervention Components
An essential first step to implementing EBP or system reform is to identify what are the core components of the intervention itself. Leadership must:
Carefully research alternatives Consider well-evaluated experiential learning from a
number of replications and Achieve a clear understanding of what of the model must
be maintained to achieve fidelity and effectiveness at the consumer level.
In other words, before considering the core components of any intervention, leadership must be clear about what is to be implemented.
19
Core Components for Implementation
The following Implementation Drivers influence staff behavior and organizational culture:
Practitioner Selection Preservice and inservice training A consultant coach Staff & program evaluation Facilitative administration and System Interventions
This is illustrated in the graphic that follows.
20
Implementation Drivers
21
Implementation Drivers
These drivers are integrated in their influence, in that strengths in one area can compensate for weaknesses in other areas.
The relative effectiveness of the implementation drivers is as important as the demonstrated effectiveness of the EBP itself.
22
Practitioner Selection
How and who are selected to carry out the EBP
Who is qualified to carry out the practices and programs?
What are the methods for recruiting & selecting practitioners?
How important are individuals skills, experience and personal characteristics to effective implementation of EBP?
23
Research on Practitioner Selection
There has been relatively little research to isolate the factors involved in selection of practitioners for EBP. Factors identified in effective practitioner selection for a national MST program included:
Responses to behavioral vignettes; Responses to role playing situations related to the
therapeutic; environment for which they were being considered; and
Responses to mini-training that requires behavior change.
Responses are rated with a rubric along several dimensions including: collaborative and strength focused, efforts to overcome barriers, ability to use behavioral language, uses of logical thinking and openness to feedback.
24
Research on Practitioner Selection
Morris & Stuart, 2002 are attempting to identify the generic skills needed by frontline practitioners in a transformed behavioral health field. Among those qualities considered:
Assessment skills; Family and support system involvement; Social and cultural engagement skills; Treatment skills; Methods to optimize recovery and empowerment; Consumer relationship skills; and Community resource and coordination skills.
It is interesting to note here and throughout this research, cultural competence was never discussed as an issue or factor.
25
Research on Staff Selection
A Housing & Urban Development study by Wanberg & Banas (2000) examined practitioner characteristics in the context of organizational change and found that personal resilience and self-efficacy were associated with greater acceptance of change in the work place.
This study would seem particularly important in SMCMH’s system transformation.
26
Research on Staff Selection
From the research we might conclude that as part of the implementation planning process, SMCMH leadership should:
Construct a rubric of ideal staff characteristics particularly as relates to operating in a transformed organization
Design an interview process that includes vignettes and role plays that require candidates
to demonstrate those characteristics
27
Preservice and Inservice Training
Training is an efficient way to provide:
Background knowledge, background information, theory, philosophy, and values;
Demonstration of new skills (through video or role play);
Opportunities to practice new skills through role plays and behavioral rehearsals; and
Feedback in a safe training environment.
28
Preservice and inservice training
Role playing & behavior rehearsals are critical for practicing new skills in training.
Role playing asks you to pretend you are someone else and try this… which builds empathy
Behavioral rehearsals asks you to be in your practitioner role and you are asked to confront a specific situation and perform your practitioner role in reaction to that situation ….which serves as direct preparation for the real thing.
29
Recommendations for Training
Model or demonstrate new skills using role play, behavior rehearsals and video tape.
Emphasize practice and use feedback on practice to teach the finer points of mapping.
Use practice sessions to help trainees integrate thinking and doing.
Provide guidance with respect to the boundaries of using the technique, describing when it may be useful and when it may not be useful.
Provide guidance on the flexible use of the core components.
Encourage peer and administrative support.
30
Limitations of Training
Numerous evaluations on the impact of training in health and human services have found that training alone rarely impacts practice.
The “train-and-hope” approach (Stokes & Baer, 1977) to implementation does not appear to work.
Kelly et al (2000) in a study of HIV service organizations reported the largest increase in adoptions of HIV service guidelines occurred when consultation was added to training.
A meta-analysis (Davis, 1995) found similar results in medicine. Davis concluded that “formal CME conferences and activities without enabling or practice reinforcing strategies, had little impact.” (page 700)
31
Limitations of Training: Spray & Pray
While training may introduce knowledge, philosophy, and new approaches, there is no research study that has demonstrated significant changes in practice resulting from an intervention that provided only training.
The limitations of training are directly related to the way in which adult learning occurs.
32
Stages of Adult Learning
Generally adult learning progresses in stages:
Orientation & new learning
Mechanical use
Routine use
Refinement
Integration
Innovation
33
Challenges of Learning New Behaviors
Based upon decades of research, Joyce & Showers (2002) concluded that
The newly-learned behavior is crude (i.e. mechanical) compared to performance by a master practitioner.
Newly-learned behavior is fragile and needs to be supported in the face of reactions from consumers and others in the service setting.
Newly-learned behavior is incomplete and will need to be shaped to be most functional in a service setting.
34
The Importance of Coaching
While most skills can be introduced in training, newly-learned behaviors are only really learned on the job with help of a consultant or coach precisely because:
The challenges of adopting new behaviors can’t be replicated in a training.
Single-point-in-time training needs to be reinforced continuously in the work setting for behavior changes to be sustained.
35
Consultant or Coach Support
Implementation of EBP requires changes in behavior at the practitioner, supervisory and administrative levels.
Training and coaching are the primary strategies in which behavior change is brought about.
In addition to training, substantial hands-on coaching and practice may be necessary before a counselor feels comfortable with a new strategy.
~ Dansereau & Dess (2002)
“ ”
36
Components of Effective Coaching
Coaching must be work-based, opportunistic, readily available, and reflective (Spouse 2001). Spouse described four roles for the coach:
Supervision
Teaching while engaged in practice activities
Assessment and feedback
Provision of emotional support.
37
Factors that Impact Coaching
Amount of time devoted to coaching (MST coaching occurs once or twice a week for 90 minutes) (Schoenwald et al., 2000)
Includes direct observation of provision of direct services (Smart et al., 1979)
Utilizes coaches who are expert in the content, techniques, and rationales of the program (Denton, Vaughn & Fletcher 2003)
Coaching relationships established during the training experience (Smart et al., 1979)
38
Other Factors Contributing to Effective Coaching
Walker, Koroloff & Schutte (2002) identified four additional factors that accounted for 62% of the variance in the perceived impact of supervision and coaching on practice. The Supervisor (coach):
Taught new skills;
Strengthened confidence;
Offered safety in sessions;
Devoted time to discipline-specific skills
39
Personal Qualities of a Coach
Encouraging & enthusiastic
Supportive
Committed
Sensitive
Flexible
Respectful & diplomatic
Willing to share information, credit and recognition.
40
Barriers to Effective Coaching
Inadequate time allotted to coaching & scheduling conflicts Role confusion due to the dual role of supervisor and coach Feelings of inadequacy on the part of coaches (hence the
importance of selecting skilled coaches) Poor match between coach and practitioner Labor relationships that don’t support observation and
feedback Resistance in the organizational culture Absence of strong leadership & commitment to
implementation Focus on paperwork compliance as opposed to changes in
practitioner behaviors Cost
41
Research on the Impact of Coaching
The study below is based on an meta-analysis of hundreds of studies in education. Note the dramatic differences in impact upon practice.
42
Staff Evaluation
Evaluation provides critical feedback to practitioners, trainers and managers related to:
Fidelity to the model being implemented
Effectiveness of training and coaching strategies
Impact of intervention upon clients
Progress of implementation itself
43
Components of Evaluation
Staff & program evaluation and fidelity seem to consist of some combination of measures of:
Context—prerequisites that must be in place to operate (staffing, qualifications, ratios)
Compliance—extent to which the practitioner uses the core intervention components as prescribed by the EBP and avoids practices proscribed by the EBP
Competence– the level of skill shown by the practitioner in using the core intervention components
44
Effective Staff Evaluation Systems
Huber et al., (2003) described a highly effective hospital management system for staff recruitment and evaluation that included:
Ongoing training and education focusing on specific skills; Cross-training on related roles, and in-services Monthly dinners for discussion; Performance evaluations based on direct observation to assess
practice knowledge, communication skills, and use of time; Prompt verbal feedback followed by a write up with
recommendations; and Quality improvement systems to keep the system on track.
45
Effective Staff Evaluation
In other words, in a highly effective system, staff evaluation is part of a sequence of supports designed to have good people well prepared to do an effective job.
46
Staff Evaluation & Fidelity to a Model
One of the most critical purposes of staff evaluation in the context of implementing EBP is to test practitioner fidelity to the EBP core components.
47
Challenges to Measuring Fidelity
McGrew et al., (1994) noted that development of fidelity measures is hampered by three factors:
Most treatment models are not well defined conceptually making it difficult to identify core intervention components;
When intervention components are identified, they are not operationally defined with agreed-upon criteria; and
Only a few models have been around long enough to study planned and unplanned variations.
48
Staff Evaluation & Fidelity to a Model
Fortunately the SMCMH transformation relies upon implementation of EBPs with effective fidelity instruments already in place.
Multisystemic Treatment (MST) utilizes the Therapist Adherence Measure, a 27-item measure used in phone interviews with parents.
The Wraparound Fidelity Index (WFI) consists of asking facilitators, parents and youth to rate 11 dimensions of services.
Assertive Community Treatment (ACT) utilizes a 73-item tool with 17-item subset used to construct a fidelity index with three subscales: staffing, organization, and service.
49
Fidelity to System Transformation
A major challenge to SMCMH is to go beyond program level (MST, Wraparound, ACT) fidelity and to construct a fidelity index that reflects the qualities of a transformed system as defined and described in our plan.
50
Theory of Change & Fidelity Models
Possible resources to explore for developing a fidelity model for the transformation…
Hodges, Hernandez, Nesman & Lipien (2002) demonstrated how a theory of change exercise can help programs clarify their strategies to develop fidelity measures.
Shern, Trochim & LaComb (1995) used concept mapping to develop fidelity measures for a mental health program.
51
Program Evaluation
Program Evaluation assesses key elements of the overall performance of the organization in facilitating the implementation of the program or practice.
52
Facilitative Administration
Facilitative administration
Provides leadership;
Makes use of data inputs to inform decision making;
Supports overall processes; and
Keeps staff organized and focused on the desired clinical outcomes.
53
Systems Interventions
Systems interventions are strategies to work with external systems to ensure the availability of the financial, organizational, and human resources required to support the work of the practitioners.
54
Relationship between Core Components & External Factors
55
Relationship between Core Components & External Factors
56
Key Elements to Organizational Change
Implementation of EBP almost always requires organizational change. Research points to 8 key elements to organizational change:
Commitment of Leadership to the Implementation Process (Hunter, Hunter & Rogers, 1993). Research indicates that leadership takes many forms, including to:
Initiate and shepherd the organization through the complex change process
Set explicit goals, communicate them clearly throughout the organization, resolve conflicts with other goals, and reinforce persistence
Help create the details of activities, processes, and tasks in order to operationalize implementation policies
Inspire, guide, and provide direction Recruit, select, train, locate, advance, promote, or dismiss
employees to further the aims of implementation policies
57
Other Key Elements to Organizational Change
Involvement of stakeholders in planning and selection of programs to implement; (Bierman et al., 2002)
Creation of an implementation task force made of consumers, stakeholders (including unions), etc. (Joyce & Showers, 2002)
Suggestions for “unfreezing” current organizational practices (Cheung & Cheng, 1997)
Resources for extra costs, effort, equipment, manuals, materials recruiting, access to expertise, re-training for new organizational roles; (Phillips, et al., 1978)
58
Other Key Elements to Organizational Change
Alignment of organizational structures to integrate staff selection, training, performance evaluation, and on-going training (Blasé et al., 1984)
Alignment of organizational structures to achieve horizontal and vertical integration (Unger et al., 2000)
Commitment of on-going resources and support for providing time and scheduling for coaching, participatory planning, exercise of leadership, evolution of teamwork and for generating and using local data (Park & Han, 2002)
59
Considerations for Effective Implementation
At a system level and with staff and consumer input, develop a rubric that depicts what a ‘transformed system’ would look like at the system and program levels.
60
Considerations for Effective Implementation Construct site-based highly structured
exercises for sites working with their consumers to identify how focusing on this rubric would require changes in daily operations.
How would you change staff meetings, staff evaluation, hours of operation, charts, reports, referrals, office operations, schedules, the waiting room?
61
Considerations for Effective Implementation Develop program level rubric that
describe the extent to which a program is achieving fidelity to operating in conformance with the new transformed system.
Build in personnel and program evaluation components that incorporate how staff performance and program structures are supporting the transformation.
62
Considerations for Effective Implementation
At a structural level, consider a wide variety strategies that reinforce fidelity to transformation and the various program EBPs.
Examine what specific practitioner and program changes are important and build a system of reinforcements that extend the impact of training.
63
Considerations for Effective Implementation Be mindful that changing behaviors is
challenging and threatening. Training alone will not generate change in
practitioner behavior or program priorities. Since implementation drivers are
integrative and compensatory, everything you do administratively to facilitate and support changes, will reinforce the other drivers and compensate for where drivers are not as strong.
64
Considerations for Effective Implementation In so many ways, implementation is
about the small things you do: Small changes in intake forms, assessment tools, client
engagement protocols, and client charts, changes that subtly cause practitioners to operate slightly differently
Notes at the top of agenda and specific standing agenda items that focus on transformation issues and allow staff dialogue and input into implementation
E-reminders to managers to observe practitioner groups Changes in personnel review tools to reinforce transformation Use of funds to foster individual staff research and
presentations to teams about issues related to implementing transformation strategies.
65
Considerations for Effective Implementation
In closing, Effective implementation of transformation is about:
Selection of the most effective, high-leverage programs;
Providing training and work-based support to reinforce the values and practices inherent in a transformed system; and
Building structures, procedures and policies that reinforce the priorities of the transformation…
And the devil is in the details. Implementation is All.