October 31, 2013 - Excellence for Child and Youth · October 31, 2013. Housekeeping notes ... Management team completed Steve Scott’s five-minute personality test: Teams across

Evaluation of implementationOctober 31, 2013

Housekeeping notes

• This webinar will be recorded and will be available on the Centre’s website as an educational resource.

• The slides have been sent to participants.

• Log in with a phone whenever possible for optimal audio quality.

• We have staff online to assist with any technical difficulties.

• There will be a short evaluation survey for all webinar attendees at the end of the webinar.

Have you got a question?

• This webinar will be followed by a question and

answer period, however questions are encouraged

throughout the presentation.

• Questions can be submitted electronically or

verbally. Specifics around this process will be

clarified prior to the question and answer period.

Welcome and introductions

Today’s presenters

Ann Barber

Program Associate, Implementation Support Services,

Ontario Centre of Excellence for Child and Youth Mental Health

Michael Tross

Director of Quality Assurance,

Quality Improvement and Standards, Youth Services Bureau

Neil Slattery

Intensive Family Support, Mental Health Services,

Youth Services Bureau

Terri Kehoe

Program Manager, Youth Services Bureau

About the Centre

We bring people and knowledge together to strengthen

the quality and effectiveness of mental health services for

children, youth and their families and caregivers.

Three strategic goals:

Learning Collaboration Leadership

Foster a culture of

organizational learning

to support agencies in

using evidence to

improve client outcomes.

Build and develop

collaborative

partnerships to sustain

capacity within mental

health services.

Be a true learning

organization and lead

by example.

The learning journey

Goals of this webinar

1. To increase knowledge of the importance

of monitoring and evaluating the process

of implementation

2. To increase awareness of tools and

strategies for evaluating implementation

3. To provide examples from the field

The gap

I M P L E M E N TAT I O NI M P L E M E N TAT I O N

(NIRN, 2011)

Evidence Practice

Why discuss evaluation of implementation?

• Gap in the literature

• Gap in the field

• Growing interest in implementation science

• Growing experience in evaluation

Implementation vs. outcome evaluations

Implementation outcomes

Outcomes evaluation

(summative)

Asks so what?Client outcomes

Implementation evaluation

(process or formative)

Asks what?

�Audience interaction:

�� Poll: Where are you in terms of evaluating your

implementation process?

1. Both implementation and outcome evaluations are in our plans.

2. We have discussed client but not implementation outcomes.

3. We usually focus on implementing first and evaluating later.

4. We are new to this process and unsure of what to evaluate.

Why evaluate implementation?

• To ensure the integrity of the EIP

• To provide direction to decision makers

• To enhance program accountability

• To differentiate theory failure from implementation failure

• To enhance sustainability

Implementation framework

Implementation framework

L E A R N I N G O R G A N I Z AT I O N

Phase 2

Doing

Training on specific

practices, implementing,

adapting and evaluating

Phase 1

Planning

Getting people and systems

ready for change

Phase 3

Sustaining

Learning to continually

use new evidence to

improve practice

(Fixsen, D., Naoom, S., Blasé, K., Friedman, R., & Wallace, F., 2005)

LEARNING ORGANIZATION

Setting up and implementing a monitoring and evaluation system

Developing evaluation

questionsEvaluate implementation

Evaluate implementation drivers and factorsEvaluate implementation drivers and factors

Phase 2

Doing




Phase 1

Planning


ready for change

Phase 3

Sustaining


use new evidence to

improve practice

Evaluate client outcomes

Evaluation activities for EIP implementation



Developing evaluation

questions

Phase 1

Planning


ready for change




Evaluate implementation


Phase 2

Doing







Phase 3

Sustaining


use new evidence to

improve practice

Evaluate client outcomes


� Audience interaction:

�� Poll: What elements of implementation

have you evaluated in the past?

1. Fidelity to the EIP model

2. Program Fit

3. Training on EIP

4. Other… please specify by typing in message box

5. Not applicable. We have not yet evaluated implementation

Examples from the field:

A conversation with

Youth Services Bureau

Background

• Youth Services Bureau (YSB) has a long tradition of exploring innovative programming and supporting a culture of continuous development. It is the agency’s goal to encourage the usage of Evidence-informed practice (EIP) wherever possible.

• When the Ontario Centre of Excellence for Child and Youth Mental Health awarded YSB a three-year grant in January 2012, we selected trauma-focused CBT and are now beginning our work with clients.

What is TF-CBT?

Trauma-focused cognitive behavioural therapy (TF-CBT)

is a components-based model of psychotherapy.

Psycho-education

Parenting skills

RelaxationCoping with

emotions

TF-CBTTF-CBT

What is TF-CBT?

Additional components include:

Cognitive processing

Trauma narrative

In vivo mastery Conjoint

parent-youth

TF-CBTTF-CBT

Youth Services Bureau – Ottawa:

Evaluation of implementation

Planning phase – preliminary

Struck an organizing committee that included:• the Executive Director and the Associate

Executive Director

• one senior representative from each clinical department

It was a small committee of six members (agency has

300 workers). Its small size and variety of roles resulted

in excellent brainstorming capacity and follow-through.

Broke down the three phases into sub-phases:

• Planning 1 (early) and 2 (late)

• Doing 1 (early) and 2 (late)

• Sustaining

Planning phase – preliminary

Planning phase 1 - early

Expected and normal challenges in implementing

a new EIP:

• EIP will be seen as restrictive or inflexible

• belief that another EIP should have been chosen

• EIP will be looked at as a criticism of existing work

• EIP may increase workload and be more time consuming

• staff will question the effectiveness of the EIP

• concern about inadequate time to prepare for EIP

• concerns about adequate supervision/consultation


• Full-day learning journey training on team-building, leadership and communication modules for all senior and middle managers provided by the Centre.

• Change management book Switch provided to all senior and middle managers.

• Ongoing short presentations about aspects of PACE and TF-CBT to the Board of Directors and management team.


Management team completed Steve Scott’s five-minute

personality test:

Teams across the agency were encouraged to use

Lencioni’s Team Assessment from his book The five

dysfunctions of a team.

Planning phase 2 - late

• Developed a vision statement and article that was posted on Y-Space (internal agency website).

• Hosted a two-day TF-CBT get acquainted EIP training for 25 managers and staff.

• Initiated planning to expand the committee following training - maximum 12 members.

Planning phase 2 - late

• Invited the Centre to facilitate a half-day training on Leading the change journey

for senior/middle managers.

• Considered National Implementation Research Network (NIRN) materials that could be adapted for fidelity measures.

Planning phase 2- late

• Constructed our first logic-model based

implementation plan with goals, indicators,

deadlines and achievement measures.

Goals Indicators

Doing phase 1 - early

• Welcomed our newly expanded

implementation team

• Further team review of

fidelity measures and

outcome measures


Regularly reviewed and updated the

implementation plan goals, indicators and

achievements. Key areas of focus included:

• staffing

• supervision

• training/coaching

• fidelity measures

• outcome measures


Developed from our implementation plan a more detailed critical components business plan document, analyzing in detail needed resources. For example:

• forecasted clinician and supervisory hours per client or staff

• hierarchy of programs for optimal EIP integration

• referral parameters

• potential communication issues

• purchase of needed materials (books and therapeutic materials)


In this phase, a supervisor of an existing EIP in our

agency provided valuable feedback and tools. We

encourage other agencies to tap into their existing

in-house EIP resources.

We organized a two-day EIP clinician training and

invited newly trained clinicians to join the

implementation team.

Doing phase 2 - late

• The implementation team began consideration of agency-wide trauma-related training.

• The implementation team reviewed and approved six fidelity measures, outcome measures and screening/assessment/referral tools.

• We invited a manager and program staff of a local long-term practitioner of TF-CBT to present to the implementation team.

Fidelity measures

• TF-CBT brief practice checklist: completed by the clinician following each session.

• Supervisor adherence measure: completed by the clinician on a quarterly basis.

• Data tracking form: completed by the clinician on a monthly basis – includes roll-up information on time spent in supervision and consultation, number of client sessions and caregiver contacts, treatment status and self-rated TF-CBT key component skills.

Youth Service Bureau of OttawaTF-CBT supervisor adherence measure

Fidelity measures

• Client-rated clinician measure: completed by the

client on a quarterly basis, rating the clinician on

fifteen key components of TF-CBT practice.

• Overall performance metrics summary: completed

by the supervisor and QA Director on a semi-annual

basis, this compares actual results with target

indicators for each metric in 31 key areas of client

data, clinician and supervision data and client

outcome data.

Not applicable

0

Never

1

A little

2

Some

3

Mostly

4

A lot

5

1) The counsellor and I worked together on my traumatic events

2) The counsellor understands my traumatic event(s)

3) The counsellor has talked to me about normal reactions to trauma

4) The counsellor has a good understanding of my strengths

5) The counsellor has talked to me about relaxation and shown me some relaxation exercises

6) The counsellor has helped me recognize my feelings

7) The counsellor has helped me figure out how to regulate my emotions by positive self-talk or

thought stopping or in other ways

8) The counsellor has helped me become aware of the connection between my thoughts, my

feelings and my behaviour

9) The counsellor has helped me come up with new ways of looking at what I've experienced that

are more helpful and that have made me feel different

10) The counsellor and I were honest and straightforward about the actual traumatic events that

happened to me

11) The counsellor has helped me figure out ways to live my life that are less stressful

12) The counselor has given me helpful strategies to feel safer than I did before

13) The counsellor has helped me learn new problem-solving skills or new social skills

14) The counsellor meets with my parent or caregiver regularly

15) The counsellor has met with my parent (caregiver) and me to talk about the traumatic events I

experienced

Youth Services Bureau of Ottawa Trauma - Focused-CBT

Quarterly Client-Rated Clinician Fidelity Measure

Clinician: Date of interview:

Interviewer: Client interviewed:

TF-CBT implementation progress

Implementation tracking graph: derived from

Fixen and Blase’s work with the National

Implementation Research Network, this method of

tracking the status of program implementation

assigns scores to certain practitioner stages – from

a score of 0 when the position is vacant, to a score

of 5 when the practitioner has met fidelity criteria

for 10 of the past 12 months.

Full

Implementation

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

13:01 13:04 13:07 13:10 14:01 14:04 14:07 14:10 15:01 15:04

I

m

p

l

e

m

e

n

t

a

t

i

o

n

Q

u

o

t

i

e

n

t

Progress Assessment Dates

YSB TF-CBT Implementation Progress

(N = 7 Positions)

Client outcome measure

• UCLA PTSD index and rating scale: completed

by the client at the assessment and case

closure stages.

• Comprises 27 screening and assessment

questions and 22 questions regarding

symptom frequency.

• The 22 symptom frequency questions will be the

source of the outcome measure data.

Doing phase 2 - late

• Our clinicians visited the neighbouring EIP

practitioner agency.

• We hired a telephone consultant.

• Initiated telephone discussions with other Ontario-

based practitioners of our selected EIP regarding

experiences and training. This broadened our

network of potential future support and resources.

Sustainability

• Our EIP meets an ongoing client profile need across

the agency.

• Our EIP was integrated into existing programs’

activities and no additional program funds

were required.

• The programs into which our EIP was integrated

were natural fits vis-à-vis client profile and staff

time-management flexibility.

Sustainability• Ongoing training and education for staff and

management are planned on the topics of TF-CBT and

change management issues.

• Ongoing soft communication with agency staff re:

TF-CBT news and outcomes.

• Quarterly and semi-annual reviews of fidelity measure

results and outcome measure results.

• Continuous review of how to integrate fidelity and

outcome results into context, compliance and

competence enhancements.

Centre tools and resources

• Learning modules

• Implementation support services capacity checklist

• Webinars (current and archived)

Tools and resources

Top three recommended articles

Chaudoir, S. R., Dugan, A. G. & Barr, C. H. (2013).

Measuring factors affecting implementation of health innovations: a systematic review of

structural, organizational, provider, patient and innovation level measures.

Implementation Science, 8, 22.

Durlak, J. A., & Dupre, E. P. (2008).

Implementation matters: A review of research on the influence of implementation on

program outcomes and the factors affecting the implementation. American Journal of

Community Psychology, 41, 327-3.

Patton, M.Q. (2008).

Utilization-Focused Evaluation (4th ed.). London: Sage Publications.

Questions?

QUESTIONS?

Questions or comments?

• To submit questions electronically, use the

question box located in your control panel.

• To submit questions verbally, use the raised

hand icon also located in the control panel.

Implementation support servicesAnn Barber, MA

Program Associate, Implementation Support Services


[email protected] ext. 3483

Sandra Huang Del Frari, M.Ed.

Program Associate, Implementation Support Services



Mark MacAulay, MA

Manager, Implementation Support Services



Laura Conroy

Research Assistant, Implementation Support Services



Upcoming webinars

Evaluation:

Conducting multi-site evaluations

November 5 at 1p.m.

Implementation:

Maximizing the lifespan and effectiveness of EIPs

over time: A webinar on sustainability

November 27 at 1p.m.

Evaluation

Please don’t forget to complete the survey at

the end of this webinar.

Your feedback is very important to us, so

we thank you for taking the time to share

your thoughts!

Documents

October 31, 2013 - Excellence for Child and Youth · October 31, 2013. Housekeeping notes ... Management team completed Steve Scott’s five-minute personality test: Teams across