Measuring Clinical Skills: A Unique Challenge in Fidelity Assessment

Kim T. Mueser

Dartmouth Psychiatric Research Center

Gary Bond: Man Without a Planet or Brother from

Another Planet?

• Innovative thinker• Math major• Beloved by students & colleagues alike• Unflappable • Force to be reckoned with in psychiatric rehabilitation• Honest, direct, modest, sensitive, humorous• Clearly not of this world, but is he without a planet or…

Fidelity: Adherence & Competence

Adherence • The extent to which the defining characteristics of a

standardized intervention are implemented in practice

Competence• The skillfulness & effectiveness with which a

standardized intervention is implemented

Potential Data Elements of Fidelity Assessments

• Record reviews• Clinician/administrator interviews• Consumer interviews• Knowledge tests• Role play tests• Direct observation• Audio/videotape observation

Methods for Assessing Fidelity Depend on:

Definition of the “treatment provider”:• Individual• Team• AgencyDefining elements of practice:• Organization & staffing (e.g., ACT team, voc unit)• Structure, aim, & scope of services (e.g., rapid job search,

services provided in community)• Clinician:consumer interactions (e.g., structure, content, & skill of

clinician working with consumer)

Practices That Have Emphasized Organization &

Structure

Assertive Community Treatment:• 1:10 staff:client ratio• Shared caseloads• Daily team meetings• Services provided in community• Direct, not brokered, service provision

Supported Employment:• Zero exclusion• Focus on competitive jobs• Rapid job search• Follow-along supports• Consumer preferences

Practices That Have Emphasized Clinician:Client

Interactions

Integrated Dual Disorder Treatment, Illness Management & Recovery:

• Psychoeducational skills• Motivational interviewing• Cognitive-behavioral teaching skills

Relevance of Clinical Skills to All Evidence-Based Practices

• Importance of the relationship between clinician & consumer

• Utility of specific skills in changing behavior or achieving specific objectives

• Differences between clinicians in their effectiveness despite adherence to organizational & structural aspects of practice

• Potential to improve quality of services through targeted training of specific skills

Assessment of Clinical Skill at Implementing a Practice

• Limited utility of interviews with clinicians or consumers as skills must be inferred

• Knowledge tests: informative about clinician’s understanding of practice, but not ability to do it

• Role play tests: informative about clinician’s ability to use specific clinical skills in simulated situations

• Direct observation: informative about clinician’s use of clinical skills during actual interactions, but may be reactive to presence of observer

• Audio/videotaped observation: informative about clinician’s use of clinical skills during actual interactions, but requires dealing with concerns about recording interactions

Advantages of Formal Assessment of Clinical Skill at

Practice• Identifies areas in need of subsequent training• Facilitates understanding of differences in effectiveness of

clinicians implementing same practice• Useful for research focused on understanding differences between

consumers in benefit from a practice• May lead to certification of clinicians in practices for which fidelity

is defined mainly in terms of the interaction between the clinician & client:

• Illness Management & Recovery• Family Psychoeducation (e.g., multi-family groups, behavioral family

therapy)• CBT (e.g., for psychosis, PTSD)

Two Examples of Clinical Skill-Based Fidelity Scales

• IMR Clinical Competency Scale

• CBT for PTSD Fidelity Scale

IMR Clinical Competency Scale

• Developed by Gingerich, Mueser, & Meyer • Initial step of brainstorming core skills necessary to implement

IMR, based on principles & specific strategies outlined in manual• Initial draft of competency items, behavioral anchors, & multiple

choice questions• Feedback on initial draft from 25 experienced IMR clinicians &

trainers• Revised draft & additional feedback obtained• Scale piloted with experienced IMR clinicians & trainers• Feedback obtained, revisions made

Overview of Scale

• 19 items, 5-point ratings, behaviorally anchored• Administered to individual providers with

experience treating at least 3 individual clients with IMR or leading/co-leading 1 group

• Ratings based on:• Observation of 2 IMR sessions • Interview with clinician, including role plays• Review of written materials

Overview of Scale (cont.)

• Administered by experienced IMR clinician, trained in implementing scale

• Requires about 6 hours to complete• Organization of scale:

• Process skills• Motivational enhancement strategies• Educational strategies• Cognitive-behavioral strategies• IMR group leader skills• IMR curriculum

Materials in the Evaluators’ Manual

1. Steps for Evaluators (in checklist form)2. Competency Items in the MN IMR Clinical

Competency Scale• Item description• Skill requirements• Primary measurement• Missed Opportunity/Comment Boxes

Materials in Manual, cont’d

3. Interview Guide• Questions for the interview section• Examples of role plays for evaluation

4. Scoring Manual• 5-point behaviorally anchored scale• Descriptors for 3 anchor points

5. Workshop materials

Steps for Evaluators Completing the Scale

1. Clinician meeting-preparation for evaluation2. Observe or listen to audiotapes of 2 sessions

(group or individual or both)3. Review written materials4. Conduct clinician interview-evaluate skills not

demonstrated in observation and follow-up any questions from observation

5. Feedback session-offer praise for strengths and suggestions for areas of improvement

11 Items to be rated using behavioral observation

• 1. Therapeutic relationship• 2. Structure of sessions• 3. Efficient use of time• 4. Motivational enhancement

strategies• 6. Goal follow-up• 7. Educational strategies

• 8. Use of reinforcement • 9. Home assignments• 11. Relaxation Training (if one of

sessions includes this topic)• 15. Group skill: Involving all

members of group• 16. Group skill: Enlisting support

between group members• 17. Group skill: Tailoring IMR

materials to experiences and goals of each group member

Items to be rated using Written Materials

• 5. Goal Setting• 6. Goal Follow-up• 10. Relapse Prevention Training• 18. Comprehensive Use of IMR Curriculum• Questions about written materials should be

followed up during the clinician interview

How to evaluate the goal tracking sheets?

• What to look for?• Recovery goal • Short-term goals-how many and are they related to the long-

term goal• Steps-how many and are they small enough• Any modifications?

• Remaining questions for Clinician Interview• See Interview Guide• Additional questions for clarification:

• Review how the client decided on the goal • Why did the client choose the short-term goals listed?

Items to be rated using the Interviewer Guide (* Role

Plays)• 11. Relaxation Training*• 12. Behavioral Tailoring and

other medication adherence strategies

• 13. Social Skills Training*• 14. Coping Skills Training• 19. Knowledge of IMR

modules

• Followed up from written materials review:

• 5. Goal Setting• 6. Goal Follow-up• 10. Relapse Prevention

Training• 18. Comprehensive Use of

IMR Curriculum

Feedback Meeting With Practitioner & Supervisor

• Conducted as soon as possible after completion of the interview

• Explain purpose of meeting is to review results of evaluation

• Give each person a copy of the Scoring Summary Sheet• Start by noting practitioner’s strengths, followed by areas

that might be improved• Briefly review ratings and the basis for them• Provide suggestions for addressing problems areas

CBT for PTSD Program• 12-16 individual sessions, for consumers with SMI & PTSD• Components of intervention:

• Orientation, crisis plan, breathing retraining (1 session)• Psychoeducation about PTSD (2 sessions)• Cognitive restructuring, taught as a skill to manage distressing feelings &

address trauma-related thoughts & beliefs (9-13 sessions)

• Supported by one RCT in SMI; multiple other studies underway in other special populations (e.g., addiction, disaster survivors, adolescents)

CBT for PTSD Fidelity Scale

• Ratings based on audio recordings• 17 items rated on 5-point Likert scales• Narrative feedback on specific ratings• Used to facilitate training & certification of

clinicians in treatment model• Used to verify fidelity to model over treatment

study

Items on CBT for PTSD Fidelity Scale

• 1. Agenda setting• 2. Homework review• 3. Overview of program• 4. Crisis plan• 5. Breathing retraining• 6. Use of Educational materials• 7. Psychoeducation• 8. Cognitive restructuring

• 9. Problem solving • 10. Trauma focus• 11. Assign homework• 12. Manual adherence• 13. Teaching effectiveness• 14. Interpersonal effectiveness• 15. Pacing & efficient use of time• 16. Reduction of client distress• 17. Overall session quality

NJ-Dartmouth PTSD Study• Kim Mueser, Dartmouth, PI• Stan Rosenberg, Dartmouth, Co-PI• Jen Gottlieb, Dartmouth, Project Manager• Steve Silverstein, UMDNJ, Site PI• Weili Lu, UMDNJ, Site Project Manager• Phil Yanos, UMDNJ, Clinical Supervisor• Stephanie Marcello, UMDNJ, Clinical Supervisor• Stephanie Dove-Williams, UMDNJ, Interviewer• Danielle Hawthorne, UMDNJ, Interviewer• Danielle Paterno, UMDNJ, Interviewer• Lindy Fox, Dartmouth, Assessment Supervisor• Rosemarie Wolfe, Dartmouth, Data Manager• Eric Slade, University of Maryland, Health Economist

Training Model

Annual Training

Weekly On-site Group Supervision

Practice Cases Assigned & Sessions Recorded

Fidelity Monitoring Via Tapes

If fidelity for sessions 4-16<3.5

Training Model: Monitoring treatment fidelity

• Frontline clinicians were provided with fidelity ratings for all sessions for their first practice cases

• If the overall fidelity ratings for sessions 4-16 was above 3.5, clinician was certified to take on protocol case

• If overall fidelity was below 3.5, then the clinician took a second practice case

• 17/20 clinicians were certified after 1 case, 3 after a second practice case

ParticipantsCompleters(n=26)

Participants(n=34)

Male 27% 29%

Age (M+/- SD) 46.8+/-10.6 45.6 +/-10.9

Race AA 27% 35%

EA 58% 50%

Hisp. & Other 15% 15%

CBT Session Progress (N=26)

Session Progress

0

10

20

30

40

50

60

1stsession

4thsession

7thsession

10thsession

13thsession

16thsession

Session Number

Score

BDI

PCL

Conclusions• Clinical skills are useful to rate for both training

purposes & to verify fidelity• Audio-based fidelity assessments are feasible to

conduct with frontline clinicians working with regular clients

• Clinical skills assessments have potential to improve other practices (e.g., ACT, supported employment)