Integrating Assessment into

CPD Programs September 26, 2017

Kate Hodgson DVM, MHSc, CCMEP

Suzan Schneeweiss MD, MEd, FRCPC

Faculty/Presenter Disclosure

• Faculty: Kate Hodgson and Suzan

Schneeweiss

• Relationships with commercial interests:

–None

Learning objectives

At the end of this session, you will be able to:

• Apply Moore’s model of outcome assessment in a

strategic curriculum

• Integrate performance and knowledge assessment

within CPD programs for Royal College or

MAINPRO+ credits

• Use a flipped classroom for deeper engagement and

relevant practice application of competency

development

Program Evaluation Versus Outcome

Measurement

• CPD Program Evaluation measures program

learning environment, speaker and content

• CPD Outcome Measurement measures

changes in learners and practice and resultant

patient and health care changes

Program Evaluation versus Outcome

Measurement

Program Evaluation

• The skill of the speaker

• Quality/relevance of

the information

presented

• The venue

• The timeframe for the

session

Outcome Measure

• Participants’

knowledge skills

attitude

• Practice Change

• Patient Care

• Health care outcomes

And now for the bigger challenge …..

Impact of CPD on Physician Performance

and Patient Outcomes

Outcome Number of Studies % positive

Knowledge 22 / 28 79

Skills 12 / 15 80

Attitude 22 / 26 85

Practice behavior 61 / 105 58

Clinical outcomes 14 / 33 42

Cervero and Gaines JCEHP 2015• Systematic reviews published since 2003• CPD has a positive impact physician performance, less

reliably on patient health outcomes

Impact of CPD on Physician Performance

and Patient Outcomes

Greater improvement in physician performance

when programs:

• Focus on outcomes considered important by

physicians

• Are more interactive

• Use multiple methods of learning

• Involve multiple exposures Cervero and Gaines JCEHP 2015

CPD Outcomes

“ … should focus on identifying,

measuring, and describing the value provided

by CME that leads to enhanced physician

performance, improved health care quality and

reduced cost.”Moore D.E. Framework for outcomes evaluations in the continuing

professional development of physicians. In Davis D et al. The

Continuing Professional Development of Physicians. Chicago. AMA

Press, 2003: Chicago. P251.

Why measure CPD outcomes?

• Leads to more effective, better-targeted

education

• Provides a road map to future education

• Demonstrates value to internal and external

clients

• Accreditation standard

Model for Assessing Outcomes in CPD

Wallace S & May SA, Vet Record 2016; Adapted from Moore 2009 and Miller 1990

Application of Moore’s Outcome

Evaluation Framework

Model combines stages of learning, instructional design

and planning, outcomes framework and assessment:

1. Start with the end in mind

2. Take stages of learning into account

3. Focus on clinical problems and

knowledge that can be used in practice

4. Provide opportunities for practice and feedback in

authentic settings.

Stages of Learning and Change

Level 3: Learning

Knowledge

(e.g. MCQ or true/false test)

Skills

(e.g. psychomotor,

decision-making,

interpersonal)

Attitude

(e.g. questionnaire)

3 domains of learning• Can be done using pre/post evaluation

Small Group Discussion

• Consider a program you are developing or

have developed.

• How might you incorporate measurement of

learning in your program?

Enhancing Learning in CPD Conference

• SickKids Annual Paediatric Update

Conference

• Integration of test-enhanced learning in

concurrent workshops

– 5 pre and post MCQ questions

– Immediate feedback on post MCQs via pop-up

– 1-month later asked new questions

• Accredited Royal College Section 3

Testing Effects

• Test-enhanced learning (TEL)

– Testing after learning

– Increases knowledge retention and transfer

• Test-potentiated learning (TPL)

– Pre-testing to enhance subsequent learning

opportunity

– Identify knowledge gaps and promote self-

regulation behaviours such as reflection, strategic

studying or learning and metacognition

Test Enhanced Learning

• Testing promotes active retrieval of information from

memory

• With repeated testing, more likely to successfully

retrieve information in future context

• Long-term retention of materials is enhanced when

learning events are separated temporally from one

another.

• Feedback enhance TEL effects

Level 4: Competence

(Shows How)

• Requires demonstration and evaluation

• Ensures that person has learned something that

could be applied accurately

• Can be done with:

– physical skills

– interpersonal skills

– reframed approaches

– assessment process

Level 4: Competence (Shows How)

Demonstration to Peers

Demonstration to Instructors

Demonstration with Simulated

Patients (Role Play)

Tested Skills

Small Group Discussion

How might you integrate competency in your

program(s)?

Clinical Skills

in Communication

▪ Work in groups of 3: • The Physician

• The Patient

• The Observer

▪ Rotate roles through each of

three cases.

▪ Physician & patient may or

may not have private

information.

Observer Checklist:

Patient Communication

As an observer, you

will have a checklist

to track physician’s

techniques in

managing challenges

presented by Patient.

Observer Checklist

Case #15: Request for Dose Escalation

Please complete the following Observer Checklist. Yes NoDid the Care Provider comply with the patient’s request for dose escalation?

□ □

Did the Care Provider non-judgmentally explore the reasons for request?

□ □

Did the Care Provider specifically track or evaluate function? □ □

Did the Care Provider explain the connection between negative mood and pain?

□ □

Did the Care Provider explore changes in pain levels? □ □

Did the Care Provider review medications trialed? □ □

Patient Communication

Abby and Dr. Doshi

Case #16:

Abby and Dr. Doshi

▪Abby is a 21 yr old female

patient who suffered an

motorcycle accident 13

months ago, resulting in a

pelvic fracture.

▪ Since this time, she has

had ongoing pain despite

normal healing.

Case #16:

Abby and Dr. Doshi

▪ She has not had a family

physician until recently;

Dr. Doshi has now taken

her on as a patient in his

family health team.

▪ She is here today to

discuss getting something

to help her pain.

Example Level 4: Competence

• SickKids Hospital competency-based

workplace learning for Paediatric

Emergency Physicians

– High-risk procedures/ resuscitation

scenarios

– Flipped classroom approach (online

videos)

– Deliberate practice and rapid cycle

debrief

– Peer coaching and feedback

– Checklist for performance assessment

Workplace Learning

• Authentic, realistic, context for learning and

assessment

• Controlled, safe learning environment with

peers

• Instruction tailored to individual or group

needs

• Repetition and deliberate practice

• Providing feedback/ drive self-awareness

Increasing Engagement with SIM

• Team competency

– Crisis resource management

– Team performance management

• Individual Competency

– Procedures

– Critical incidents

– Leadership

– Communication

– Decision analysis

In-Situ Simulation in CPD

• Team Training

• In-house training in clinical setting

• Blend of simulation in real working

environments

• Assessment of teamwork effectiveness

– TEAM (Team Emergency Assessment Measure

(TEAM)Cooper et al. Resuscitation 2010;81:446-452.

TEAM Scale

Teamwork observational scale to assess performance of emergency medical teams4 Domainso Leadership,

teamwork, task management, global score

Level 5: Performance (Does)

• The learner does what he he has learned

– Follow up of practice change

• Subjective or self-report (e.g. post activity

survey, commitment to change)

• Direct observation in practice

• Indirect objective measures – quality and

utilization measures

Methods for Capturing Self-Reported

ChangeQualitative

• Online or mailed paper surveys

• Phone interviews

• Change reported (and recorded) at subsequent

meetings or educational sessions

• Focus groups

Quantitative

• Online or mailed paper surveys

Level 5: Performance (Does)

Products sent in for feedback

(e.g. videotape)

Case Study

(e.g. chart stimulated recall)

Databases

(e.g. length of stay, return visits)

Chart Audits

Commitments to Change (CTC)

Three Stages

• Participant asked to write 1 – 5 changes they plan to

make a change as a result of activity

• Asked to indicate a level of commitment utilizing a

Likert scale ranging from 1-5.

• 30 – 45 days participant is sent a list of these changes

and asked to indicate if a change occurred, partially

occurred or did not occur & why

Commitment to Change

• Allows participants opportunity to reflect on

salient pieces of information and extract

meaning in a personalized fashion

• Rating level of commitment is a mechanism

for how strongly one feels the goals should be

actualized

• Set up sense of accountability

Commitment to Change (CTC)

• Rates of compliance to CTC between 47-87%

• Dose response relationship between time in course

and number of changes reportedLockyer et al. JCEHP 2001

• Physicians who generated their own CTC, more

likely to implement changes in observed practiceWakefield et al. JCEHP 2003

• CTC can be based on predefined set of suggested

changes and still have impact on reported behavior

change Domino FJ et al. Med Teach 2011

• What questions would you consider for a

commitment to change feedback questionnaire

45 – 60 days later?

Questions to consider using for

commitment to change

• Did you make any change or apply learning to your practice?

– Yes: considering, already doing this, doesn’t apply to me

• Describe 2 things your tried or did differently

• Did you tell colleagues about X?

• Has this course changed how you interact with the team?

• What was easiest to change? Why?

• What are the barriers to making changes you would like to

make?

• What could help you to make changes?

Safer Opioid Prescribing

Guided Self-Assessment

• Chart Review:

• Support guided self-assessment by comparing

to standard guidelines

• Adapted from UHN Chart Review Checklist

for Opioid Prescription

Safer Opioid Prescribing

Guided Self-Assessment

• Had already implemented before the

webinar/workshop

• Improved since webinar/workshop

• Improvement still needed- high priority

• Improvement still needed- low priority

• Not applicable

Safer Opioid Prescribing Guided Self-AssessmentImplemented

before the webinars/workshop

Improved Since

Webinars/Workshop

Still Needs Improve-mentHigh Priority

Still Needs Improve-ment Low Priority

1. Clear assessment of the pain condition:

2. Clear assessment of psychiatric history

3. Clear assessment of substance use disorder

4. Clear assessment of psychosocial history

5. Clear assessment of co-existing use of alcohol or illicit substances

6. Clear assessment of contraindications to opioid prescribing (including pregnancy)

Evaluating CE in the Context of Individual

Performance Improvement

• Participants audit set of charts using data collection

survey

• Intervention based on gaps identified through chart

review

• Participants asked to develop performance

improvement plan

• 3 months later participants asked to re-audit different

set of charts to see if changes implemented

Chart Audit

Question

Select Performance

Metrics

Access and Collect Data

Analyze Data

Obtain Feedback

Conclusion & Document

45

Clinical Audit Tool

• Royal College clinical audit tool and guideline

for physicians to conduct audit with feedback

to assess performance

• http://www.royalcollege.ca/rcsite/cpd/moc-

program/moc-support-tools-resources-e

Level 6: Patient Health Status

Observed

Patient health records

Administrative records

Self-reports

Physician questionnaire

Patient questionnaire

• Individual patient health status

Example of Performance Assessment and

Reported Patient Outcomes

• Safer Opioid Prescribing Webinar

• Brief Pain Inventory Patient Assessment Tool

• Application to Practice exercises

– Use of Practice Tools

– Report on Patient outcomes

Brief Pain Inventory

▪ Have patients complete a

Brief Pain Inventory.

• At initial work-up.

• At each subsequent

visit.

▪ Refer to Practice Tool

Brief Pain Inventory.

(sent by email)

Determine average Pain score/10

Sum score for 9a to 9i=Functional impairment score/90high score = low function

Application to Practice

• Have 5 chronic pain patients complete the Brief Pain

Inventory.

• For each of these patients, indicate:

– Whether this is an initial or follow-up assessment.

– The patient’s pain score out of 10.

– The patient’s functional impairment score out of 90.

– Your management decisions.

• A tracking sheet will be provided.

• Submit electronic report by DATE.

Application to Practice

Patient

Brief Pain Inventory Scores Patient Management Plans

Initial Examor

Follow-up Pain /10

Functional Impairment

/90

Non-pharmacological

treatments recommended

Medications prescribed (if any)

1o Initialo Follow-

up

o Physical Activity

o Self-management program

o Psychological therapy

o Physical therapy

Level 7: Community Health

• Ultimate goal of health care

Quantitative Sources

• Morbidity and mortality rates

– E.g. Incidence of secondary complications

• Adequate control of underlying disease

• Hospitalization and re-hospitalization rates

• Community public health data

Outcome Assessment in

Continuing Professional Development

• Outcome assessments at all levels can be

embedded throughout CPD Programs:

– Before- preparatory work

– During- active learning and assessed outcomes

– After- practice application exercises

Flipped Classroom

• Pre-classroom activities

– Support cognitive work (knowledge and

comprehension)

– Tailoring program to learner needs (survey)

• In-class activities

– Facilitate higher levels of learning (application and

analysis)

– Frees up time for interactive engagement and

introduction of innovative educational models

Flipped Classroom

• Time and resources to create pre-classroom

materials (e.g. videos)

• Educator readiness

– Training or faculty development

– Use of technology

• Learner readiness

– Active vs passive participant

– Perception of increase workload for learners

PRECEDE Model

• Predisposing, Reinforcing, Enabling Causes in

Educational Diagnosis and Evaluation

• Predisposing:

– Getting attention and identification of gaps before the

program

• Reinforcing

– Selecting methods to promote learning and retention

• Enabling

– Planning for application to practice

Multiple Interventions in

Safer Opioid Prescribing• For each webinar:

1. Pre-work – directed reading and guided practice audit based on Managing Chronic Pain Toolkit

2. During - Active learning to Share Practice Experience

3. Postwork - submit tracking sheets describing patient care- Brief Pain Inventory, Calculate MME/day, and Initiate Tapering

Multiple Interventions in

Safer Opioid Prescribing• For the workshop:

1. Pre-work - prepare a challenging case to share

with class

2. During - active learning include calculation of

MME/d and Switching, team consults and

role play

3. Post-work – Implement Structured Opioid

Therapy and (2) Survey of Improved

Implementation of Recommendations

An Active Approach to Learning

with Outcome Measures

15 distinct educational

interventions with outcome

measures over 4 months

Small Group Discussion

Using the PRECEDE model, how would you

embed higher level outcome measures before,

during and after your program?

• Predisposing

• Reinforcing

• Enabling

Take Home Messages

• Good outcomes assessment provides

opportunity for overall program improvement

• Measuring outcomes leads to enhancement in

learning, competence, performance and

potentially patient outcomes

• Use of PRECEDE model in curriculum

planning to link to outcomes

• Consider integrating one new form of

assessment in your next program

•

Selected References

• Lowe, M, Hebert,D and Rappolt, S. Occupational Therapy Now, ABCx of

CTCs: An introduction to Commitments to Change. 2009:11: 20

• Moore D et al. Achieving desired results and improved outcome.

Integrating planning and assessment through learning activities. JCEHP

2009:29:1-15.

• Lockyer, J. M. et al. (2001). Commitment to change statements: A way of

understanding how participants use information and skills taught in an

educational session. JCEHP 2001:21:82-89.

• Cervero RM and Gaines JK. Effectiveness of continuing medical

education: updated synthesis of systematic reviews. JCEHP 2015;

35(2):131–138.

• Cooper et al. Rating medical emergency teamwork performance:

development of the Team Emergency Assessment Measure (TEAM)

Resuscitation 2010;81:446-452.

•