Upload
changezkn
View
1.444
Download
1
Embed Size (px)
DESCRIPTION
Citation preview
Educational Outcomes Evaluation:
The Future for CME
James C. Leist, EdD
Staff Consultant
Alliance for Continuing Medical Education
Disclosure
Part time CME consultant
This format and some of the content were developed by Donald Moore, Vanderbilt University, Robert Addleton, Medical Association of Georgia and Derek Dietze, Improve CME
Plan for Session
Challenge-based Approach– Scenario– Independent Reflection– Small and Large Group Work– Mini Lecture– Further Reflection and Integration
Challenge Scenario
In her recent meeting with the Hospital CEO, Maureen O. Measure, BSN, MEd, the CME Director for Hopewell General Hospital, was asked about the value of CME. Maureen noted that they have done many programs for all physicians in the hospital system.
The CEO asked whether CME improved physician practice and patient care.
Maureen said that she didn’t have any data.
Challenge Scenario
The CEO asked if QI data were needed and agreed to appoint her to the QI Committee for the Hospital. She agreed anxiously.
In her third meeting with the QI Committee, Maureen listened to a report about the varied levels of physician prescriptions of ACE inhibitors for patients with CHF. How could the physician behavior be changed to assure that at least 80% of patients received the appropriate dosage of ACE I?
Challenge Scenario Questions
What can a CME provider contribute to this scenario?
If a CME intervention is provided, what outcomes should be evaluated and how?
ChallengeIndividual Reflection
For the next 5 minutes, reflect on the challenges listed below:– What are Outcomes?– How do you evaluate the outcomes of a
CME activity for this scenario?
ChallengeSmall Group
In your small groups in the next 10 minutes, share the results of your individual reflection on the definition of outcomes and how you would address the outcomes of a CME activity for this scenario.
Designate one member of the small group to share your results with the large group, time permitting.
Outcomes Evaluation Mini-Lecture
Learning Objectives for session– Define Educational Outcomes– Describe how CME providers in the
Hospital/Health Systems Venue can contribute to more sophisticated educational outcomes
– Apply measurement of educational outcomes to one CME activity when you return to your venue
Outcomes Defined
An outcome is defined as– the result, or effect, of an event– the consequences of an action
In CME, an educational outcome is the result, or effect, of CME activities
Evaluation is the process for measuring outcomes
Key to Outcomes Evaluation is to plan for the results that are desired – NEEDS ASSESSMENT DATA ARE CRITICAL
Yogi Berraism
“If you don’t know where you’re goin’, you ain’t gonna get there”
“Yogi Berra”
Process for Evaluating Outcomes
Conduct a needs assessment - practice gap Determine if education can address the gap Identify the results of the educational intervention Develop/Implement the intervention/s - target
audience and Faculty Measure the results/outcomes Share the results with appropriate stakeholders Repeat and Improve the process
Practice Gap - Individual and/or Organizational
Current Practice(Problem/Need) - Current level Desired Result - Practice/Performance level
desired Gap - The Difference between the desired result
and the current practice Will an educational intervention/interventions
address the identified gap.
A Continuum, or Levels, of Educational Outcomes/Results
Attendance Satisfaction Learning - KSA (Knows) Competence (Knows/Shows How) Performance/Behavior in Practice -
Individual and Organizational (Does) Patient Outcomes Community Health Status
Evaluation - A Continuum of Educational Outcomes
Current Practice - Generally– Attendance– Satisfaction– Learning (Knowledge, Skills and Attitudes)
Most Desirable, More Difficult and Required– Competence– Performance/Behavior in Practice - Individual and
Organizational– Patient Outcomes
Outcome Evaluation by Venues
Venues
Results
Office-based
Practice
Hospital/
Health System
Specialty
Societies
Med. Ed.
Comm. Co.
Medical School
Attendance/
SatisfactionX X X X X
Learning - KSA
X X X X X
Competence X X X X X
Practice X X X? X? X?
Patient Outcomes
X X X? X?
Community X X X X?
How do we Evaluate Outcomes?
Accurate data-based needs assessment of desired/current practice required to document need for change and provide baseline for measurement of improvement in practice.
Educational Formats Data Collection Strategies
– Indirect – Self-report by physicians– Direct – Actual performance data that describes
performance, or result of the performance
Educational Formats
Conferences– Advantages
– Disadvantages
Regularly Scheduled Series - RSS
– Advantages
– Disadvantages
Data Collection Strategies - Indirect
Outcome – Physicians will prescribe ACE inhibitors in the appropriate dosage for patients with CHF in every case when it is indicated
Measure – Prescriptions for ACE inhibitors
Methods – Survey; focus group; interview cardiologists
Data Collection Strategies - Direct
Outcome – Physicians will prescribe ACE inhibitors in the appropriate dosage for patients with CHF in every case when it is indicated
Measure – Prescriptions for ACE inhibitors
Methods – Chart audit; chart recall; prescription pads; observation; QI/PI data
Learning (Knows)
Methods for Measuring Knowledge, Skills and Attitudes– Post program evaluations
Listing of what you learned (most important thing that you learned)
– Pre- and Post - Tests– Others?
Competence (Knows/Shows How)Minimum evaluation standard
Methods for Measuring Competence– Post Test with Case Study/Scenario-Asks how– Commitment to Change with Explanation - What
do you plan to change and how?– Demonstration of how to apply a skill learned in
the CME activity– Action Plan for use of knowledge learned.– Others?
Performance/Behavior in Practice
Methods for Measuring Performance in Practice– Indirect (1-3 Months)
Follow up Survey - Self Report Observed practice Others?
– Direct (3-12 Months) Monitor Clinical Data - organizational Chart Audit - individual and organizational Others?
Patient Outcome
Methods for measuring patient outcomes– Monitor health status data for target audience– Monitor patient practices related to need addressed– Others?
Value of CME Formats(Conference/RSS) in Each Outcome Level
Learning
Competence
Performance
Patient Outcome
Challenge Scenario Questions
What can a CME provider contribute to this scenario?
If a CME intervention is provided, what outcomes should be evaluated and how?
How Do We Evaluate CME in Changing Physician Practice/Performance?
Conduct effective Needs Assessment to establish a baseline of performance, or health status, and identify the gap between what is and what should be. (EXAMPLE)
Numerous RCTs have shown that in patients with CHF, ACE Inhibitors
– Reduce mortality by 20-25%– Increase functional status and quality of life– Decrease hospitalization and death by 35-40%
ACE Inhibitors are cost effective Currently, ACE-I are prescribed for only 50% of pts.
How Do We Evaluate CME in Changing Physician Practice/Performance?
Re-measure the specific practice after the educational intervention to assure desired result. (EXAMPLE)
Measure performance after CME/activity/s– Method
Direct Measurement of performance Chart Audit
– Results: 75% of the prescriptions written by physicians for their CHF patients were ACE I, when indicated
Role for CME Provider - 1
Ask the right questions when planning CME activity. – Are there data to reflect a need and provide a
baseline for performance of individual/organization?– What result do you want from the CME intervention?
Work directly with your defined population, such as a physician practice, or your institution, to identify the gap in practice/health status.
Partner with units that have data that reflect need for change in practice/health status and that continually re-measure performance. (P&T Comm., QI/PI Comm.)
Role for CME Provider - 2
Learn about performance measurement and measure and share results
Learn to do chart audits, or teach physician office staff to conduct audits, or gather other data, for assessment of performance/health status as a standard procedure.
Role for CME Provider - 3
Make physicians Aware of gap, Agree on need to change, Adopt the new practice and Adhere to new practice.(Committee members or Influential physicians)
Use educational practices of – Predisposing – Create awareness of need to change– Enabling – Provide tools (education) to make change– Reinforcing – Follow up to support needed change
CME Profession Competencies
Adult/Organizational Learning Principles
Educational Interventions
Performance Measurement
Systems Thinking
Partnering
Leadership
Administration/ Management
Self Assessment and Lifelong Learning
CME Profession Competencies
Performance Measurement – Use appropriate data to assess educational and administrative performance of the CME Program
Partnering – Identify and collaborate with key partners and stakeholders in accomplishing their CME mission
Systems Thinking – Continually assess individual and organizational performance and make improvements through relevant learning experiences.
Lessons Learned
Accurate Needs Assessment Data a Necessary Foundation For Outcomes Evaluation
Re-measure Interventions Use 3 CME Profession
Competencies in Outcomes Evaluation
– Performance Measurement – Partnering– Systems Thinking
CME must provide outcomes evaluation to contribute to effective health care in the future
Role of CME Provider is expanding from a meeting planner to learning facilitator, or learning consultant
DATA, DATA, DATA
Other Lessonsor
QUESTIONS?
What should YOU do when you return to work in your setting/venue?
Identify one area where performance or health status is necessary for improvement
Ask the right questions in the planning process to assure need and desired results
Re-measure performance or health status after educational intervention/s to document results
Develop at least one new competency Promote the effectiveness and value of CME
References/Resources
ACME Evaluating Educational Outcomes-An Electronic Workbook for CME Providers www.acme-assn.org/workbook.shtml (Available Spring 2009)
CMExchange AMA Physician Consortium on Performance
Development, www.ama-assn/ama/pub/category/2946.html