Upload
raphael-ford
View
19
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Standardizing Learner Surveys Across the Enterprise. Francis Kwakwa, MA, Radiological Society of North America Valerie Smothers, MA, MedBiquitous. Disclosure. We have no financial relationships to disclose. Objectives. At the completion of this session, you will be able to: - PowerPoint PPT Presentation
Citation preview
Standardizing Learner Standardizing Learner Surveys Across the Surveys Across the
EnterpriseEnterpriseFrancis Kwakwa, MA, Francis Kwakwa, MA,
Radiological Society of North Radiological Society of North AmericaAmerica
Valerie Smothers, MA, Valerie Smothers, MA, MedBiquitousMedBiquitous
DisclosureDisclosure
We have no financial We have no financial relationships to disclose.relationships to disclose.
ObjectivesObjectives
At the completion of this session, you At the completion of this session, you will be able to:will be able to:
Adopt strategies to improve the Adopt strategies to improve the collection of consistent evaluation collection of consistent evaluation data from learners data from learners
Adopt strategies to improve the Adopt strategies to improve the analysis of evaluation data across analysis of evaluation data across the CME enterprise the CME enterprise
OverviewOverview
1.1. Challenges in analyzing learner Challenges in analyzing learner surveyssurveys
2.2. MedBiquitous and MEMSMedBiquitous and MEMS
3.3. RSNA’s Implementation of a RSNA’s Implementation of a Standardized SurveyStandardized Survey
4.4. Results of RSNA course evaluationResults of RSNA course evaluation
5.5. Challenges faced by RSNAChallenges faced by RSNA
6.6. Key strategies for improving data Key strategies for improving data collection and analysiscollection and analysis
Challenges in Analyzing Challenges in Analyzing Learner SurveysLearner Surveys
Most of us use Most of us use surveyssurveys
Surveys often differ Surveys often differ based on activitybased on activity
Survey data may be Survey data may be in different systems in different systems or formatsor formats
The result: it’s hard The result: it’s hard to analyze results to analyze results across activitiesacross activities
RSNARSNA
Radiological Society of North AmericaRadiological Society of North America ““to promote and develop the highest to promote and develop the highest
standards of radiology and related standards of radiology and related sciences through education and sciences through education and research”research”
Over 40,000 membersOver 40,000 members Online and in-person CME activitiesOnline and in-person CME activities Member of MedBiquitousMember of MedBiquitous Francis Kwakwa, Chair of the Francis Kwakwa, Chair of the
MedBiquitous Metrics Working Group MedBiquitous Metrics Working Group
MedBiquitousMedBiquitous
Technology standards developer for Technology standards developer for healthcare educationhealthcare education ANSI AccreditedANSI Accredited Develops open XML standardsDevelops open XML standards 60 members (societies, universities, 60 members (societies, universities,
government, industry)government, industry) 7 working groups7 working groups
The Focus on MetricsThe Focus on Metrics ““Without the creation Without the creation
of a standard data of a standard data set for reporting set for reporting CME program CME program outcomes … it is outcomes … it is difficult to obtain difficult to obtain consistent metrics of consistent metrics of those outcomes. And those outcomes. And if you can’t measure if you can’t measure it, you can’t improve it, you can’t improve itit.” .”
Medical Education Medical Education Metrics – MEMSMetrics – MEMS
Ross Martin, MD, Director Healthcare Informatics Group, Pfizer
Another PerspectiveAnother Perspective
““I need this to I need this to better better understand how understand how my program as a my program as a whole is doing.”whole is doing.”
Nancy Davis, American Academy of Family Physicians
The MedBiquitous Metrics The MedBiquitous Metrics Working GroupWorking Group
Mission:Mission: to develop XML standards … for the exchange of aggregate evaluation data and other key metrics for health professions education.
Originally a subcommittee of the Originally a subcommittee of the Education Working GroupEducation Working Group
Became a working group in April Became a working group in April 20052005
We’re all using the same measuring stick…
--Francis
Who is Involved?Who is Involved? Francis Kwakwa, Francis Kwakwa,
RSNA, ChairRSNA, Chair Linda Casebeer, Linda Casebeer,
Outcomes Inc. Outcomes Inc. Nancy Davis, AAFPNancy Davis, AAFP Michael Fordis, Baylor Michael Fordis, Baylor
College of MedicineCollege of Medicine Stuart Gilman, Stuart Gilman,
Department of Department of Veterans AffairsVeterans Affairs
Edward Kennedy, Edward Kennedy, ACCME *ACCME *
Jack Kues, University Jack Kues, University of Cincinnatiof Cincinnati
Tao Le, Johns Hopkins Tao Le, Johns Hopkins UniversityUniversity
Ross Martin, PfizerRoss Martin, Pfizer Jackie Mayhew, PfizerJackie Mayhew, Pfizer Mellie Pouwels, RSNAMellie Pouwels, RSNA Andy Rabin, CE CityAndy Rabin, CE City Donna Schoonover, Donna Schoonover,
Department of Department of Veterans AffairsVeterans Affairs
* Invited experts
What’s in MEMSWhat’s in MEMS Participation Metrics Participation Metrics
how many participantshow many participants Learner Demographics Learner Demographics
profession, specialtyprofession, specialty Activity Description Activity Description
name, typename, type Participant Activity Evaluation Participant Activity Evaluation
survey resultssurvey results Other types of evaluation metrics planned Other types of evaluation metrics planned
for future versionsfor future versions
For more information:For more information:
Metrics Working Group PageMetrics Working Group Pagehttp://www.medbiq.org/working_grohttp://www.medbiq.org/working_groups/metrics/index.htmlups/metrics/index.html
MedBiquitous WebsiteMedBiquitous Websitehttp://www.medbiq.orghttp://www.medbiq.org
DiscussionDiscussion
Describe the learner surveys that Describe the learner surveys that you are using and how they differ you are using and how they differ from or are similar to the survey from or are similar to the survey described. What are the benefits or described. What are the benefits or drawbacks of using a standardized drawbacks of using a standardized survey? survey?
RSNA’s Project…RSNA’s Project…
Adoption of MEMS survey instrument Adoption of MEMS survey instrument coincided with implementation of a coincided with implementation of a new Learning Management Systemnew Learning Management System
Currently MEMS is used to evaluate Currently MEMS is used to evaluate over 300 online coursesover 300 online courses
RSNA’s Project…RSNA’s Project…
Types of online courses using MEMSTypes of online courses using MEMS
Cases of the Day (COD)Cases of the Day (COD)
RadioGraphics CME Tests/Education RadioGraphics CME Tests/Education Exhibits (EE)Exhibits (EE)
Refresher Courses (RSP)Refresher Courses (RSP)
COD-45 (N = 24)COD-45 (N = 24)The course achieved its learning objectivesThe course achieved its learning objectives
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
EE-355 (N = 32)EE-355 (N = 32)The course achieved its learning objectivesThe course achieved its learning objectives
0
10
20
30
40
50
60
70
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
RSP-2904 (N = 43)RSP-2904 (N = 43)The course achieved its learning objectivesThe course achieved its learning objectives
0
10
20
30
40
50
60
70
80
90
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
COD-45 (N = 24) COD-45 (N = 24) The course was relevant to my clinical The course was relevant to my clinical
learning needslearning needs
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
EE-355 (N = 32)EE-355 (N = 32)The course was relevant to my clinical The course was relevant to my clinical
learning needslearning needs
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
RSP-2904 (N = 43)RSP-2904 (N = 43)The course was relevant to my clinical The course was relevant to my clinical
learning needslearning needs
0
10
20
30
40
50
60
70
80
90
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
COD-45 (N = 24) COD-45 (N = 24) The course was relevant to my personal The course was relevant to my personal
learning needslearning needs
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
EE-355 (N = 32)EE-355 (N = 32)The course was relevant to my personal The course was relevant to my personal
learning needslearning needs
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
RSP-2904 (N = 43)RSP-2904 (N = 43)The course was relevant to my personal The course was relevant to my personal
learning needslearning needs
0
10
20
30
40
50
60
70
80
90
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
COD-45 (N = 24) COD-45 (N = 24) The online method of instruction was The online method of instruction was
conducive to learningconducive to learning
0
10
20
30
40
50
60
70
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
EE-355 (N = 32)EE-355 (N = 32)The online method of instruction was The online method of instruction was
conducive to learningconducive to learning
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
RSP-2904 (N = 43)RSP-2904 (N = 43)The online method of instruction was The online method of instruction was
conducive to learningconducive to learning
0
10
20
30
40
50
60
70
80
90
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
COD-45 (N = 24) COD-45 (N = 24) The course validated my current practiceThe course validated my current practice
0
10
20
30
40
50
60
70
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
EE-355 (N = 32)EE-355 (N = 32)The course validated my current practiceThe course validated my current practice
0
10
20
30
40
50
60
70
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
RSP-2904 (N = 43)RSP-2904 (N = 43)The course validated my current practiceThe course validated my current practice
0
10
20
30
40
50
60
70
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
COD-45 (N = 24) COD-45 (N = 24) I plan to change my practice based on what I I plan to change my practice based on what I
learned in the courselearned in the course
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
EE-355 (N = 32)EE-355 (N = 32)I plan to change my practice based on what I I plan to change my practice based on what I
learned in the courselearned in the course
0
5
10
15
20
25
30
35
40
45
50
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
RSP-2904 (N = 43)RSP-2904 (N = 43)I plan to change my practice based on what I I plan to change my practice based on what I
learned in the courselearned in the course
0
5
10
15
20
25
30
35
40
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
COD-45 (N = 24) COD-45 (N = 24) The faculty provided sufficient evidence to The faculty provided sufficient evidence to
support the content presentedsupport the content presented
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
EE-355 (N = 32)EE-355 (N = 32)The faculty provided sufficient evidence to The faculty provided sufficient evidence to
support the content presentedsupport the content presented
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
RSP-2904 (N = 43)RSP-2904 (N = 43)The faculty provided sufficient evidence to The faculty provided sufficient evidence to
support the content presentedsupport the content presented
0
10
20
30
40
50
60
Strongly Agree Agree Neutral Disagree StronglyDisagree
%
COD-45 (N = 24) COD-45 (N = 24) Was the course free of commercial bias Was the course free of commercial bias
towards a particular product or company?towards a particular product or company?
Yes100%
EE-355 (N = 32)EE-355 (N = 32) Was the course free of commercial bias Was the course free of commercial bias
towards a particular product or towards a particular product or company?company?
Yes100%
RSP-2904 (N = 43)RSP-2904 (N = 43) Was the course free of commercial bias Was the course free of commercial bias
towards a particular product or towards a particular product or company?company?
Yes98%
No2%
COD-45 (N = 24) COD-45 (N = 24) Did the course present a balanced view of Did the course present a balanced view of
clinical options?clinical options?
Yes96%
No4%
EE-355 (N = 32)EE-355 (N = 32) Did the course present a balanced view Did the course present a balanced view
of clinical options?of clinical options?
Yes100%
RSP-2904 (N = 43)RSP-2904 (N = 43) Did the course present a balanced view Did the course present a balanced view
of clinical options?of clinical options?
Yes98%
No2%
Group DiscussionGroup Discussion
What challenges to survey data What challenges to survey data collection and analysis have you collection and analysis have you faced?faced?
Challenges Faced by Challenges Faced by RSNARSNA
Survey is optional; little data Survey is optional; little data available for some coursesavailable for some courses
Little variation in the dataLittle variation in the data Some disconnect with educators on Some disconnect with educators on
how the data is usedhow the data is used Difficult to get data out of the LMSDifficult to get data out of the LMS Surveys for live events are not Surveys for live events are not
includedincluded
Key StrategiesKey Strategies
Data CollectionData Collection Common core set of survey questionsCommon core set of survey questions Common format for evaluation dataCommon format for evaluation data
Data AnalysisData Analysis Compare within type and modalityCompare within type and modality Compare across type and modalityCompare across type and modality Look for trends and variationLook for trends and variation Look for red flagsLook for red flags
An Added BenefitAn Added Benefit
Assists with program analysis and improvement required by the ACCME “The provider gathers data or information
and conducts a program-based analysis on the degree to which the CME mission of the provider has been met through the conduct of CME activities/educational interventions.”
--ACCME Updated Accreditation Criteria,
September 2006