Educational Outcomes Report · website’s program registration pages, CME test/evaluation pages, or other areas interferes with your ability to access the information, please contact

EducationalOutcomes Report

November 17, 2011

Accredited provider of medical and professional education

8201 W McNab Road, Tamarac, FL 33321 Phone: (954) 718-6055 Fax: (954) 718-6013 www.primeinc.org

Copyright © 2011 PRIME Education, Inc. All Rights Reserved.

This activity was provided by PRIME® through an independent educational grant from Teva Neuroscience.

Educational Outcomes Report: Essential Elements in the Nursing Process to Improve Multiple Sclerosis Patient Outcomes 2Copyright © 2011 PRIME Education, Inc. All Rights Reserved.

Grant # 7994

Table of ContentsExecutive Summary and Program Description ....................................................................................................... 3

Expert Faculty ...................................................................................................................................................................... 4

Level 1: Participant Demographics ............................................................................................................................. 5

Level 2: Participants’ Assessments of the Educational Activity ......................................................................... 6

Level 3: Knowledge Outcomes ...................................................................................................................................... 9

Continuous Needs Assessment .................................................................................................................................... 11

Summary, Conclusions, and Future Directions ....................................................................................................... 13

Appendix:

Educational Needs Assessment, Activity Development, and Outcomes Assessment at PRIME®:

Applications of Established Conceptual Frameworks and Principles of Adult Learning ......................... 14


Grant # 7994

Executive Summary and Program DescriptionSpecialty nurses who manage patients with multiple sclerosis (MS) have knowledge needs pertaining to emerging oral therapies. Specific foci include teaching patients about these therapies and addressing individualized therapy needs among the MS patient population, including efficacy, safety, and adherence. Nurses also have needs in assimilating the immunopathophysiology of MS, as well as applying this complex information to the daily routines of patient management and therapeutic management. Reinforced learning about the symptoms of MS, side effects of treatment, and special health care needs is also important for nurses. To address these educational needs, PRIME® sponsored a series of 3 educational meetings focused on key topics in the field of MS nursing.

This report summarizes the educational outcomes of these meetings. The body of the report is organized by distinct levels of educational outcomes, with focused assessments of:

• Participant demographics

• The teaching effectiveness of the faculty

• The quality of the learning experience

• Gains in declarative and procedural knowledge

• Activity-influenced gains in understanding and confidence

• Activity-influenced expectations of clinical performance improvement

Explanations of these outcome domains, along with their evidence-based rationale, are presented in the appendix document titled Educational Needs Assessment, Activity Development, and Outcomes Assessment at PRIME®: Applications of Established Conceptual Frameworks and Principles of Adult Learning.


Grant # 7994

Expert Faculty

Stephanie Agrella, MSN, ANP-BC, MSCNDirector of Clinical ServicesMultiple Sclerosis Clinic of Central TexasRound Rock, TX

Brian Stellinga, BSRN, MSCNMS Nurse CoordinatorGreat Plains MS Center at Neurology AssociatesSioux Falls, SD

Gina Remington, RN, BSN, MSCNClinical Research ManagerClinical Center for Multiple SclerosisUniversity of Texas Southwestern Medical CenterDallas, Texas

Registration web page

Print invitation (cover)

Print invitation (inside)

Program Title:

Essential Elements in the Nursing Process to Improve Multiple Sclerosis Patient Outcomes

Accreditation:

1.5 ANCC

Program Dates and Locations:

September 12, 2011; Peoria, IL

September 22, 2011; Sioux Falls, SD

October 13, 2011; Fort Worth, TX

AccreditationNurse Accreditation Statement

PRIME Education, Inc. (PRIME®) is accredited as a provider of continuing nursing education by the American Nurses Credentialing Center’s Commission on Accreditation.

PRIME® designates this activity for 1.5 contact hours.

CE awards require registration for the program, completion of the presurvey, attendance during the entire program, and completion of the program evaluation at PRIME®’s Credit Center: primeinc.org/credit, after which you will be able to print your CE certificate. Conflict of interest and disclosure information can be found on the program registration page at www.primeinc.org.

PRIME® endorses the standards of the ACCME and ANCC that require everyone in a position to control the content of a CME/CE activity to disclose all financial relationships with commercial interests that are related to the activity content. CME/CE activities must be balanced, be independent of commercial bias, and promote improvements or quality in healthcare. All recommendations involving clinical medicine must be based on evidence accepted within the medical profession. Conflict of interest and disclosure information is available on the program registration page at www.primeinc.org and will be available in the course syllabus.

Intended AudienceNurse practitioners and nurses who care for patients with multiple sclerosis (MS)

Learning Objectives• Assess strategies to improve patient outcomes

in multiple sclerosis, with a focus on symptom management, medication management, psychosocial support, and quality of life

• Examine contemporary approaches in immunopathophysiology of multiple sclerosis and how to apply the evidence in practice

• Discuss the role and utility of novel therapeutic targets in ongoing patient treatment and patient education

• Apply skills for effective communication in ongoing multiple sclerosis nursing management

Program AgendaI Current therapeutic management of patients with

MS — where we are todayII Immunopathophysiology of MS and how it relates

to emerging therapeutic optionsIII Nursing implications with novel therapy — how

we can assist the patientIV Addressing ongoing special health care needs of

patients with MS

Program OverviewSpecialty nurses managing patients with multiple sclerosis (MS) have knowledge needs pertaining to emerging oral therapy, with specific foci on how to teach patients about these therapies and how to address individualized therapy needs among the MS patient population, including efficacy, safety and adherence. Nurses also have needs in assimilating the immunopathophysiology of MS, as well as applying this complex information to the daily routines of patient management and therapeutic management. Reinforced learning in symptoms of MS, side effects of treatment, and special health care needs is also important for nurses. This program is specifically tailored to address these topics for nurses who manage patients with MS.

Locations and Expert Presenters

Monday, September 12, 2011309

5832 N Knoxville AvenuePeoria, IL

Stephanie Agrella, MSN, ANP-BC, MSCN Director of Clinical Services

Multiple Sclerosis Clinic of Central TexasRound Rock, TX

Thursday, September 22, 2011Foley’s

2507 South Shirley AvenueSioux Falls, SD

Brian Stellinga, BSRN, MSCNMS Nurse Coordinator

Great Plains MS Center at Neurology AssociatesSioux Falls, SD

Thursday, October 13, 2011The Worthington Renaissance

Fort Worth Hotel200 Main StreetFort Worth, TX

Gina M Remington, RN, BSN, MSCN Clinical Research Manager

Clinical Center for Multiple SclerosisUniversity of Texas Southwestern Medical Center

Dallas, TX

Registration & dinner will commence at 6:30 PMEducational program: 6:45 PM – 8:15 PM

To register go to: www.primeinc.org/RNMS

PRIME® is committed to providing access to our live CME programs for individuals with disabilities as identified in Title III of the Americans with Disabilities Act (ADA). If you use assistive technology (ie, Braille reader, screen reader, TTY) and the format of any material on the PRIME® website’s program registration pages, CME test/evaluation pages, or other areas interferes with your ability to access the information, please contact [email protected] or 954-718-6055 for assistance.

Parking fees are the responsibility of the attendee.

35LV1100RMS Invitation.indd 2 7/12/2011 1:36:45 PM

First Name: _______________________ Last Name: _____________________________

E-mail: ______________________________________________________________

Discipline: ____________________________________________________________

Organization: __________________________________________________________

Mailing Address: ________________________________________________________

City: ______________________________ State: _____________________________

Zip: _______________________ Phone: ( ) _____________________________

Practice Setting: ______________ Location attending: _____________________________

Call Us

2

954-718-6055

Online

1

www.primeinc.org/RNMS

3

Fax this Form

954-718-6013

4

Mail this Form

8201 W McNab Rd.Tamarac, FL 33321

4 easy ways to register:

There is no fee for this activity as it is sponsored by PRIME® through an independent educational grant from Teva Neuroscience.

1.5 hour LIVE CE program specifically tailored for nurses who manage patients with MS

35LV1100RMS Invitation.indd 1 7/12/2011 1:36:43 PM


Grant # 7994

Level 1: Participant Demographics 53 Registered Attendees

34 Program Participants

29 Earning CME/CE Credit

By discipline, the distribution was as follows:• 1 Neurologist• 7 Nurse Practitioners• 26 Specialty Nurses

*PRIME®’s Planning Committee convened to discuss the drop off in attendance. Comments are noted below.

1. PRIME® clearly missed the mark on educational gaps in 1 of the 3 cities. Given the large success of the first round of cities, this low turnout tells us this topic is no longer valid for neurology nurses; therefore we will not submit future grants on this topic area for neurology nurses.

2. The need to have the programs in hotels due to the very low budget is a known deterrent. Restaurants have proven to get better turnouts, but they cost more than this budget allowed.

3. In primary recruitment outreach, some nurses stated they worked in Dallas, and it took too long to get to Ft. Worth in rush hour traffic. They would never be able to arrive at the hotel in time for the program. It should be noted that there were 16 pre-registrations for this program, but only 3 program participants.

4. All nurses who received personal reminder phone calls the day before the program were excited about attending and confirmed, but for some reason did not show. Messages were left for the remaining registrations.

Result: This is the constellation of factors identified by the Planning Committee in their analysis of the low

turnout. Registrations and intended audience were adequate for all 3 program locations, and typical followup processes were engaged. An email was sent to the no-shows to see if they would like to receive a pdf of the slide content, and to ask them what future topics would meet their educational gaps. Results are provided in the Continuous Needs Assessment section of this report.

Before the educational activity, to assess baseline clinical practices regarding

communication with patients, we asked participants how often they assess patients’ health literacy and learning challenges. As illustrated in Figure 1, 34% and 32% of participants indicated “occasionally” and “regularly” carrying out these assessments, respectively.

In our assessment of demographics, we asked participants, “In a typical month, how many patients with MS do you currently provide care for?” As shown in Figure 2, 78% of participants indicated

Figure 1. Participants’ self-reported frequency of assessing patients’ health literacy/learning challenges to improve their communication skills.

Figure 2. Self-reported numbers of MS patients for whom participants provide care in a typical month.

14%

34%32%

20%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Never Occasionally Regularly AlwaysPa

rtici

pant

s Cho

osin

g Re

spon

se (%

)

Reported Prior to Activity

78%

4% 4%

14%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

1 to 10 11 to 20 21 to 30 Over 30

Part

icipa

nts C

hoos

ing

Resp

onse

(%)

Reported Immediately After Activity


Grant # 7994

providing services for 1-10 patients over this time period.

Level 2: Participants’ Assessments of the Educational ActivityFollowing the educational activity, participants evaluated various aspects of its effectiveness and accessibility. PRIME® uses these outcomes to determine the extent to which its activities fill identified educational gaps and address critical learning objectives. In addition, participants’ assessments of educational activities inform our future needs assessments and curriculum development. Evaluations were completed by 34 participants.

Effectiveness in Meeting Learning Objectives

This educational activity was designed to support participants in achieving the following learning objectives.

1. Assess strategies to improve patient outcomes in multiple sclerosis, with a focus on symptom management, medication management, psychosocial support, and quality of life

2. Examine contemporary approaches in immunopathophysiology of multiple sclerosis and how to apply the evidence in practice

3. Discuss the role and utility of novel therapeutic targets in ongoing patient treatment and patient education

4. Apply skills for effective communication in ongoing multiple sclerosis nursing management

Participants assessed the educational activity’s effectiveness in meeting the learning objectives. On a 5-point scale (with 5 indicating the highest ranking), the combined mean average was 4.5 (Figure 3).

Figure 3. Participants’ assessments of how effectively the educational activity met its learning objectives. Rating scale: 1 = Most strongly disagree that learning objective was met 5 = Most strongly agree that learning objective was met

4.5

4.5

4.5

4.4

1 2 3 4 5

Learning objective #4




Average Ratings


Grant # 7994

5

5

5

5

1 2 3 4 5

Overall evaluation of total program

PRIME's assistance in preperation to presentprogram

Travel accommodations provided by PRIME

Initial contact from PRIME adequate forpreparation

Average Ratings

As illustrated in Figure 4, participants gave high ratings of the educational activity’s accessibility, time allotment, rigor, and adherence to adult learning principles.

Participant Ratings of Faculty & Program

Figure 5 presents participants’ ratings of the knowledge, expertise, and presentation skills of the faculty. On a 5-point scale (with 5 indicating the highest rating), the average rating for these measures of faculty teaching effectiveness was 4.5.

Faculty Rating of Program Experience

The 3 faculty completed surveys pertaining to their experience in working with PRIME®. Results are provided in Figure 6. Open-ended responses to questions about their experience are presented as follows.

1. How would you rate the overall audience response to the program?

• Averypositiveaudience

• Forthemostpart,aprettygoodaudience

• Goodregistrations,butfewshowedupforprogram.Notsurewhy.

2. What were some of the more interesting questions/issues asked by the audience?

• QuestionsregardingvitaminD

• QuestionsaboutthepossiblecausesofMS

• QuestionsonPMLtreatment

• Patientassistancequestions

• Questionsregardinghoworaltherapieseffectadherencetomedication

• TheimpactofheatwithMS

3. Please provide any comments regarding your involvement with PRIME® which will help us improve the overall program process in the future.

• PRIMEstaffwasveryhelpfulandfriendly

Figure 4. Participants’ assessments of the accessibility, effectiveness, and rigor of the educational activity. Rating scale: 1 = Lowest rating; 5 =Highest rating

Figure 5. Participants’ assessments of faculty teaching effectiveness. Rating scale: 1 = Lowest rating; 5 =Highest rating

Figure 6. Faculty evaluation of program experience. Rating scale: 1 = Lowest rating; 5 =Highest rating

4.6

4.5

4.5

4.5

4.6

1 2 3 4 5

Time allotted forprogram

Principles of adultlearning

Registration process

Scientific rigor

User friendliness

Average Ratings

4.7

4.3

4.4

1 2 3 4 5

Knowledge andexpertise

Ability to conveysubject matter

effectively

Ability to deliverobjective and balanced

program

Average Ratings


Grant # 7994

Fair Balance

In a post-activity questionnaire, 100% of participants indicated that the educational content was objective and unbiased.

Participants’ Open-Ended Comments

In an online survey delivered following the educational activity, participants were given the opportunity to comment freely on its strengths and weaknesses. Representative comments are reproduced as follows.

Excellent program management and speaker. I was brought up to date with new treatments. — Specialty Nurse

The speaker was excellent and I enjoyed listening to her. — Specialty Nurse

I really enjoyed this program. I felt the speaker really tried and succeeded in speaking on a level that was understandable. She gave us a great update and answered all my questions adequately. — Specialty Nurse

I had hoped for a more in-depth lecture in new therapies, but it was a good general over view of MS. — Nurse Practitioner

The program was very well done. It would be nice if we could have a copy of the power points that were presented. — Specialty Nurse

The speaker was excellent and very knowledgeable on the subject discussed. Would have been helpful if we could have had a printout of the slides. It is very difficult to remember the various drugs and their mechanism of action. — Specialty Nurse

Thanks for the new updates on MS treatments. The speaker was outstanding. — Nurse Practitioner

Very technical and presentation focused on medications - somewhat hard to retain information. — Specialty Nurse

Given the time frame, I found myself scrambling to take notes, particularly related to the medications. A handout with meds listed would have made that process easier. — Specialty Nurse

It was an interesting topic and the facility was nice. I would attend again. Thanks. — Nurse Practitioner

The presenter was new but was diligent in trying to make her presentation meaningful. Less reading of the slides would strengthen her presentation. The slides were visually attractive and helpful. — Nurse Practitioner

I felt the speaker was very knowledgeable. She was easy to listen to and understand. I have attended many MS meetings and I felt she did an excellent job. — Specialty Nurse


Grant # 7994

Level 3: Knowledge OutcomesBefore and after the educational activity, participants answered a series of multiple-choice questions designed to evaluate gap-targeted knowledge outcomes. These assessments focused on activity-influenced changes in declarative knowledge and learning insights, values, and behaviors.

Declarative Knowledge

Based on the program’s learning objectives, the following questions were asked to assess participant’s gains in declarative and procedural knowledge (correct answers are indicated by asterisks).

1. Which MS symptom is primarily due to lesions of the brainstem?

a. Cognitive decline

b. Ataxia

*c. Diplopia

2. What is the most common reason that patients with MS stop taking disease-modifying therapies?

*a. Perceived lack of efficacy

b. Fear of needles

c. Difficulty with self-injection

d. Side effects

3. Knowing that patient education is positively related to medication adherence, you are delighted to hear that one of your patients wishes to learn more about how MS therapies work. Which of the following does NOT accurately match an emerging oral MS therapy to its primary proposed mechanism of action?

a. Teriflunomide: Inhibits DNA synthesis in autoreactive T cells

b. Cladribine: Suppresses T cell proliferation by inhibiting DNA synthesis

*c. Fingolimod: Selectively depletes T cells

d. Laquinimod: shifts balance of lymphocyte and cytokine profiles to anti-inflammatory pathways

Figure 7 presents the percentages of participants who answered the declarative knowledge questions correctly before and after the educational activity. Across the 3 questions, the average number of correct answers increased by 8.7%.

48%

45%

28%

36%

39%

20%

0% 10% 20% 30% 40% 50% 60%

Question 3

Question 2

Question 1

Participants Answering Correctly (%)

Pre-activity

Post-activity

48%

45%

28%

36%

39%

20%

0% 10% 20% 30% 40% 50% 60%

Question 3

Question 2

Question 1

Participants Answering Correctly (%)

Pre-activity

Post-activity

Figure 7. Pre-activity and post-activity assessments of declarative knowledge.


Grant # 7994

Figure 8. Participants’ self-reported understanding of the roles and utility of emerging oral agents and monoclonal antibodies for treating patients with MS.

Figure 9. Participants’ self-reported expected improvement in communication skills with patients as a result of knowledge gained from the educational activity.

Learning Insights, Values, and Intentions to Change Practices

As reflected by the following pre-activity/post-activity questions, this sub-level of knowledge outcomes addresses the influences of educational interventions on participants’ self-reported understanding of essential topics. In addition, at this sub-level of knowledge outcomes, we assess learners’ activity-influenced intentions to change clinical and/or management practices.

1. How would you describe your current level of understanding of the roles and utility of emerging oral agents and monoclonal antibodies for treating patients with MS?

a. Limited

b. Adequate

c. Good

d. Excellent

Figure 8 presents the percentages of participants reporting different levels of understanding regarding the roles and utility of emerging oral agents and monoclonal antibodies for treating patients with MS. The greatest magnitudes of pre-activity to post-activity change were reported for “limited” understanding, which decreased from 86% to 77% of participants and “adequate” understanding, which increased from 5% to 11% of participants.

To assess expectations for performance improvement, we asked participants how much they anticipated that, over the next 6 months, the knowledge gained from the educational activity would improve their skills in communicating with their patients. As illustrated in Figure 9, 52% of participants indicated that they expected “some” improvement and 21% expected “major” improvement.

86%

5% 7%2%

77%

11% 10%2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Limited Adequate Good ExcellentPa

rtici

pant

s Cho

osin

g Re

spon

se (%

)

Reported Prior to Activity Reported Immediately After Activity

0%

27%

52%

21%

0%

10%

20%

30%

40%

50%

60%

No improvementsexpected

Minimalimprovements

expected

Some improvementsexpected

Major improvementsexpected

Part

icipa

nts C

hoos

ing

Resp

onse

(%)

Reported Immediately After Activity


Grant # 7994

Continuous Needs Assessment Our continuous needs assessment is based partly on questions that participants asked the expert faculty presenters during and after their presentations. Representative questions are listed as follows:

What is the relationship between internuclear ophthalmoplegia (INO) and MS?

How do you do your differential diagnosis?

What does benign MS really mean?

What are the key pharmacoeconomic issues with MS drugs?

How would you characterize a severe relapse?

For patients with cognitive function, are there management options?

Can you comment on women with MS who want to have children - how do you advise them about breast feeding, postpartum follow-up, etc?

Is there any ethnic predisposition to getting MS?

Have you personally seen any cases of PML? What did you do?

Do you screen for JCV prior to initiating natalizumab?

What kind of lab work do you order for a patient with suspected CIS?

Have you ever seen any patients with MS who experience excessive sweating as a symptom?

How will new oral MS therapies impact adherence?

Does dalfampridine have any affect on body temperature?

Post-Impact Analysis

To assess further performance improvement, participants were asked a series of questions 60 days following the educational activity. A total of 11 program participants responded to the following 60-day post-impact questions.

How would you describe your current level of understanding of the roles and utility of emerging oral agents and monoclonal antibodies for treating patients with MS?

a. Limited

b. Adequate

c. Good

d. Excellent

Figure 10 presents the percentages of participants reporting different levels of current understanding regarding the roles and utility of emerging oral agents and monoclonal antibodies for treating patients with MS. At 60 days after the program, 50% of respondents indicated a “limited” level of understanding, while there was a 15% increase in the number of participants reporting “good” understanding.

77%

11% 10%

2%

50%

25% 25%

0%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Limited Adequate Good Excellent

Part

icipa

nts C

hoos

ing

Resp

onse

(%)

Reported Immediately After Activity Reported 60 Days After Activity

Figure 10. Participants’ self-reported understanding of the roles and utility of emerging oral agents and monoclonal antibodies for treating patients with MS.


Grant # 7994

In your current practice, how often do you assess patients’ health literacy and learning challenges in order to improve your communication skills?

a. Never

b. Occasionally

c. Regularly

d. Always

Figure 11 illustrates participants’ self-reported frequency of assessing patients’ health literacy and learning challenges. There was a 13% increase in the number of participants reporting “regularly” assessing patients’ health literacy and learning challenges at 60 days post program.

In caring for patients with MS, which primary action are you performing now that you were not performing 60 days ago?

a. Educating patients about the efficacy and safety profiles of current and emerging therapies

b. Applying evidence-based strategies for promoting medication adherence and addressing other individualized health care needs

Figure 12 presents the percentages of participants reporting the primary action they are performing now, which they were not performing before the educational program. Results show that 75% of participants are applying evidence-based strategies for promoting medication adherence and addressing other individualized health care needs at 60 days post program.

14%

34%32%

20%

0%

35%

45%

20%

0%5%

10%15%20%25%30%35%40%45%50%

Never Occasionally Regularly AlwaysPa

rtici

pant

s Cho

osin

g Re

spon

se (%

)

Reported Prior to Activity Reported 60 Days After Activity

Figure 11. Participants’ self-reported frequency of assessing patients’ health literacy/learning challenges to improve their communication skills.

Figure 12. Participant’s responses to the question, “In caring for patients with MS, which primary action are you performing now that you were not performing 60 days ago?”

25%

75%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Educating patients about efficacy/safetyprofiles of current/emerging therapies

Applying evidence base strategies forpromoting medication

adherence/addressing individualizedhealth care needs

Part

icipa

nts C

hoos

ing

Resp

onse

(%)

Reported 60 Days After Activity


Grant # 7994

Summary, Conclusions, and Future DirectionsThis CME/CE activity was successfully provided to health care professionals involved in the nursing management of patients with MS. The main outcomes of the educational activity are summarized as follows.

• The participants gave high ratings for the educational activity’s accessibility, time allotment, rigor, and adherence to adult learning principles, and also indicated the material was objective and unbiased.

• Participants commented that the speakers were excellent and knowledgeable about the topic.

• As indicated in the pre-activity/post-activity multiple-choice test, participants reported increased levels of understanding regarding the roles and utility of emerging oral agents and monoclonal antibodies for treating patients with MS.

• The 60-day post-activity assessment also indicated that participants are applying evidence-based strategies for promoting medication adherence and addressing other individualized health care needs.

Participants’ open-ended comments and questions to the faculty presenters reflected ongoing educational needs in the areas of methods for treating MS, conducting differential diagnoses, and understanding key pharmacoeconomic issues with MS drugs.


Grant # 7994

Laurence Greene, PhD

Senior Medical Writer, PRIME Education, Inc.

Chris R Prostko, PhD

Scientific Program Director, PRIME Education, Inc.

Established in the 1970s, the field of continuing medical education (CME) addresses the concern that the last 30 to 40 years of physicians’ and other healthcare providers’ careers may occur without any formal course of study.1,2 This issue is currently compounded by several factors, including (1) ongoing advancements in biomedical science and technology; (2) the continual development of novel therapies for emerging diseases; (3) evolution of models for collaborative and interprofessional medical practices; and (4) major changes in the infrastructure of our healthcare system. Indeed, the knowledge and skills that healthcare professionals acquire during their formative education may be obsolete within a matter of years or, in some cases, even months. To ensure the most successful outcomes for their patients, healthcare professionals must therefore engage in progressive, high-quality, and career-long education and skill training. Thus, CME can be an important element in continuing professional development.

In serving the vital mission of continuing education (CE) and CME for healthcare professionals, PRIME Education, Inc. (PRIME®) operates on established conceptual frameworks and sound principles of adult learning. This article describes PRIME®’s theory-guided and evidence-based processes for educational needs assessment, activity development, and outcomes assessment. The processes are summarized in the schematic overview in Figure 1.

Educational Needs Assessment and Activity Development Informed by Gap AnalysisThe success of any CME/CE activity depends on an initial comprehensive assessment of learners’ needs.3,4 A logical and productive approach to needs assessment is gap analysis, the systematic process of identifying differences between:

(1) The healthcare professional’s current knowledge, competence, and performance skills; and

(2) Established standards and criteria that must be achieved to promote the highest quality clinical performance and optimal patient outcomes.

As depicted in steps 1 and 2 of Figure 1, gap analysis directly informs the development of learning objectives for educational activities. Gap analysis is also fundamental to devising the methods and tools for outcomes assessment and for developing effective curricular strategies, media, and content. PRIME® operates on the principle that these two processes—the design of outcomes methodology and the development of educational activities—must be tightly integrated. This complementary approach, depicted in steps 3 and 4 of Figure 1, is essential for serving learners’ needs and ensuring successful outcomes of CME/CE activities.

Principled Approaches to Outcomes AssessmentToday’s leading approaches to outcomes assessment in CME/CE have been largely shaped by conceptual frameworks developed by Donald Kirkpatrick,5 George Miller,6 and Donald Moore and colleagues.7 Among other shared features these frameworks are based on the principle that the highest goals of adult education are achieved when learners successfully apply new knowledge to solve problems and master skills in their practice settings. Thus, outcomes assessment in CME/CE must account for the extent to which health professionals:

(1) Acquire essential information, ideas, and procedural skills that target identified gaps and serve an educational activity’s learning objectives;

(2) Demonstrate competent applications of the knowledge within the educational setting; and

(3) Skillfully transfer the newly acquired knowledge to practical settings, effectively closing the gaps that initially motivated the educational intervention.

PRIME®’s pyramid model of outcomes assessment, adapted largely from the recently refined framework of Moore et al,7 is presented in Figure 2. The base of the pyramid represents outcomes of participant demographics (level 1) and participants’ assessments of the quality and effectiveness of educational activities (level 2). Level 2 assessments are implemented through post-activity questionnaires in which participants rate the effectiveness, scientific rigor, and objectivity of the curriculum as well as the knowledge, expertise, and presentation skills of the faculty. In PRIME®’s continuous assessment model, data derived from level 2

Educational Needs Assessment, Activity Development, and Outcomes Assessment at PRIME®: Applications of Established Conceptual Frameworks and Principles of Adult Learning


Grant # 7994

Figure 1. An overview of PRIME®’s approach to needs assessment, educational activity development, and outcomes assessment. Post-activity assessments (steps 9 and 10) are conducted immediately following educational programs and up to 60–180 days later.


Grant # 7994

evaluations are essential for guiding future gap analyses and for informing the development of new educational activities that enable learners to achieve higher levels of knowledge, competence, and performance. These applications are reflected in the feedback loop from steps 9 and 10 to step 1 in Figure 1.

At level 3 of PRIME®’s pyramid model, pre-activity and post-activity tests are administered to assess changes in declarative (factual) knowledge and procedural knowledge, the latter of which is defined as an expressed understanding of the steps involved in carrying out healthcare practices. Learners’ gains in declarative and procedural knowledge are obviously prerequisites to improving performance skills and, ultimately, to ensuring successful patient health and community health outcomes. This pivotal role of core knowledge is especially pertinent for contemporary healthcare professionals, who continually face the challenges of grasping complex new information and techniques in the biomedical sciences. In addition to assessing learners’ declarative and procedural knowledge acquisition, PRIME® evaluates the extent to which CME/CE activities influence self-reported learning insights, values, and behaviors. Outcomes are assessed, for example,

on the influences of CME/CE interventions on participants’ attitudes about educational topics, their intentions to change practices in ways that meet established standards, and their subsequent self-directed learning behaviors. Positive changes in these important subjective learning domains are very often correlated with improved clinical performance and a deeper engagement in lifelong education.8-10

In keeping with the recent outcomes framework developed by Moore et al,7 PRIME® defines competence (level 4) by how successfully learners apply knowledge within the context of an educational activity. For healthcare professionals, gains in competence are thus reflected by such actions as direct applications of knowledge to diagnosing disease; selecting, administering, and adjusting therapies; and counseling and monitoring patients to ensure medication adherence and to prevent medication-related problems. In live and web-based educational settings, PRIME® assesses learner competence through such educational design strategies as performance simulations, practice-feedback sessions involving patient encounters, peer-to-peer virtual town-hall symposia, and technology-driven (eg, Unique Critique®) programs in which expert faculty provide individualized, branching feedback

Figure 2. A model for assessing outcomes of continuing education programs for health professionals. Adapted from Kirkpatrick,5 Miller,6 and Moore et al.7


Grant # 7994

to learners in a case-based question-and-answer format.11

PRIME® also applies elements of these strategies to assess outcomes at the level of performance (level 5).

The most pressing challenge for CME/CE providers is to support healthcare professionals in transferring newly acquired knowledge from educational contexts to practice settings, to promote improvement in performance. The extent to which CME/CE influences performance can be evaluated partly through subjective measures, including post-activity surveys. For example, in a questionnaire administered 60-180 days after an educational activity, PRIME® assesses participants as to how frequently they have applied gap-targeted knowledge in their recent practice, as well as what new actions and interventions they are regularly performing in the clinical setting that they were not performing prior to the CME/CE activity. Though subjective measures, these responses assist PRIME® in tracking the participant’s journey toward performance improvement and in identifying potential new barriers that may thwart the journey, lending important information in the gap analysis.

The ideal approaches to assessing performance outcomes demand direct and objective measures. However, CME/CE providers have traditionally faced many logistical barriers, including patient-privacy issues, in efforts to measure the

effects of educational activities on clinical performance. A major initiative, called performance improvement CME (PI-CME), is currently underway to address this problem, engaging maintenance of certification programs.12,13 Through application of this technology, PRIME® is evaluating the extent to which performance can be measured and level 5 learning can be achieved. The ability to track patient health outcomes (level 6) is an intended result of this technology through patient registry data. The success of this technology will hinge on many factors, not the least of which is the significant time commitment required of learners to fully engage in the PI-CME activity. As a result, PRIME® is also establishing business partnerships with physician member societies and government organizations, where outcomes of patient health and community health may be identified and measured.

The ultimate goal of CME/CE is to support healthcare professionals in closing targeted learning gaps to improve patient health (level 6) and community health (level 7). At present, logistical matters usually prohibit objective assessments of outcomes at these highest levels. The potential impact of PI-CME in providing data to assess patient and community health outcomes remains to be determined through future applications and associated educational research.

References1. Mansouri M, Lockyer J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof. 2007;27:6-15.

2. Mazmanian PE. Reform of continuing medical education in the United States. J Contin Educ Health Prof. 2005;25:132-133.

3. Kaufman DM. Applying education theory in practice. BMJ. 2003;326:213-216.

4. Grant J. Learning needs assessment: assessing the need. BMJ. 2002;324:156-159.

5. Kirkpatrick D. Revisiting Kirkpatrick’s four-level model. Training and Development. 1996;50:54-59.

6. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63-67.

7. Moore DE, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1–15.

8. Stahl SM, Grady M, Santiago G, et al. Optimizing outcomes in psychopharmacology continuing medical education (CME): measuring learning and attitudes that may predict knowledge translation into clinical practice. Focus. 2006;4(4):487-495.

9. Wakefield J, Herbert CP, Maclure M, et al. Commitment to change statements can predict actual change in practice. J Contin Educ Health Prof. 2003;23:81–93.

10. Delcourt JL. Commitment to change: a strategy for promoting educational effectiveness. J Contin Educ Health Prof. 2000;20:156–163.

11. PRIME® Education Inc. http://www.primeinc.org.

12. American Academy of Family Physicians. Performance Improvement in Practice. http://www.aafp.org/online/en/home/cme/cmea/cmeapplying/perfimprovement.html. Accessed April 5, 2010.

13. Stowell SA, Karcher RB, Carter RD, et al. Outcomes measurement design for a performance improvement initiative in diabetes care. CE Measure. 2009;3:76-83.

Documents

Educational Outcomes Report · website’s program registration pages, CME test/evaluation pages, or other areas interferes with your ability to access the information, please contact