Student USMLE Step 1 Preparation and Performance

Advances in Health Sciences Education 9: 291–297, 2004.© 2004 Kluwer Academic Publishers. Printed in the Netherlands.

291

Student USMLE Step 1 Preparation andPerformance

CHI ZHANG1∗, ABBY RAUCHWARGER2, CYNTHIA TOTH3 andMARK O’CONNELL1

1Department of Medical Education, University of Miami School of Medicine, P.O. Box 016960(R53), Miami, Florida 33101, USA; 2Department of General Surgery, New York University Schoolof Medicine, New York, NY, USA; 3Department of Internal Medicine, Duke University MedicalCenter, Durham, NC, USA (∗author for correspondence, e-mail: [email protected])

Abstract. Objectives: Many medical students take commercial preparation courses to prepare for theUSMLE Step 1. This investigation examined the relationships among Step 1 performance, prepara-tion method, and academic achievement in medical school.Method: A survey was conducted on 148 junior students at a medical school in July 2000 regardingtheir preparation methods for Step 1. Additional data was collected on student Step 1 scores, attend-ance record on preparation courses, and course grades in medical school.Results: One hundred students (68%) responded to the survey including 32 preparation course parti-cipants. Course participants had significantly lower Step 1 scores and second-year GPAs than thoseof the non-participants (p < 0.05). However, the effect of preparation method was not significant byusing ANCOVA when the second-year GPA was used as a covariate (p = 0.71).Conclusion: Performance on Step 1 is related to academic performance in medical school and notthe type of preparation methods.

Key words: academic performance, medical curriculum, medical education, medical student,preparation courses, USMLE Step 1, undergraduate

Introduction

The United States Medical Licensing Examination (USMLE) is currently thesole examination pathway to initial licensure for allopathic physicians in theUS. The three Steps of the USMLE have different foci and form a continuum.The USMLE Step 1 emphasizes basic mechanisms and principles (United StatesMedical Licensing Examination, 2001), which are typically covered within thebasic science and pre-clinical curriculum during the first two years of medicalschool. Over three fourths of US medical schools require medical students to passthe USMLE Step 1 for promotion and/or graduation (Barzansky, Jonas and Etzel,1997). Residency program directors also view USMLE Step 1 scores as a nation-ally standardized, objective report of information in the student profile (Bowles,Melnick and Nungester et al., 2000). Many medical students take commercialpreparation courses to help them prepare for Step 1 because of its importance to

292 CHI ZHANG ET AL.

their careers. However, a 1998 survey on a random sample of Step 1 examinees inthe US and Canada suggested that participation in such courses had little effecton their performance (Thadani, Swanson and Galbrath, 2000). A recent studyalso concluded that students who took commercial coaching courses to improveperformance on Step 1 did not achieve higher scores than those who studied ontheir own (Werner and Bull, 2003). Furthermore, two earlier studies concurredthat commercial review courses had little or no impact on the medical board examperformances (Scott, Scott and Palmisano et al., 1980; Lewis and Kuske, 1978).On the other hand, previous studies have suggested a strong correlation betweenstudent performance on Step 1 and academic achievement in medical school(Thadani, Swanson and Galbrath, 2000; Swanson, Ripkey and Case et al., 1996).Other reported predictors for Step 1 performance included basic science pathways(Way, Biagi and Clausen et al., 1999) and MCAT scores (Thadani, Swanson andGalbrath, 2000).

This investigation explored factors associated with medical students’ perfor-mance on the USMLE Step 1. The three major objectives of this study were to:(1) examine the relationships between student performance on Step 1, academicachievement in medical school, and the preparation method for Step 1; (2)examine students’ perception on the efficacy of the medical school curriculum inpreparing them for Step 1; and (3) compare preparation course participants andnon-participants on their perceptions of preparedness for Step 1. This study wasconducted at the University of Miami School of Medicine (UMSM), a privateschool in the southeast of the United States with an enrollment of approxi-mately 600 medical students for the 2000–2001 academic year. Up until the2001–2002 academic year, the UMSM curriculum was a discipline-based model(Swanson, Ripkey and Case et al., 1996), consisting of two years of basic sciences,mechanisms of disease, and clinical skills, and two years of clinical clerkships.

All second-year medical students at the UMSM are required to pass Step 1for promotion to the junior year. The UMSM students have approximately eightunscheduled weeks between the second and third years of medical school. Studentsmust sit for the USMLE Step 1 prior to the beginning of the third year clinicalclerkships. Several available commercial preparation courses for Step 1 are offeredto students in May and June in the region. The preparation course of choice in2000 was offered by Kaplan, which provided three weeks of full-day lecture formatinstruction. All of the students that participated in a preparation course took theKaplan course that year.

Methodology

A survey questionnaire was distributed to the junior medical students (N = 148)in July 2000 after they took the USMLE Step 1 but before they received theExam results. The gender makeup of the junior class included 71 (48%) maleand 77 (52%) female. The questionnaire contained 43 multiple-choice questions

STUDENT USMLE STEP 1 PREPARATION AND PERFORMANCE 293

including: (a) demographic data and the number of weeks they spent preparing forStep 1; (b) factors influencing their decision on preparation course participation;(c) assessment of the medical school curriculum; and (d) preparation methods forStep 1.

Student academic records were obtained including Step 1 scores, UMSM coursegrades, UMSM weighted grade point averages (GPAs) for the first two years ofmedical school, class GPA quartile standing, as well as attendance status on prepar-ation course participation. The UMSM uses a numeric grading system, assigningcourse grades on a scale of 0–100. Weighted GPAs were calculated based on theindividual course grades weighted by the number of credits.

Descriptive and inferential statistical techniques were used to analyze the data.An α level of 0.05 was used in determining statistical significance for inferentialstatistical analyses.

Results

One hundred medical students (68%) responded to the survey including 44 malesand 56 females, among which 32 students claimed to be preparation course parti-cipants. Among the sample of 148 junior medical students of this study, 135 recordsof first year academic achievements (GPAs, quartiles & course grades) and 140academic records for the second year were obtained from the medical school. Thegaps for student records were due to multiple factors: dropouts, transfers and leavesof absence. Records from the school indicated that a total of 43 (29%) studentsparticipated in the preparation course for Step 1. To compare with the targetedsample, the respondents had slightly higher percentages of females (56% vs. 52%)and preparation course participants (32% vs. 29%).

The 2000 USMLE Step 1 passing score was 179. Ninety-seven percent ofUMSM juniors (n = 143) passed the 2000 Step 1 on their first attempt. One studentwho failed initially soon dropped out of medical school due to other academicreasons. The other four students who failed initially passed Step 1 on their laterattempts. Among these five juniors, two took the preparation course and threestudied on their own. The UMSM mean score for the 2000 USMLE Step 1 was215 ± 21, which was very similar to the national average (215 ± 23).

Associations between the following factors and methods of preparation for StepOne were evaluated: “personal learning habits”, “advice from other students”, “theneed for an organized schedule”, “deficiency of the curriculum”, “reputation of thecourse”, “confidence in your abilities”, and “money”. “Personal learning habits”and “advice from other students” were the two most important factors affecting allthe participants (100%) and nearly all the non-participants (97%) in their decisionas to how to prepare for Step 1. “The need for an organized schedule” was animportant factor to all student participants but only to 59% of the non-participants(Table I).


Table I. Factors influencing students’ decision on preparation course participation

Factor Participants Non-participants

Advice from other students 94% 88%

Money 40% 60%

Reputation of the course 97% 57%

Personal learning habits 100% 97%

Deficiency of the curriculum 85% 39%

Confidence in your abilities 71% 84%

The need for an organized schedule 100% 59%

Note: The numbers shown are the percentages of students who responded either“very important” or “important” to each listed factor.

There was no significant difference in first-year GPAs between the prep courseparticipants (85.8 ± 4.6) and non-participants (87.2 ± 4.4) (t = 1.8, p = 0.08).Non-participants had a higher second-year GPA (87.8 ± 4.3) than participants(85.4 ± 4.6) (t = 3.1, p < 0.05). Chi-square test results suggested that there wasa significant relationship between the first year class quartile standing and prepar-ation course participation (χ2 = 11.4, p < 0.05). A significant relationship wasalso found between the second year class quartile standing and preparation courseparticipation (χ2 = 9.8, p < 0.05). A greater proportion of students from the lowestquartile of the first- and second-year GPAs took preparation courses.

The Step 1 scores correlated significantly with both the first-year GPAs (Pearsonr = 0.78, p < 0.01) and the second-year GPAs (Pearson r = 0.80, p < 0.05). Usinga t-test, the difference on Step 1 mean scores between participants (206.0 ± 18.5)and non-participants (218.7 ± 21.4) was found statistically significant (t = 3.4,p < 0.05). Non-participants had a significantly higher Step 1 mean score thanparticipants. However, due to the significant difference in the second year GPAmeans between participants and non-participants, the pre-existing difference inGPAs should be adjusted for a meaningful comparison of the Step 1 scores betweenthe two groups. By using an analysis of covariance (ANCOVA) with the second-year GPA as a covariate, the effect of preparation course participation was notstatistically significant (F = 0.1, p = 0.71). This suggests that participation in thepreparation course had no significant effect on student Step 1 performance whenthe effect of second-year GPA was controlled.

Overall, 84% of the survey respondents perceived the first two years of themedical school curriculum to be helpful in preparing them for the USMLE Step 1.Among the specific segments of the medical school curriculum, “pathology”(94%), “physiology” (90%), “anatomy/neuroanatomy/embryology” (88%), and“microbiology/immunology” (85%) were thought to be most helpful; “behavioralsciences” (43%), “biochemistry/genetics” (47%), and “pharmacology” (58%) werethought to be least helpful (Table II). There was a strong correlation between


Table II. Student perceptions of the medical school curriculum and preparation course inpreparing them for step 1

Course segment Medical school curriculum Prep

All Non- Prep course course

respondents participants participants

Pathology 94% 93% 97% 68%

Physiology Portion 90% 90% 91% 94%

Anatomy/Neuroanatomy/Embryology 88% 82% 90% 87%

Microbiology/Immunology 85% 85% 84% 94%

Pharmacology 58% 63% 47% 97%

Biochemistry/Genetics 47% 42% 56% 90%

Behavioral Sciences 43% 42% 47% 81%

Note: The numbers shown are the percentages respondents who perceived that the medical schoolcurriculum or the preparation course “definitely” or “somewhat” prepared them well for Step 1.

preparation course participants and non-participants on the perceptions of themedical school curriculum (Spearman ρ = 0.88, p < 0.05).

Ninety-seven percent of preparation course participants perceived that thepreparation course either “definitely” or “somewhat” helped them in preparing forthe USMLE Step 1. Most participants would “definitely” (55%) or “somewhat”(27%) recommend the course. Only 18% would not recommend the course “atall”. Most participants found the course materials (books, CD-ROMs) to be “defin-itely” (53%) or “somewhat” (41%) helpful. The live lectures were perceived tobe “definitely” (69%) or “somewhat” (22%) helpful. Similarly, many participantsbelieved that the course “definitely” (55%) or “somewhat” (27%) prepared themwell with regard to “test format/test-taking strategies”. Approximately 28% of theparticipants felt that the “amount of time in lecture” was “definitely” appropriate;and 53% of the participants thought it was “somewhat” appropriate.

Non-participants perceived that the self-study method was either “definitely”(84%) or “somewhat” (15%) an efficient use of their time. About 59% of non-participants reported that they studied alone, while 31% reported studying withanother person, and 10% reported studying with a group.

In terms of specific segment of the preparation course, “pharmacology”,“physiology”, “microbiology/immunology”, “biochemistry/genetics”, “anatomy/neuroanatomy/embryology”, and “behavioral sciences” were thought to be veryhelpful. However, only 68% of the participants were in favor of the “pathology”segment (Table II). There was a negative correlation between the perceived qualityof the medical school curriculum and the perceived quality of correspondingsegments of the preparation course (Spearman ρ = –0.32, p < 0.05).


Discussions

The results of this study indicate that academic achievement during the first twoyears in medical school is an important factor related to student performance onthe USMLE Step 1, which concurs with previous findings (Thadani, Swansonand Galbrath, 2000; Swanson, Ripkey and Case et al., 1996). This suggests thatstudents’ early academic performance in medical school can be used as a predictorfor future USMLE Step 1 performance. Remedial studies or tutorials in the weaksubject areas should be recommended and offered to those students with low GPAs.

Commercial preparation courses such as Kaplan’s were well received by theparticipants in general. These courses served well for those students who neededan organized schedule. However, student performance on Step 1 was not enhancedafter participating in the preparation course. Kaplan’s preparation course appearedto offer no advantage as a remedial course to the medical students. In addition,this study showed that self-study can be an equally effective method to prepare forStep 1 for those students who prefer to study alone or in pairs.

The medical students in this study felt that the school curriculum prepared themwell for Step 1. Most students felt well prepared for Step 1 regardless of theirpreparation methods. Student expectations for the preparation course appeared tobe based on their coursework experiences in the medical school curriculum. Therewas an inverse relationship between the perceived course quality of the curriculumand the perceived quality of corresponding segments of the preparation course. Forexample, pathology was the most favored course within the UMSM curriculumaccording to the preparation course participants; however, it was the least favoredsegment in the preparation course. Pharmacology was the most favored segment ofthe preparation course according to participants; yet it was one of the least favoredcourses with the UMSM curriculum by the same group. It can be postulated thatpreparation course participants were looking for added value to complement theperceived weaknesses in the medical school curriculum.

The junior students at the University of Miami School of Medicine did well onthe USMLE Step 1 in 2000. The vast majority of the junior students passed Step 1on the first attempt, and the remainder passed it in later attempts. A few conclusionscan be reached based on the findings of this study: (1) performance on Step 1 isrelated to academic performance in medical school and not the type of prepara-tion methods; (2) students perceived value in whatever method of preparation theyused; and (3) the strengths of the commercial preparation course were in areas ofperceived curriculum weaknesses.

References

Barzansky, B., Jonas, H. & Etzel, S. (1997). Educational programs in US medical schools, 1996–1997. Journal of American Medical Association 278: 744–749.


Bowles, L.T., Melnick, D.E., Nungester, R.J., Golden, G.G., Swanson, D.B., Case S.M., Dillon G.F.,Henzel T.R., Orr, N.A. & Thadani, R.A. (2000). Review of the score-reporting policy for theUnited States Medical Licensing Examination. Academic Medicine 75(5): 426–431.

Lewis, L.A. & Kuske, T.T. (1978). Commercial National Board Review Programs: a case study atthe Medical College of Georgia. Journal of American Medical Association 240: 754–755.

Scott, L.K., Scott, C.W., Palmisano, P.A., Cunningham, R.D., Nass, J.C. & Brown, S. (1980). Theeffects of commercial coaching for the NBME Part I examination. Journal of Medical Education55: 733–742.

Swanson, D., Ripkey, D., Case, S. & the Validity Study Group for USMLE Step 1/2 Pass/Fail Stand-ards (1996). Relationship between achievement in basic science coursework and performance on1994 USMLE Step One test administrations. Academic Medicine 71(10 Supplement): S28–S30.

Thadani, R.A., Swanson, D.B. & Galbrath, R.M. (2000). A preliminary analysis of differentapproaches to preparing for the USMLE Step 1. Academic Medicine 75(10 Supplement):S40–S42.

United States Medical Licensing Examination (2001). 2001 Step One Content Description andSample Test Materials.

Way, D.P., Biagi, B., Clausen, K. & Hudson, A. (1999). The effects of basic science pathway onUSMLE Step 1 scores. Academic Medicine 74(10 Supplement): S7–S9.

Werner, L.S. & Bull B.S. (2003). The effects of three commercial coaching courses on Step OneUSMLE performance. Medical Education 37: 527–531.

Documents

Student USMLE Step 1 Preparation and Performance