PREDICTORS OF ACHIEVEMENT ON USMLE STEP I FOR STUDENTS AT UNIVERSITY OF PUERTO RICO

SCHOOL OF MEDICINE: IMPLICATIONS FOR ACADEMIC LEADERSHIP

Irma L. Rivera, EdD1 Claudia X. Alvarez, PhD2

Ricardo R. González-Méndez, PhD3

1Evaluation and Research on Medical Education Office, School of Medicine, Medical Sciences Campus, University of Puerto Rico, San Juan, PR 2Faculty of Education, University of Puerto Rico, Río Piedras Campus 3Department of Radiological Sciences, School of Medicine, Medical Sciences Campus, University of Puerto Rico, San Juan, PR

Disclosure

This study is part of the dissertation work of Irma L. Rivera, as part of the requirements towards the degree of Education Doctor at the University of Puerto Rico, Río Piedras Campus.

The authors do not have any financial interests and

do not have any conflicts of interest to declare.

Purpose

This study has two major objectives: Identify key performance variables predictive for passing

USMLE Step I for students at University of Puerto Rico School of Medicine.

*Characteristics before being admitted to Medical School *Performance of students during first and second year of Medical School

Understand how useful this knowledge is for the educational

leadership of the UPRSOM in order to develop early intervention strategies.

Background

93 92 90 91 90 92 93 93 94 93 93 91

83 90

68

87 83

78

89 85 88

77 77 76

50 55 60 65 70 75 80 85 90 95

100

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Perc

ent

Percent Passing U.S & Canada

Percent Passing P.R

Year Prepared by: Evaluation and Research on Medical Education Office

Percent Passing for USMLE Step I First Time Takers P.R. vs. USA-Canada, 1999-2010

University of Puerto Rico School of Medicine has been fully accredited by the LCME since 1954 Puerto Rico’s population are US citizens or legal resident aliens In national studies we are considered as minorities Being a US school we are not considered as a Caribbean School of Medicine

Background

National Studies identify the Medical College Admission Test (MCAT) as a good predictor of performance on USMLE Step I (Julian, 2000).

MCAT Biological Sciences score combined with General GPA were good predictors for performance on USMLE Step I (Collin, 2005).

MCAT Biological Sciences score combined with grades obtained during the first two years at medical school were significant predictors for performance on USMLE Step I (Coumarbatch, 2010).

Background

The practice of medicine is a shared social contract between the medical profession and the public. Educational leaders have to be aware about the constant state of change on medical education (McMahon, 2010).

Leaders keep things simple and make information accessible. This

helps individuals understand what’s going on, hence they are better able to achieve success (Rich, 2008).

Top performing leaders seem to be self-effacing team builders who focus on strategic and incremental changes that will gradually transform their organizations (Naylor, 2006)

Methods

IRB approval: (having advisors and involvement from two campuses at UPR,

approval was obtained from both IRB’s) University of Puerto Rico, Río Piedras Campus (00000944) IRB Medical Sciences Campus (A9170111).

Mixed Method Explanatory Sequential Design (Creswell, 2007)

Phase I

QUAN

Phase II

qual Interpretation based on QUAN qual results

Methods Phase I (Quantitative)

Graduates from years 2001 to 2011 (N=1,113) Frequencies, Chi-Square, Linear and Logistic Regression Logistic Regression, Backward Likelihood Ratio (Kleinbaum, 2002)

( )( ) ( )( ) ∑+=

−

=i

ii XXPXPXP βα

1lnlogit

Phase II (Qualitative) Dean, Associate Deans, Committees Coordinators (N=7)

Focus Group (1 Hour) 8 key questions on Educational Leadership

( ) ieXOR iβ=

Results

48% 52%

Female Male

Gender Age Mean 23 (SD = 2.1)

N=1,113

Results Statistically Significant Variables

Variable Chi

Square df p

GPA sciences 157.67 131 0.056

GPA general 136.63 106 0.024

Age 753.32 666 0.010

Gender 4.36 1 0.037

MCAT total 50.60 21 0.0001

MCAT Physical Sciences 39.83 9 0.0001

MCAT Biological Sciences 36.00 9 0.0001

High School 17.07 5 0.004

1st Year Courses Chi

Square df p

Anatomy 72.11 2 0.0001

Embriology 64.96 2 0.0001

Histology 62.97 2 0.0001

Biochemistry 99.80 2 0.0001

Physiology 101.19 2 0.0001

Neurosciences 105.69 2 0.0001

Human Development 44.64 2 0.014

2nd Year Courses Chi Square df p

Pathology 159.83 2 0.0001

Infectious Diseases 132.47 3 0.0001

Pharmacology 152.36 2 0.0001

Fundamentals of Clinical Diagnosis 56.53 2 0.0001

Psicopathology 30.61 2 0.0001

Mechanisms of Diseases 109.77 2 0.0001

Public Health 48.26 2 0.0001

Results Logit Model (Admissions)

Predicting Variable Estimate of Parameter

Standard Error

Wald df p OR

GPA sciences 1.176 1.176 15.843 1 .000 3.241

MCAT Physical Sciences 0.337 0.085 15.863 1 .000 1.400

MCAT Biological Sciences 0.264 0.078 11.435 1 .001 1.302

Constant -6.743 1.293 27.212 1 .000

Variables entered: GPA sciences, GPA general, MCAT Physical Sciences, MCAT Biological Sciences. R² = 6% Power = 99%

Results Logit Model (Combined)

Predicting Variable Estimate of Parameter

Standard Error

Wald df p OR

MCAT Biological Sciences .220 0.086 6.498 1 0.011 1.24

Physiology .792 0.336 5.552 1 0.018 2.207

Neurosciences .430 0.197 4.787 1 0.029 1.538

Biochemistry .500 0.218 5.274 1 0.022 1.648

Pathology .962 0.230 17.545 1 0.000 2.616

Pharmacology .900 0.239 14.179 1 0.000 2.458

Constant -9.164 1.052 75.877 1 0.000

Variables entered: GPA sciences, MCAT Physical Sciences, MCAT Biological Sciences, Physiology, Neurosciences, Pathology, Pharmacology and Biochemistry. R² = 19% Power = 100%

Results

Educational Leadership: Consensus of the Focus Group

The University of Puerto Rico School of Medicine has a reliable study that will be helpful to make appropriate decisions regarding admissions and students at risk of academic performance problems.

The school needs to review: the admissions formula, the course

content and the assessment methodology.

The school should continue to strengthen faculty development.

The leadership of the Second Year made a decision to develop an early intervention program to address the students at risk of failing USMLE Step 1 based on the factors identified in this study.

New Intervention Strategies

Postpone the taking of the USMLE Step 1 Counseling by Psychologist, Counselors and Faculty (individualized study plan) Faculty development to improve questions in tests. Improve the integration of basic and clinical sciences Open Forum in which third year students talk about their experiences in taking the exam NBME practice test recommended (score >200 desired) Early access during 2nd year to USMLE World question bank.

93 92 90 91 90 92 93 93 94 93 93 91 94 96

83 90

68

87 83

78

89 85

88

77 77 76

92 98

50

60

70

80

90

100

110

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Perc

ent

Percent Passing U.S & Canada

Percent Passing P.R

Year Prepared by: Evaluation and Research on Medical Education Office

Percent Passing for USMLE Step I First Takers P.R. vs USA-Canada, 1999-2011

Conclusions

The PREDICTIVE variables identified by our logistic model at ADMISSION for passing the USMLE Step I were: GPA sciences, MCAT Biological Sciences, MCAT

Physical Sciences

The PREDICTIVE variables identified by our logistic model for passing the USMLE Step I were: MCAT Biological Sciences Grades on courses of Physiology, Neurosciences,

Biochemistry, Pathology and Pharmacology

Conclusions

Leaders reflected about their leadership. They concluded that transformational leadership style

will be more effective in the development of strategies that will promote the incremental changes needed for continuous improvement at the UPRSOM.

The leadership of the Second Year designed an early intervention program that has substantially improved the passing rate of first time takers of the USMLE Step 1 over the last two years.

Leaders think and talk about the solutions. Followers think an talk about the problems. —Brian Tracy

Thank you!

References

Collin, V.T. (2005). Predictive validity for the medical college admission test using multiple regression and latent variable path analysis (LVPA). (Disertación doctoral). Universidad de Calgary, Alberta, Canadá.

Coumarbatch, J., Robinson, L., Thomas, R., Bridge, P.D. (2010). Strategies for identifying students at risk for usmle step 1 failure. Family Medicine, 42, 105-110.

Creswell, J. W. (2009). Research design: qualitative, and mixed methods approaches. California: SAGE Publications. Julian, E.R. (2000). The predictive validity of the medical college admission test. Association of American Medical

Colleges, 3, 1-3. Kleinbaum, D.G., Klein, M. (2002). Logistic regression a self-learning text (second edition). New York: Springer-Verlag. McMahon, G.T., Tallia, A.F. (2010). Anticipating the challenges of reforming the united states medical licensing

examination. Academic Medicine, 85, 453-456. Naylor, D. (2006). Leadership in academic medicine: reflections from administrative exile. Clinical Medicine, 6, 488-492. Rich, E.C., Magrane, D., Kirch, D.G. (2008). Qualities of the medical school dean: insights from the literature.

Academic Medicine, 83, 483-487.

Additional Slides

Graduated Class

53 54 10749.5% 50.5% 100.0%

53 53 10650.0% 50.0% 100.0%

50 48 9851.0% 49.0% 100.0%

57 58 11549.6% 50.4% 100.0%

37 63 10037.0% 63.0% 100.0%

35 54 8939.3% 60.7% 100.0%

54 61 11547.0% 53.0% 100.0%

48 52 10048.0% 52.0% 100.0%

50 42 9254.3% 45.7% 100.0%

47 46 9350.5% 49.5% 100.0%

52 46 9853.1% 46.9% 100.0%

536 577 111348.2% 51.8% 100.0%

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

Total

M FGender

Total

Average size graduating classes =102 (SD=8.8)

Focus Group Key Questions

Strategic Leadership 1-As with the results of phase 1 of this study a plan could be designed to achieve the mission and vision of

University of Puerto Rico School of Medicine? Didactic Leadership 2-Which educational practices can be integrated into the courses to strengthen the performance of medical

students on USMLE Step I and facilitate students will approve it on the first attempt? 3-How technology (computer programs, simulators, electronic data banks and so on) could be integrated

into the curriculum to strengthen the teaching and learning process, that has an impact for a better medical students performance on USMLE Step I and promoting students will approve it on the first attempt?

4- How faculty could be developed professionally so can be strengthening the teaching and learning process and thus the performance of medical students on USMLE Step I?

Administrative Leadership 5- With the results obtained in phase 1 of the study. Additional fiscal, human resources and technology are

required to strengthen the performance of medical students on USMLE Step I and facilitate students will approve it on the first attempt?

6- How you will use the results from phase 1 of the study to prepare action plans directed on how you as a leader can effectively manage fiscal and human sources that have an impact on students’ success?

Community Leadership 7-How important are the results of this study so you as a leader can respond to changes and challenges that

have today medical education and train doctors that respond to society needs? Implications for leadership 8-You as a leader in the school of medicine, How important is to knowing the predictive variables for

medical students performance on USMLE Step 1, and how helps you to exercise educational leadership at University of Puerto Rico School of Medicine to facilitate that students can pass USMLE Step 1 on the first attempt?