AN 8-WEEK CLINICAL RESEARCH

EDUCATION COURSE FOR NEUROLOGY RESIDENTS

Educating Future Clinical Research Investigators:

Jeanie McGee, MSHS, CCRC

Acknowledgements

Dissertation ChairWarren McDonald, PhD

Dissertation CommitteeAlireza Minagar, MD, FAANRebecca Nolan, PhD, MP

Special ThanksRobert Schwendimann, MDDebra E. Davis, MD

Background

The physician-investigator has been labeled as an endangered species (Armstrong, DeCherney, Leppert, Rebar, & Maddox, 2009; Goldhammer et al., 2009; Guelich, Singer, Castro, & Rosenberg, 2002; Johnson, Subak, Brown, Lee, & Feldman, 2010; Lowe et al., 2007; Raghavan & Sandham, 2007; Siemens, Punnen, Wong, & Kanji, 2010; Teo, 2009; Wyngaarden,1979)

Lack of research fundingLack of time and interestRegulatory requirementsLack of adequate exposure, education, and

training in research during residency (Armstrong et al., 2009; Clancy, 2008; Krousel-Wood, 2006; Teo, 2009)

Background (cont.)

Consequences:Delays in translational researchIndustry hesitance to invest time and moneyDecrease in competent and effective

mentorsDevastating effects on patient outcomes

Review of the Literature Medline (EBSCOHost) (Exploded with MeSH options)

Pubmed Google Scholar 2002 to 2012 (some historical literature) Terms:

research education, residency research education, clinical research education, self-perceived competency, residency education, and neurology residency education

Research during residency is a critical topic. Limited evaluation of research education during

residency, especially neurology

Review of the Literature (cont.)

Institutional neurology residency program accredited in 2005

3 residents have participated in research elective (out of 27 graduated)

Mostly used as study time, so revised to require research plan with specific topic and protocol – 0 residents participated since

(Dr. Davis, personal communication, August 28, 2012)

Problem Statement

This researcher will develop, implement, and evaluate an 8-week clinical research educational course for residents in a neurology department at a Louisiana university medical center to prepare neurology residents for participation as clinical research investigators by increasing knowledge of fundamental clinical research methodology, design, and analysis, as well as by increasing the residents' self-perceived competency in these areas.

First Major Aspect of Problem Statement

Improving the quality of patient care by filling the research education gapPhysicians with research interests and skills

needed for successful translation of new treatments and therapies

Solution: Require research education during residency training

Second Major Aspect of Problem Statement

National recognition of the problem National Institutes of Health (NIH)

○ Clinical Research Training Program – 1997○ Medical Research Scholars Program – 2012○ Resident Research Career Day - 2012○ Office of Clinical Research Training and Medical Education (OCRTME)

Summer internships, research electives for medical students, residents, fellows, and high school students

United States Food & Drug Administration (FDA)○ Critical Path Initiative - 2009○ Clinical Investigator Training Course – 2009

Accreditation Council for Graduate Medical Education (ACGME)○ Require curriculum that advances “"residents' knowledge of the basic

principles of research, including how research is conducted, evaluated, explained to patients, and applied to patient care" (ACGME, 2010, p.14).

Solution: Incorporate research education at the local level

Third Major Aspect of Problem Statement

Research education in neurology residencyOnly 10% of ACGME- accredited neurology

residency programs have a research requirementOver 50% of neurology residency programs require

research, but offer no specific rotation or educational curriculum, nor protected research time

If available, research rotation has shortest average rotation time (mean = 0.3; range = 0 – 2 months) (Schuh et al., 2009)

Solution: Implement a mandatory, structured clinical research curriculum for neurology

residents

Fourth Major Aspect of Problem Statement

Extinction of the clinical research investigatorDownward trend since late 1970s (Wyngaarden, 1979)

Currently, only 4% of physicians engage in research activities (CISCRP, 2012)

Need to preserve the integrity of clinical researchCNS therapeutics are the most prevalent area

addressed by pharmaceutical research (second only to oncology and immunomodulators, which are also common neurological therapies) (CISCRP, 2012)

Solution: Promote interest and encourage clinical research activity involvement during neurology residency

Fifth Major Aspect of Problem Statement Effects of research education for residents in order to build

research competency 87% of physicians believe statistics knowledge is important in patient

care delivery, but only 17% report their statistics knowledge to be sufficient (West & Ficalora, 2007)

88% report biostatistics and a strong research design foundation is necessary to effectively and accurately appraise published literature (West & Ficalora, 2007)

75% of medicine residents reported they understand all statistics reported in the literature, while their mean score was 41.4% correct responses when tested on statistical concepts found in the literature (Windish, Huot, & Green, 2007)

Solution:

Implement curriculum that includes evaluation that effectively assessing discrepancies in self-assessed

competency and actual

level of knowledge

Sub-Problems

Sub-Problem 1.Do neurology residents who have taken a prior elective or required clinical research course(s) have a higher level of self-perceived competency in fundamental clinical research methodology, design, and analysis than neurology residents who have not taken a prior elective or required clinical research course(s) as determined by correlating results of a pre-test measure of self-perceived competency in fundamental clinical research with resident prior exposure to research education?

Sub-Problems

Sub-Problem 2.Does the neurology resident’s future career plans have an impact on their level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Sub-Problems

Sub-Problem 3.Does the resident’s age influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Sub-Problems

Sub-Problem 4.Does the resident’s gender influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Sub-Problems

Sub-Problem 5.Does the resident’s post-graduate year level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Purpose of the Study

The purpose of this dissertation research was to evaluate the effectiveness of an 8-week clinical research education course designed for neurology residents. The research curriculum was designed to include topics significant to research methodology, design, and analysis.

Study Goals

to provide neurology residents in a single department at a university medical center in Louisiana information to increase the knowledge level of residents in the topic areas of research methodology, design, and analysis

Study Objectives (1) measure baseline levels of research knowledge

and residents' self-perceived level of competency in research,

(2) measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention,

(3) evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research, and

(4) determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure

Theoretical Framework

Cognitivist Approach and Competency-Based Model Jerome Bruner’s cognitivist approach

Self-discoveryLeveled learningPrior knowledgeInformation acquisition (derived from existing

knowledge)Information transformation (processing and analyzing

new information)Information evaluation (self-assessment of information

accuracy)

Theoretical FrameworkCognitivist Approach and Competency-Based

Model G. E. Miller’s framework for clinical

assessmentDoes

(Action)

Shows How (Performance)

Knows How

(Competence)

Knows

(Knowledgeable)

Research Design

Quantitative one-group pre-test post-test Dependent variables:

Resident level of knowledge in clinical research methods, design, and analysis

Resident self-perceived competency in clinical research methods, design, and analysis

Sample:Neurology residents in PGY II, III, and IV

training

Curriculum / Intervention Weekly sessions for 8 consecutive weeks 1 hour Session topics:

Introduction to Research: Basic Research Concepts and Terminology

The Research Question and HypothesisResearch Study Design and Sampling Introducing Variables and Scales of Measurement Introduction to BiostatisticsResearch Ethics, Privacy, and Responsible Study ConductBasic Regulatory Requirements for Clinical ResearchReview of Research Methodology, Design, and Analysis

(Overview)

Methodology Measure baseline levels of research knowledge and

residents' self-perceived level of competency in research

Measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention

Evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research

Determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure

Methodology (cont.) Measure baseline levels of research knowledge and

residents' self-perceived level of competency in research

Measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention

Evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research

Determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure

Instruments Developed by the researcher:

Evaluation Tool to Capture Neurology Resident Descriptive Data demographic assessment tool

Self-Assessed Level of Knowledge and Competency in Clinical Research (pre-assessment) (38 items – score 1 to 4 from least competent to most competent)

Level of Knowledge in Clinical Research Methodology, Design, and Analysis (pre-assessment) (50 multiple choice items with a possible score = 100%)

Self-Assessed Level of Knowledge and Competency in Clinical Research (post-assessment) (38 items – score 1 to 4 from least competent to most competent)

Level of Knowledge in Clinical Research Methodology, Design, and Analysis (post-assessment) (50 multiple choice items with a possible score = 100%)

Instruments

Considerations in the design of instruments:American Academy of Neurology (AAN)

evidence-based medicine Association of Clinical Research Professionals

(ACRP) Certified Clinical Research Investigator mock examination

Miller’s pyramid of competence

Data Collection

Collect demographic data Collect baseline self-competency data Collect baseline level of knowledge data Implement curriculum intervention Collect post self-competency data Collect post level of knowledge data

Data Analysis

Data coded and entered into Microsoft Excel Cleaned, verified Data moved to SPSS v. 21 for statistical

analysis

Predictions1. Residents with prior research education will have

higher mean scores on both knowledge and self-perceived competency at baseline

2. Residents with future career plans involving research and/or academia will have higher mean scores on both knowledge and self-perceived competency at baseline

3. Higher PGY levels will have higher mean scores on both knowledge and self-perceived competency at baseline

4. Male residents will have higher mean scores on both knowledge and self-perceived competency at baseline

5. Older will have higher mean scores on both knowledge and self-perceived competency at baseline

Data Analysis Descriptive statistics

Average age = 35.5 yearsPGY Levels:

○ PGY2: 30% (n=3)○ PGY3: 40% (n=4)○ PGY4: 30% (n=3)

70% reported prior research education (only 1 during medical school or residency)

Length of time of prior education: range = less than 1 week (n=3) to three years or more (n=1) (others were less than one month (n=1), 3m to 6m (n=1), 6m to 1y (n=1)

Most common topics:○ Developing and writing research question and hypothesis

(n=5)○ Research study design and sampling (n=5)

Data Analysis (cont.) Descriptive statistics

20% (n=2) reported career plans involving clinical research

20% (n=2) reported career plans involving basic science research

20% (n=2) reported career plans involving writing and publication

30% (n=3) reported career plans involving work at an academic medical center

20% (n=2) reported career plans NOT involving any type of research

30% (n=3) have not yet decided whether or not they will engage in research after residency

Data Analysis (cont.) Self-assessed Competency at Baseline

Mode = 1 “knows about the skill or concept or has heard about it, but is unable to do it”

Mode range 1 -2:○ “knows about the skill or concept or has heard

about it, but is unable to do it”○ "knows how to do a skill/concept, but has never

done it and is unable to show or explain how to do it“

No item had a mode of 3 or 4 at baseline ○ 3: “knows how, can show or explain how to do it, but has

never done it”○ 4: “knows how, has done, and can do at any time”

Data Analysis (cont.) Self-assessed Competency at Baseline

Possible total score = 152 (all 4s, representing highest level of competency)

Percentage scores calculated, mean = 63% at baseline (range = 42 – 125)

Data Analysis (cont.) Level of Knowledge at Baseline

Mean score = 53 (range = 36 – 70)

Resea

rch

Term

s

Resea

rch

Quest

ion &

Hyp

othe

sis

Study

Des

ign &

Sam

pling

Variab

les &

Sca

les o

f Mea

sure

men

t

Statis

tics

Ethics

, Priv

acy,

Res

pons

ibility

Regula

tory

Req

uirem

ents

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

52.546.7

53.8 48.845.6

61.7 62.0

Sub-Problem ResultsSub-Problem 1. Do neurology residents who have taken a prior

elective or required clinical research course(s) have a higher level of self-perceived competency in fundamental clinical research methodology, design, and analysis than neurology residents who have not taken a prior elective or required clinical research course(s) as determined by correlating results of a pre-test measure of self-perceived competency in fundamental clinical research with resident prior exposure to research education?

Pre-Knowledge Mean Pre-Competency Mean Percentage0

10

20

30

40

50

60

44.748.6

56.651.1

No Prior Research Educa-tion

Prior Research Education

Had an effect, but did not reach statistical significance.

Sub-Problem Results

Sub-Problem 2. Does the neurology resident’s future career plans have an impact on their level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Self-perceived competency: Career interest involving clinical research (F=9.142; p=.016)* Career interest involving basic science research (F=8.652; p=.019)* Career interest involving writing and publishing scientific

manuscripts (F = 8.652; p = .019)*

Level of knowledge: No statistically significant effect

Sub-Problem ResultsSub-Problem 3. Does the resident’s age influence level of knowledge in

fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Self-perceived competency: No statistically significant effect

Level of knowledge: No statistically significant effect

Sub-Problem ResultsSub-Problem 4. Does the resident’s gender influence level of knowledge

in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Self-perceived competency: No statistically significant effect, but males had a higher mean (67,

compared to 63 for females)

Level of knowledge: No statistically significant effect, but males had a higher mean

(54.57, compared to 49.33 for females)

Sub-Problem ResultsSub-Problem 5. Does the resident’s post-graduate year (PGY)

level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Self-perceived competency: No statistically significant effect

Level of knowledge: No statistically significant effect on overall score, but statistics

domain differences statistically significant (F=5.067; p=.044)*

PGY2 PGY3 PGY40

10

20

30

40

50

60

70

80

90

100

33.3

52.8 51.87 PreStats Domain

Sub-Problem ResultsSub-Problem 5. Does the resident’s post-graduate year (PGY)

level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

Level of knowledge: No statistically significant effect on overall score

Pre-Knowledge Mean Pre-Competency Mean Percentage0.0

20.0

40.0

60.0

80.0

100.0

48.0

58.956.0 56.254.0

33.8

PGY2

PGY3

PGY4

Final Results Self-perceived competency baseline to follow-up

(t = -2.558; p =.034)*Baseline percentage mean = 48.4Follow-up percentage mean = 68.4Baseline modes = 1 – 2Follow-up modes = 3 – 4Four items: mode increased from 1: Does Not Know How

to 4 Knows How, Has Done, and Can Do at Any Time: ○ Item 11 “Identify the difference between descriptive and analytic

research study designs”○ Item 21 “Identify the measures of central tendency”○ Item 27 “Identify what information a confidence interval (CI) says

about a statistical result”○ Item 37 “Determine when a research study should be closed

through the local Institutional Review Board (IRB)”.

Final Results Self-perceived competency baseline to follow-up

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

0

1

2

3

4

Pre-Assessment

Post-Assessment

Final Results Level of knowledge baseline to follow-up

(t = -3.210; p=.012)* Baseline mean = 52.67 Follow-up mean = 65.78 Knowledge level scores increased from 12.5% to 94.44% per

resident

1 2 3 4 5 6 7 8 90

10

20

30

40

50

60

70

80

90

100

Pre-Knowledge Score

Post-Knowledge Score

Final ResultsLevel of knowledge baseline to follow-up by domains

Pre-Assessment

Means

Post-Assessmen

tMeans

Difference in Means

t = p =

Overall raw score 52.67 65.78 -13.11 -3.210 .012*

Terminology and concepts 54.17 63.89 -9.722 -1.360 .211

Research question and hypothesis

46.28 66.60 -20.38 -3.051 .016*

Study design and sampling 52.78 68.06 -15.28 -1.846 .102

Variables and scales of measurement

48.61 72.22 -23.61 -3.688 .006*

Statistics 43.21 50.61 -7.40 -1.260 .243

Ethics, privacy, and responsible study conduct

61.12 64.82 -3.69 -.449 .666

Basic regulatory requirements 62.22 80.00 -17.78 -2.530 .035*

Final ResultsSelf-perceived competency and level of knowledge

PreKnow PreComp PostKnow PostComp0

10

20

30

40

50

60

70

80

90

100

8.37% Difference

3.88% Difference

Final ResultsSub-problem 1. Differences in change in level of knowledge based on

prior research education (F = 13.44; p = .008)*

PreKnow Mean PostKnow Mean0

10

20

30

40

50

60

70

80

90

100

44.67

70.00

56.57

63.67

No Prior Research Education

Prior Research Education

No prior education:56.71% improvement

Prior education: 12.55% improvement

Final ResultsSub-problem 4. Differences in change in level of knowledge based on

gender (F=15.028; p=.006)*

Male:24.08% improvement

Female: 19.60% improvement

(no statistical difference)PreKnow Mean PostKnow Mean0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

54.57

67.71

49.33

59.00 Male

Female

Attendance Per resident

Range = 0 sessions attended to 8 attended (100%)

Mean total attendance = 66.7%

PGY2

PGY3

PGY4

0 10 20 30 40 50 60 70 80 90

53.2

58.3

87.5

Mean Attendance

(x2 (df = 2) = 10.33; p = .005)*

Attendance Per session

Range = 30% in attendance to 90% in attendanceMean = 60% in attendance

1 2 3 4 5 6 7 80.00

20.00

40.00

60.00

80.00

100.00

% of Neurology Residents in Attendance

Session 6. Ethics, Privacy, and Responsible Study Conduct:Lowest attendance and lowest mean percentage change from baseline

Statistically significant

improvement from baseline

Implications and Sustainability

Implications Level of knowledge and self-perceived competency BOTH

significantly increased regardless of inconsistent attendance Any exposure is better than no exposure Results suggest:

○ Residents still consider research careersPromote and prepare

○ Most residency programs serve a small numberIndividual improvements are important; educate and study

population anyway Sustainability:

Develop an environment fostering research Identify key players for dedication to continued education Departmental investment (time, space, personnel) to continue

education throughout residency

Limitations

Small sample size (n=10), although census sample

Conflicting responsibilities interrupt attendance

Recommendations for Future Research Potential benefits to providing a

curriculum longer than 8 weeks Collaborative projects among residency

training programs to determine effects of education in a larger sample

Consideration of “leveling” research courses and topics based on PGY level

Identified a great need for more intensive statistical education

Thank You

Any questions?