Developing a Validated Responsible Conduct of Research

Test Bank

Holly Phernetton, MPH, MSJames DuBois, PhD, DSc

Saint Louis University

Planned Collaboration with CITI

With ORI funding, DuBois, Mumford, Phernetton, Antes have developed a RCR test bank: >125 multiple-choice items aimed at assessing

basic knowledge of RCR 50 “pick 2” ethical decision-making items that

test ability to choose optimal responses to ethical problems described in vignettes

Planned Collaboration with CITI

All items will be available for CITI to use at no cost

This presentation describes how we have already validated the test items and how we propose to work with CITI for final construct validation using classical test theory

Background

Office of Research Integrity (ORI) identified 9 core areas that RCR courses should address (2000) However, there was no guidance or consensus

on what content should be taught within these core areas, nor what the overarching objectives of RCR training should be assessed

Further, there is no standard validated test that measures RCR education objectives

Process for Test Bank Development

3 Stages of Test Development1. Delphi Survey to validate objectives and

content areas2. Development of the Items

Multiple choice items Pick-2 items

3. Pilot-testing of items and calculation of psychometric properties

Validation Process

Content Validity Delphi survey: identify areas to assess Item development: evidence-based answers Expert review and revision

Construct Validity Pilot-test items Classical test theory

Content Validity

Level 1: Delphi survey Define objectives and determine important RCR

content to be taught and knowledge assessed by consensus panel

Level 2: Item development According to assessment recommendations from

Delphi panel Evidence-based answers from well-recognized RCR

curricula and text books Level 3: Expert review

Delphi Survey

Funded by ORI 2006-2007: four panels of experts convened

to develop a consensus: identify RCR overarching goals for instruction and assessment

Delphi process: anonymous process Minimizes groupthink and domineering group

members Maximizes benefits of group decision-making

Delphi Survey Formed 4 separate panels: each worked independently

and simultaneously Survey conducted online in 3 rounds Round 1: Open-ended items asking what should be the

objectives and content of RCR training Round 2: Presented lists of edited responses from

round 1—participants asked to rate the importance of teaching each topic

Round 3: Presented lists of edited responses from round 2—participants asked to rate importance of teaching and assessing each area

Delphi Survey

4 Panels 1. Objectives (n=18)

Overarching objectives and assessed objectives 2. Scientific Data (n=13)

Data management and research misconduct sections 3. Scientific Relationships (n=14)

Mentor-trainee responsibilities, collaborative science, conflicts of interest and commitment

4. Publication and Peer Review (n=13) Publication practices/responsible authorship and peer review

Delphi Survey Results

7 Core Areas of RCR instruction examined Panel Consensus: 2/3 of panelists supporting

a rating of “important” or “very important” Consensus on teaching 43 main topics Consensus on assessing learning in 21 of the 43

main topics

Content Validity Level 1: Delphi survey

Define objectives and determine important RCR content to be taught and knowledge assessed by consensus panel

Level 2: Item development According to assessment recommendations from Delphi panel Evidence-based answers from well-recognized RCR curricula and

text books Level 3: Expert review

Item Development Multiple Choice Questions (MCQ)

Developed by Saint Louis University(Dr. James DuBois, Holly Phernetton)

Stem and lead-in question: 1 correct answer and 3 distractors

Pick 2 Items (P2) Developed by University of Oklahoma

(Dr. Michael Mumford, Allison Antes) Stem: vignette presenting ethical problem Answer: pick 2 “optimal responses” from list of 8

options

MCQ Item Development

Follow National Board of Medical Examiner Guidelines

Rationale for NBME guidelines. Good test items: Discriminate between those who know and

those who do not know material Are constructed to reduce correct guessing by

“test savvy” participants Avoid being “tricky”

Rationale Behind the NBME Guidelines

Items contain only critically important content Must have 1 clearly best answer No negative questions such as

“each of the following is correct except” “which of the following is NOT correct”

No absolute terms such as “never” or “always” Avoid “all of the above” and “none of the

above answers” All responses are of roughly equal length and

form

Item Example: MCQA researcher commits research misconduct at a large

university which receives federal funding for research from the Public Health Services. Which of the following is correct regarding institutional obligations in such cases?

A. Academic institutions are legally obligated to report all allegations of misconduct to the Office of Research Integrity prior to investigation.

B. Institutions must have policies in place for the inquiry, investigation, and adjudication of allegations of misconduct.

C. The federal government has to be immediately contacted in every case of alleged misconduct.

D. Researchers found guilty of misconduct can never receive PHS funding again.

Item Example: P2Vignette:Dr. Davis has been culturing a strain of Escherichia coli in an

oxygenated broth. For 12 years he has been gradually reducing glucose concentration in the medium. As a mid-career evolutionary biologist, he is interested in the possible evolution of the strain over thousands of generations. He is targeting an extremely unlikely adaptation-the ability to take up and metabolize citrate, something that E. coli strains are notably unable to do. After about 30,000 generations, the adaptation appears. Indeed, the bacteria, which also shows some morphological changes, can thrive in a totally glucose-free medium, and phenotypes with the trait rapidly dominate the population.

Item Example: P2I

Vignette, cont’dAn investigation revealed a new set of genes in Davis’ E. coli

culture consistent with contamination by the common soil bacterium, Serratia marcascens. Yet, analysis could not identify a single living S. marcescens in the culture. S. marcescens readily metabolizes citrate, and the almost certain conclusion is that his E. coli acquired the adaptive trait through lateral gene transfer. This, itself, is an exciting finding, and Davis writes up and distributes a report for prepublication review by the research team. Two members object to Davis’ procedure section where he says that the E. coli culture was experimentally inoculated with S. marcescens. How should Davis respond? Choose two from the following:

Item Example: P2I A. Remind them that this is his laboratory and he alone makes the final

decisions about the content of publications from his laboratory. B. Agree that the students have a good and valid point, but that the machinery

of science is intolerant of accidental findings. C. Explain to his students that the intention behind a finding is irrelevant to the

fact being presented. D. Revise the paper’s method section to explain how S. marcescens found its

way into the culture. E. Explain that full disclosure would only taint the reputation of the laboratory

by admitting the breach in original procedure. F. Listen to the objections of his students, accept that he is outvoted, and change

the procedure section of the paper. G. Do not offer any explanation other than a reminder that solid publications are

what make a career and that if they are uncomfortable with this, they should consider leaving the lab.

H. Acknowledge to the students that they are probably correct and use the data as a basis for a follow-up study on lateral gene transfer under more tightly controlled conditions.

Content Validity Level 1: Delphi survey

Define objectives and determine important RCR content to be taught and knowledge assessed by consensus panel

Level 2: Item development According to assessment recommendations from Delphi panel Evidence-based answers from well-recognized RCR curricula and

text books Level 3: Expert review

Expert Review

3 Experts to review RCR test items; provide revision, comments MCQ (all three experts to review) P2 (one expert to review each area)

Biological Sciences Social Science Health Science

Construct Validity

Pilot-test Items CITI Program:

Implement test bank Each item will be completed by a minimum of 200

CITI participants A participant will complete only 20% of total battery of

items: 5 MCQ per 7 core areas, 2-3 P2 vignettes with 3 items each (41-44 items total)

Total of 1,000 CITI participants needed

CITI & Construct Validation

CITI program pilot-testing All items will be pilot-tested on the CITI training

program While participants can re-take test until they pass, only

their initial response will be considered in construct validation

Will use classical test theory to validate items: Will identify top 20% and bottom 20% of test performers Any items missed more frequently by the top 20% than the

bottom 20% will be treated as invalid. Such items will be discarded or revised

CITI Logistics CITI will upload test items using our Excel file Program test items to appear in random subsets After participants have completed all test items,

CITI will send de-identified data for statistical analysis

Test bank team will revise items and send CITI a finalized test bank for use by Jan 2010

CITI—Ethical Issues Participants can re-take the test until passing score

achieved Most items tested are covered by CITI educational

content A few test items will be piloted of which are not

directly covered in the CITI training program These will not be scored, but feedback will be

provided This will ensure fairness and will provide

educational value

CITI—Ethical Issues Will obtain IRB permission CITI will provide participants with

introductory statement and request for permission to use test data

Our approach distinguishes between: Consent to take test (not needed, because testing is

a required component of training and items are already content validated)

Consent to use data (will be requested) in any activities that extend beyond normal QI

Introductory Statement for Participants

“Most of the test items you will see address material directly covered in our training program. However, some items also test your ability to generalize to new ethical situations, and still others test your ethical problem-solving skills. In each case, you will be told the correct answer after you complete an item and will have the chance to re-take the test until you achieve a passing score.”

Upon Completion of the Test

“CITI is committed to quality improvement and sometimes analyzes de-identified test and survey data to identify ways to improve test items and course content. We may also describe our quality improvement efforts in publications or at conferences using analyses of de-identified test data. Do we have your permission to use your de-identified test data in this manner?

Yes No