Subtitle

…from a Misconduct- to an Integrity-centered Universe

Nicholas H. Steneck, PhD

Promoting Research Integrity Historical Background & Current Trends

Workshop on Best Practices for Ensuring Scientific Integrity & Preventing Misconduct

22-23 February, 2007

Mita Conference Hall, Tokyo, Japan

2

Historical Perspective

Research misconduct is not new Galileo fabricated & falsified data Piltdown hox and other scandals Some things do not change over time

Major changes between 1600 and 2000

http://ww

w.talkorigins.org/faqs/piltdow

n/pskull.gif

Science ca. 1600•Privately supported

•Limited public benefit

Science ca. 1600•Privately supported

•Limited public benefit

Science today•Publicly supported

•Essential to technological growth & public welfare

Science today•Publicly supported

•Essential to technological growth & public welfare

By 1980, when first cases of misconduct emerge, science (research) is a public activity

Over time, scientists (researchers) became public servants!

http://en.wikipedia.org/w

iki/Image:G

alileo.arp.300pix.jpg

3

Financial support 1930s, under 1% GDP 1950 ca. 1% GDP Currently 2-3%

http://ww

w.nsf.gov/statistics/databrf/sdb99357.pdf

http://ww

w.nsf.gov/statistics/databrf/sdb99357.pdf

1 of 100 public dollars in US spent on health research

Research is a public activity

Major growth since 1950

4

Some doubts along with the growth

Science linked with unpopular events and problems: Cold War links science with “military-industrial complex

Clock, Bulletin of the Atomic Scientist (1947) War in Vietnam (1960s) Environmental impact of nuclear power

Concerns about human & animal experimentation US, Tuskegee experiments Declaration of Helsinki (1964)

Global energy crisis (1970s) New worldviews compete with scientific worldview

By 1980 (first major US misconduct cases) public is taking a closer look at the way research is conducted

http://ww

w.thebulletin.org/m

inutes-to-midnight/

5

Response to research misconduct, 1980 ff.

Events have been driven by major cases & media Story breaks in the news Local institution responds Pressure for an official/government response

Official response Gather information ~ committees, hearings, reports Try to resolve the immediate problem ~ the major cases Develop policies and procedures to avoid similar problems in the

future

Timing US begins in early 1980s, policy development still in process Europe, early 1990s, policy development in process Asia, late 1990s, policy development in process

6

The misconduct-centered universe

First priority ~ major cases Define misconduct Assign authority Develop procedures for investigation

Definitions focus on deliberate reporting false data & information Careful to not confuse with scientific disagreement Not the same as waste and sloppiness

Policies protect researchers from improper charges Working assumption: pursuing individual cases of misconduct

is the best way to protect the integrity of publicly supported research

7

Misconduct-based universe rested on 5 assumptions

Serious misconduct in research is rare Self-regulation keeps improper behavior in check Research misconduct is difficult to detect Research misconduct cannot be prevented Apart from misconduct, standards for integrity in research

are high

Assumptions were based on common perceptions, not empirical evidence

All five can be questioned!

8

Research misconduct is rare?

Martinson, Nature (June 2005) Goal: factors that influence research behavior Method:

Developed peer-based list of major offenses Survey to 6,000+ researchers (3,000+ response) Major question: “have you done … in last three years?”

Results Major offenses, ca. 0.3% Questionable Research Practices (QRP) ca. 5-15% or higher

9

Data from other recent studies JM Ranstam, Control Clin Trials (2000)

Survey, 442 biostatisticians, 37% response 51% knew about fraud in medical research

26% involved FF 31% directly involved in projects with misconduct

Estimates of rate, .69% –> .80% (.25% standard)

Geggie, J Med Ethics (2001) Survey, 305 new medical consultants, 64% response

55.7% observed misconduct (FF lower) 5.7% committed misconduct in the past 18% would commit in future 17% had research ethics training

10

Studies continued Gardner, Contemporary Clinical Trials (2005)

Authors pharmaceutical clinical trials (64% response) 1% reported target article misrepresented the research 5% reported fabrication in a study they had participated in over the

last 10 years 17% knew personally of fabrication in a study over the last 10 years

Rossner, Journal of Cell Biology 11 in 1,100 papers had serious improper digital image manipulation

11

Realistic estimates: Rough approximation:

Evidence ~ 1/1,000+ Assume ~ 1/10,000

Cases predicted US ~ 1,500 EU ~ 1,000 Japan ~ 600 Other OECD ~ 400

Cases seen US ~ 20/year EU ~ 10/year

Lesson #1 ~ policy makers have always under-estimated the amount of misconduct in research

Science &

Engineering Indicators 2006, F

igure 3-36

12

Other assumptions about misconduct? Self-regulation keeps improper behavior in check?

Researchers do not report suspected misconduct (20-40%) Journals often do not report misconduct

Research misconduct is difficult to detect? Hwang could not have completed work in the time reported Sudbø, trial not started, patients did not exists, data repeated

Research misconduct cannot be prevented? Schön’s co-author/mentor did not check experiments or data Pohlmann’s MD co-author did not oversee clinical results

Lesson #2: Policy makers did not understand the strengths & weaknesses of self-regulation in research If they understood, did not honestly and accurately report

13

Integrity in research otherwise high?

Martinson study, self-reported misbehaviors:15.5% Changing the design, methodology or results of a study in response to

pressure from a funding source

12.5% Overlooking others' use of flawed data or questionable interpretation

7.6% Circumventing certain minor aspects of human-subject requirements

6.0% Failing to present data that contradict one’s own previous research

1.7% Unauthorized use of confidential information

1.4% Using another’s ideas without obtaining permission or giving due credit

1.4% Relationships with students, research subjects or clients that may be interpreted as questionable

0.3% Not properly disclosing involvement in firms whose products are based on one‘s own research

0.3% Ignoring major aspects of human-subject requirements

0.3% Falsifying or ‘cooking’ research data

14

Al-Marsouki, Contemp Clin Trials 26(2005)

Practices felt likely to occur and adversely impact research83% Over-interpretation of “significant" findings in small trials80% Selective reporting based on p-values76% Selective reporting of outcomes in the abstract 75% Subgroup analyses done without interaction tests68% Negative or detrimental studies not published68% Putting undue stress on results from subgroup analysis64% Inappropriate subgroup analyses64% Selective reporting of (i) subgroups (ii) outcomes (iii) time points60% Selective reporting of positive results/omission of adverse events data60% Failure to report results or long delay in reporting59% Post-hoc analysis not admitted56% Giving incomplete information about analyses with non significant results 54% Analysis conducted by the sponsor of the trial

15

How do researchers behave?

QRP ~ 10% <–> 50%

High standards ??

Misconduct ~ 0.1% <–> 1%

Res

earc

h be

havi

or

Lesson #3. Significant gap between ideal (high standards) and actual standards for integrity in research

16

How should researchers behave?

Integrity-centered

Misconduct-centered

17

Slow change to an integrity-centered universe

More emphasis on prevention and improving integrity First major change ~ improve education/training US, efforts to require training/education

1989 Institute of Medicine Report called for training 1990/92, National Institutes of Health required for trainees 1997, National Science Foundation required for trainees 2000, Public Health Service proposed general requirement

Strong objections raised by research community Requirement has been suspended

Global initiatives Finland, national requirement for graduate students Elsewhere, growing number of courses and resources

18

Training covers more than misconduct Areas developed over time:

1994 Training Grant Requirement Conflict of Interest Responsible authorship Policies for handling misconduct Data management Human & animal subjects

2000 HHS RCR requirement 9 areas

Other areas could be added: Grant and financial management Workplace rules Laboratory safety & special rules (biological materials…)

Nine Core Areas1. Data acquisition,

management, sharing, and ownership

2. Mentor/trainee responsibilities

3. Publication practices and responsible authorship

4. Peer review

5. Collaborative science

6. Human subjects

7. Research involving animals

8. Research misconduct

9. Conflict of interest and commitment

Nine Core Areas1. Data acquisition,

management, sharing, and ownership

2. Mentor/trainee responsibilities

3. Publication practices and responsible authorship

4. Peer review

5. Collaborative science

6. Human subjects

7. Research involving animals

8. Research misconduct

9. Conflict of interest and commitment

19

Current status of “RCR” training

Abundant resources for teaching

Textbooks

Web pages

Train-the-trainer programs

No standards for content or approach

Coverage depends on the instructor

Training not integrated with research

Minimal testing or follow-up

Little assessment of effectiveness

Major challenge:

Research is global; research teams and laboratories are international

RCR training is local, inconsistent, and for the most part inadequate

20

Second change ~ Better understanding of behavior Misconduct-based universe ~ cause of misconduct?

Too few cases/too much variation to draw “scientific” conclusions Suggested some areas for further attention:

Quality of mentoring, supervision, peer review

Integrity-centered universe ~ many issues to study Social processes

Authorship - who is listed and why? Peer review - weaknesses and how to correct? Data -how do researchers collect and record data?

Institutional role and influences Policies - how institutions develop and promote policies Good management - how administrators and committees work Conflict of interest - how institutions manage their own conflicts

Much better prepared to implement effective policies

21

Four crucial challenges

“Misconduct” policies can have many objectives:A. Establish procedures for responding to misconduct in research

B. Detect and eliminate/correct fraudulent information

C. Protect public from consequences of flawed research findings

D. Maintain/restore public confidence in research

US Policy (OSTP, 2000) has three general objectives Protect “reliability of the research record” Maintain public “confidence in the research record” Achieve policy unity

Reason for objectives Clear understand of goals to be achieved Standard for measuring success and improving policy

Establish clear objectives

22

Fair, effective procedures for responding

A. Establishing national policies is a starting point Comprehensive definitions that cover all serious misbehavior Assure reporting and accountability Protect informants and insulate from bias and conflicts of interest

Global harmonization and communication are next step Research is no longer local or national Laws of nature and scientific methods are not local Standards for reporting, investigating, and judging misconduct should

be global Some accommodation for differences in law and government as long as

good research practices are not compromised

The globalization of research requires the globalization of policies and best research practices

23

Take steps to make policies more effective Detect and eliminate/correct fraudulent information??

Investigating 1 of every 100 cases has little impact on the reliability of the research record

Some FFP has is trivial Current policies have minimum impact on the research record

Steps that would eliminate/correct current system Improved education, emphasizing professional responsibility Clearer rules for data management, mentoring, and peer review Random audits of publications and supporting data Institutional climate surveys to assess reliability of self-regulation Extend responsibility for reporting to journals

Investigating major cases is essential but has little impact on the overall reliability of the research record

24

Take a serious look at QRPs Viewed from a public perspective, questionable practices are

more significant than research misconduct Occur more frequently ~ 10x or more Have serious impacts

Poor literature reviews have led to harm of research subjects Bias reporting/duplicate publication impact health-care decisions Ineffective public decisions due to improper statistical analysis

How would you explain the following to the public? 25% of researchers reported recording results in loose-leave notebooks 40% of abstracts misrepresent findings reported in the article Funding makes researcher 3-5 times more likely to report results

favorable to the funding source

Protecting the integrity of the research record requires more than simply responding to cases of misconduct

25