Research Integrity, The Importance of Data Acquisition and Management Ralph H. Hruban, M.D. December 12, 2012

Research Integrity, The Importance of Data

Acquisition and Management

Ralph H. Hruban, M.D.

December 12, 2012

Conflict of Interest

• I receive royalty payments from Myriad Genetics for the PalB2 invention

Aristotle “We become just by performing just actions, temperate by performing temperate actions, brave by performing brave actions”

Nicomachean Ethics

http://www.gap-system.org/~history/PictDisplay/Aristotle.htm

Henry L. Mencken“Science, at

bottom, is really anti-intellectual. It always distrusts pure reason, and

demands the production of

objective fact.”http://www.toptenz.net

“The [pirate] code is more what you’d call ‘guidelines’

than actual rules”

Screenrant.com

Barbossa, Pirates of the Caribbean

The PHS regulation (42 C.F.R. 93) defines research misconduct as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results.

(a) Fabrication is making up data or results and recording or reporting them.(b) Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.(c) Plagiarism is the appropriation of another person's ideas, processes, results, or words without giving appropriate credit.(d) Research misconduct does not include honest error or differences of opinion.

PHS 42 C.F.R. 93

Research Misconduct1. How common is research

misconduct?

2. Misconduct harms patients, the institution and the investigator

3. What leads to misconduct?

4. Examples of research misconduct

5. How can you prevent research misconduct in your own lab?

2010 survey of 40,000 high-school students• 59% admitted to cheating on a test the previous year

• One in three admitted they had plagiarized using the Internet

• One in four admitted they lied on the survey itself!

New York Times, Sept. 26, 2012Josephson Institute of Ethics

http://www.psychologytoday.com/blog

Fraud in High School

Lake WobegonFurthermore, although…• 3/5th admitted to having

cheated in the last year• 4/5th said that their own ethics

are above average

Josephson Institute of Ethics http://www.psychologytoday.com/blog

Harvard College• 279 students took a

take-home final exam• Similarities in test

papers suggest that nearly half broke the rules against plagiarism and working together

• Some of the accused students said their behavior was innocent, or fell into gray areas http://www.thecrimson.com/article/

2012/8/30/academic-dishonesty-ad-board/

• Fraud at a prominent NY high school

• “Writing on your hand, that’s kiddie stuff…the way we do it is to take a picture…then the whole class has it”

New York Times, Sept. 26, 2012http://www.prlog.org

Technological Advances

At the Faculty Level

• Meta-analysis of surveys of scientific misconduct

• 2% of scientists admitted to have fabricated, falsified or modified data or results at least once

Fanelli PLoS One 2009; 4:e5738

Fanelli PLoS One 2009; 4:e5738

Many More Were Aware of Misconduct by Others than Admitted to it Themselves!

QRP=questionable research practices

Fang et al, PNAS, 2012

Fraud is Underestimated• Retraction read: “results were derived from

experiments that were found to have flaws in methodological execution and data analysis.”

• In fact, the Harvard University report to the ORI on this study stated that “many instances of data fabrication and falsification were found.”

• “Incomplete, uninformative or misleading retraction announcements have led to a previous underestimation of the role of fraud in the ongoing retraction epidemic”

Fang et al, PNAS, 2012

Web Siteshttp://retractionwatch.wordpress.com

and http://abnormalscienceblog.wordpress.com/

Retractionwatch.wordpress.com/

Fraud is More Common Than You Think, and

Most People Don’t Acknowledge it

Data Integrity1. How common is data fraud?

2. Fraud harms patients, the institution and the investigator


4. Examples of inappropriately published data

5. How can you prevent fraud in your own lab?

Fraud Harms Patients• Analyzed 180 retracted articles that involved human

subjects or “freshly derived human material,” along with 851 published studies citing that research

• The retracted papers were cited over 5,000 times• According to Steen, 6,573 patients received treatment in

studies eventually retracted because of fraud. One study alone, published in 2001, included 2,161 women being treated for postpartum bleeding

• The downstream studies included more than 400,000 subjects, with 70,501 receiving treatment

R. Grant Steen, Journal of Medical Ethics, 2011

Fraud Harms the Institution and the Investigator

“But the research at Duke turned out to be wrong. Its gene-based tests proved worthless, and the research behind them was discredited. Ms. Jacobs died a few months after treatment, and her husband and other patients’ relatives have retained lawyers.”

Gina Kolata, on Anil Potti, New York times, July 7, 2011

Anil Potti, MD

Potti ScandalThe defendants named in the suits include:• Duke University • Duke University Health System, Inc. • Private Diagnostics Clinic PLLC • Joseph Nevins, PhD • Anil Potti, MD • Michael Cuff, MD • Sally Kornbluth, MD • John M. Harrelson, MD • Cancer Diagnostics, Inc.

Deception at Duke Scott Pelley reports on a

Duke University oncologist whose supervisor says he manipulated the data in his

study of a breakthrough cancer therapy

http://www.cbsnews.com/8301-18560_162-57376073/deception-at-duke

http://www.cbsnews.com/8301-18560_162-57376073/deception-at-duke/?tag=contentMain;cbsCarousel

http://en.wikipedia.org/wiki/File:New60minutes.jpg

Fraud Causes Harm• Research misconduct can have a devastating impact on a lab • If an individual involved in NIH funded research is found to have

committed research misconduct, the administrative actions PHS/HHS may take against them include, but are not limited to: – Debarment from eligibility to receive Federal funds for grants

and contracts, – Prohibition from service on PHS advisory committees, peer

review committees, or as consultants, – Imposition of supervision on the respondent by the institution, – Submission of a correction of published articles by the

respondent, and – Submission of a retraction of published articles by the

respondent.

Office of Research Integrity

Fraud Causes Harm• In addition, NIH may take further administrative

action, including:– Suspension or termination of an award, – Recovery of funds, and – The institution (university) may impose

additional penalties:– Loss of employment – Reassignment of personnel

Office Research Integrity

Research Misconduct Harms

Patients, the Investigator and

the Institution

Research Misconduct

Destroys Careers






What Leads to Misconduct?

There is a pattern:• Ambition• Poor mentorship• Failure to take responsibility for

one’s actions• A firm belief one knows the

answerhttp://www.thenation.com/article/165313/disgrace-marc-hauser?page=full#

What Leads to Misconduct?1. Ambition

A bright and ambitious young person working in an elite institution in a rapidly moving and highly competitive branch of modern biology or medicine, where results have important theoretical, clinical or financial implications.

(Sounds a lot like Hopkins)http://www.thenation.com/article/165313/disgrace-marc-hauser?page=full#

• The University of Connecticut said Wednesday that it was returning two new grants to Dr. Das, worth a total of $890,000, to the federal government

• 19 papers retracted as of November 30, 2012

• Dr. Das’ published research articles were found to contain 145 instances of fabrication and falsification of data.

Dipak K. Das

New York Times, Jan. 11, 2012Retraction Watch

Ambition• “Right from the start, there is

a huge emphasis on his fame and power, his multiple degrees, how many times his papers have been quoted and how wonderful he is. There is not a hint of humility or any acceptance that mistakes might have been made”

• YouTube: http://www.youtube.com/watch?v=3zNJNB5qn54


http://www.youtube.com/watch?v=3zNJNB5qn54



What Leads to Misconduct?2. Poor Mentorship

He has been mentored and supported by a senior and respected establishment figure who is often the co-author of many of his papers but may have not been closely involved in the research

http://www.thenation.com/article/165313/disgrace-marc-hauser?page=full#

Poor Mentorship

• ‘I didn’t actually do it – the students and fellows did’

Dipak Das New York Times, Jan. 11, 2012Retraction Watch

What Leads to Misconduct?3. Failure to Take Responsibility

http://academy.justjobs.com/stop-whining-take-ownership/

Failure to Take Responsibility

• Dr. Das concedes that Western Blot images were altered, but only at the demand of journal editors


Hyung-In Moon• South Korean plant compound

researcher faked email addresses so he could review his own studies

• The names of real people were provided (so if an editor Googled them, they would appear to exist) but he created email accounts for them to which he or associates had access

http://retractionwatch.wordpress.com/

Hyung-In Moon• The review comments

submitted by these “reviewers” were almost always favorable but still provided suggestions for how the paper could be improved

• “Of course, authors will ask for their friends, but Editors are supposed to check they are not from the same institution or coauthors on previous papers.”

http://retractionwatch.wordpress.com/

• Scientists admit more frequently to have ‘‘modified’’ or ‘‘altered’’ research to ‘‘improve the outcome’’ than they do to having reported results they ‘‘knew to be untrue’’

• In other words, many did not think that the data they ‘‘improved’’ were falsified

Scientific Ethics 1(1): 53-58, 2006Fanelli, 2009, PLoS

What Leads to Misconduct? 4. A firm belief that one knows the answer

Knows the Answer• ‘Even if the

Western blots were fabricated, this doesn’t change the fact that resveratrol protects the heart’ New York Times, Jan. 11, 2012

Retraction Watch

I think what happened is that you are betting on football, and what’s after football is basketball, and then the

NCAA tournament. The next thing that follows is betting on baseball…

I wish I could take it all back.

Pete Rose

What ever initiates misconduct, it is a slippery slope!

Once Started it is Hard to Stop

• “We’re up to 31: retractions keep coming for Diederik Stapel”

• Yoshitaka Fujii holds the record with 172 http://retractionwatch.wordpress.com/

Diederik Stapel






Examples• Fabricated data• Falsified data• Selective reporting of data• Image manipulation

Example 1:Trial of 3 Drugs- Actual Results

Series10

2

4

6

8

10

12

Drug 1Drug 2Drug 3

Trial of 3 Drugs- Results Reported

Time 1 Time 2 Time 3 Time 40

1

2

3

4

5

6

Drug 1Drug 2Drug 3

Data Fabrication

Fabrication is making up data or results and recording or

reporting them

Jon Sudbø- Fabrication• Medical researcher at

the Radium Hospital, Oslo, Norway

• 2005 article in the Lancet suggested that Ibuprofen reduces oral cancer in smokers

The Lancet, 366 (9494): 1359–1366;http://www.vg.no/nyheter/innenriks/artikkel.php

Jon Sudbø- Fabrication• Suspicion aroused because the data were

supposedly from a cancer patient database which had not yet opened

• Of the 908 subjects in the Lancet study 250 had the same date of birth

• Sudbø later acknowledged that he used fictional data in at least two more papers, published in the New England Journal of Medicine and Journal of Clinical Oncology

http://en.m.wikipedia.org/wiki/Jon_Sudb%C3%B8#cite_note-2


Series10

2

4

6

8

10

12

Drug 1Drug 2Drug 3



1

2

3

4

5

6

Drug 1Drug 2Drug 3

But you were just a co-author

International Committee of Medical Journal Editors

• An “author” is generally considered to be someone who has made substantive intellectual contributions to a published study…. An author must take responsibility for at least one component of the work, should be able to identify who is responsible for each other component, and should ideally be confident in their co-authors’ ability and integrity.

• When a large, multicenter group has conducted the work, the group should identify the individuals who accept direct responsibility for the manuscript

http://www.icmje.org/ethical_1author.html



0.5

1

1.5

2

2.5

3

3.5

Drug 1Drug 2Drug 3



1

2

3

4

5

6

Drug 1Drug 2Drug 3

Data Falsification

Falsification is manipulating research materials, equipment, or processes, or

changing or omitting data or results such that the research is not accurately represented in the research record

Example 4:Trial of 3 Drugs-Actual ResultsResults Statistically Significant


1

2

3

4

5

6

Drug 1Drug 2Drug 3

Trial of 3 Drugs-Reported Results(Results still Significant)


1

2

3

4

5

6

7

Drug 1Drug 2Drug 3

It is Still Data Falsification


changing or omitting data or results such that the research is not accurately represented in the research record

(No qualifier here that falsification is ok so long as the results were originally

statistically significant!!!!)

Example 5:Trial of 3 Drugs-Actual Results:Results Statistically Significant


1

2

3

4

5

6

Drug 1Drug 2Drug 3

The PI Tells the Post-Doc

“These two data points seems off. I would expect there to be a greater difference”

Trial of 3 Drugs-Reported Results(Results still Significant)


1

2

3

4

5

6

7

Drug 1Drug 2Drug 3

It is found out later that the Post-Doc

Changed the data in question

Does the P.I. have any responsibility for

what happened?

What Environment Did the PI Create?

We All Have a Responsibility to Maintain Integrity in Our Labs and to Create an Environment Where Research Integrity is Emphasized Above all Else

Dipak K. Das

NY Times, January 11, 2012, retractionwatch.wordpress.com,http://www.youtube.com/watch?v=3zNJNB5qn54

• The University of Connecticut report alleges Dr. Das “defunded” the work of a student in his lab because she did not produce results that he wanted

• “Never terminated that student…he simply removed her from his budget”


Example 6:5-Month Trial of 3 Drugs-Actual

Results of a 5-Month Design

Time 1 Time 2 Time 3 Time 4 Time 50

1

2

3

4

5

6

Drug 1Drug 2Drug 3

Trial of 3 Drugs-Results Reported


1

2

3

4

5

6

Drug 1Drug 2Drug 3

Could be Falsification


changing or omitting data or results such that the research is not accurately

represented in the research record

OK Only if Clearly Documented in the

Paper“…not accurately represented in

the research record”

Example 7: Results (4 Day Expt., but Technician

ran the Experiment too long)


1

2

3

4

5

6

Drug 1Drug 2Drug 3

Trial of 3 Drugs-Results Reported


1

2

3

4

5

6

Drug 1Drug 2Drug 3

Probably OK if the study design was

shorter, but it ought to get you

thinking!

The most exciting phrase to hear in science, the one

that heralds new discoveries, is not

‘Eureka,’ but ‘That’s funny…’

Isaac Asimov

Example 8:Trial of 3 Drugs- Results First Run


0.5

1

1.5

2

2.5

3

3.5

Drug 1Drug 2Drug 3

Results Second Run


1

2

3

4

5

6

Drug 1Drug 2Drug 3

Results Third Run


1

2

3

4

5

6

Drug 1Drug 2Drug 3

Reported (The Third Run)


1

2

3

4

5

6

Drug 1Drug 2Drug 3

Likely Falsification


changing or omitting data or results such that the research is not accurately


If there were genuine reasons the first two runs didn’t work you ought to document why, fix them

and repeat the study a 4th time!

“…not accurately represented in the research record”

Example 9: PowerPoint Presentation Within Hopkins

Falsified data are presented at a meeting within the

Hopkins community. The data are not published. Is this research misconduct?

Yes, it is research misconduct even if

the data are not published

42.CFR §93.103

• Research results do not need to be published before they fall within the definition of research misconduct.

• The research record is defined as “the record of data or results that embody the facts resulting from scientific inquiry, including but not limited to, research proposals, laboratory records, both physical and electronic, progress reports, abstracts, theses, oral presentations, internal reports, and journal articles...”

Image Manipulation

Falsification is manipulating research materials, equipment, or

processes, or changing or omitting data or results such that

the research is not accurately represented in the research

record

http://www.labtimes.org/

M. Rossner and K. Yamada, JCB, 2004

It‘s so easy to add and subtract with Photoshop

Can Be Detected


“Your X-ray showed a broken rib, but we fixed it in Photoshop”

http://www.glasbergen.com

Science 2005Woo-Suk Hwang

http://news.naver.com/main/read.nhn

Time, Dec. 15, 2005

Woo-Suk Hwang

Reusing Images- Potti

Augustine et al., 2009, ClinCan Res, 15:502-10, Fig 4A.Temozolomide, NCI-60.

Hsu et al., 2007, J ClinOncol, 25:4350-7, Fig 1A.Cisplatin, Gyorffy cell lines.

http://videolectures.net/keith_baggerly/


Altering ImagesIf you misrepresent your data, you are deceiving your colleagues, who expect and assume basic scientific honesty—that is, that each image you present is an accurate representation of what you actually observed. In addition, an image usually carries information beyond the specific point being made.

Altering Images


For every adjustment that you make to a digital image, it is important to ask yourself, “Is the image that results from this adjustment still an accurate representation of the original data?” If the answer to this question is “no,” your actions may be construed as misconduct.

Image Manipulation

Falsification is manipulating research materials, equipment, or processes, or changing or omitting

data or results such that the research is not accurately







http://randysright.files.wordpress.com

Prevention!!• Establish a culture of honesty

above all in your lab• Inform and educate• Screen- Periodically ask to see

lab books• Detect problems by working

closely with primary data

Establish a Culture of Honesty

• “We need this difference to be significant or I won’t get my grant”

• “These data points don’t fit the results I expected”

These small things can add up and can quickly become the norm

Establish a Culture of Honesty

vs.

From day one; “All that matters to me is that the results you present are

100% honest”

Inform and Educate

• Dedicate some journal clubs or lab group meetings to educating those under you on the importance of academic integrity

• Encourage members of your lab to attend lectures such as this one!

Even so we have to screen for problems!

Picture of a PowerPoint presentation

We are not going to detect fraud if we only look at PowerPoint presentations

of finished results

We need to carefully review and question primary data

Henry L. Mencken

“Conscience is the inner voice that

warns us somebody may be looking”

If it is Too Good to be True…

• Blind the samples and ask the person to rerun the experiment

• Have someone else in the lab rerun the experiment

http://econsultancy.com/

Tools for detecting misconduct

• Anti-plagiarism software (eTBLAST, CrossCheck, Turnitin)

• Screening images (PhotoShop)- Pioneered by J Cell Biology. See M. Rossner and K. Yamada, JCB 2004; 166:11-15- found 1% unacceptable manipulation

• Data Review (digit preference)

Liz Wager, Council of Scientific Editors

Strong and repeated messages about what is allowed and what is

prohibited

Conclusions

What Happened to Dr. Das?

• October 2012 issue of the Journal of Cellular and Molecular Medicine contained two more retractions (#18 and 19)

• And…a new paper from Dr. Das that cites three of his now-retracted papers!

Retractionwatch.wordpress.com

http://retractionwatch.files.wordpress.com/2012/11/j-cell-mol-med.gif

ConclusionsPreventing research misconduct will

save careers from ruin

One of those careers could be yours

Everyone has a responsibility to promote a culture in which research misconduct does not occur

Harold C. Sox, Annals of Internal Medicine

If you are the first or last author on a paper

You are responsible:1. For making sure all of the other authors have

read and approved the manuscript2. For everything in the manuscript- make sure

the images included have not been manipulated, that the text isn’t copied from somewhere else, that the data weren’t manipulated, that you have appropriate IRB protocols, and that the protocols were followed

We need to be aware that at Johns Hopkins

people may feel enormous pressures

Take Home Message #1

Science should be our

“touchstone”

Take Home Message #2

• Chris Iacobuzio-Donahue, M.D.• Scott Kern, M.D.• Anirban Maitra, M.B.B.S.

• Sheila Garrity, J.D., M.P.H., M.B.A.– Moderator

Panel Discussion

Documents

Research Integrity, The Importance of Data Acquisition and Management Ralph H. Hruban, M.D. December 12, 2012