How Do We Know What We Know?

  • View

  • Download

Embed Size (px)

Text of How Do We Know What We Know?

  1. 1. How Do We KnowWhat We Know?UTPABioethics: Creating and Challenging Knowledge in Health Edinburg, Texas, April 2013Ivan Oransky, MD Executive Editor, Reuters HealthCo-Founder, Retraction Watch@ivanoransky
  2. 2. Retractions on the Rise -The Wall Street Journal
  3. 3. How Often Are Studies Wrong?
  4. 4. Winner Takes All IncentivesScientific American, August 2012
  5. 5. Winner Takes All IncentivesThe winner-take-all aspect of the priority rulehas its drawbacks, however. It can encouragesecrecy, sloppy practices, dishonesty and anexcessive emphasis on surrogate measures ofscientific quality, such as publication in high-impact journals. -- Fang and Casadevall, Scientific American
  6. 6. Anonymous Whistleblowers Step Up
  7. 7. Blogs Get Aggressive
  8. 8. Blogs Get Aggressive
  9. 9. Blogs Get Aggressive
  10. 10. Blogs Get Aggressive
  11. 11. Journals are Listening
  12. 12. Retraction Watch
  13. 13. Post-Publication Peer Review Nature (22 Dec 2011) doi:10.1038/480449a
  14. 14. Post-Publication Peer Review
  15. 15. Post-Publication Peer Review
  16. 16. Post-Publication Peer Review
  17. 17. Post-Publication Peer Review
  18. 18. Alt Metrics
  19. 19. Alt Metrics
  20. 20. How Often Are Medical Studies Wrong? Ioannidis JPA. PLoS Med 2005; 2(8): e124
  21. 21. How Often Are Medical Studies Wrong?
  22. 22. Does The Literature Reflect Reality?
  23. 23. Does The Literature Reflect Reality?Publish a trial that will bring US$100,000of profit or meet the end-of-year budgetby firing an editor. -- Former BMJ editor Richard Smith
  24. 24. Positive Publication Bias
  25. 25. Positive Publication BiasThe overall frequency of positive supportshas grown by over 22% between 1990 and2007, with significant differences betweendisciplines and countries.the strongest increase in positive resultswas observed in disciplineslike ClinicalMedicine, Pharmacology & Toxicology,Molecular Biology Fanelli, Scientometrics 2012.
  26. 26. Publish All Data?
  27. 27. Publish All Data?
  28. 28. FDA: Black-or-White Approval
  29. 29. FDA: Black-or-White Approvalabandon the current black-or-white approvalprocess in favor of an incremental, conditional one.In such a process, drugs could be provisionallyapproved after promising early-stage data, with theFDA retaining the option to revoke that approvallater on, should unexpected data come to light.A conditional approval approach would grantlimited marketing authorization to new drugs aftersuccessful Phase II trials.
  30. 30. Confirmation BiasesFacts do not accumulate on the blank slates ofresearchers minds and data simply do not speak forthemselves. Good science inevitably embodies atension between the empiricism of concrete dataand the rationalism of deeply held convictions.awareness of the systematic errors that can occurin evaluative processes may facilitate the selfregulating forces of science and help producereliable knowledge sooner rather than later. -- Kaptchuk, BMJ, 2003;326:14535
  31. 31. Believers vs. Snails-- Kaptchuk, BMJ, 2003;326:14535
  32. 32. Contact/ @ivanoranskyThanks to Nancy Lapid, Reuters Health