Making sense of prior probabilities in research

Lex (ALB) Rutten1, Robert T. Mathie2, and Raj K. Manchanda3

1 Homeopathic physician and independent researcher, Aard 10, 4813 NN Breda, The Netherlands2 British Homeopathic Association, Hahnemann House, 29 Park Street West, Luton LUI 3BE, UK3 Central Council for Research in Homoeopathy, Ministry of Health and Family Welfare, Government of India; Secretary for

Research, Liga Medicorum Homoeopathica Internationalis (LMHI), 61–65 Janak Puri Institutional Area, Delhi 110058, India

Letters

Table 1. Updating beliefs in eight steps after new evidence:the posterior chance after the first evidence is the prior chancebefore the second evidence, and so on [6]

RCT number Prior chance Posterior chance LR+a

1 0.000001 0.000016 16

2 0.000016 0.000256 16

3 0.000256 0.004079 16

In a recent article, Gorski and Novella state that priorprobabilities can be so low that putting them to the testmakes no sense [1]. A few decades ago the randomisedcontrolled trial (RCT) was demanded because of the lowprior probability of clinical methods such as homeopathy.Interestingly, the mention of prior chance and its updat-ing using Bayes’ theorem arose after a considerablenumber of RCTs of homeopathy were subjected tometa-analysis; this concluded that the results were notcompatible with the placebo hypothesis for homeopathy[2]. Extremely low priors, based on theory, are nowproposed as sufficient reason to disregard scientific evi-dence.

It is stated that a prior probability is infinitesimally lowif it ‘violates well-established laws of physics and chemis-try’. Quantum mechanics seemed at one time to violatewell-established laws; it did not overturn all existingknowledge, but supplemented it. The Copernican world-view was once ridiculed: the daily turn of the world wouldcause a devastating wind. Nowadays homeopathy is ridic-uled because sometimes ‘there is no molecule in it’, thoughit is now importantly recognised that nanoparticles mayremain [3]. We cannot yet conclude which mechanismmight explain homeopathy, but over 2000 basic researchexperiments indicate an effect of ultra-molecular dilutions(http://www.carstens-stiftung.de/hombrex/index.php).

Prior chances must be updated by new evidence usingBayes’ theorem. This is not a one-step process, and allpossible evidence should be used for sequential updating[4]. In sequential updating, the posterior odds afterone piece of evidence serve as prior odds for the followingcalculation based on new information. Bayes’ formulais:

Posterior odds ¼ likelihood ratio � prior odds (1)

where odds = chance/(1 – chance), and chance = odds/(1 +odds). Evidence from RCTs can be translated into alikelihood ratio (LR). Rosendaal and Bouter assignedone significantly positive RCT an LR of 16 [5]. Theirestimate of the prior that homeopathy ‘works’ was 1 in106. Sequential updating after eight positive trials gen-erates the results shown in Table 1. Very low priors areupdated surprisingly quickly by Bayes’ formula [6]. In

1471-4914/

� 2014 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.molmed.2014.09.007

Corresponding author: Rutten, L. (lexrutten@gmail.com)Keywords: prior chance; homeopathy; Bayes’ theorem.

eight trials the probability of efficacy increased from onein a million to more than 99%.

The only existing comparative review of homeopathyand conventional medicine showed similar indications ofefficacy, with better quality in the homeopathy trials[7]. The funnel plot of homeopathy trials shows no differ-ence from the comparable plot of conventional trials. Mostspots on the graph indicate beneficial effects for bothhomeopathy and conventional medicine, and the mediansample sizes are the same (65.5 vs 65.0). Gorski andNovella, however, stated that the conclusions of the reviewwere negative or inconclusive. This was based on testing apost hoc hypothesis of different efficacy of ‘larger trials ofhigher methodological quality’, with unspecified subsets ofeight homeopathy and six conventional medicine trials.These subsets appeared to be incomparable and highlyheterogeneous [8]. The negative conclusion depended onvarious subjective choices.

Before RCT evidence in complementary and alternativemedicine (CAM) became available, this type of researchwas requested by opponents of CAM. Declaring now thatthis evidence is pseudoscientific, based only on plausibility,is suggestive of post hoc reasoning. Is there perfection inthe evidence-based medicine (EBM) paradigm, based onbasic science, cell culture, in vitro studies, animal models,and clinical trials? Ezzo concluded, after reviewing160 Cochrane reviews, that 43.8% of conventional treat-ments were (possibly) efficacious [9]. Daily reality illus-trates that CAM helps many patients after conventionalmedicine has failed [10]. It is unlikely that the presentphysicochemical model is complete; how unlikely, there-fore, is a future explanation for homeopathy?

A low theoretical prior is a questionable argumentfor rejecting CAM trials. The theory is imprecise; it isnot about whether homeopathy acts in the same way as

4 0.004079 0.061505 16

5 0.061505 0.511856 16

6 0.511856 0.943748 16

7 0.943748 0.996289 16

8 0.996289 0.999767 16

aLR+, likelihood ratio for positive results.

Trends in Molecular Medicine, November 2014, Vol. 20, No. 11 599

Letters Trends in Molecular Medicine November 2014, Vol. 20, No. 11

conventional medicine, but reflects the possibility of an-other mechanism of action. The statement that acceptinghomeopathy would overthrow existing knowledge is unsci-entific. The conventional paradigm is incomplete, leavingroom for additional mechanisms of action that could fillgaps in medicine. Patients experiencing effectiveness afterconventional medicine has failed should be taken serious-ly. Existing RCT evidence seems to be incompatible withthe placebo hypothesis, and quality bias does not explainthis finding. Prior chance combined with Bayes’ theoremcannot be used as a one-step method to interpret scientificevidence because, when used correctly, it demonstratesthat extremely low priors are consecutively increasedby new positive evidence. To accept the possibility thathomeopathy works because of an as-yet unestablisheddifferent mechanism of action is more compelling thanto ignore the experience of patients, basic research, andRCT evidence.

References1 Gorski, D.H. and Novella, S.P. (2014) Clinical trial of integrative

medicine: testing whether magic works. Trends Mol. Med. 20, 473–476

1471-4914/

� 2014 Published by Elsevier Ltd. http://dx.doi.org/10.1016/j.molmed.2014.10.001

Corresponding author: Gorski, D.H. (gorskid@med.wayne.edu)Keywords: evidence-based medicine; science-based medicine; clinical trials; comple-mentary and alternative medicine; integrative medicine.

600

2 Vandenbroucke, J.P. and de Craen, A.J.M. (2001) Alternativemedicine: a ‘mirror image’ for scientific reasoning in conventionalmedicine. Ann. Intern. Med. 135, 507–513

3 Bell, I.R. and Schwartz, G.E. (2013) Adaptive network nanomedicine:an integrated model for homeopathic medicine. Front. Biosci. (Schol.Ed.) S5, 685–708

4 Gill, C.J. et al. (2005) Why clinicians are natural bayesians. BMJ 330,1080–1083

5 Rosendaal, F.R. and Bouter, L.M. (2002) Errors in methodology(conclusion). XXXIX. The ultimate truth. Ned. Tijdschr. Geneeskd.146, 304–309 (article in Dutch)

6 Rutten, A.L.B. (2008) How can we change beliefs? A Bayesianperspective. Homeopathy 97, 214–219

7 Shang, A. et al. (2005) Are the clinical effects of homeopathy placeboeffects? Comparative study of placebo-controlled trials of homeopathyand allopathy. Lancet 366, 726–732

8 Lu dtke, R. and Rutten, A.L. (2008) The conclusions on the effectivenessof homeopathy highly depend on the set of analyzed trials. J. Clin.Epidemiol. 61, 1197–1204

9 Ezzo, J. et al. (2001) Reviewing the reviews. How strong is the evidence?How clear are the conclusions? Int. J. Technol. Assess. Health Care 17,457–466

10 Marian, F. et al. (2008) Patient satisfaction and side effects in primarycare: an observational study comparing homeopathy and conventionalmedicine. BMC Complement. Altern. Med. 8, 52

Considering prior plausibility in clinical trials does notmean ignoring scientific evidence

David H. Gorski1,2 and Steven P. Novella3

1 Michael and Marian Illitch Department of Surgery, Wayne State University School of Medicine, 3990 John R St., Detroit,

MI 48201, USA2 Molecular Therapeutics Program, Wayne State University, Barbara Ann Karmanos Cancer Institute, 4100 John R St., Detroit,

MI 48201, USA3 Department of Neurology, Yale University, 40 Temple St., Suite 6C, New Haven CT 06510, USA

The response of Rutten et al. to our recent article [1], inwhich we contend that it is unscientific and unethical totest highly implausible treatments such as homeopathyand reiki in randomized clinical trials (RCTs), represents acommon and misguided complaint by advocates of alterna-tive medicine against interpreting clinical trials throughthe lens of prior plausibility. It is a complaint that is, from ascientific viewpoint, unjustified and relies on a misunder-standing of the history of science, a straw man characteri-zation of arguments for science-based medicine, and anincorrect interpretation of Bayes’ theorem applied to clini-cal trials of homeopathy. They mischaracterize our coreargument as stating that ‘extremely low priors, based ontheory, are now proposed as sufficient reason to disregardscientific evidence’; yet nowhere did we write that priorplausibility alone is sufficient reason to disregard scientificevidence. Rather, we argued that prior plausibility, when

so low as to be indistinguishable from zero (as is the casefor homeopathy), can be sufficient reason to conclude thatclinical trials are so unlikely to be informative that carry-ing out such trials would be both unethical and scientifi-cally unproductive.

The extremely low prior probability of homeopathy isnot based solely on theory, as implied by Rutten et al. [1]. Ascientific theory, after all, is not merely a hunch, but rather‘a well-substantiated explanation of some aspect of thenatural world that can incorporate facts, laws, inferences,and tested hypotheses’ (http://www.nap.edu/openbook.ph-p?record_id=6024&page=2). A theory, such as the theory ofrelativity or evolution, is accepted by the community ofrelevant scientists based on overwhelming observationaland experimental evidence supporting it. Contrary to howit is portrayed by Rutten et al., quantum theory wasnecessary to explain compelling observations that thethen-current theory could not account for. Currently, thereare no such compelling observations that can only beexplained by homeopathy. Moreover, the underpinningsof physics and chemistry, in both theory and evidence, thatconclude that homeopathy is about as close to impossible asone can imagine are so well established, rooted in 200 years