Hunting side effects and explaining them: should we reverse evidence hierarchies upside down? Barbara Osimani Catholic University of Milan Evidence and

Hunting side effects and explaining them: should we reverse evidence

hierarchies upside down?

Barbara Osimani

Catholic University of Milan

Evidence and Causality in the SciencesCanterbury, 5-7 September 2012

Hunting side effects and explaining them: should we reverse evidence hierarchies upside down?

In their comparative analysis of RCTs andobservational studies, Papanikolau et al. (2006)assert:

“it may be unfair to invoke bias and confounding to discredit observational studies as a source of evidence on harms” (p. 640, my emphasis).

Hunting side effects and explaining them: should we reverse evidence hierarchies upside down?

• Recent contributions by philosophers and health scientists have acknowledged the role of so called "lower level" evidence as a valid source of information contributory to assessing the risk profile of medications both on theoretical (Aronson and Hauben, 2006; Howick et al. 2009) and on empirical grounds (Benson and Hartz, 2000; Golder et al. 2011).

• Nevertheless current practices have difficulty in assigning a precise epistemic status to this kind of evidence and in amalgamating it with standard methods of hypothesis testing.

Topics

• Recent proposals to amend evidence hierarchies especially Vandenbroucke’s suggestion to reverse hierarchies when addressing the issue of risk discovery and assessment;

• The requirement of total evidence as applied to this specific context: how it is and how it should be interpreted (vs. lexicographic rule implicit in ranking);

• I will illustrate some of these points by reference to the recent debate on the causal association between paracetamol and increased asthma prevalence/exhacerbation.

Hierarchy reversal for risk assessment

Vandenbroucke J.P. (2008) Observational Research, Randomised Trials, and Two Views of Medical Science. Plos Medicine, 5 (3): 339-43

Hierarchy of study designs for intended effects of therapy

Hierarchy of study designs for discovery and explanation

i. Randomised controlled trials i. Anecdotal: case report and series, findings in data, literature

ii. Prospective follow-up studies ii. Case-control studies

iii. Retrospective follow-up studies iii. Retrospective follow-up studies

iv. Case-control studies iv. Prospective follow-up studies

v. Anecdotal: case report and series v. Randomised controlled trials

Vandenbroucke’s defence of hierarchy reversal (I)

1. Methodological point:Observational studies concerning adverse reactionswill not suffer from confounding in the same way asobservational studies for intended effects do. selection bias is less likely to affect observational studies with respect to adverse reactions.This because unintended effects, qua unintended, are not knownin advance, and thus also not known by the drug prescriber, whocannot take them into consideration and thus bias treatmentallocation.

Ignorance of possible effect = “natural masking”

Vandenbroucke’s defence of hierarchy reversal (II)

2. Epistemological point:

Context of discovery vs. context of evaluation:

Discovery is focused on explanation and hypothesisgeneration;Evaluation instead on hypothesis testing/confirmation.

And research methods differ in the opportunities they offerwith respect to either of these goals.

Vandenbroucke’s defence of hierarchy reversal (III)

• Vandenbroucke (2008) formalizes the contrast between the context of evaluation and the context of discovery in terms of different priors assigned to hypotheses of benefits and of adverse reactions.

• High priors for intended effects• Low priors for unintended ones


1. It is the higher priors which make the results more robust, not the method (Vandenbroucke, 2008: 16-17).

2. The reason why we accept uncertain results for risks rather than for benefits is that evaluation and discovery studies are associated with different loss functions:

1. evaluation is related to the approval of health technologies and is required to assure stakeholders about their efficacy and safety,

2. whereas discovery is more related to the context of research for its own sake, which might explain why certain study designs are preferred to others in different circumstances.


1. Priors are quickly swamped by data2. Stakes are not lower for detecting risks than

for testing the drug benefit: adverse drug reactions might be so severe as to reverse the safety profile of the drug and determine its withdrawal.

Prior knowledge about drug’s general capacity to produce unintended adverse reactions

• The acceptability of anecdotal evidence or of uncontrolled studies for assessing risk has to do with a high prior about the general capacity of the drug to bring about side-effects.

• Whereas there is total ignorance as to some specific side effects which might be possibly caused by the drug, still there is almost certainty about the fact that the drug will indeed cause side-effects beyond the ones already detected in the pre-marketing phase.

• This high prior derives from historical knowledge and past experience with pharmaceutical products and is also strongly reflected in the regulation which introduced the notion of “development (or potential) risk”, the pharmacosurveillance system, and the precautionary principle.

Vandenbroucke’s defence of hierarchy reversal on abductive grounds

• “For discoveries, the original case reports, lab observations, data analysis, or juxtaposition in literature may be so convincing that they stand by themselves, either because of the magnitude of the effect or because the new explanation suddenly and convincingly makes the new finding fall into place with previous unexplained data or previous ideas”. (Vandenbroucke, 2008: 6).

Case Study: hypothesis of causal connection between paracetamol and asthma

Asthma increase in the United States and inWestern countries in the last 3 decades: up to a 75% increase among adultsand to a 160% among children in the sameperiod.(Burr et al., 1989; Eneli et al., 2005, Ninan and

Russel, 1992; Mannino et al., 1998, 2002, Seaton et al. 1994).

Explanatory hypotheses for asthma epidemic1) increased exposure to outdoor and indoor pollutants;

2) decreased exposure to bacteria and childhood illnesses during infancy (the “hygiene hypothesis”);

3) increased obesity incidence and prevalence;

4) changes in diet and oxidant intake;

5) cytokine imbalance as a reaction to environmental allergens in early childhood leading to lifelong T-helper type 2 (allergic) dominance over T-helper type 1 (nonallergic) reactions, thus increasing the risk for atopic disease

Eneli et al., 2005; Seaton et al. 1994, Shaheen et al. 2000.

How suspicion fell upon paracetamol• Varner and colleagues (1998) detected a precise correspondence between increase of

asthma incidence and increased paracetamol use as a substitute for aspirin (following the recognition of an association between aspirin and Reye’s syndrome).

• The trend levelled off in the 1990s, i.e. at a time when paracetamol had already become one of the most widespread analgesics.

• Varner and colleagues tentative explanation was however that asthma increase was due to aspirin avoidance, for the reason that aspirin may protect from asthma through inhibition of prostaglandins.

• However, this hypothesis was soon discounted on grounds that, if this had been the case, then one should have observed a decrease of asthma incidence when aspirin was first introduced (Shaheen et al. 2000).

• Thus the suspicion finally fell upon paracetamol itself and subsequent investigations explicitly aimed to examine the hypothesis of causal connection between paracetamol and asthma.

Evidence for causal association between paracetamol and asthma

“Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative:

1) consistency of the association across geography, culture and age;2) strength of the association (comparative studies); 3) the dose-response relationship between paracetamol exposure and asthma; 4) the coincidence of the timing of increasing asthma prevalence and increasing

paracetamol use;5) the relationship between per-capita sales of paracetamol and asthma

morbidity across countries; 6) our inability to identify any other abrupt environmental change that could

explain this increase in asthma morbidity; 7) plausible mechanism: glutathione depletion in airway mucosa caused by

paracetamol”.

McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).

Consistency of the association across geography, culture and age (I) Source Year of

study Study objective Population Results

Beasley et al. Cross-cultural study

2008 Examine the risk of asthma rhynoconjunctivitis and eczema in children using paracetamol

122 centers in 54 countries 200,000 children 6-7 yr

Dose dependent increase in prevalence and severity of asthma > once per year: OR 1.61 (95% CI 1.46-1.77) ≥ once per month: OR 3.23 (95% CI 2.91-3.60) Association identified at almost all sites regardless of geography, culture, stage of development

Beasley et al. Cross-cultural study

2011 Examine the risk of asthma rhynoconjunctivitis and eczema in adolescents using paracetamol

122 centers in 54 countries 320,000 children 13-14 yr old

Dose dependent increase in prevalence and severity of asthma > once per year: OR 1.43 (95% CI 1.33-1.53) ≥ once per month: OR 2.51 (95% CI 2.33-2.70) Association identified at almost all sites regardless of geography, culture, stage of development

Systematic review and meta-analysis of epidemiol ogic studies Etminan et al.

2009 Quantify the association between acetaminophen use and the risk of asthma in children and adults.

Thirteen cross-sectional studies, four cohort studies, and two case-control studies comprising 425,140 subjects

Pooled odds ratio (OR) for asthma among subjects using acetaminophen was 1.63 (95% CI, 1.46 to 1.77). The risk of asthma in children among users of acetaminophen in the year prior to asthma diagnosis and within the first year of life was elevated (OR: 1.60 [95% CI, 1.48 to 1.74] and 1.47 [95% CI, 1.36 to 1.56], respectively). Only one study reported the association between high acetaminophen dose and asthma in children (OR, 3.23; 95% CI, 2.9 to 3.6). There was an increase in the risk of asthma and wheezing with prenatal use of acetaminophen (OR: 1.28 [95% CI, 1.16 to 41] and 1.50 [95% CI, 1.10 to 2.05], respectively).

Consistency of the association across geography, culture and age (II)

Longitudinal birth-cohort study Amberbir et al.

2011 Investigate the independent effects of paracetamol and geohelminth infection on the incidence of wheeze and eczema in a birth cohort.

population-based cohort of 1,065 pregnant women from Butajira, Ethiopia,

Paracetamol use was significantly associated with a dose-dependent increased risk of incident wheeze (adjusted odds ratio = 1.88 and 95% confidence interval 1.03-3.44 for one to three tablets and 7.25 and 2.02-25.95 for ≥ 4 tablets in the past month at age 1 vs. never), but not eczema.

Wickens et al. Birth cohort study

2011 investigate the associations between infant and childhood paracetamol use and atopy and allergic disease at 5-6 years.

New Zealand Paracetamol exposure between birth and 15 months in Christchurch (n=505) and between 5 and 6 years for all participants (Christchurch and Wellington) (n=914). Outcome data collected at 6 years for all participants. Logistic regression models were adjusted for potential confounders

Paracetamol exposure before the age of 15 months was associated with atopy at 6 years [adjusted odds ratio (OR)=3.61, 95% confidence interval (CI) 1.33-9.77]. Paracetamol exposure between 5 and 6 years showed dose-dependent associations with reported wheeze and current asthma but there was no association with atopy. Compared with use 0-2 times, the adjusted OR (95% CI) were wheeze 1.83 (1.04-3.23) for use 3-10 times, and 2.30 (1.28-4.16) for use >10 times: current asthma 1.63 (0.92-2.89) for use 3-10 times and 2.16 (1.19-3.92) for use >10 times: atopy 0.96 (0.59-1.56) for use 3-10 times, and 1.05 (0.62-1.77) for use >10 times.

Cross-sectional analysis McKeever et al.

2005 To investigate the associations between use of pain medication, particularly paracetamol, and asthma, COPD, and FEV1 in adults.

Data from the Third National Health and Nutrition Examination Survey (U.S.) Participants aged between 20 and 80 years, with complete data for relevant exposures, outcomes

Dose–response association of paracetamol use and asthma (adjusted odds ratio, 1.20; 95% CI, 1.12–1.28; p value for trend 0.001).

Shaheen et al. Case-control study

2000 To investigate whether frequent use in humans was associated with asthma.

Adults aged 16–49 years registered with 40 general practices in Greenwich, South London. Frequency of use of paracetamol and aspirin was compared in 664 individuals with asthma and in 910 without asthma.

After controlling for potential confounding factors OR for asthma, compared with never users, was 1.06 (95% CI 0.77 to 1.45) in infrequent users (<monthly), 1.22 (0.87 to 1.72) in monthly users, 1.79 (1.21 to 2.65) in weekly users, and 2.38 (1.22 to 4.64) in daily users (p (trend) = 0.0002). This association was present in users and nonusers of aspirin.

Shaheen et al. Multicentric case-control study

2008 To examine whether or not frequent paracetamol use is associated with adult asthma across Europe.

The network compared 521 cases with a diagnosis of asthma and reporting of asthma symptoms with 507 controls with no diagnosis of asthma and no asthmatic symptoms across 12 European centres.

Weekly use of paracetamol, compared with less frequent use, was strongly positively associated with asthma after controlling for confounders. OR 2.87 95% CI 1.49-5.37 No association was seen between use of other analgesics and asthma.

Consistency of the association across geography, culture and age (III)




paracetamol use;5) our inability to identify any other abrupt environmental change that could

explain this increase in asthma morbidity; 6) the relationship between per-capita sales of paracetamol and asthma

morbidity cross countries; 7) plausible mechanism: glutathione depletion in airway mucosa caused by

paracetamol”.


Comparative studies

Source Year of

study Study objective Population Results

Case-control study Shaheen et al.

2000 Determine if frequent paracetamol use is a risk factor for asthma.

Adults aged 16-51 yr in South London Cases: n = 720 (51% response rate) Controls: n = 980 (49% response rate).

Never users: OR 1.06 (95% CI 0.77-1.45); Monthly users: OR 1.22 (95% CI 0.87-1.72); Weekly users: OR 1.79 (95% CI 1.21-2.65); Daily users: OR 2.38 (95% CI 1.22-4.64); P value for trend = 0.0002

Prospective cohort study Shaheen et al.

2002 Examine the relationship between prenatal paracetamol use and wheezing in offspring at 6 mo.

9400 women Increased risk of wheezing before 6 mo for offspring of frequent paracetamol users over 20-32 wk prenatally: OR 2.34 (95% CI 1.24-4.40).

Prospective cohort study Barr et al. Nurses’ Health Study

2004 Examine the relationship between paracetamol use and new onset of asthma

73,321 women (44-69 yr)

Increased risk of diagnosis of new-onset asthma with frequency of use Adjusted RR 1.63, 95% CI 1.11-2.39 Dose dependence: p value for trend = 0.006

Randomized double blind trial without placebo Boston University Fever Study

2002 Compare the incidence of adverse reactions among children administered paracetamol or ibuprofen

84,000 febrile children Age ≤ 12 yr Randomly assigned paracetamol or low –dose ibuprofen, or high dose ibuprofen

Among 1879 children with pre-existing asthma, outpatient visits for asthma were lower in the ibuprofen arm than the paracetamol arm (RR 0.56 95% CI 0.34-0.95); + dose-dependence Hospitalizations were nonsignificantly lower (RR 0.63 95% CI 0.25-1.60).




paracetamol use;5) the relationship between per-capita sales of paracetamol and asthma

morbidity and across countries; 6) our inability to identify any other abrupt environmental change that could

explain this increase in asthma morbidity; 7) plausible mechanism: glutathione depletion in airway mucosa caused by

paracetamol”.


Varner et al. Systematic review

1998 Investigate relationship between substitution of aspirin with paracetamol and increased asthma prevalence in developed countries.

Systematic review of U.S. studies

Epidemiologic trends, known biologic effects of cytokines and PGE2 on allergic sensitization, and a potentially relevant pharmacologic effect of aspirin used to explain a component of the increasing prevalence of childhood asthma in the United States.

Coincidence of the timing of increasing asthma prevalence and increasing paracetamol use (4)

“Although other changes in the environment have been suggested that might explain an increase in childhood asthma, none so easily explains the rapid increase in asthma in the 1980s and the subsequent levelling off of asthma

prevalence over the last 15 years. Furthermore, the prevalence of childhood wheezing in 36 countries around the world is predicted by each country’s per-

capita sales of paracetamol”. McBride (2011) (5)

Ecologic Study Newson et al.

2000 Examine the rate of Asthma and aggregate consumption of acetaminophen in 1994-95.

English speaking countries in the ECHRIS study.

Prevalence of wheeze increased by 0.52% for 13-14 yr olds; By 0.26% for young adults, For each gram increase in per capita paracetamol sales. Prevalence of childhood wheezing in 36 countries around the world is predicted by each country’s per-capita sales of paracetamol.

Relationship between per-capita sales of paracetamol and asthma morbidity and across countries (5)

Possible mechanisms

Paracetamol

Low glutathione level

Lack of suppression of cyclooxigenase

pathway

**IgE mediated immune response

Inability to counteract oxydative

stress

Defective antigen

processing

**Toxic effects of n.acetyl-p-benzoquinone

mine

Lung Injury with Bronchoconstriction

Source: Eneli et al. 2005

Proof onus

• McBride (2011) explicitly warns against the use of Paracetamol in children with asthma or at risk for asthma and claims that if further evidence is required, then this is for documenting product safety rather than the contrary.

• This explicitly addresses the reluctance of sceptical commentators to accept such evidence as a sufficient basis for practice change and for establishing a causal relationship between paracetamol and asthma, on grounds that it does not result from randomized clinical trials (Eneli et al. 2005, Allmers et al. 2009, Johnson and Ownby, 2011; Karimi et al., 2006, Wickens et al. 2011, Chang 2011).

Hierarchy reversal = chronological1. Non-controlled observational studies (e.g.

ecological studies)

2. comparative studies (case-controlled retrospective)

3. prospective studies

+ evidence coming from cellular and molecular studies.

• However, the justification of this hypothesis is neither exclusively focused on the exclusion of confounders (Vandenbroucke’s point 1), nor based on utility functions(Vandenbroucke’s point 2).

• Instead, several independent pieces of evidence jointly support a given hypothesis and are considered to do at least as good a job as the one-shot proof presumably provided by an RCT (or a meta-analysis of RCTs).

1. Both according to Vandenbroucke’s point 1, as well as to the recent contribution by Aronson and Haube, (or Howick’s contribution to the topic) case reports and observational data are considered sufficient evidence for causal claims to the extent that possible confounders (i.e. alternative causes for the experimental result) can be confidently excluded.

• This kind of reasoning also guides the general framework of evidence hierarchies: The higher the likelihood that the study design rules out more confounders than others, the higher it is settled in the ranking.

• And it is a straightforward consequence of the method of hypothesis testing, which is an hypothetico/deductive mode of investigation, for which the evidence is supposed to refute the null hypothesis.

2. Vandenbroucke’s call on priors instead (point 2) invokes a Bayesian epistemology, where hypotheses are assigned probabilities and these are updated in the light of new data.

3. In McBride’s, the causal hypothesis is assessed abductively, by putting things together and inferring the implications of their joint occurrence. The hypothesis of causal connection provides a unified explanation of the different pieces of evidence, which would otherwise need a series of distinct explanatory facts.

Epistemological paradigmsEpistemology Method Assumptions Justification of “lower level”

evidence

Unificationist (qualitative abduction)

Connection of data in light of explanatory hypothesis

Connectedness (ontological) Explanatory power of hypothesis in light of data.(see also recent proposal of integration of Bradford-Hill criteria: Stegenga, 2011; Howick et al. 2009).

Inductive-Bayesian (quantitative abduction)

Bayes theorem Principle of total evidence - coherence

Probability of hypothesis given evidence

Hypothetico-deductive (statistical mode)

Hypothesis testing: likelihood of evidence if H0 = true (p-value)

Homogeneous populations with regard to all possible relevant causal factors(randomization)

Only if alternative hypotheses (confounders) can be safely excluded, or treatment effect swamps them by a statistically significant amount (Howick, 2011).

• The unificationist paradigm regards hypotheses as explanatory factors for the observed data. In order to be really explanatory they must accommodate as many data as possible. Data which fail to be taken into account are left unexplained, thus making the hypothesis less virtuous from a theoretical point of view.

• In the Bayesian paradigm hypotheses can be associated with any probability in the unit interval. The main requirement is coherence (in the mathematical sense of standard probability calculus) and that all available evidence is used: this because, bayesian epistemology tracks inductive uncertainty and all non-deductive logics are non-monotonic.

Epistemological paradigms

Non-monotonicity Principle of total evidence

Nonmonotonicity means that an addition to the premises may invalidate some previous conclusion.

The principle of total evidence responds to this issue and is an essential principle of uncertain inference (Carnap, 1947).

Linda is getting out of the bank It is 4 pm.Linda is getting out of the bank and she is going to an

antinuclear demonstration It is earlier than 4 pm.

Hypothetico-deductive method (statistical mode)

• Instead statistical hypothesis-testing is a kind of approach which admittedly follows a Popperian hypothetico-deductive method of scientific enquiry. And being this paradigm inherently deductive, it does not feel urged to address the issue of non-monotonicity.

• The very idea of hierarchies follows from the assumption that if you have a study which has the capacity to eliminates more confounders than another, than the former should trump the latter. Trumping means that higher level evidence discards any evidence of inferior ranking, and also makes it irrelevant.

Lexicographic rule for hierarchy implementation

Higher level studies trump lower level ones:1. when two studies of different levels deliver

contradictory findings, then the higher in the evidence study is considered more reliable and is allowed to discard the lower level one

2. lower level evidence adds nothing to higher level one and thus it can be neglected without loss of information.

• The strongest way to interpret hierarchies is to claim that, because it is assumed that randomization provides a guarantee for causality, than it should follow that there is no guarantee of causality without randomization.

• “No randomization in, no causes out”

Lexicographic rule for hierarchy implementation

Cause

• I think all instruments of investigation play an important role in scientific investigation.

• Analytical approaches work with truth conditions (RCTs set strict desiderata for considering sthg to be a cause for sthg else) but have strong methodological (only certain kind of evidence is at all meaningful) and epistemological limitations (failure to account for important phenomena such as causal interaction - see Muller).

• Inductive approaches work with imperfect indicators (P(E/H); dose-response relationship; specificity of the association); or theoretical virtues such as the explanatory power), thus not only they can, but they must take into account all evidence (non-monotonicity). Nevertheless they can give you no sure-fire guarantee that the delivered result demonstrates the truth of the hypothesis.

1. Probabilistic Hypothesis of Causal Connection

• From the time a risk is not known, to the moment in which it is incontrovertibly proven to be causally associated with the drug, there is a period of evidence accumulation which constitutes a state of partial and imperfect (but continuously increasing) knowledge.

• In this period it cannot be claimed that there is a causal link between the drug and the detected risk; but neither can we behave as if we knew nothing about it. Still, the latter attitude is precisely the only possible policy allowed by an epistemology grounded on hypothesis rejection.

2. Implementation of the precautionary principle

• Following the precautionary principle, you are not supposed to wait for the causal connection between harm and suspected drug to be certain, before you take adequate countermeasures, but instead, you should act as soon as the probability of causal connection is high enough to recommend countermeasures because of a negative risk/benefit balance.

• This probability might be also very low, in case the risk magnitude is considerably big with respect to the expected benefit.

• The frequentist mode of summarizing statistical data, following which hypotheses may only be accepted or rejected, cannot be of any use to this purpose.

• The problem of external validity and causal interaction is dramatic in the case of side-effects.

3. Problem of external validity (ontological

Conclusion• My aim is to raise awareness about the different

epistemological paradigms underlying the distinct evidence policies we may intuitively endorse.

• The take-home lesson is that different epistemologies grant different methodological actions and impose different, which in turn bring about relevant practical implications.

• Thus it is worthy to bear in mind the criteria underlying our evidence constraints whether we want to rank it or not, or else to reverse rankings.

Documents

Hunting side effects and explaining them: should we reverse evidence hierarchies upside down? Barbara Osimani Catholic University of Milan Evidence and