ECRAN – EBM guide for healthcare journalists

WP6

European Communication on Research Awareness

Needs

WP6 – EBM guide for healthcare journalists to the

critical interpretation of clinical trials

WP6

ECRAN – EBM guide for healthcare journalists

What is quality of scientific information?What is quality of scientific information?

Let’s sort out the wheat from the chaff …

The huge amount of research studies is a big challenge for health care journalists, who have to sort out the wheat of scientific excellence from the chaff of low quality or interested driven science. The purpose of this guide is to ensure excellence in medical journalism by means of:

•applying the core principles of journalistic coverage and investigation in the field of science and medicine;•training journalists to properly cover news events, trends, and issues in all aspects of health care, sorting out the tangle of data about clinical trials and transferring them to medical practice in a comprehensive and multifactorial perspective; …

WP6

WP3 task 3.2 – Others (1)

…

• enabling properly trained journalists to better convey the complex dimensions of health care challenges from an overall point of view;

• ensuring an independent coverage about health and research, analyzing health related data with fairness, accuracy and balance;

in order to:• increase the capacity of individuals and policymakers to

adopt health care attitudes and policies that can improve the health of all the citizens;

• improve the ability of citizens to make wise decisions about behaviors that promote health, treatment options, and their choice of health care providers.


WP6

WP3 task 3.2 – Others (2)ECRAN – EBM guide for healthcare journalists

What is science?What is science?«A system of certain knowledge in a period of time built up, arranged and ensured by research, teaching and accumulated data».

(translation from the German definition in Brockhaus 2003:2193)

Which is the goal of science?Which is the goal of science?The ultimate goal of science must always be to verify or reject existing (content-specific or methodical) knowledge and to replace it with new knowledge. Science is a cumulative process. The touchstone for assessing new knowledge is the comparison with existing knowledge. Systematic reviews, that collect and evaluate knowledge of several scientific studies, have been developed and implemented for 20 years as methodical tool for assessing research.

WP6

WP3 task 3.2 – Others (3)ECRAN – EBM guide for healthcare journalists

How should good science look like?How should good science look like?Assessing good science according to its neutrality and objectivity is unrealistic, because of its several infrastructural, financial and performance-related implications. In the last decade the call of transparency has been playing a central role in this process, in order to ensure the integrity of research. Assessment of the framework conditions for research should be therefore integrative part of each journalistic investigation, in order to set an indicator for a preliminary assessment of the reliability of research outcomes.

How should good journalistic investigation look like?How should good journalistic investigation look like?Basic scientific understanding of scientific studies and their weak points enables journalists to facilitate orientation and to avoid lurking pitfalls in this field (e.g. look at the section “Tips for understanding studies – Common pitfalls” of this guide).

WP6


Toolkit for journalistsToolkit for journalistsThere are many challenges to achieving accuracy, balance and completeness in health care journalism. For a reporter, there are deadlines, editors, and corporate financial pressures. Though quality in health care journalism is far more than this, because it plays a major role in educating consumers! Consumers need to be well informed to make sound choices that affect their health and well being.

The following rating criteria are an internationally validated methodology, that helps conjugate general and medical journalistic criteria to review stories claiming about specific treatments, tests, products or procedures (i.e. drugs or devices, vitamins or nutritional supplement, diagnostic and screening tests, dietary recommendations, surgical procedures).

WP6


Rating criteria for healthcare journalists

1 Quantification of benefits

(treatment/test/product/procedure)

6 Establishing true novelty of approach

4 Further independent source and declaration of

conflicts of interest

2 Quantification of harmsof intervention

7 Comparing new approach with existing

alternatives

5 Story solely or largely based on news release?

3 Proper evaluation of quality of evidence

8 Establishing availability of treatment/test/product/

procedure9 Adequate estimate of costs of intervention

10 Is there potential for disease mongering?

WP6

ECRAN – Rating criteria for healthcare journalists

Quantification of benefits (treatment/test/product/procedure)

Stories are expected to describe the effectiveness of a benefit. Few stories provide helpful numbers to back the real benefit of a new treatment, test, product or procedure. Empirical investigations have shown that news reports, drug ads and some studies coverage tend to report benefits as the relative reduction instead of the absolute risk reduction in the frequency of failures.

Another challenging question may be: «How many patients have to be treated in order to prevent the problem for one patient?». This is the Number Needed to Treat (NNT): it is defined as the inverse of the absolute risk reduction and is the average number of patients who need to be treated to prevent one additional bad outcome.

1

WP6


What does “significantly increased” mean? How was this measured? Apart from this, statistical significance may not be equal to clinical relevance. The real question is: «What difference does it make in peoples’ lives?» Reports indicating surrogates and intermediate endpoints, e.g. changes in blood test scores, may not influence individual health scores (increased life expectancy or lower mortality rate) or quality of live.

Moreover, anecdotal storytelling makes engaging reading but is not scientific data. Glowing personal stories or single unchallenged exaggerated success stories may not be a representative example of how a benefit works and require a critical appraisal for a balanced framing of it.

1

WP6


Data of possible benefit is not available at all. Using only relative, not absolute, risk reduction or benefit

data. Relying too heavily on what may be unrepresentative

patient anecdotes about benefits. Citing statistics appropriately, but allowing even a single

unchallenged exaggerated quote to throw the story out of balance.

Question to clear at the end of the day

Is the benefit really as much more or less significant as it is presented?

Criterion will not be fulfilled, in case of …

1

WP6

Quantification of harms of intervention

Be aware of appearances! When you hear someone saying that a new approach «appears to be safe», look for evidence supporting it.

Stories may emphasize or exaggerate potential benefits of interventions, while minimizing or completely ignoring their potential harms. For some patients slight side effects may even have dramatic impact on their lives. An overall healthcare report should always consider possible – even the slightest – adverse effects of interventions, ideally reporting their frequency and severity.

Any healthcare decision implies tradeoffs, all treatments may have side effects, risks and complications.

ECRAN – Rating criteria for healthcare journalists 2

WP6

Even a screening test may turn out to be harmful to the individual getting it: in fact, it may produce a false positive result, which causes anxiety, and will probably lead to more

testing (i.e. biopsy) and possibly treatment (i.e. surgical intervention) that comes with its own harms. Moreover, enthusiasm for early tests in people without symptoms is often not supported by evidence.

PSA testing, for example, often turns out to cause more harms than benefits, including false positive results and overdiagnosis, both of which could lead to unnecessary treatments associated with significant side effects.


WP6

Criterion will not be fulfilled, in case of …Failing to mention potential harms.Failing to quantify potential harms.Failing to describe the severity of potential harms.Failing to account for “minor” side effects that could have a significant impact on a patient’s life.Relying too heavily on a patient anecdote about safety.Relying too heavily on a researcher’s comment that an approach appears to be safe – without supporting data.

Question to clear at the end of the day

Which are the risks and the side effects of this intervention or product?


WP6

Proper evaluation of quality of evidenceHealthcare coverage should critically evaluate the evidence and not merely take published or presented research as gospel. Quality of evidence has to be classified according to evidence hierarchy:


WP6

Science works cyclically with ebbs and flows: knowledge stream keeps flowing and keeps changing in the course of time. So, ask yourself:

At which point in the stream did I grasp the evidence?

Not all evidence is bullet-proof, even if it is published in high impact journals, because not all studies are equal. Eminence is not evidence and study stories often turn out to be ongoing stories.

A solid journalistic investigation is expected to scrutinize limitations of evidence, number of probands, duration of the study, relevance of outcomes; to assess whether what appears as the latest story is already existing in larger and better-designed stories on the same topic or whether the statements generalize small group data (e.g. a single medical center with experienced surgeons).


WP6

Criterion will not be fulfilled, in case of …Failing to dig out the limits of the study design (i.e. lack of control group, lack of blinding, missing drop out rate).Not cautioning on the interpretation of uncontrolled data.Conflating causation and association (i.e. limitations of observational studies).Generalizing small group data (i.e. at one medical center with experienced surgeons).Confounding surrogates with real health outcomes (illness or death vs. markers or scores).Not mentioning that scientific meeting outcomes lack adequate peer review.Not pointing out that findings from animal or lab experiments are not directly applicable to human health.Presenting anecdotic data as evidence for intervention.

Question to clear at the end of the dayDoes this study design justify these allegations?


WP6

Further independent source and declaration of conflicts of interest

Claims about new treatments, tests, products, and procedures should be based on evidence, not on beliefs or business hopes. In health care journalism, it is therefore essential that second opinions be sought from independent experts who have no vested interest in the approach being discussed. For this scope two things must occur: there must be an independent expert source quoted (someone not directly connected with the research), there must be some attempt to let the public know about potential conflicts of interest.


WP6

Hyperbolic healthcare claims made by drug companies, medical device manufacturers and academic research centers are to be confronted with this question:

Do they have something to gain?


May be that: a trial is paid for by the drug manufacturer; researchers are getting fees from a drug company; claims are made by a spokesman for a device manufacturer; doctors are being early adapters of a new device; there are underlying inventors’ interests.

4

WP6

A study funded by industry doesn’t necessarily mean that findings lack integrity, but evidence from recent studies shows that potential conflicts might lurk around every corner in health care. So, the next time you go to your doctor’s office that is cluttered with drug company promotions, it is reasonable for you to ask him if he/she had any financial relationship with the makers of drugs, devices or test advertised for. Moreover, seek independent perspectives and beware of statements like: «Experts believe …, doctor think …». Instead

Give them a name, because vagueness is nameless!


WP6

Criterion will not be fulfilled, in case of …Failing to identify the source of the story (news release, journal article, editorial, scientific meeting presentation, etc.) and failing to make obvious the extent to which that source is likely to be conflicted (e.g. «A PR consultant working for the company said…» or «Dr. Smith, who received a company grant to perform the study, said…»)Failing to include the input of a trusted independent source with expertise on the subject to comment on the claims made (ideally, more than one such independent source would be used).

Question to clear at the end of the dayTo what extent are the information independent and how plausible is the expertise?


WP6

Is the story solely or largely based on news release?

It is simply not sound journalistic practice to lift material directly from a news release without stating that this is the case. If we know that a news release was sent out and if we can get a copy of that release, we will review it along with the story.Many news organizations facing financial pressures may be lead to become more open to using news releases as a way to publish content while cutting costs. Although news releases can be valid sources of some information, journalism is being asked to collect independent information with the consequence that it is unacceptable to rely on a news release as the sole source of information.


WP6

Particularly in health care, there are many vested interests trying to influence consumer choices. Journalists are expected to use independent verification relying neither on news releases nor on company spokesmen!

A team from Dartmouth Medical School and the Veterans Affairs Outcomes Group published in 2009 an analysis on the quality of press releases by medical centers concluding that «Press releases from academic medical centers often promote research that has uncertain relevance to human health and do not provide key facts or acknowledge important limitations.»


WP6

Criterion will not be fulfilled, in case of …There is clear evidence that the exact same wording from a news release appears in a story.Lacking evidence of a press release and the worst a story can be rated is “not applicable”.

Criterion will be fulfilled, in case of …A story has several independent sources and proves NOT to rely only on a news release.

Question to clear at the end of the dayIs the news based only on a press release?


WP6

Establishing true novelty of approachMany alledgedly new treatments, tests, products or procedures are not really novel. The agent may be another member of a well-established therapeutic class of drug. Even if it represents a new class, it may offer no more than the drugs that are widely available. In the press release for a new drug this essential information may be lost in the hype and the drug is portrayed as “novel” in order to increase initial sales. So ask yourself:

Is it novel or simply new?

Journalists’ reporting should be able to make these distinctions on the real novelty of a product by researching or asking independent sources, e.g. specifiying that there are previous studies on this agent for the same indication.


WP6

In health care, newer isn’t always better, the first in a new class of drugs may offer no more than the drugs that are widely available. Drug companies are very good in promoting the alledged novelty of a drug in order to increase initial sales. Moreover, some claims for new drugs are made for “off-label use”, i.e. for not officially approved uses and under doctors’ careful monitoring. Off-label uses might also result in wasting money and harming patients and the respective costs may not be covered by the state or private insurance. By focusing on a new idea journalists should tell how well it works putting it into the context of existing alternatives.


WP6

Mostly it is rare that a study appears out of nowhere and the “big picture”, where the “novelty” comes from, is often missing. To find it out journalists should search for the opinion of independent experts or reliable scientific sources like.


WP6

Criterion is will not be fulfilled, in case of …Failing to accurately describe the new product/intervention and, as a consequence of this, misleading the consumer.Failing to explain the real novelty of the product/intervention (preferably comparing it to existing alternatives on the market).Failing to mention that the described new product/intervention is, in fact, for off-label uses.

Question to clear at the end of the dayIs the product/intervention a real novelty?


WP6

Comparing new approach with existing alternatives

It is mandatory for a healthcare journalist to put the new treatment into the context of existing alternatives that have a longer, more proven track record. Whatever the new approach, the report should try to put it into the context of already available interventions, in order to compare the effectiveness of the various options. News stories on new treatments, tests, products and procedures should also take into consideration the value for money ratio.

In case a treatment should have been developed from scratch, the journalist is expected to explain that there are no alternative treatments or only supportive or soothing remedies.


WP6

“To wait and see” is also a plausible alternative to the proposed treatment/test/product/intervention. “Active surveillance”, declining immediate aggressive therapy, is a viable and rationale option.

It is wise for patients to know their treatment options. They may also consider to make lifestyle changes rather than taking a drug, or else they may pursue a drug or non-surgical option rather than having surgery.

Promotions of screening tests should also consider the option for individuals to choose NOT to be screened.


WP6

Criterion will not be fulfilled, in case of …Discussing a surgical approach without mentioning non-surgical alternative approaches.Discussing a new test without mentioning other tests that are available, including the option of not being screened in the case of a screening test.Failing to discuss the advantages/disadvantages of the new idea compared with existing approaches.Failing to discuss how the new treatment, test, product or procedure fits into the realm of existing alternatives.

Question to clear at the end of the dayAre there any other treatment options besides this procedure/product?


WP6

Establishing availability of

treatment/test/product/procedureThis criterion shall mostly be applied to coverage about a new drug in clinical trials (or in a trial on the new use of an established drug). A journalist is expected to tell his/her readers/public, if the drug/agent is available at the chemist’s (or may be soon) or if it is really only in an early-phase trial (e.g. pilot study or stage before approval).

It is important for a journalist to know that there may be a lot of marketing hype that suggests that something is well-studied and ready for use.

Questions should also be raised by the fact that something should have been approved in the U.S. and not or not yet in Europe.


WP6

Some early studies focus on safety and not on how well something works. Some other focus on early efficacy and may not be large enough or designed to show safety of a drug/treatment in the long term.

Predictions about «something to be approved by …» in a certain period should also be taken with a grain of salt, they usually come from someone who stands to benefit.

New approaches, devices or operations should be scrutinized for their availability (no one wants to be patient number one with a new device/operation).

Finally, have the personnel required to deliver the new approach properly been trained and are there enough of them? What is the learning curve for this?


WP6

Criterion will not be fulfilled, in case of …Making not clear that the drug is not available now.The approval of a drug is treated as a closed issue.Something «expected to be approved by …» and its availability on the market sounds more like a crystal ball prediction.Time predictions about the market availability of the drug are solely based on declarations sourced to a company spokesperson.Failing to describe to what extent a treatment is available (in every hospital or only in specialty clinics).Promoting a special intervention without providing a sense of how many trained doctors there are in the field (e.g. a hi-tech surgical intervention).

Question to clear at the end of the dayIs this item available now or not?


WP6

Adequate estimate of costs of intervention Journalists are expected to appropriately report on intervention/product costs, though it may be difficult to estimate costs of an experimental approach early in its development. Even if the real cost is probably known at the time of market entry, health reporters could at least cite costs of existing alternatives. Journalists should be able to get a reasonable estimate from their source/experts on the topic.

The rule of thumb is:

If it is not too early to talk about how well something might work,

then it is not too early to start discussing what it may cost


WP6

A large amount of healthcare stories do not appropriately report on the cost issue. Healthcare takes a large part of the Gross Domestic Product in developed countries and we all better start thinking about costs of treatments, tests, products and procedures, we all pay for.

So, it is not enough to say The cost is much lower than …

but you should better askWhat does this actually cost?

How much lower?What about the cost-effectiveness, if compared to …?

Journalists should also consider additional expenses for psychologists, dieticians, physiotherapists involved in the intervention or that the convalescence and rehabilitation may be more time-consuming.Moreover, they should investigate, if insurances cover the intervention/drug or it is affordable for the patients needing them.


WP6

Criterion will not be fulfilled, in case of …Costs of an intervention are not mentioned at all.Costs are played down without giving a satisfactory explanation therefore.Not comparing the costs of a procedure with other available alternatives (more or less expensive ones).Not mentioning, if the new procedure/drug will be covered by public/private health insurance.

Criterion will not apply, in case of …An experimental procedure in early development with no mention of the related costs.An expert claiming that costs are quite unimportant (e.g. speaking of well known products like aspirin or weight-loss products).

Question to clear at the end of the dayIs it covered by health insurance or must I pay out of my pocket?

Question to clear at the end of the day:


WP6

Is there potential for disease mongering?This criterion assesses whether a condition is exaggerated or oversold. There are different forms of “mongering”: turning risk factors into diseases (e.g. low bone mineral density becomes osteoporosis); misrepresentation or inaccurate description of the natural history and/or severity of a disease (e.g. early-stage low-grade prostate cancer); medicalization of minor or transient variations in function (e.g. temporary erectile dysfunction or female ‘sexual dysfunction’); medicalization of normal states (baldness, wrinkles, shyness, menopause); exaggeration of the prevalence of a disease/disorder (e.g. using rating scales to ‘diagnose’ erectile dysfunction).

ECRAN – Rating criteria for healthcare journalists10

WP6

Some statements are trying to make all of us feel like there’s something wrong with us – even if there isn’t. That’s what we call disease-mongering. This is dangerous because it is an unnecessary emotional and financial burden for society and may distract us and precious public resources from important healthcare issues that should be addressed. Also turning a normal state or variation of health into a medical problem is disease mongering, like baldness, wrinkles, shyness or menopause. Pseudodiseases, that lower the threshold for common diseases but, in fact, only open new markets, may turn millions of people overnight into patients needed to be treated with drugs or vitamins against high cholesterol or osteoporosis. Often, women are targeted because of premenstrual dysphoric disorders, female sexual dysfunctions, overactive bladders or menopause; men equally get their share being hammered because of low testosterone levels, balding etc.


WP6

Identifying disease mongering is a matter of judgment. Sometimes it is obvious. Sometimes there’s a fine line about whether a statement on irritable bowel disorder, erectile dysfunction, restless leg syndrome or osteoporosis (all of which can be serious for some sufferers) is misrepresenting the condition to the public. Sometimes there may be “multiple mongering” with many hyperboles for the same issue.


WP6

Criterion will not be fulfilled, in case of …Spurious statistics are presented. Question prevalence estimates!Statistics inflate the seriousness of a condition (e.g. «millions of people `suffering` in silence with toenail fungus…») Someone “selling sickness” wants to scare you into testing or treatment.Risk factors are treated as diseases to convince us that we should be concerned about “a score” (i.e. surrogate markers, e.g. changed blood parameters) instead of outcomes like illness or death.Holding up “worst-case patients experiences” as representative examples of all with this conditions.


WP6

Criterion will not apply, in case of …

A therapy/procedure/test/drug possibly being overestimated. There is no «treatment mongering» as such. This item should be best scrutinized under criterion Quantification of benefit or Proper evaluation of quality of evidence.

Question to clear at the end of the dayShould I really be concerned about this disease/circumstance?


WP6

Choosing the right issueChoosing the right issueIs it a current, relevant and original contribution?

Topicality of an issue may be related to the event and the frequency of medium publication, the importance or the originality (i.e. curiosity) of its contents. Relevance may occur for instance, if the issue directly or indirectly regards a large part of the population, if a therapy determines high costs for the health system, if a new development has an ethical dimension, if the topic has political importance or an exemplary nature. If not relevant, an issue may instead be curios, humorous, surprising etc. Novelty of an issue may be characterized by latest data published e.g. today, yesterday, this week in a scientific journal, or outcomes of recently held conferences.

The rule of thumb is:

If the issue should not be current, then it must be relevant. If not relevant, then it must be at least curios/original!

ECRAN – General journalistic criteria 11

WP6

Delivery of messageDelivery of messageHas the issue been successfully delivered or

does it even have model character for its format?A successful message must be delivered in an intelligible language, with a plain structure avoiding long and complicated sentences. Specialist or technical terms should only be used in exceptional cases and, anyway, accordingly explained by means of context or examples. Formal factors also play an important role: the story must follow dramaturgical principles (personalisation, narrative elements, relation to everyday life); form and content must harmonise; abstract thinking must be explained by pictures, illustrations, clear specimen texts; good sound clips in a radio report should invite to listen to medical issues. In case of reports, WH questions should all be answered.

Rule of thumb: delivery standard is conform with the format


WP6

Factual accuracyFactual accuracyDoes the story report essential facts?

Factual accuracy means reporting historical facts without manifest or essential errors, e.g. accurate reporting of study conclusions or data, correct description of study design (i.e. number of participants enrolled/allocated/followed up/analysed in clinical trials). Accuracy is neither synonymous with objective truth nor completeness, e.g. a report must not comprise each and every aspect of the study but the described facts must be true. A single name error is not enough to invalidate this criterion, but many little mistakes may create the impression of a real sloppy work!

Apart from purely data errors or hyperboles in the headlines or teaser beware of surrogate markers, association vs. correlation, absolute vs. relative risks, NNT, off-label use, phases of clinical

trials, confidence interval etc ...


WP6

Surrogate markersA surrogate marker is an event or a laboratory value that researchers hope can serve as a reliable substitute for an actual disease. A common example is blood cholesterol levels which are surrogates, or substitutes, for heart disease. As a matter of fact, surrogate studies are indirect by definition and a journalist should be skeptical before celebrating the last medical breakthrough, because what is sound medical dogma today may fade over time. Surrogate research is an important research tool, that can open important therapeutic avenues and stimulate additional research. Those who earn their living and reputation from medical research may have a different view on surrogacy, but it is important to understand that exaggerations may damage patient care by highlighting surrogate outcomes instead of important results such as death, quality of life or functional capacity.

ECRAN – Tips for understanding studies – Common pitfalls

WP6

Association vs correlation and language useLook at this wording describing the results of an observational study:

«Frequent fish consumption was associated with a 50% reduction in the relative risk of dying from a heart attack.»

And look at the editor’s streamline “adaption” detrimental to meaning:

«Women who ate fish five times a week cut their risk of dying later from a heart attack by half.»

The new wording suggests that the subjects’ fish consumption is responsible for their dying less frequently from heart attacks, but the original study does not support a conclusion of cause and effect.

Epidemiologic – or observational – studies examine the association between what is known in epidemiologic jargon as an exposure (e.g. a food, environmental influences, lifestyle) and an outcome (often a disease or death). Because of all the other exposures occurring simultaneously in the complex lives of free-living humans that can never be completely accounted for, such studies cannot provide evidence of cause and effect; they can only provide evidence of some relationship (between exposure and outcome) that a stronger design could explore further.


WP6

The only study design involving humans that does rise to the level of demonstrating cause and effect is a randomized trial. In this design, study subjects are assigned an exposure (or a control condition) at random, irrespective of any other exposures in their lives, and all such other exposures are then assumed to even out between the treated group and the control group of subjects (and this can be demonstrated). As a result, the only difference between the groups is whether they receive the exposure under study or the control condition. This approach is a true experiment. Any difference in outcome seen between the control and the experimental group should be due to the one factor or variable that differs.

A subtle trap occurs in the transition from the cautious, nondirectional, noncausal, passive scientifical language in reporting the results of observational studies to the active language used in mass media. Active language is fine in general (who wants to write like a scientist?), but problems can arise when the use of causal language is not justified by the study design. In practice, a shift to causal language can occur at any stage: writing, editing, or headline composing, with similar effects on meaning, that could lead readers to overestimate the meaning of a given study and possibly even make life choices that the evidence does not warrant. Sometimes even scientists and press-release writers slide into causal language in expressing results of observational studies.


WP6

Absolute vs relative risksResearchers, clinicians or journalists often report only relative differences in making claims about a new idea, but they should tell the rest of the story!

Absolute differences make the real sense out of such claims.

Look at this example about blindness in diabetic patients in a 5-year-period:

If the risk for blindness is 2 in 100 (2%) in a group of patients treated conventionally and 1 in 100 (1%) in patients treated with a new drug, the absolute difference is derived by simply subtracting the two risks: 2% – 1% = 1%.Expressed as an absolute difference, the new drug reduces the 5-year risk for blindness by 1%.The relative difference is the ratio of the two risks. Given the data above, the relative difference is: 1% ÷ 2% = 50%Expressed as a relative difference, the new drug reduces the risk for blindness by half.

Data are accurate in both cases, but: if you are a marketing manager for the new drug, you are likely to only use the relative risk reduction; if you are a journalist, you would better do your job by citing the “rest of the story” (absolute risk reduction) often missing in news releases and direct-to-consumer prescription drug ads.


WP6

Number needed to treat (NNT)The NNT is defined as the inverse of the absolute risk reduction or the number of patients who need to be treated to prevent one additional bad outcome, calculated as 1/absolute risk reduction.

Applied to our hypothetical diabetes blindness drug example:the risk for blindness in a patient with diabetes over a 5-year period is 2 in 100 (2%) in a group of patients treated conventionally and 1 in 100 (1%) in patients treated with a new drug. So the absolute difference is derived by simply subtracting the two risks: 2% – 1% = 1%.the number needed to treat would be 1 / 1% (or .01) = 100.so you would need to treat 100 people with diabetes for 5 years in order to prevent one case of blindness.

You can see that this is an important way to look at new claims about new drugs.


WP6

Off-label usePrescription drugs are officially approved by regulatory agencies for specific indications to treat specific conditions or diseases. Physicians may prescribe a drug for a use that is not described in the approved labeling if it seems reasonable or appropriate to them. This is what’s called off-label use. For example: it is not uncommon for physicians to prescribe low doses of beta blocker drugs to help people overcome jitters before public speaking. Beta blockers are not formally approved for this use, but in such circumstances doctors are advised that they have the responsibility to be well informed about the product, to base its use on firm scientific rationale and on sound medical evidence, to inform the patients and get its consent for the use of the drug.A study in the Archives of Internal Medicine found that about 21% of all estimated uses for commonly prescribed medications were off-label, and that 15% of all estimated uses lacked scientific evidence of any useful or beneficial effect. Authors of the study recommended more careful monitoring of off-label uses that might waste money and harm patients. Another problem of the off-label prescribing is that its cost may not be reimbursed by public or private health insurances.Finally, whilst off-label use is permitted, marketing of a drug for an off-label use is not.


WP6

Phases of clinical trialsJournalists who report on drugs while they are still in clinical trials need to understand the distinction between phases I, II, and III of drug trials. It is misleading to report bold or conclusive statements about how well a drug works when it is only in phase I trials, since the primary goal of phase I trials is to evaluate how safe a drug is, not how well it works. Many times journalists report on early phase drug trials as if all the evidence is in hand.

Advice for healthcare journalists:

Give accurate portrayals of the status of investigational drugs, devices and procedures, including significant caveats and explanations of hurdles, unknown and potential problems.

Without such caveats consumers would be better advised to have doubts about the accuracy and balance of the story.


WP6

Confidence intervalIn the interpretation of trial results scientists divide errors into two classes: a type I error is the mistake of thinking something is true when it is not (also known as a false positive); a type II error is thinking something is not true when in fact it is (a false negative). When testing a specific hypothesis, scientists run statistical checks to work out how likely it would be for data which seem to support the idea to have come about simply by chance. If the likelihood of such a false positive conclusion is less than 5%, they deem the evidence that the hypothesis is truly statistically significant. They are thus accepting that one result in 20 will be falsely positive – but one in 20 seems a satisfactory low rate.


WP6

Valuable papers fur further reading

This list has been updated at November 2013To find further contents, visit ECRAN website www.ecranproject.eu

- Opportunities for research and NIHhttp://www.sciencemag.org/content/327/5961/36.summary- Demystifying trial networks and network meta-analysishttp://www.bmj.com/content/346/bmj.f2914- Independent drug testing to ensure drug safety and efficacyhttp://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328348- Editorial policies for clinical trials and the continued changes in medical journalismhttp://jama.jamanetwork.com/article.aspx?articleid=1699471- Informing the uninformed: optimizing the consent message using a fractional factorial designhttp://www.ncbi.nlm.nih.gov/pubmed/23700028


WP6

- The Women’s Health Initiative—A victory for women and their healthhttp://jama.jamanetwork.com/article.aspx?articleID=1745653- Better reporting of scientific studies: why it mattershttp://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.1001504- Quantification of harms in cancer screening trials: literature previewhttp://www.bmj.com/content/347/bmj.f5334- Risks (and benefits) in comparative effectiveness research trialshttp://www.nejm.org/doi/full/10.1056/NEJMp1309322- The right to participate in high-risk researchhttp://philpapers.org/rec/SHATRT-2- Who’s afraid of peer review?http://www.sciencemag.org/content/342/6154/60.full- Why we can’t trust clinical guidelineshttp://www.bmj.com/content/346/bmj.f3830- Practices and impact of primary outcome adjustment in randomized controlled trials: meta-epidemiologic studyhttp://www.bmj.com/content/347/bmj.f4313


WP6

- US study criticized for experimentation with premature infantshttp://www.bmj.com/content/347/bmj.f4198- Do clinical trials work?http://www.nytimes.com/2013/07/14/opinion/sunday/do-clinical-trials-work.html?pagewanted=all&_r=2&- Understanding clinical trials: appreciating medical heroeshttp://doc.mediaplanet.com/all_projects/4921.pdf - Dispelling the many myths about clinical trialshttp://www.newswise.com/articles/dispelling-the-many-myths-about-clinical-trials


Documents

ECRAN – EBM guide for healthcare journalists