Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013

Gary SchwitzerPublisher

Adjunct Assoc. Prof., Univ. of Minnesota School of Public Health

Knight Science Journalism Fellowships at MITMedical Evidence Boot Camp

December 3, 2013

Journalistic peaks of excellence

But the valleys between these occasional peaks are becoming wider and deeper… …the daily drumbeat of dreck

Every day for 7 years3,500 media messages reviewed

Two elements to our work – common theme

Review news stories that make claims about treatments, tests, products, procedures. • Objective• Systematic• Criteria-driven

Comment on health care advertising, marketing, public relations, broader communications issues at intersection of medicine & media• Like most blogs - more

subjective, opinionated• Allows us to show broader

influence of industry - pharma, device manufacturers, hospitals – on the public dialogue

Our criteria: Does the story explain…

• What’s the total cost?

• How often do benefits occur?

• How often do harms occur?

• How strong is the evidence?

• Are there alternative choices?

• Is the condition exaggerated?

• Is this really a new approach?

• Is it available?

• Who’s promoting this?

• Do they have a financial conflict of interest?

69%

66%

65%

61%

57%

Percent unsatisfactory after 1,900 story reviews – 7 years

News stories often paint a kid-in-candy-store picture

of U.S. health care

TerrificRisk-freeWithout a price tag

Exaggerate or emphasize benefits

Ignoreor

Minimize Potential Harms

Common flaws: too much stenography – not independent vetting of studies in journals

Glorifying journals - Publication in a medical journal does not guarantee the findings are true (or even important).

Not ready for prime time. Never intended to be sources of daily news – journals meant for conversation among scientists

So if you’re going to feed off them daily, you simply must be aware of the landscape:

• Publication bias for positive results. Rose-colored view of progress in research.

• Retractions, research fraud, fabrication, falsification of data

• Unpublished data (BMJ special edition on “the extent, causes and consequences of unpublished evidence”)

• Ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)

Tip: Set up Google Alert for John Ioannidis about pitfalls of steady diet of journal stories

PLoS Med 2005; 2(8): e124

Journals complicit in miscommunication

• Editors of the HEART Group journals stated that “inappropriate word choice to describe results can lead to scientific inaccuracy.”

– J AM COLL CARDIOL, Vol. 60, No. 23, 2012

• “Are we making a mountain out of a mole hill? A call to appropriate interpretation of clinical trials and population-based studies”

– Am J Obstet Gynecol, published online 11/29/12

• “Spin and Boasting in Research Articles.”- Commentary in Arch Pediatr Adolesc Med:

[published online October 2012]

Risk reduction must be stated in absolute, not just relative, terms

• NBC News reported, “In women, aspirin reduces the rate of stroke blood clots in the brain, the most common type, by 24% but had little effect on heart attack risk. In men, the daily aspirin cuts heart attack risk by 32 percent but had little effect on stroke risk.”

– HealthNewsReview.org comment: “24 percent of what? And 32

percent of what? The journal article upon which the story is based did state the absolute values for both benefits and risks. Those absolute values show that, over about a six and half year period, daily aspirin is predicted to result in 2 fewer strokes and 2.5 more major bleeds per 1000 women; and 8 fewer heart attacks and 3 more major bleeds per 1000 men. To many viewers, those numbers will mean more than 32% or 24%.”

The limits of observational studies should always be stated clearly

• The Associated Press reported “lots of red meat increases mortality risk.” HealthNewsReview.org noted: “The language of the story suggests that cause-and-effect has been established when this kind of observational study simply can not establish causation. It can only point to statistical association.”

• When CNN reported, “Coffee may cut risk for some cancers,” frustrated readers reacted with these comments posted online:– "The statistics book in a class I’m taking right now uses coffee as an example of

statistics run amok. It seems coffee has caused all the cancers and cures them at the same time."

– "…correlation IS NOT causation!!!! So people that drink 4 or more cups of coffee have a lower incidence of two certain types of head and neck cancers, and this is supposed to mean that coffee is actually ‘warding off’ these cancers???”

The tyranny of the anecdote• Stories may use only positive, glowing patient anecdotes but fail to capture trial

dropouts, compliance problems, patient dissatisfaction or the choice to pursue less aggressive options.

• In one example, ABC, CBS, NBC, WebMD all used the same “breakthrough” patient anecdote of one man’s reported improvement from a heart failure stem cell experiment.

– This suggests another common problem: public relations people spoon-feed journalists the patient anecdotes that put an intervention in the most positive light.

Single source stories & journalism via news release are unacceptable

• ABC’s Good Morning America asked, “Could a cure for obesity be just a pill away?” HealthNewsReview.org analyzed the sourcing in this story: – “In a bizarre twist, the reporter used the potential conflicts of interest of the doctor he

interviewed in the studio as evidence of his expertise. He noted that the doctor is a consultant to the manufacturer of the pill and then said, “You’re the right man to talk to.” Really? In a recent journal publication, this researcher listed financial ties to 17 drug and device makers, including the maker of the (pill in question)… viewers were not clearly told that all of the results reported in the story came from a trial controlled by the company that has not been independently reviewed.”

• The Wall Street Journal reported, “Depression treatment Cymbalta effective as pain reliever, Lilly says.” HealthNewsReview.org wrote: – “This 189-word story failed on almost every count: – no independent perspective;– a quote apparently lifted from a company news release;– The company news release actually did a better job of explaining the study than did this

news story.”

Failure to explain limits of surrogate markers

• USA Today: “New drug ‘may turn back the clock on heart disease.’ ”– HealthNewsReview.org analysis: “The story focuses excessively on the

‘unprecedented’ changes in cholesterol achieved with the drug. … these are surrogate markers that may not reflect any real benefit on the outcomes that matter to patients.”

• WebMD: “Beet Juice Good for Brain: Drinking Beet Juice Increases Blood Flow to Brain and May Fight Dementia.” – HealthNewsReview.org analysis: “Surrogate endpoints do not necessarily

reflect on overall health outcomes. And especially not from a study of 14 people over four days. And from this we get the headline that ‘drinking beet juice may fight dementia’ ???”

• HealthNewsReview.org linked to 10 stories that framed a tiny short-term study of a biochemical marker of bone turnover as if it were definitive advice to women that, for example, “Two glasses of wine a day ends menopause misery.”

Screening stories should explain the tradeoffs of benefits and harms

• The worst single episode was news coverage of the November 2009 revised mammography recommendations from the U.S. Preventive Services Task Force. We wrote: “There is an undeniable and clear bias in many news stories, reporters and news organizations for promoting screening – evidence be damned.”

• The HealthNewsReview.org story, “A clear pattern of imbalanced screening stories by ABC News,” identified 10 examples in recent years involving cardiovascular screening, and screening for lung, prostate and breast cancer

• The HealthNewsReview.org story, “Prevention magazine editor promotes ovarian cancer screening despite lack of evidence,” explained that neither the American Cancer Society nor the National Cancer Institute support the transvaginal ultrasound screening advocated by the magazine.

• We have criticized the Chicago Sun-Times, the New York Daily News, the NBC station in Minneapolis, and other news organizations for promoting public prostate cancer screening events, disregarding evidence-based recommendations.

• The project has also identified imbalanced screening stories by CNN, CBS, Fox News, NBC News, Minneapolis Star Tribune, Saint Paul Pioneer Press, Chicago Sun-Times, Washington Post, and HealthDay among others.

Unquestioning news coverage of disease awareness campaigns

• Prostate cancer awareness is one focus of the “Movember” publicity campaign subtitled “Changing the Face of Men’s Health.” In the WCCO-TV (CBS owned-and-operated) television station newsroom, several members of the news staff grew mustaches “in support of the cause.” Exactly which cause, and how growing a mustache helps, is unclear. This campaign has spawned countless prostate cancer screening events, conflicting with many evidence-based recommendations.

Mo’ Movember “awareness”

• The NBC station in Atlanta had newsmen wear johnnies to promote prostate screening.

• The Chicago Sun-Times promoted Movember with misinformation from Dr. Laura Berman, self-proclaimed “star” of “In The Bedroom with Dr. Laura Berman.”

Fawning coverage of new technologies (proton beam therapy & robotic surgery)

• KERA public radio in Dallas reported on both technologies in its “Battlefield Breakthrough” series. – Headline: “Zapping Cancer With A Ray-Gun.” We wrote: “The

story presents not one word of caveats, concerns or limitations. There was no independent source, no data about efficacy.” Instead, there was fascination: “looks like something out of Star Wars.”

– Headline: “Robots Slice Time Under The Knife.”

We wrote: “Star Wars was a theme in that one as well. In fact, the formula was the same as for part one with proton – more battlefield breakthroughs. … Maybe we need to wait a bit to see how this arms race plays out with evidence and data before proclaiming battlefield breakthroughs just yet.”

Going easy on health business stories• Reuters

– Headline: Medtronic tests stents for erectile dysfunction. Our comment: “An initial feasibility study in 30 men is worthy of news coverage? With a company-sponsored investigator as the only source? Not in our view.”

• Chicago Tribune– Headline: Area firm’s pneumonia treatment passes test. Our comment: “It features the company president’s

positive assessment of the study results but does not seek independent comment. … It fails to make clear that the results of the trial have not been peer-reviewed or published and that results of a related trial have not been released at all.” 56

• New York Times– Headline: “Test of Eye Drug Is Said To Show Success in Elderly.” Our comment: “The piece … should’ve

slapped a bigger warning label on this cocktail of leaked summaries of secret results, speculation, and an inadequate dash of independent evaluation.”

• Wall Street Journal– Headline: “AstraZeneca Says Brilinta Beats Plavix in Clinical Trial.” Our comment: “… lets a drug company get

away with making superiority claims without releasing data. … All information in the story appears to come from company sources.”

• Los Angeles Times– Headline: “Drug for menstrual cramps in the works.” Our comment: “This story reports that data were

presented at a scientific meeting. But it didn’t tell readers what the data showed. But it did allow a company VP – the only person quoted – to say this could be a ‘breakthrough.’ Wow.”

• Minneapolis Star Tribune – HealthNewsReview.org found four stories within six weeks in which the Star Tribune established a clear

pattern of writing nice things about local medical device industries, but did not lead in reporting negative stories.

Whom to believe? Framing matters.

• Dueling news stories on brain wave scan for ADHD. Important tool? Or waste of money?

– NJ Star-Ledger, Fox News, NY Daily News, Bloomberg – all gushing praise such as “this should be an important tool” quote from MD in the study

vs.

– NY Times quoted independent expert “skeptical about the test”

– LA Times quoted independent expert “The current scientific research really doesn’t support (this) as a diagnostic tool. I would caution people.”

– ABC News quoting independent expert: “I don’t know that this going to help...it’s going to make people spend money needlessly. They can charge for it and it gives you a pseudo-scientific basis for diagnosis.”

Whom to believe? Framing matters.Dueling news stories on melanoma detection device. Helping doctors? Or causing dissent/polarizing the field?

• Cleveland Plain Dealer: “helps doctors detect deadly cancer”• WTVF Nashville & KFSN Fresno – among the stations using a syndicated news

service’s story that claimed “the Cleveland Clinic named the device one of its top medical innovations of 2013.”

• KDVR Denver allowed a doctor to call it the biggest advance in melanoma in his 26-year career

vs.• NY Times quoted 3 independent experts:

1. “This should still be considered to be in the developmental stage.”2. A dermatologist on the FDA panel who voted against approval3. Biostatistician: “(The test) just says everything is positive. I don’t think it

helps an aggressive doctor and unaggressive doctors could do just as well if they were more diligent without the device.”

Whom to believe? Framing matters.

• Dueling news stories on meds for mild-moderate depression. Give pause about prescribing? Or no reason to avoid meds?

– HealthDay: “Severely depressed gain most from antidepressants: But that’s no reason to avoid meds for moderate cases.”

vs.

– LA Times: “Study finds medication of little help to patients with mild, moderate depression”

– USA Today (quoting independent expert): Such findings “demonstrate a failure in the system: These drugs were not thoroughly tested in mild to moderate depression prior to their approval.”

Whom to believe? Framing matters.

• Dueling news stories on Alzheimer’s drug. BY THE SAME NEWS SOURCE! Reverses disease in mice? Or a flop?

– MedicalDaily.com: “Cancer drug may help reverse Alzheimer’s.”

AND

– “Targretin a flop: Potential Alzheimer’s drug fails retrial.”

Whom to believe? Framing matters.

• Dueling news stories on limitations of colonoscopy. Miss many cancers? Or concerns overblown?

– NY Times: “Colonoscopies miss many cancers, study finds.”

VS.

– ABC News: “The Case for Keeping Colonoscopy: Cancer Experts Say Concerns Over Colonoscopy Effectiveness May Be Overblown.”

– CBS News’ Katie Couric: “Don’t use this study as an excuse not to get screened.”

Whom to believe? Framing matters.

• Dueling news stories on prostate cancer screening study. Reduces deaths? Or isn’t saving lives/value questioned?

– MSNBC: “Regular prostate screening reduces deaths”– HealthDay: “PSA testing cuts cancer death risk”– Bloomberg: “PSA screening reduces deaths”

vs.

– Associated Press: “prostate cancer screening isn’t saving lives, study finds”

– CNN.com: “Value of mass prostate cancer screenings questioned.”

Framing

• Star Tribune: “Boy, 12, dies after historic transplant to treat HIV, leukemia.” – “He was in line to become the second person in

the world to be cured of both deadly illnesses by the extraordinary type of bone marrow transplant, doctors said.”

– Maybe. But in fact he was in line to die from a transplant that was not historic because it did not treat his HIV & leukemia.

“Media’s most troublesome trend” award

THE WINNER IS:

SCREENING TESTS(often involving this gland)

Award for best supporting roles:• Breasts• Lungs• Arteries

I am not anti-screening

• But I do criticize: – Stories that don’t discuss tradeoffs involved in all

screening test decisions– Framing screening as a mandate, not a choice– Stories/journalists taking an advocacy stance– Stories that are inaccurate, imbalanced,

incomplete

“All screening programs do harm; some do good as well.” - Dr. Muir Gray

All screening stories would be better if they simply stated:

Anecdotes of those who claim their life was saved by screening

Anecdotes of those who make a rational choice

not to be screened – or

those who were screened and

regret it

The tyranny of the anecdotein screening news stories: Whom do we model?

Prizes for Prostates

Dr. Otis Brawley, chief medical officer, American Cancer Society

“I’m very concerned. There’s a lot of publicity out there – some of it by people who want to make money by recruiting patients – that oversimplifies this – that says that ‘prostate cancer screening clearly saves lives.’ That is a lie. We don’t know that for sure……We’re very concerned about a number of clinics that are offering mass screening where informed decision making is not happening. Many of these free screening things are designed more to get patients for hospitals and clinics and doctors than they are to benefit the patients. That’s a huge ethical issue that needs to be addressed. We’re not against prostate cancer screening. We’re against a man being duped and deceived into getting prostate cancer screening.”

“Duped & deceived”

Tim Glynn, lawyer, age 47 when his doctor “decided I’d have a PSA test without consulting me…..Men should be aware of the truly terrible consequences. As a screening tool, you could do as well by throwing dice on a table.”

Sharon Begley in Newsweek:

“Dr. Stephen Smith, Professor emeritus of family medicine at Brown University School of Medicine, tells his physician not to order a PSA blood test for prostate cancer or an annual electrocardiogram to screen for heart irregularities, since neither test has been shown to save lives. Rather, both tests frequently find innocuous quirks that can lead to a dangerous odyssey of tests and procedures.”

“I can only hope my story will …inspire every woman who hears it to get a mammogram, to take a self-exam. No excuses. It is the difference between life and death.”

1. “No excuses” promotes theme of mandate, not a choice. 2. No, it is not the difference between life and death for every

woman.3. There are potential harms from mammography. They are

clear, well-known, supported by data.

Mammography screening also finds slowly growing or less aggressive tumors which would never be life-threatening. But because that cannot be predicted, about 10 of 2,000 women in the screening group had their breast completely or partially removed, though this would not have been necessary.

For years I’ve been writing about why it’s unethical for network television journalists to use their celebrity status to talk about their own health care decisions and to suggest

what other people should do.

• NBC’s Mike Taibbi had lung CT and endorsed the scans in his piece.

• NBC’s George Lewis reported on his own prostate cancer diagnosis/treatment and ended recommended all men have PSA tests

• NBC’s Andrea Mitchell reported on her own breast cancer diagnosis and treatment by giving advice on the air to other women

• NBC’s Matt Lauer & Al Roker had digital rectal exams on the air to promote prostate cancer screening and failed to discuss guidelines which don’t recommend such screening.

• CBS’s Harry Smith followed colleague Katie Couric’s example from when she was at NBC and had a colonoscopy on the air, endorsing the practice while not mentioning other colon cancer screening options which many people might prefer and which the evidence supports.

Andrea Mitchell: “For you women out there and for the men who love you, screening matters. Do it. This disease can be

completely curable if you find it at the right time.”• Katherine O’Brien – who published the ihatebreastcancer blog – wrote:

“Early detection is not a cure. …”Completely curable” is a like a fat man wearing a hockey jersey. It covers a lot of ground. You have access to the top medical experts in the world-ask them what “cured” means in the setting of breast cancer.”

• Breast cancer patient Katie Ford Hall of the UneasyPink blog wrote to Ms. Mitchell:“Wishing you the best Ms Mitchell. When you feel settled, I’d love to talk to you about the realities of breast cancer “caught early” … What you said about breast cancer is harmful and untrue. You should correct it immediately.”

• One expert pointed out the misuse of the overused “1 in 8” statistic, as when she said:

“I am now among the 1 in 8 women in this country — incredibly 1 in 8 — who have had breast cancer.” • As the National Cancer Institute explains:

Women born now have an average risk of 12.2 percent (often expressed as “1 in 8″) of being diagnosed with breast cancer at some time in their lives. On the other hand, the chance that they will never have breast cancer is 87.8 percent (expressed as “7 in 8″).• But 1 in 8 is a lifetime risk. Risk increases with age, so the NCI provides a more helpful way of looking at

it – for all of those women watching who are of different ages: • A woman’s chance of being diagnosed with breast cancer is:

from age 30 through age 39 . . . . . . 0.43 percent (often expressed as “1 in 233″)from age 40 through age 49 . . . . . . 1.45 percent (often expressed as “1 in 69″)from age 50 through age 59 . . . . . . 2.38 percent (often expressed as “1 in 42″)from age 60 through age 69 . . . . . . 3.45 percent (often expressed as “1 in 29″)

What is the quality of the shared decision-making encounter…

…as mammograms are promoted with wine, cheese, fondue, chocolate fountains, pink treats, pink lemonade, massages, facials, manicures/pedicures, paraffin hand treatments, beauty consultations, etc. ?

Sharon Begley reported for Newsweekmagazine:

“Dr. Rita Redberg, professor of medicine at the University of California, San Francisco, and editor of the prestigious Archives of Internal Medicine, has no intention of having a screening mammogram even though her 50th birthday has come and gone. That’s the age at which women are advised to get one. But, says Redberg, they detect too many false positives (suspicious spots that turn out, upon biopsy, to be nothing) and tumors that might regress on their own, and there is little if any evidence that they save lives.”

• Reported on woman with DCIS – ductal carcinoma in situ• Reported on her choice to pursue active surveillance rather

than immediate aggressive intervention such as bilateral prophylactic mastectomy

• Reported on the shared decision-making program at UCSF that helped her understand the tradeoffs.

THAT IS GOOD JOURNALISM

• Began with virtual colonoscopy – after routine physical.

• But scan also showed something on kidney, liver, and lungs.

• Kidney and liver issues benign after liver biopsy, PET scan and more CT scans. But lung questions led to major lung surgery

“I awoke in the recovery room after 5 hours, with a chest tube, a Foley catheter, a subclavian central venous catheter, a nasal oxygen catheter, an epidural catheter, an arterial catheter, subcutaneously administered heparin, a constant infusion of prophylactic antibiotics, and patient-controlled analgesia with intravenously administered narcotics…..Excruciating pain.”

• No malignant disease – all “incidentalomas” • Total cost > $50,000

• All precipitated by a screening test

Repeat: it’s not anti-screening….

• …when you cite the evidence.• …when you educate people about tradeoffs.• …when you help people avoid “avoidable

ignorance.”

• We shouldn’t be telling anyone not to be screened just as we shouldn’t be telling anyone to be screened. We should inform about the tradeoffs so people can make informed decisions.

If we don’t improve discussion on screening, we may never improve public dialogue on expensive new technologies – “Gizmo Idolatry”

Proton beam or IMRT

JOSH BILLINGS (PEN NAME OF HUMORIST HENRY WHEELER SHAW, 1818 – 1885)

“I honestly believe it is better to know nothing than to know what ain’t so.”

Why?

Because there are harms in making people more hopeful or more frightened than evidence and rational thinking would warrant.

@Thank you

gary@healthnewsreview.org