Understanding and Reporting on Journal Articles

8/20/2019 Understanding and Reporting on Journal Articles

1/29

Useful mathematical tools for journalists

www.stats.org

Understanding and Reportingon Journal Articles

Rebecca Goldin, Ph.D. January 20, 20!"ational Press #oundation


2/29

$ cou%le words about our organi&ation'

• Princi%al aim is to increase statistical sa((y inthe media

• Pro(ide wor)sho%s, %ublic lectures

• *rite critical %ieces at +$+.org and

occasionally other (enues• $d(isory -oard of statisticians to hel% journalists

• #oundation funded. "o cor%orate s%onsorshi%%ermitted.


3/29

+tatistical conce%ts in

scientic journal articles/ean, median, mode

+tandard de(iation

ondence inter(als

1rders of magnitudeonfounding factors

Percentages

$bsolute (s. relati(e ris)

+cientic methods+tudy Design

ausation (ersus correlation


4/29

In the beginning: a pressreleasePress Releases

Don’t Tell the WholeStory

Abstracts Don’t Tellthe Whole Story

Designed to get attention by the

%ress

Present the results in the rosiestterms %ossible

Dont %ut the results in conte3t of

other research

+hy away from concrete descri%tions

4ow were subjects recruited5

*hat was the design of the

e3%eriment5

*hat methods were used to analy&e

the data5

*hat are the wea)nesses of the

conclusions5


5/29

1(er(iew of scientic

journals and their (alue toscientic consensus

How many scientifc jornals are there! Pub/ed is theweb interface of the "ational 6ibrary of /edicine. 7t has justunder 89,000 titles.

How "o I #now i$ a jornal is peer re%iewe"! hec) the journals website. :ou can also chec) on Ulrich;s PeriodicalsDirectory, but this reand websites that mirror it? will gi(eyou ran)ings so you ha(e a sense. :ou can google + "otlisted5 "ot cited@

http://0-www.ulrichsweb.com.libcat.lafayette.edu/ulrichsweb/http://0-www.ulrichsweb.com.libcat.lafayette.edu/ulrichsweb/http://0-www.ulrichsweb.com.libcat.lafayette.edu/ulrichsweb/http://0-www.ulrichsweb.com.libcat.lafayette.edu/ulrichsweb/


6/29

1(er(iew of scientic

journals and their (alue toscientic consensus

What’s the process o$ pblication!

Peer re(iewed %ublications go through a %eerAre(iew%rocess. 7f theres no re(iew, anything goes.

Peer re(iew can be timeAconstrainedBwea). /any%a%ers retracted.

Researchers should re%ort their a=liations and su%%ort

>industry, go(ernment, conCicts?.

Is there %ale in non&peer re%iewe" science at aprestigios con$erence! +ure, but there has been noe3amination of the


7/29

/oral and statisticalcollision$bortion >and ancer?

1besity >and ancer?

"ursing (s. -ottle#eeding

+mo)ing >and ancer?#oodB$lcohol >and

ancer?

"aturalE (ersus

hemicalE >and ancer?Pollution >and ancer?

rime


8/29

ausation or orrelation


9/29

7ts easy to be fooled4eight correlates with reading s)ills in children under

0

4eight correlates with cancer

hocolate consum%tion is correlated with "obel Pri&es

Digit ratio >2D'8D? and %rostate cancer7ncome correlates with success in college

Par)ing a(ailability correlates with dri(ing

#aceboo) correlates with %oor grades

#aceboo) correlates with good grades

Doing heroin correlates with doing marijuana

$lcoholism correlates with less gray matter in the%refrontal corte3


10/29

ancer Fbrea)throughs that costtoo much and do too littleNewsweek and The Daily Beast, /arch 20

Article on “targeted therapies,” cancer therapies that try totarget the tumor of patients and are often used as part of alast eort to beat the cancer, or to delay death. Perjeta is a

drug used for late stage breast cancer patients.“Perjeta gives the average woman only about six months more ofcalm before her disease starts to stir again. Given the limitedbenet, the price was startling. For most women, a full course ofthe drug combination will cost !"",###$.

Article is about cost compared to average increased life

span.But who is the “average woman” !he studies refer to the

median length of increased life. "alf of women die within #months, and half of women die more than # months later.


11/29

ancer Fbrea)throughs that costtoo much and do too littleNewsweek and The Daily Beast, /arch 20

+omeone who li(es 9H years more>considered curedE? would count the same

in this calculation as women who li(e !.months more.

he median increased lifes%an tells usalmost nothing about the drugs

eIecti(eness.

!ime # months $ years


12/29

Doctors o%inion on median lifee3tension' The New York Times

1%Aed by 20 %restigious oncologists' $ Plan to #i3 ancer areEom%lained that .9 %ercent of cancer %atients are consuming 9

%ercent of the countrys health care s%ending.

Pointed to the fact that the bestE of these recent thera%iesincreased the lifes%an of cancer %atients by a median of !months. wo only increased lifes%an by 8A! wee)s.

1lder drugs as good as newer drugs costing K9,L00 %er month.

What’s wrong with this reasoning!

"o one would bal) at the %rice if these drugs really worked.

But with no sense o how long the !etter hal li"es, how cananyone determine how #good$ these drugs are%

7f a drug were to cure 29M of %eo%le on cancer >and do nothingfor the others?, it would be a miracle, but could fail this test.

#a(oring older drugs may ignore realities about which %eo%le theywor) for, (ersus their a(erage rate of hel%.


13/29

>Nasy?


14/29

>Nasy? this is es%eciallyim%ortant in sur(ey, o%inion, and food studies?

• Are the reslts statistically (signifcant)! >more onsignicance in a bit?

• Are the reslts clinically signifcant! Results thatclaim signicanceE are not necessarily going to im%act

our li(es that much.• 7f a study s%ea)s in increased ris)E, what is the

absolute increased risk?

• *hy might there be a correlation without a causalrelationshi%5


15/29

Jum%ing from correlation to cause

:ou dont always ha(e to )now why it may not becausal. -e wary of any claims of causality.

+ome common reasons that a correlation could loo)causal when its not, include' not adjusting forconfounders, misunderstanding the mechanism,ha(ing an un)nown confounder.

$ causal relationshi% might be reasonable to sus%ect

when the statistics are1(erwhelming

1bser(ed in many diIerent conte3tsRe%eated tests show the same eIect, on large

numbers of %eo%leDouble blind caseAcontrol studies.


16/29

*hat the hec) is that p&"alue%

$ p&"alue is often re%orted in anabstract, claiming the results arestatistically signicant, with %O.09.E

:ou Ci% a coin many times, and you

thin) this coin is biased, because youarent getting close to heads and tails. 4ow can you


17/29

$ Gra%hic on oin #li%%ingProbabilities


18/29

he normal cur(e


19/29

*hat ha%%ens in the lab'N3%eriments Galore...


20/29

*hat therest of theworld

sees


21/29

4ard to get our heads aroundrandomness

+y +agic Penny How I $on" it,*

Predicts the %residency' 4 for

Democrats, for Re%ublicans 7t was correct 9 out of Q of the

%re(ious elections.

Just .M chance of that ha%%eningby chance, so it truly is magic.

7f it werent a coin, but were some

indicatorE li)e the economy, the%arty of the incumbent, Junea%%ro(al ratings, you would

%robably thin) theres a reason forsuccess.

7 as)ed 000 of my best friends to Ci%

a %enny Q times, writing 4 or foreach trial. hen 7 com%ared to the list

of correct %residents.

hen 7 %ic)ed the best coin.

7n a room of 000 Ci%%ers, there is a

Q0M chance of nding a magicE coin.


22/29

Do you see any

%atterns5417N $'

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

417N -'

4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 44 4 4 4 4 4 4 4 4 4 4 4 4

*hich is random5


23/29

+ome Ci% data

0

2

8

!R

0

2

8

!

R20

here is an M %robability that a

se


24/29

ancer clustersancer clusters are

real' sus%ected clustersare geogra%hicalregions where a higherthan e3%ected numberof cancers occur.

1(er 000 %ossible cancer clusters are re%orted each yearto state health de%artments.

ancer clusters can occur by chance. 4ow to tell if its not5

+tatistics hel%' what is the e3%ected number of cancers5

Just li)e coin Ci%s with many in a row' e(en if its allrandom, there will be %laces with more cancers thane3%ected.

+eries of other >nonAstatistical? factors in(ol(ed, such asconsistent ty%es of cancer, su=cient number of cases, and

biologically %lausible mechanisms.


25/29

e3as +har%shooter #allacyS


26/29

-asic $d(ice for a Journalist with6imited ime and 7deas

I$ yo ha%e -.mintes

I$ yo ha%e /.mintes

Rea" the smmary0 theabstract0 an" the conclsion*/any writers ignore conclusions.

The abstract will tell yo thereslt. 7t hardly e(er hints at

limitations.

The 1conclsion’ or 1"iscssion’o$ten shares ca%eats* he

conclusion is ty%ically at the end ofthe %a%er.

A%oi" most conclsions o$casality* Read as a s)e%tic at all

times.

2oo# to 3anti$ying the ris#**hat is the actual ris)5 *hat does

it mean for most %eo%les li(es5

4hec# $or signifcance* $re the %A(alues small5 $re the diIerences

clinically meaningful5

As# the athors 3estions* linicalim%act5 Generali&able to which

%o%ulations5 *ea)nesses of methods5

5et another pro$essional opinion**hat is the


27/29

Research is routinely %lagued

%esearch is plagued &hat can a journalist do'

%ow levels of signi&cance

'ultiple testing (o ac)nowledgement of

randomness in research design

%ac) of conte*t+repeatede*periments

cientists don-t )now how to tal) to journalists. ou can help by as)ing

good /uestions and reiteratingwhat you thin) the person

said+sending /uotes for edits.

But if you are loo)ing to &nd onescientist willing to tal), you may not

get the mainstream opinion.

0rite about the levels of signi&cance,bias, caveats

As) the researchers about multipletesting. 1id they adjust for them

0rite about absolute ris)s.

%oo) for a body of research ratherthan one speci&c paper

2ite your sources3

14(-! 5(152A!6 2A7A!54(0"6( A 24886%A!54( "A B66(

"40(3


28/29

4ow to get hel%$s) the corres%onding author and disinterested %arties for

information li)e'*hats the clinical signicance5 >N3am%les include' 4ow many li(es will

be sa(ed in a grou% of 000 %eo%le5 Does the im%ro(ement describedin a %a%er mean that %eo%les li(es will im%ro(e in a %ractical way5Does the measurement that changed when %eo%le are e3%osed to ato3in matter for their li(es5?Do you see any %ossible bias based on data you collected or study

design5

*hat other %eerAre(iewed research by other laboratories ts in withwhat you ha(e done5

+$+.org1n our website, theres a lin) for journalists to get hel%

$d(isory board consists of statisticians who are familiar with thechallenges journalists faceT they answer


29/29

han) you@

Documents

Understanding and Reporting on Journal Articles