Introductiontometa-analysis

Outlinefortoday

• Metaanalysiscomparedwithtraditionalreviewarticle

• Quantitativesummariesvs.vote-counting

• Howtocarryoutameta-analysis

• Effectsize

• Fixedandmixed-effects

• Associatingeffectsizeswithrelevantvariables

• Publicationbias

• Makeyourresultsaccessibletometa-analysis

Scientificstudiesonatopicareoftenrepeated

Newstudiesimprove/expandonpreviousstudies,orexaminethesameissueinadifferentstudysystem,orusingdifferentmethods

• Schoeneretal.(1983)found164publishedfieldexperimentsoninterspecificcompetition.

• Gardneretal.(2003)obtainedresultsfrom51separatestudiesreportingcoralcoverfrom294sitesfromacrosstheCaribbean.

• Belletal.(2009)found759publishedestimatesoftherepeatabilityofbehavior,from114studiesof98species.

• Vilàetal(2011)reviewed199articlesreporting1041fieldstudiesdescribingtheecologicalimpactsof135alienplanttaxa.

Amethodisneededtosummarizeresultsfrommultiplestudies

• Dr.BenjaminSpocksold50millioncopiesofBabyandChildCare1950s--1990s.Inithewrote“Ithinkitispreferabletoaccustomababytosleepingonhisstomachfromthebeginningifheiswilling”.Otherpediatriciansmadesimilarrecommendations.

• Fromthe1950sintothe1990s,morethan100,000babiesdiedofsuddeninfantdeathsyndrome(SIDS).

• Intheearly1990s,researchersrealizedthattheriskofSIDSdecreasedbyatleast50%whenbabieswereputtosleepontheirbacksratherthanfacedown.

• SubsequenteducationcampaignsledtoadramaticdropinthenumberofSIDSdeaths.

• However,researchwasavailablefrom1970thatsleepingonthestomachwashazardoustobabies.Anearliersynthesisofthedatacouldhavegottheanswermuchsooner.

Traditionalapproachisthereviewarticle

Anexpertinthefieldassemblesthestudiespublishedonatopic,thinksaboutthemcarefullyand(hopefully)fairly,andthenwritesareviewarticlesummarizingtheoverallconclusionsreached.Afirst-ratereviewarticleadvancesafieldfarbeyondameresummary.Itreviewsandcommentsonthecurrentstateofthoughtandknowledgeaboutaparticulartopic.Suchareviewwillproposenewhypotheses,uncoverpreviouslyunnoticedrelationships,andpointtonewpathsofresearch.

Thetraditionalreviewlacksaquantitativemethod

Thismightleadtotwoproblems• Bias.Inhis1986bookHowtoLiveLongerandFeelBetter,LinusPauling(theonlypersontobeawardedtwounsharedNobelPrizes)cited30studiessupportinghisideathatlargedailydosesofvitaminCreducestheriskofcontractingthecommoncold,butcitednostudiesopposingtheidea,eventhoughanumberhadbeenpublished.Notallreviewsaresobiased,buttherearefewrulesregardingselectionofstudiesforreview.

• Lackofaquantitativesummaryofresearchfindings.Reviewsdon’ttellusabouthowlargetheeffectis.

“Vote-counting”isastepintherightdirection

Dividestudiesintotwocategories:thosethatyieldedastatisticallysignificantresultsupportingtheresearchhypothesis,andthosethatdidnot.Theproportionsofstudies‘voting’fororagainstthehypothesisarethencounted.

Limitationsofvote-counting

• Bycountingonlythestatisticallysignificantstudiesvote-countingignoresallthequantitativeinformationaboutthemagnitudesofeffects.

• Tooconservative.“Votes”areaffectedbythepowerofindividualstudies,whichmaybeweak.

• Significancelevelbyitselfdoesn’tindicatewhethertwoormorestudiesobtainedthesameoutcome.

• Themagnitudeoftheeffectisdownplayed.

• Itisdifficulttoquantifytheeffectsofpublicationbias.

• Methodisunabletoweightheeffectofstudiesdifferinginsamplesize,andthereforepower.

Limitationsofvote-counting

• TheAntiplateletTrialists’Collaboration(1994)conductedameta-analysisof142randomizedexperimentstestingwhethertakingaspirinorotherantiplateletmedicationfollowingastrokeormyocardialinfarction(“heartattack”)reducedtheriskoffuturestroke.TotalN>70,000.

• Thevote:19of142studiesshowedastatisticallysignificantlybetterresultforpatientsonantiplatelettherapythanforthecontrolpatients.Twoofthe142studiesshowedasignificantlyworserateofvasculareventswithaspirintreatment.

• Yet14.7%(5400/36,711)ofpatientsinthecontrolgroupshadsubsequentvascularevents,comparedwith11.4%(4183/36,536)inthetreatedgroup.Smalleffectbutreal,accordingtometa-analysismethods.Thisconclusionsavedmanylives.

Meta-analysis,the“analysisofanalyses”

Meta-analysisreferstothestatisticalsynthesisofresultsfromaseriesofstudies(Borensteinetal2009).Themethodinvolvescompilingallknownscientificstudiesestimatinganeffectandquantitativelycombiningthemtogiveanoverallestimateoftheeffect.Meta-analysisallowsustogeneralize.Itletsusdeterminehowfrequent,howimportant,andhowconsistenteffectsareacrossavarietyofsystems. Meta-analysisgetspasttheoccasionalsensationalresult(theoneyoureadaboutinthenewspaper)toanobjectiveassessmentofalltheevidence.

Meta-analysis,the“analysisofanalyses”

Camefrommedicalresearch,inwhichallstudiesareallofthesamespecies(humans).Hereisa“forestplot”fromtheAntiplateletTrialists’Collaboration.

Meta-analysis,the“analysisofanalyses”

Ecologistsandevolutionarybiologistsattempttogeneralizeacrossamuchwiderrangeofspeciesandsystems.Thisismorechallengingthanstudiescarriedoutonasinglespecies(e.g.,humans).

Example1:Meta-analysisoftheTransylvaniaeffect

• Manypeoplebelievethatafullmooncanaffecthumanbehavior.ThewordlunacyisderivedfromtheLatinluna,moon.

• Legendsofstrangehappenings,suchaswerewolvesandvampires,havebeenconnectedtofullmoonsforcenturies.

• LordBlackstone,an18th-centuryEnglishjurist,wasthefirsttodefineaconditionofmadnessexacerbatedbythelunarcycle:“Alunatic,ornoncomposmentis,isproperlyonewhohathlucidintervals,sometimesenjoyinghissensesandsometimesnotandthatfrequentlydependinguponthechangesofthemoon.”

• RottonandKelly(1985)showedthat50%ofuniversitystudentsbelievedthatpeopleactstrangelyduringafullmoon.

• Vance(1995)reportedthatasmanyas81%ofmentalhealthprofessionalsbelievedthatthefullmoonaltersindividualbehaviour.

Example1:Meta-analysisoftheTransylvaniaeffect

RottonandKelly(1985)carriedoutameta-analysisofstudiescorrelatinghomiciderates,psychiatrichospitaladmissions,suiciderates,crisiscalls,etc.Theaverageeffectsizerwassmallerthan0.01.

Howtocarryoutameta-analysis

1. Definethequestionandscope.

• Anarrowquestionappliedtoahomogeneousgroup?“Doesaspirinreduceincidenceofmyocardialinfarction?”

• Oraheterogeneoussetofstudiesorvariables?“Howmuchgeneticvariationexistsinpopulationsforbehavioraltraits?”

• Onlyexperimentswithcontrolsandrandomization?Onlyreplicatedexperiments?Onlyexperimentswithblinding?

• Itmaybebesttoadoptareasonablywidescopeandinvestigatelaterwhetherdifferencesbetweenmethodsleadtodifferenteffectsoverall.

Howtocarryoutameta-analysis

2. Literaturesearch,gatherdata.

• Makeitexhaustivetoavoidbias.• Easily-foundstudiesaredifferentfromthosethatwecannotfindeasily.Studiesfindinglarge,statisticallysignificanteffectsaremorelikelytobepublished,morelikelytobein“first-rate”journals,andmorelikelytobereferencedinotherarticles.

• Statisticaltechniquesexisttoaccountpartiallyforpublicationbias(funnelplots)buttheydonotreplaceanexhaustivesurvey.

• Decidewhetherto(holdyournoseand)includestudiesofapparentlypoorquality.Failuretohavewell-definedcriteriacanleadtobias(wearemorelikelytodiscardapoorstudyifitdisagreeswithourpethypothesis).

Howtocarryoutameta-analysis

• Ideally,thedataobtainedshouldallbeindependent,butnon-independenceofvarioussortscreepsin(e.g.multiplestudiesbythesamelab).

• Asinglestudymayprovidemeasurementsonmultiplespecies,ormeasurementsofmultipleresponsesonthesamespecies.Includethemallortakeasummarymeasure?

• Oneorasmallnumberofspecies(e.g.,greattit)orsystems(e.g.,intertidalzone)maybeoverrepresentedintheliterature.Treatthemallasindependent?

• Itmaybeworsetoleavedataout,ortakesummarymeasures,thantothroweverydatapointintotheanalysis.

Example2:Testosteronevsaggression

Booketal.(2001)asked“Aretestosteronelevelsandaggressioncorrelatedinhumanmales?”Itincludedahugediversityoftypesofstudies:

• levelsoftestosteroneinprisonersconvictedofviolentcrimescomparedtothoseofprisonersconvictedofpropertycrimes

• levelsoftestosteroneinuniversitystudentscomparedwiththeiranswerstoquestionnairesthataskedthemforlevelsofagreementtostatementslike“Ifsomebodyhitsme,Ihitback.”

• levelsofaggressionin!KungSanmalesasdeterminedbycounting“theirscarsandsometimesstillopenwoundsintheheadregion.”

• drunkenFinnishspouse-abuserscomparedtodrunkenFinnsdrinkingquietlyinabar.

• membersof“rambunctious”fraternitiescomparedto“responsible”fraternities

Example2:Testosteronevsaggression

Below is the “funnel plot” of studies comparing human aggression to levels oftestosterone. The curves show the approximate boundaries of the critical regionsthatwouldrejectthenullhypothesisinanyonestudywithα=0.05.

Howtocarryoutameta-analysis

3. Calculateaneffectsizethatcanbecombinedacrossstudiestoproduceaquantitativesummaryofthefindings.

• Correlationcoefficientriscommonlyusedthoughnotalwaysideal,becauseeffectsizedependsontherangeofthedata.

• Oddsratio–usedinhighlyhomogeneousstudies(oddsratiosintestsofaspirinandmyocardialinfarctions).

• Responseratio:𝑅 = 𝑌$/𝑌& orlogofresponseratio:ln(𝑅)

• Standardizedmeandifference,Cohen’sdorHedges’g:

𝑔 =𝑌$ − 𝑌&𝑠pooled

𝐽(𝑚)

sisthepooledsamplevarianceandJ(m)isasmall-samplebiascorrection.

Example3:Effectivenessofmarinereserves

Halpern(2003)usedthelogofresponseratiotocomparemarinereservestocomparisonareas(orthesameareabeforereserveestablishment)inabundanceanddiversityoffishand/orinvertebrates

Howtocarryoutameta-analysis

4. Statisticalinferenceonaverageeffectsize.

Fixedeffectsmodels

• Mostcommonlyusedinmedicalstudies.

• Assumesthatthemultiplestudieshavethesamemean,differingonlybecauseofsamplingerror.Ifeverystudywereinfinitelylarge,everystudywouldyieldanidenticalresult.Noheterogeneityamongthestudies.

• Perhapsneverjustifiedunlessallstudiesconductedsimilarlyandonthesamespecies.Thisisrarelythecaseinecologyandevolution.

Howtocarryoutameta-analysis

4.Statisticalinferenceonaverageeffectsize.

Random(mixed)effectsmodels

• Randomvariationispresentamongmeansofstudiesinadditiontosamplingerror.

• Individualstudiesarethereforeestimatingdifferenttreatmenteffects.

• Mostinterestisfocusedonthecentralvalue,ormean,ofthedistributionofeffects.

• Buttheideaofarandomeffectsmeta-analysisistounderstandthedistributionofeffectsacrossdifferentstudies.

Howtocarryoutameta-analysis

Fixedeffectmodel

Effectsizeofeachstudyiis

where

€

Θistheone“true”effectsize.

Randomeffectmodel

Effectsizeofstudyiis

whereμisthegrandmeanand

€

ζi isthedeviationofthe“true”effectsizeofstudyifromthegrandmean.

Thedifferenceaffectshoweachstudyisweighedwhencalculatingtheaverageeffectsizeoverallstudies.We’lldothisintheworkshop.

Unfortunately,lmecan’tbeusedforrandomeffectsmeta-analysisinR,becauseitwon’tcalculatethenecessaryweights.Otherpackages(e.g.,meta)areavailable.

€

Yi =Θ + ε i

€

Yi = µ + ζi + ε i

Howtocarryoutameta-analysis

5. Lookforeffectsofstudyquality.Forexample,areeffectsizesdifferentonaveragebetweenstudiesthatincludedblindingandthosethatdidnot?

6. Lookforassociationswithvariablesthatmightexplainheterogeneityofeffect

sizesamongstudies.Forexample,doestheaverageeffectsizedifferbetweenstudiescarriedoutonwomensubjectsandthoseonmalesubjects?

Example4:Meta-analysisofcompetitioninfieldexperiments

Gurevitchetal(1992)studyofinter-andintra-specificcompetition,lookingonlyatstudiespublishedin1980’s

Example4:Meta-analysisofcompetitioninfieldexperiments

Theylookedforeffectsofstudyquality

Example4:Meta-analysisofcompetitioninfieldexperiments

Theylookedforassociationswithvariablesthatmightexplainvariationineffectsize

File-drawerproblemInmeta-analysis,thedifficultiescausedbypublicationbiasarecalledthefile-drawerproblem,inreferencetotheunknownstudiessittingunavailableinresearchers’filedrawersorhiddeninobscurejournals.Thefile-drawerproblemisthepossiblebiasinestimatesandtestscausedbypublicationbias.

FunnelplotFunnelplotscangiveanindicationofthebiasresultingfromsmallstudies.SomaandGaramszegi(2011)usedtheTrimfillalgorithmtofillinhypotheticalmissingstudiesinthefunnelplottoachievetheoreticalsymmetry.

Fail-safenumberThefail-safenumbercalculateshowmanymissingstudieswouldbeneededtochangetheoverallresultofthemeta-analysis.Vilàetal(2011)estimatedthenumberofstudiesthatwouldhavetobeaddedtochangetheresultsoftheirinvasiveplantmeta-analysisfromsignificanttonon-significantas37,689.Thiswastooimplausible,sotheyconcludedthattheirestimateswerereliable.(Fig1a:toplinereferstototalplantproduction;otherlinesareeffectsonnativeplants)

Makeyourresultsaccessibletometa-analysisManypublishedpapersdonotreportenoughinformationformeta-analyststoextractthenumbersthattheyneed.Asaresult,manyotherwiserelevantpapershavetobediscarded.Don’tletthishappentoyourwork.• Alwaysgivesizesofeffectsandtheirstandarderrors.AP-valuebyitselfisuseless.

• Giveestimatesofthemeansandstandarddeviationsoftheimportantvariables.

• Alwaysindicateyoursamplesizesordegreesoffreedom.• Makethedataaccessible.PublishtherawdatainthepaperordeposittoanonlinearchivesuchasDryad.

Considerameta-analysisforyourfirstthesischapter

Often,thefirstchapterofathesisisareviewoftheliterature.Ifyourreviewisthorough,andyoukepttrackoftheimportantquantitiesandfeatureofeachstudy,youmayhaveenoughforaquantitativereview–yourownmeta-analysis.

Bestpracticesformeta-analysis

Bestpracticesformeta-analysis

Rtoolforbestpractices

Discussionpapernextweek:

RichmanandPrice:Evolutionofecologicaldifferencesinwarblers(multivariateanalysis)

Downloadfrom“Handouts”taboncoursewebsite.

Presenters:Vince&Rafael

Moderators:Sarah&Takuji