Embed Size (px)
Citation preview
ReproducibleComputationalWorkflowswithContinuousAnalysis
BrettK.Beaulieu-Jones1,andCaseyS.Greene2,+
1GenomicsandComputationalBiologyGraduateGroup.PerelmanSchoolofMedicine.UniversityofPennsylvania.2DepartmentofSystemsPharmacologyandTranslationalTherapeutics.PerelmanSchoolofMedicine.UniversityofPennsylvania+CorrespondingAuthoremail:[email protected]:3400CivicCenterBlvd.10-131SCTRPhiladelphia,PA19103
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
Abstract
Reproducingexperimentsisvitaltoscience.Beingabletoreplicate,validateandextendpreviousworkalsospeedsnewresearchprojects.Reproducingcomputationalbiologyexperiments,whicharescripted,shouldbestraightforward.Butreproducingsuchworkremainschallengingandtimeconsuming.Intheidealworldwewouldbeabletoquicklyandeasilyrewindtotheprecisecomputingenvironmentwhereresultsweregenerated.Wewouldthenbeabletoreproducetheoriginalanalysisorperformnewanalyses.Weintroduceaprocesstermed“continuousanalysis”whichprovidesinherentreproducibilitytocomputationalresearchataminimalcosttotheresearcher.ContinuousanalysiscombinesDocker,acontainerservicesimilartovirtualmachines,withcontinuousintegration,apopularsoftwaredevelopmenttechnique,toautomaticallyre-runcomputationalanalysiswheneverrelevantchangesaremadetothesourcecode.Thisallowsresultstobereproducedquickly,accuratelyandwithoutneedingtocontacttheoriginalauthors.Continuousanalysisalsoprovidesanaudittrailforanalysesthatusedatawithsharingrestrictions.Thisallowsreviewers,editors,andreaderstoverifyreproducibilitywithoutmanuallydownloadingandrerunninganycode.Exampleconfigurationsareavailableatouronlinerepository(https://github.com/greenelab/continuous_analysis).
TheCurrentStateofReproducibility
Leadingscientificjournalshavetargetedreproducibilitytoincreasereaders’
confidenceinresultsandreduceretractions1–5.Inarecentsurvey,90%ofresearchersacknowledgedareproducibilitycrisis6.Researchthatusescomputationalprotocolsshouldbeparticularlyamenabletoreproducibleworkflowsbecauseallofthestepsarescriptedintoamachine-readableformat.Butwrittendescriptionsofcomputationalapproachescanbedifficulttounderstandandmaylackrequiredparameters.Evenwhenresultscanbereproduced,theprocessoftenrequiresasubstantialtimeinvestmentandhelpfromtheoriginalauthors.Garijoetal.7estimateditwouldtake280hoursforanon-experttoreproduceapaperdescribingacomputationalconstructionofadrug-targetnetworkforMycobacteriumtuberculosis8.Thesearethegoodscenarios:theresultsbehindmostcomputationalpapersarenotreadilyreproducible7,9–11.
Thepracticeof“openscience”hasbeendiscussedasameanstoaidreproducibility3,12.Inopensciencethedataandsourcecodeareshared.Sharingcanalsoextendtointermediateresultsandprojectplanning13.Sharingdataandsourcecodeiscurrentlynecessarybutnotsufficienttomakeresearchreproducible.Evenwhencodeanddataareshared,itremainsdifficulttoreproduceresultsduetodifferingcomputingenvironments,operatingsystems,librarydependenciesetc.Itiscommontouseoneormoreopensourcelibrariesonaproject,andresearchcodequicklybecomesdependentonoldversionsoftheselibrariesassoftwareadvances14.Theseoldorbrokendependenciesmakeitdifficultforreadersand
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
reviewerstorecreatetheenvironmentoftheoriginalresearchers,whethertovalidateorextendtheirwork. Anexampleofwheresharingdatadoesnotautomaticallymakesciencereproducibleoccursinthemoststandardofplaces:differentialgeneexpressionanalysis.Suchanalysesareroutine.Ourunderstandingofthegenome,includingtranscriptomeannotations,haveimprovedandupdatedprobesetdefinitionsareavailable15.Analysesrelyingonunspecifiedprobesetdefinitionscannotbereproducedusingcurrentdefinitions.
WeanalyzedthefifteenmostrecentlypublishedpapersthatciteDaietal.,acommonsourceforcustomchipdescriptionfiles(CustomCDF),thatwereaccessibleatourinstitution16–31.WeidentifiedthesemanuscriptsusingWebofScienceonMay31,2016.WerecordedthenumberofpapersthatcitedaversionofCustomCDF,aswellaswhichversionwascited.Weexpectthisanalysistoprovideanupperboundonreproduciblework:thesepapersspecificallycitedthesourceoftheirCDFs.Ofthesefifteenpapers,nine(60%)specifiedwhichversiontheyused.Thesenineusedversions11,15,16,17,18,and19oftheBrainArrayCustomCDF.
Thisinitialanalysiswasperformedbasedonarticlerecencywithoutregardtoarticleimpact.Todeterminetheextenttowhichthisissueaffectshighimpactpapers,weperformedaparallelevaluationforthetenmostcitedpapers32–41thatciteDaietal.WedeterminedthetenmostcitedpapersusingWebofScienceonMay31,2016.Ofthesetenpapers,one38(10%)specifiedwhichversionoftheCustomCDFwasused.Thatpaperusedversion11oftheBrainArrayCustomCDF. Wesoughttodeterminewhichversionswerecurrentlyinuseinthefield.Weaskedthreeindividualswhoperformedmicroarrayanalysisrecently,andweaccessedandevaluatedtwoclustersystemsusedforprocessingdata.Wefoundthateachindividualhadoneofthethreemostrecentlyreleasedversionsinstalled(18,19,and20),andversions18and19wereinstalledonclustersystems. ToevaluatetheimpactofdifferingCDFversions,wedownloadedarecentlypublishedpublicgeneexpressiondataset(GEOSeriesAscensionnumberGSE47664).ThisexperimentexamineddifferentialexpressionbetweennormalHeLacellsandHeLacellswithTIA1andTIARknockeddown42.Weperformedaparallelanalysisusingeachofthethreeversionsthatwefoundinstalledonmachinesthatwecouldaccess(18,19,and20).Eachversionidentifiesadifferentnumberofsignificantlyalteredgenes(Figure1A),demonstratingthechallengeofreproducibleanalysis.WesimulatedaparallelanalysisofdifferentialexpressionusingDockercontainersonmismatchedmachines43.ThisspecifiestheCDFversionandproducesthesamenumberandsetofdifferentiallyexpressedgenesforagivenversionacrossmachines(v18exampleinFigure1B).HadcontinuousanalysisbeenusedforpaperscitingtheBrainArrayCustomCDFtheircomputationalresultswouldbeeasilyreplicated.
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
Figure1.Currentstateofresearchcomputingvs.container-basedapproaches.A.)Thestatusquorequiresareaderorreviewertofindandinstallspecificversionsofdependencies.Thesedependenciescanbecomedifficulttofindandmaybecomeincompatiblewithnewerversionsofothersoftwarepackages.Differentversionsofpackagesidentifydifferentnumbersofsignificantlydifferentiallyexpressedgenesfromthesamesourcecodeanddata.B.)Containersdefineacomputingenvironmentthatcapturesdependencies.Incontainer-basedsystems,theresultsarethesameregardlessofthehostsystem.
ContinuousAnalysisinComputationalWorkflows
Wedevelopedcontinuousanalysistoproduceaverifiableend-to-endrunofcomputationalresearchwithminimalstart-upcosts.Incontrastwiththestatusquo,continuousanalysispreservesthecomputingenvironmentandmaintainstheversionsofdependencies.Wedescribedthebenefitsofcontainerizedapproachesabove,butmaintaining,runninganddistributingDockerimagesmanuallywouldbecometimeconsuming.IntegratingDockerintoacontinuousscientificanalysispipelinemeetsthreecriteria:(1)anyonecanre-runcodeinacomputingenvironmentmatchingtheoriginalauthors(SupplementalFigure1);(2)readersandreviewerscanfollowexactlywhatwasdoneinan“audit”fashionwithoutrunningcode(SupplementalFigure2&3);and(3)thesolutionimposeszeroto
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
Figure2.Continuousanalysiscanbesetupinthreeprimarysteps(numbered1,2,and3).(1)TheresearchercreatesaDockercontainerwiththerequiredsoftware.(2)TheresearcherconfiguresacontinuousintegrationservicetousethisDockerimage.(3)Theresearcherpushescodethatincludesascriptcapableofrunningtheanalysesfromstarttofinish.ThecontinuousintegrationproviderrunsthelatestversionofcodeinthespecifiedDockerenvironmentwithoutmanualintervention.ThisgeneratesaDockercontainerwithintermediateresultsthatallowsanyonetorerunanalysisinthesameenvironment,producesupdatedfigures,andstoreslogsdescribingeverythingthatoccurred.Exampleconfigurationsareavailableinthesupplementarymaterialsaswellasouronlinerepository(https://github.com/greenelab/continuous_analysis).Becausecodeisruninanindependent,reproduciblecomputingenvironmentandproducesdetailedlogsofwhatwasexecuted,thispracticereducesoreliminatestheneedforreviewerstore-runcodetoverifyreproducibility.
minimalcostintermsoftimeandmoneyontheresearcher,dependingontheircurrentresearchprocess. Continuousanalysisextendscontinuousintegration44,acommonpracticeinsoftwaredevelopmentanddeployment.Continuousintegrationisasoftwaredevelopmentworkflowthattriggersanautomatedbuildprocesswheneverdeveloperschecktheircodeintoasourcecontrolrepository.Thisautomatedbuildprocessrunstestscriptsiftheyexist.Thesetestscancatchbugsintroducedintosoftware.Softwarethatpassestestsisautomaticallydeployedtoremoteservers.
Forcontinuousanalysis(Figure2),werepurposetheseservicesinordertoruncomputationalanalyses,updatefigures,andpublishchangestoonlinerepositorieswheneverrelevantchangesaremadetothesourcecode.Whenanauthorisreadytoreleasecodeorpublishtheirworktheycanexportthemostrecentcontinuousintegrationrun.Becausethisprocessgeneratesresultsinacleanandclearlydefinedcomputingenvironmentwithoutmanualintervention,reviewerscanbeconfidentthattheanalysesarereproducible.
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
Ineachprojectwemaintaindependencieswiththefreeopen-source
softwaretoolDocker45.Dockerdefinesan“image”thatallowsuserstodownloadandrunacontainer,aminimalistvirtualmachinewithapredefinedcomputingenvironment.Dockerimagescanbeseveralgigabytesinsize,butoncedownloadedcanbestartedinamatterofsecondsandhasminimaloverhead14.Inaddition,Dockerimagescanbeeasilytaggedtocoincidewithsoftwarereleasesandpaperrevisions.Atthetimeofsubmission,authorscanrunthe`dockersave`commandtoexportastaticfilethatcanbeuploadedtoservicessuchasFigshareorZenodotoreceiveaDOI.Forexample,wehaveuploadedourcontinuousanalysisenvironmentfortheexamplesinthispaper46.
Tosetupcontinuousanalysis,aresearcherneedstodothreethings.First
theymustcreateaDockerfile,whichspecifiesalistofdependencies.Second,theyneedtoconnectacontinuousintegrationservicetotheirversioncontrolsystemandprovidethecommandstoruntheiranalysis.Finally,theyneedtocommitandpushchangestotheirversioncontrolsystem.Manyresearchersalreadyperformthefirstandthirdtasksintheirstandardworkflow.
Thecontinuousintegrationsystemwillautomaticallyrerunthespecified
analysiswitheachchange,preciselymatchingthesourcecodeandresults.Itcanalsobesettolistenandrunonlywhenchangesaremarkedinaspecificway,e.g.bycommittingtoaspecific‘staging’branch.Forthefirstproject,thisprocesscanbeputintoplaceinlessthanaday.Forsubsequentprojects,thiscanbedoneinunderanhour.SettingupContinuousAnalysis
WehavecreatedaGitHubrepositorywithinstructionsforpaid,local,and
cloud-basedcontinuousanalysissetups47.Thesearefullydetailedinthesupplementarymaterialsandonlinerepository.HerewedescribehowcontinuousanalysiscanbesetupusingthefreeandopensourceDronesoftwareonaresearcher’spersonalcomputerandconnectedtotheGitHubversioncontrolservice.Thissetupisfreetousers.
1. InstallDockeronthecomputer.2. PulltheDroneimageviadocker:
sudodockerpulldrone/drone:latest
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
Figure3.RegisteranewapplicationfortheDronecontinuousintegrationserver.SetthehomepageURLtobetheIPaddressoftheDronecomputer.SetthecallbackURLtothesameIPaddressfollowedby/authorize.
3. CreateanewapplicationinGitHub(Figure3).
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
Figure4.RegisteranewapplicationfortheDronecontinuousintegrationserver.ThepayloadURLshouldbeintheformatofyour-ip/api/hook/github.com/client-id
4. AddawebhooktotheGitHubproject(Figure4).Thiswillnotifythecontinuousintegrationserverofanyupdatespushedtotherepository.
5. CreateaconfigurationfileontheDronecomputerat/etc/drone/dronerc
fillingintheclientinformationprovidedbyGitHubREMOTE_DRIVER=githubREMOTE_CONFIG=https://github.com?client_id=....&client_secret=....
6. Runthedronecontainer
sudodockerrundrone/drone:latest
Continuousanalysiscanbeperformedwithdozensoffullserviceproviders
oraprivateinstallationonalocalmachine,clusterorcloudservice47.Fullserviceproviderscanbesetupinminutesbutmayhavecomputationalresourcelimitsormonthlyfees.Privateinstallationsrequireconfigurationbutcanscaletoalocalclusterorcloudservicetomatchthecomputationalcomplexityofallwalksofresearch.Withfree,open-sourcecontinuousintegrationsoftware48,computingresourcesaretheonlyassociatedcosts.UsingContinuousAnalysis Aftersetup,runningcontinuousanalysisissimpleandfitsintoexistingresearchworkflowsthatusesourcecontrolsystems.Wehaveusedcontinuous
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
analysisinourownwork49.Wehavealsopreparedthreeexamplerepositories(detailedinsupplementalmaterials):
1. Anexampledemonstratingthesetupofcontinuousanalysiswithawidevarietyofservicesandconfigurations(highlightedbelow).
2. Aneasytofollowbasicphylogenytreebuildingexample,combiningsequencealignmentusingMAFFT50,formatconversionusingEMBOSSSeqret51,andtreecalculationanddrawingusing.
3. AnRNAexpressionanalysisworkflowexaminingorganoidmodelsofpancreaticcancerinmicebasedonworkfromBojetal.52usingdetailsandsourcecodepublishedbyBalli53.Thisexampleshowstheabilityofcontinuousanalysistoscaletolargecomputations.Thisexampleuseskallisto54,limma55,56,andsleuth57toanalyze150GBofgeneexpressiondataandapproximately480millionreads.
Todemonstratethesetupprocessanddifferentconfigurationsofcontinuousanalysisweshowasimpleexampleofcontinuousanalysiswithkallisto.TherecentlypublishedsoftwaretoolkallistoquantifiestranscriptabundanceinRNA-seqdata.Ourexamplere-runstheexamplesprovidedinkallistowitheachcommittoarepository.
1. Addascriptfiletore-runcustomanalysis.ForDrone,thisisa.drone.ymlfilethatspecifiescommandstoruneachstepoftheanalysis.AnexampleconfigurationisavailableinthecontinuousanalysisGitHubrepositoryaswellasthesupplementalmaterials.
2. Commitchangestothesourcecontrolrepository.3. PushchangestoGitHub.
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
Figure5.AuditlogsfromacontinuousintegrationrunwiththeserviceShippableforthekallistoexample. Theconfiguredcontinuousintegrationserviceautomaticallyrunsthespecifiedscript.Weconfiguredthistoreruntheanalysis,regeneratethefigures,andcommitupdatedversionstotherepository.Theserviceprovidesacompleteauditlogofwhatwasruninthecleancontinuousintegrationenvironment(Figure5).Bygeneratingandpushingupdatedfigures,thisprocessalsogeneratesacompletechangelogforeachresult(Figure6).Interactivedevelopmenttools,suchasJupyter58,59,RMarkdown60,61andSweave62canbeincorporatedtopresentthecodeandanalysisinalogicalgraphicalmanner.Forexample,werecentlyusedJupyterwithcontinuousanalysisinourownpublication63andcorrespondingrepository49.
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
Figure6.ResultingfiguresfromtherunarecommittedbacktoGithubwherechangesbetweenrunscanbeviewed.A.)Theeffectofaddinganadditionalgene(HumanTw2)toaphylogenetictree-buildingexample.B.)Theeffectofaddinganadditionalgene(mt8)toanRNA-seqdifferentialexpressionexperimentPCAplot.
Insummary,continuousanalysisprovidestheresultsofaverifiableend-to-
endrunina“clean”environment.Becausecontinuousanalysisrunsautomaticallyinthebackground,notransitionisneededbetweentheexplorationandpublicationphasesofascientificproject.Theaudittrailprovidedbycontinuousanalysisallowsreviewersandeditorstoprovidesoundjudgmentonreproducibilitywithoutalargetimecommitment.Ifreadersorreviewerswouldliketore-runthecodeontheirown(e.g.tochangeaparameterandevaluatetheimpactonresults),theycaneasilydosowiththeDockercontainercontainingthefinalcomputingenvironmentandintermediateresults.Versioncontrolsystemsprovidethecapabilitytowatchforupdates.Readerscan“star”or“watch”arepositoryonservicessuchasGithub,Gitlab,andBitbuckettobeautomaticallynotifiedofchangesandupdatedruns.Wideadoptionofthesesystemsthroughoutthepublicationprocesscouldallowreviewersandeditorstoautomaticallybenotifiedofupdatedresults. Continuousanalysisprovidesanaudittrailforreproducibleanalysesofcloseddata. Continuousanalysiscanbeevenmorepowerfulwhenworkingwithcloseddatathatcannotbereleased.Withoutcontinuousanalysis,reproducing
C. D.
AdditionalSample
A. B.AdditionalSample
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
computationalanalysesbasedoncloseddataisdependentontheoriginalauthorscompletelyandexactlydescribingeachstep,aprocessthatmaybeanafterthoughtandrelegatedtoextendedmethods.Readersmustthendiligentlyfollowcomplexwritteninstructionswithoutintermediateconfirmationtheyareontherighttrack.Thecontainersproducedduringcontinuousanalysisincludeamatchingenvironmentforreplicationaswellasintermediateresults.Thisallowsreaderstodeterminewheretheirresultsdivergefromtheoriginalworkandtodeterminewhetherdivergenceisduetosoftware-basedordata-baseddifferences.Bestpracticeswithcontinuousanalysis Wesuggestadevelopmentworkflowwherecontinuousanalysisrunsonlyonasinglebranch(SupplementalFigure4).Researcherscanpushtothisbranchwhentheybelievetheyarereadyforareleasetoavoidrunningthefullprocessduringincompleteupdates.Iftheupdatestothisbranchsucceed,thechangesarethenautomaticallycarriedovertothemasterorproductionbranchandreleased.WerecommendexportingboththebeforeandafterprocessingDockerimagesanduploadingtoanarchivalservicelikeFigshareorZenodo.Thearchivedimagescanthenbecitedtoguidereaderstotheversionusedinthemanuscript46.Forconvenience,theimagescanalsobesharedthroughtheDockerHubregistry. Itmaycurrentlybeimpracticaltousecontinuousanalysisforgenericpreprocessingstepsinvolvingverylargedataoranalysesrequiringparticularlyhighcomputationalcosts.Inparticular,stepsthattakedaystorunorincursubstantialcostsincomputationalresourcesmaynotbeamenablewithexistingproviders64.Oneday,continuousanalysissystemsspecificallydesignedforscientificworkflowsmayfacilitatereproducibleworkflowsinthesesettings.Fornow,researchersmayneedtousediscretionwhenpreprocessingviacontinuousanalysis,asitmaybecomputationallyintractabletoreanalyzeaftereachcommittoastagingbranch.Researchersmayelecttorunonlythefinalworkflowthroughthisprocess,ormayelecttoemploycontinuousanalysisafterstandardbutcomputationallyexpensivepreprocessingstepsarecompleted. Forsmalldatasetsandlessintensivecomputationalworkflowsitiseasiesttouseafullservicecontinuousintegrationservice.Theseserviceshavethesmallestsetuptimes.Withprivatedataorwhendatasizeandcomputationalcomplexityscaleitbecomesnecessarytosetupalocalprivatelyhostedcontinuousintegrationserver.Clusterorcloudbasedcontinuousintegrationserverscanhandlethelargestworkflows.Theimpactofreproduciblecomputationalresearch
Reproducibilitycanhavewide-reachingbenefitsfortheadvancementofscience.Forauthors,easilyreproducibleworkisasignofqualityandcredibility.Continuousanalysisaddressesthereproducibilityofcomputationallyanalysesinthenarrowsense:generatingthesameresultsfromthesameinputs.Itdoesnot
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
solvereproducibilityinthebroadersense:howrobustresultsaretoparametersettings,startingconditionsandpartitionsinthedata.Continuousanalysislaysthegroundworkneededtoaddressreproducibilityandrobustnessoffindingsinthebroadsense.AcknowledgementsThisworkwassupportedbytheGordonandBettyMooreFoundationunderaDataDrivenDiscoveryInvestigatorAwardtoCSG(GBMF4552)andsupportedbyaCommonwealthUniversalResearchEnhancement(CURE)ProgramgrantfromthePennsylvaniaDepartmentofHealth.WewouldliketothankDavidBalliforprovidingtheRNA-seqanalysisdesign,KatieSiewertforprovidingthephylogeneticanalysisdesign,andAlexWhanforcontributingaTravis-CIimplementation.References1. Rebootingreview.NatBiotech.2015;33(4):319.
http://dx.doi.org/10.1038/nbt.3202.2. Softwarewithimpact.NatMeth.2014;11(3):211.
http://dx.doi.org/10.1038/nmeth.2880.3. PengRD.ReproducibleResearchinComputationalScience.Science(80-).
2011;334(6060):1226-1227.doi:10.1126/science.1213847.4. McNuttM.Reproducibility.Science(80-).2014;343(6168):229.
http://science.sciencemag.org/content/343/6168/229.abstract.5. Illuminatingtheblackbox.Nature.2006;442(7098):1.
http://dx.doi.org/10.1038/442001a.6. BakerM.1,500scientistsliftthelidonreproducibility.Nature.
2016;533(7604):452-454.doi:10.1038/533452a.7. GarijoD,KinningsS,XieLL,etal.Quantifyingreproducibilityincomputational
biology:Thecaseofthetuberculosisdrugome.PLoSOne.2013;8(11).doi:10.1371/journal.pone.0080278.
8. KinningsSL,XieLL,FungKH,JacksonRM,XieLL,BournePE.TheMycobacteriumtuberculosisdrugomeanditspolypharmacologicalimplications.PLoSComputBiol.2010;6(11).doi:10.1371/journal.pcbi.1000976.
9. BellAW,DeutschEW,AuCE,etal.AHUPOtestsamplestudyrevealscommonproblemsinmassspectrometry-basedproteomics.NatMethods.2009;6(6):423-430.doi:10.1038/nmeth.1333.
10. IoannidisJPA,AllisonDB,BallCA,etal.Repeatabilityofpublishedmicroarraygeneexpressionanalyses.NatGenet.2009;41(2):149-155.doi:10.1038/ng.295.
11. HothornT,LeischF.Casestudiesinreproducibility.BriefBioinform.2011;12(3):288-300.doi:10.1093/bib/bbq084.
12. GrovesT,GodleeF.Openscienceandreproducibleresearch.BMJ.2012;344.doi:10.1136/bmj.e4383.
13. ThinkLab.https://thinklab.com/.AccessedJanuary1,2016.14. BoettigerC.AnintroductiontoDockerforreproducibleresearch,with
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
examplesfromtheRenvironment.ACMSIGOPSOperSystRevSpecIssueRepeatabilitySharExpArtifacts.2015;49(1):71-79.doi:10.1145/2723872.2723882.
15. DaiM,WangP,BoydAD,etal.Evolvinggene/transcriptdefinitionssignificantlyaltertheinterpretationofGeneChipdata.NucleicAcidsRes.2005;33(20):e175.doi:10.1093/nar/gni179.
16. KopljarI,GallacherDJ,DeBondtA,etal.FunctionalandTranscriptionalCharacterizationofHistoneDeacetylaseInhibitor-MediatedCardiacAdverseEffectsinHumanInducedPluripotentStemCell-DerivedCardiomyocytes.StemCellsTranslMed.2016;5(5):602-612.doi:10.5966/sctm.2015-0279.
17. KarpińskiP,FrydeckaD,SąsiadekM.Reducednumberofperipheralnaturalkillercellsinschizophreniabutnotinbipolardisorder.Brain,Behav.2016.http://www.sciencedirect.com/science/article/pii/S0889159116300265.AccessedMay31,2016.
18. BrummelmanJ,RaevenR,HelmK.TranscriptomesignaturefordampenedTh2dominanceinacellularpertussisvaccine-inducedCD4+TcellresponsesthroughTLR4ligation.Scientific.2016.http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4846868/.AccessedMay31,2016.
19. BilgrauA,EriksenP,RasmussenJ.GMCM:Unsupervisedclusteringandmeta-analysisusinggaussianmixturecopulamodels.JStat.2016.https://www.jstatsoft.org/article/view/v070i02/v70i02.pdf.AccessedMay31,2016.
20. GandinV,MasvidalL,CargnelloM,GyenisL.mTORC1andCK2coordinateternaryandeIF4Fcomplexassembly.Nature.2016.http://www.nature.com/ncomms/2016/160404/ncomms11127/full/ncomms11127.html.AccessedMay31,2016.
21. KilleenA,DiskinM,MorrisD.Endometrialgeneexpressioninhigh-andlow-fertilityheifersinthelatelutealphaseoftheestrouscycleandacomparisonwithmidlutealgeneexpression.Physiological.2016.http://physiolgenomics.physiology.org/content/48/4/306.abstract.AccessedMay31,2016.
22. CollettiN,LiuH,GowerA,AlekseyevY.Tlr3signalingPromotestheinductionofUniquehumanBDca-3DendriticcellPopulations.Front.2016.http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4789364/.AccessedMay31,2016.
23. LeeM,HuangR,TongW.Discoveryoftranscriptionaltargetsregulatedbynuclearreceptorsusingaprobabilisticgraphicalmodel.ToxicolSci.2015.http://toxsci.oxfordjournals.org/content/early/2015/12/07/toxsci.kfv261.abstract.AccessedMay31,2016.
24. TroyN,HollamsE,HoltP.Differentialgenenetworkanalysisfortheidentificationofasthma-associatedtherapeutictargetsinallergen-specificT-helpermemoryresponses.BMCMed.2016.http://bmcmedgenomics.biomedcentral.com/articles/10.1186/s12920-016-0171-z.AccessedMay31,2016.
25. ManiéE,PopovaT,BattistellaA.Genomichallmarksofhomologous
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
recombinationdeficiencyininvasivebreastcarcinomas.JCancer.2016.http://onlinelibrary.wiley.com/doi/10.1002/ijc.29829/full.AccessedMay31,2016.
26. DekkersB,HeH,HansonJ,WillemsL.TheArabidopsisDELAYOFGERMINATION1geneaffectsABSCISICACIDINSENSITIVE5(ABI5)expressionandgeneticallyinteractswithABI3duringArabidopsis.ThePlant.2016.http://onlinelibrary.wiley.com/doi/10.1111/tpj.13118/full.AccessedMay31,2016.
27. HoltP,StricklandD,BoscoA,BelgraveD.DistinguishingbenignfrompathologicTH2immunityinatopicchildren.JAllergy.2015.http://www.sciencedirect.com/science/article/pii/S0091674915013342.AccessedMay31,2016.
28. LückS,WestermarkP.CircadianmRNAexpression:insightsfrommodelingandtranscriptomics.CellMolLifeSci.2016.http://link.springer.com/article/10.1007/s00018-015-2072-2.AccessedMay31,2016.
29. BoscoA,WiehlerS,ProudD.Interferonregulatoryfactor7regulatesairwayepithelialcellresponsestohumanrhinovirusinfection.BMCGenomics.2016.http://bmcgenomics.biomedcentral.com/articles/10.1186/s12864-016-2405-z.AccessedMay31,2016.
30. FauteuxF,HillJ,JaramilloM,PanY,PhanS.Computationalselectionofantibody-drugconjugatetargetsforbreastcancer.Oncotarget.2015.http://europepmc.org/abstract/med/26700623.AccessedMay31,2016.
31. NapolitanoF,SirciF,CarrellaD,BernardoDdi.Drug-setenrichmentanalysis:anoveltooltoinvestigatedrugmodeofaction.Bioinformatics.2016.http://bioinformatics.oxfordjournals.org/content/32/2/235.short.AccessedMay31,2016.
32. CarrollJ,MeyerC,SongJ,LiW,GeistlingerT.Genome-wideanalysisofestrogenreceptorbindingsites.Nature.2006.http://www.nature.com/ng/journal/v38/n11/abs/ng1901.html.AccessedMay31,2016.
33. LupienM,EeckhouteJ,MeyerC,WangQ,ZhangY.FoxA1translatesepigeneticsignaturesintoenhancer-drivenlineage-specifictranscription.Cell.2008.http://www.sciencedirect.com/science/article/pii/S0092867408001189.AccessedMay31,2016.
34. WangQ,LiW,ZhangY,etal.Androgenreceptorregulatesadistincttranscriptionprograminandrogen-independentprostatecancer.Cell.2009.http://www.sciencedirect.com/science/article/pii/S0092867409005170.AccessedMay31,2016.
35. LefterovaM,ZhangY,StegerD.PPARγandC/EBPfactorsorchestrateadipocytebiologyviaadjacentbindingonagenome-widescale.Genes&.2008.http://genesdev.cshlp.org/content/22/21/2941.short.AccessedMay31,2016.
36. TuupanenS,TurunenM,LehtonenR,HallikasO.ThecommoncolorectalcancerpredispositionSNPrs6983267atchromosome8q24conferspotentialtoenhancedWntsignaling.Nature.2009.
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
http://www.nature.com/ng/journal/v41/n8/abs/ng.406.html.AccessedMay31,2016.
37. ObadS,SantosCdos,PetriA,HeidenbladM.SilencingofmicroRNAfamiliesbyseed-targetingtinyLNAs.Nature.2011.http://www.nature.com/ng/journal/v43/n4/abs/ng.786.html.AccessedMay31,2016.
38. HeH,MeyerC,ShinH,BaileyS,WeiG,WangQ.Nucleosomedynamicsdefinetranscriptionalenhancers.Nature.2010.http://www.nature.com/ng/journal/v42/n4/abs/ng.545.html.AccessedMay31,2016.
39. OzsolakF,SongJ,LiuX,FisherD.High-throughputmappingofthechromatinstructureofhumanpromoters.NatBiotechnol.2007.http://www.nature.com/nbt/journal/v25/n2/abs/nbt1279.html.AccessedMay31,2016.
40. ZuoT,WangL,MorrisonC,ChangX,ZhangH,LiW.FOXP3isanX-linkedbreastcancersuppressorgeneandanimportantrepressoroftheHER-2/ErbB2oncogene.Cell.2007.http://www.sciencedirect.com/science/article/pii/S0092867407005454.AccessedMay31,2016.
41. EnardW,GehreS,HammerschmidtK,HölterS.AhumanizedversionofFoxp2affectscortico-basalgangliacircuitsinmice.Cell.2009.http://www.sciencedirect.com/science/article/pii/S009286740900378X.AccessedMay31,2016.
42. NunezM,Sanchez-JimenezC,AlcaldeJ,IzquierdoJM.Long-termreductionofT-cellintracellularantigensrevealsatranscriptomeassociatedwithextracellularmatrixandcelladhesioncomponents.PLoSOne.2014;9(11).doi:10.1371/journal.pone.0113141.
43. Beaulieu-JonesB,GreeneC.ContinuousAnalysisBrainArray:SubmissionReleaseContinuousAnalysisBrainArray:SubmissionRelease.August2016.doi:10.5281/zenodo.59892.
44. DuvallP,MatyasS,GloverA.ContinuousIntegration:ImprovingSoftwareQualityandReducingRisk.;2007.http://portal.acm.org/citation.cfm?id=1406212.
45. Docker.Docker.https://www.docker.com.46. Beaulieu-JonesBK,GreeneCS.ContinuousAnalysisExampleDockerImages.
2016.10.6084/m9.figshare.3545156.v1.47. Beaulieu-JonesBK,GreeneCS.ContinuousAnalysis.GitHubrepository.
https://github.com/greenelab/continuous_analysis.Published2016.48. Drone.io.https://drone.io/.49. Beaulieu-JonesBK.DenoisingAutoencodersforPhenotypeStratification
(DAPS):PreprintRelease.Zenodo.January2016.doi:10.5281/zenodo.46165.50. KatohK,MisawaK,KumaK,MiyataT.MAFFT:anovelmethodforrapid
multiplesequencealignmentbasedonfastFouriertransform.NucleicAcidsRes.2002;30(14):3059-3066.doi:10.1093/nar/gkf436.
51. RiceP,LongdenI,BleasbyA,etal.EMBOSS:theEuropeanMolecularBiologyOpenSoftwareSuite.TrendsGenet.2000;16(6):276-277.doi:10.1016/s0168-
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
9525(00)02024-2.52. BojSF,HwangC-I,BakerLA,etal.OrganoidModelsofHumanandMouse
DuctalPancreaticCancer.Cell.2015;160(1):324-338.doi:10.1016/j.cell.2014.12.021.
53. BalliD.UsingKallistoforexpressionanalysisofpublishedRNAseqdata.https://benchtobioinformatics.wordpress.com/2015/07/10/using-kallisto-for-gene-expression-analysis-of-published-rnaseq-data/.Published2015.AccessedAugust1,2016.
54. BrayNL,PimentelH,MelstedP,PachterL.Near-optimalprobabilisticRNA-seqquantification.NatBiotechnol.2016;34(5):525-527.doi:10.1038/nbt.3519.
55. RitchieME,PhipsonB,WuD,etal.limmapowersdifferentialexpressionanalysesforRNA-sequencingandmicroarraystudies.NucleicAcidsRes.2015;43(7):e47.doi:10.1093/nar/gkv007.
56. SmythGK.Linearmodelsandempiricalbayesmethodsforassessingdifferentialexpressioninmicroarrayexperiments.StatApplGenetMolBiol.2004;3:Article3.doi:10.2202/1544-6115.1027.
57. PimentelHJ,BrayN,PuenteS,MelstedP,PachterL.DifferentialanalysisofRNA-Seqincorporatingquantificationuncertainty.bioRxiv.2016.doi:10.1101/058164.
58. PérezF,GrangerBE.{IP}ython:aSystemforInteractiveScientificComputing.ComputSciEng.2007;9(3):21-29.doi:10.1109/MCSE.2007.53.
59. Jupyter.http://jupyter.org/.Published2016.AccessedJanuary8,2016.60. RStudio.RStudio:IntegrateddevelopmentenvironmentforR(Version
0.97.311).JWildlManage.2011;75(8):1753-1766.doi:10.1002/jwmg.232.61. BaumerB,Cetinkaya-RundelM,BrayA,LoiL,HortonNJ.RMarkdown:
IntegratingAReproducibleAnalysisToolintoIntroductoryStatistics.TechnolInnovStatEduc.2014;8(1):20.doi:10.5811/westjem.2011.5.6700.
62. FriedrichLeisch.Sweave:Dynamicgenerationofstatisticalreportsusingliteratedataanalysis.Compstat2002-ProcComputStat.2002;(69):575-580.doi:10.1.1.20.2737.
63. Beaulieu-JonesBK,GreeneCS.Semi-SupervisedLearningoftheElectronicHealthRecordwithDenoisingAutoencodersforPhenotypeStratification.bioRxiv.February2016.http://biorxiv.org/content/early/2016/02/18/039800.abstract.
64. SouilmiY,LancasterAK,JungJ-Y,etal.Scalableandcost-effectiveNGSgenotypinginthecloud.BMCMedGenomics.2015;8(1):64.doi:10.1186/s12920-015-0134-9.
.CC-BY 4.0 International licenseacertified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under
The copyright holder for this preprint (which was notthis version posted August 11, 2016. ; https://doi.org/10.1101/056473doi: bioRxiv preprint
