• Home

  • Documents

  • Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) •...
21
Pipelines! CTB 9/23/13

Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Pipelines!  

CTB  9/23/13  

Page 2: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

A  pipeline  view  of  the  world  

Remove adapters

Discard/trim low quality

Assemble

Sequence E. coli2x110

Genome!

Page 3: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Each  computa@onal  step  is  one  or  more  commands  

Remove adapters

Discard/trim low quality

Assemble

Sequence E. coli2x110

Genome!

Trimmoma@c  

fastx  

Velvet  

Page 4: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

The  breakdown  into  steps  is  dictated  by  input/output…  

Remove adapters

Discard/trim low quality

Assemble

Sequence E. coli2x110

Genome!

In:  reads;  out:  reads  

In:  reads;  out:  reads  

In:  reads;  out:  con@gs  

Page 5: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

The  breakdown  into  steps  is  driven  by  input/output  and  “concept”  

Remove adapters

Discard/trim low quality

Assemble

Sequence E. coli2x110

Genome!

In:  reads;  out:  reads.  Trimmoma@c  OR  scythe  OR  …  

In:  reads;  out:  reads.  FASTX  OR  sickle  OR  ConDeTri  OR  …  

In:  reads;  out:  con@gs  Velvet  OR  SGA  OR  …  

Page 6: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Generally,  I  don’t  include  diagnos@c  steps  as  part  of  the  main  “flow”.  

Remove adapters

Discard/trim low quality

Assemble

Sequence E. coli2x110

Genome!

Evaluate with FastQC

Evaluate with FastQC

Evaluate with mapping, BLAST,

etc.

Page 7: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Generally,  I  don’t  include  diagnos@c  steps  as  part  of  the  main  “flow”.  

Remove adapters

Discard/trim low quality

Assemble

Sequence E. coli2x110

Genome!

Evaluate with FastQC

Evaluate with FastQC

Evaluate with mapping, BLAST,

etc.

Page 8: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

…but  there  isn’t  exactly  a  standard  :)  

Remove adapters

Discard/trim low quality

Assemble

Sequence E. coli2x110

Genome!

Evaluate with FastQC

Evaluate with FastQC

Index contigs

Map reads

Calculate mismatch profile

Page 9: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

What  is  a  pipeline,  anyway?  

•  Conceptually:  series  of  data  in/data  out  steps.  

•  Prac@cally:  series  of  commands  that  load  data,  process  it,  and  save  it  back  to  disk.  – This  is  generally  true  in  bioinforma@cs  – You  can  also  have  programs  that  do  mul@ple  steps,  which  involves  less  disk  “traffic”  

•  Actually:  a  bunch  of  UNIX  commands.  

Page 10: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

“Shell  scrip@ng”  

•  The  shell  (bash,  csh,  etc)  is  specialized  for  exactly  this:  running  commands.  

•  Shell  “scrip@ng”  is  puang  together  a  series  of  commands  –  “scrip@ng  ac@ons”  to  be  run.  

•  Scrip@ng  vs  programming  –  fuzzy  line.  –  Scrip@ng  generally  involves  less  complex  organiza@on.  –  Scrip@ng  typically  done  w/in  single  file  

Page 11: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Wri@ng  a  shell  script:  It’s  just  a  series  of  shell  commands,  in  a  file.  

# trim adapters!… Trimmomatic …!!# shuffle reads together!Interleave.py …!!# Trim bad reads!fastx_trimmer!!# Run velvet!velveth...!velvetg…!!

trim-­‐and-­‐assemble.sh  

Page 12: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Back  to  pipelines  •  Automated  pipelines  are  good  things.  

–  Encode  each  and  every  step  in  a  script;  –  Provide  all  the  details,  incl  parameters;  

 •  Explicit:  each  command  is  present.  •  Reusable:  can  easily  tweak  a  parameter,  re-­‐run  &  re-­‐evaluate.  

•  Communicable:  you  can  give  to  lab  mate,  PI,  etc.  •  Minimizes  confusion  as  to  what  you  actually  did  :)  •  Automated:  start  &  walk  away  from  long-­‐running  pipelines.  

 

Page 13: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Why  pipelines?  

•  Automa@on:  – Convenience  – Reuse  – Reproducibility  

 Pipelines  encode  knowledge  in  an  explicit  &  executable  computa@onal  representa@on.  

 

Page 14: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Reproducibility  

•  Most  groups  can’t  reproduce  their  own  results,  6  months  later.  

•  Other  groups  don’t  even  have  a  chance.  

•  Limits:  – Reusability  – Bug  finding/tracking/fixing  

Both  convenience  and  correctness.  

Page 15: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Cita@ons  re  importance  of  reproducibility  

Go  read  this:    

hip://www.motherjones.com/poli@cs/2013/09/austerity-­‐reinhart-­‐rogoff-­‐s@mulus-­‐debt-­‐ceiling    And  this:    hip://boscoh.com/protein/a-­‐sign-­‐a-­‐flipped-­‐structure-­‐and-­‐a-­‐scien@fic-­‐flameout-­‐of-­‐epic-­‐[email protected]    And  this:    hip://pharma.about.com/b/2012/04/06/researchers-­‐vast-­‐majority-­‐of-­‐preclinical-­‐cancer-­‐studies-­‐not-­‐reproducible.htm  

Page 16: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Some  nonobvious  corollaries  •  Each  processing  step  from  the  raw  data  onwards  is  interes@ng;  so  you  need  to  provide  close-­‐to-­‐raw  data.  

•  Making  the  figures  is  part  of  the  pipeline;  but  Excel  cannot  be  automated.  

•  Keeping  track  of  what  exact  version  of  the  pipeline  script  you  used  to  generate  the  results  now  becomes  a  problem…  

Page 17: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

hip://www.phdcomics.com/comics/archive.php?comicid=1531  

Page 18: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

This  is  what  version  control  is  about.  

•  Version  control  gives  you  can  explicit  way  to  track,  mark,  and  annotate  changes  to  collec@ons  of  files.  

•  (Git  is  one  such  system.)  •  In  combina@on  with  Web  sites  like  github.com,  you  can:  – View  changes  and  files  online  – Download  specific  marked  versions  of  files  

Page 19: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

An  actual  pipeline  

•  The  results  in  our  digital  normaliza@on  paper  are  about  80%  automated.  –  Raw  data  –  Single  command  to  go  from  raw  data  to  fully  processed  data.  

–  Single  IPython  Notebook  to  go  from  raw  data  to  figures.  

–  (Super  special)  single  command  to  go  from  figures  +  paper  source  to  submission  PDF.  

–  Figures  &  text  are  @ed  to  a  specific  version  of  our  pipeline  =>  100%  reproducible.  

Page 20: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

IPython  Notebook  

Page 21: Pipelines! - 2013-uw-zero-entry.readthedocs.io · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Tips  &  tricks  

•  Develop  a  systema@c  naming  scheme  for  files  =>  easier  to  inves@gate  results.  

•  Work  with  a  small  data  set  first  &  develop  the  pipeline;  then,  once  it’s  working,  apply  to  full  data  set.  

•  Put  in  friendly  “echo”  commands.  

•  Advanced:  use  loops  and  wildcards  to  write  generic  processing  steps.  


Huong Dan Encode Bang Xvid4psp

Huong Dan Encode Bang Xvid4psp

Documents
Pipelines! - Read the Docs · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Pipelines! - Read the Docs · Back)to)pipelines) • Automated)pipelines)are)good)things.) – Encode)each)and)every)step)in)ascript;) – Provide)all)the)details,)incl)parameters;))

Documents
ENCODE Ensembl Workbook - genome.gov

ENCODE Ensembl Workbook - genome.gov

Documents
ENCODE Portal and Uniform Processing Pipelines : Open Web

ENCODE Portal and Uniform Processing Pipelines : Open Web

Documents
Happyweb8 Encode

Happyweb8 Encode

Technology
3.0 kV RMS/3.75 kV RMS Quad Digital Isolators.../ADuM140E Functional Block Diagram DECODE ENCODE ENCODE DECODE ENCODE DECODE ENCODE DECODE V DD1 GND 1 V IA V IB V IC V OD DISABLE 1

3.0 kV RMS/3.75 kV RMS Quad Digital Isolators.../ADuM140E Functional Block Diagram DECODE ENCODE ENCODE DECODE ENCODE DECODE ENCODE DECODE V DD1 GND 1 V IA V IB V IC V OD DISABLE 1

Documents
15.1 Many Genes Encode Proteins

15.1 Many Genes Encode Proteins

Documents
ENCODE enhancers

ENCODE enhancers

Documents
Should We Encode Rain Streaks in Video as Deterministic or ...openaccess.thecvf.com/content_ICCV_2017/...Encode...Should We Encode Rain Streaks in Video as Deterministic or Stochastic?

Should We Encode Rain Streaks in Video as Deterministic or ...openaccess.thecvf.com/content_ICCV_2017/...Encode...Should We Encode Rain Streaks in Video as Deterministic or Stochastic?

Documents
ENCODE ANALYSIS PIPELINES - National Human … Goals J. Seth Strattan, PhD ENCODE DCC 3 • Introduce the ENCODE Analysis Pipelines. • Run the transcription factor ChIP-seq pipeline

ENCODE ANALYSIS PIPELINES - National Human … Goals J. Seth Strattan, PhD ENCODE DCC 3 • Introduce the ENCODE Analysis Pipelines. • Run the transcription factor ChIP-seq pipeline

Documents
Encode Sequence

Encode Sequence

Documents
encode project

encode project

Documents
Shell scripting and system variables...Variables in Shell Scripting •Variables arecontainers that store a value. •All variables created in ascript are shell variables. •Ascript

Shell scripting and system variables...Variables in Shell Scripting •Variables arecontainers that store a value. •All variables created in ascript are shell variables. •Ascript

Documents
Large scale computational motif finding · ENCODE project: GENCODE consensus human gene set GENCODE [ENCODE] Transcription Tom Gingeras/ENCODE Structural Biology EU Biosapiens Nomenclature

Large scale computational motif finding · ENCODE project: GENCODE consensus human gene set GENCODE [ENCODE] Transcription Tom Gingeras/ENCODE Structural Biology EU Biosapiens Nomenclature

Documents
Encode RNA Dashboard

Encode RNA Dashboard

Technology
ENCODE ANALYSIS PIPELINES - Genome.gov · Workshop Goals J. Seth Strattan, PhD ENCODE DCC 3 • Introduce the ENCODE Analysis Pipelines. • Run the transcription factor ChIP-seq

ENCODE ANALYSIS PIPELINES - Genome.gov · Workshop Goals J. Seth Strattan, PhD ENCODE DCC 3 • Introduce the ENCODE Analysis Pipelines. • Run the transcription factor ChIP-seq

Documents
ENCODE and Epigenomics Roadmap Workshop...Introduction, Overview of ENCODE and Roadmap Epigenomics data Author: J. Michael Cherry and Eurie L. Hong Subject: ENCODE/Roadmap Epigenomics

ENCODE and Epigenomics Roadmap Workshop...Introduction, Overview of ENCODE and Roadmap Epigenomics data Author: J. Michael Cherry and Eurie L. Hong Subject: ENCODE/Roadmap Epigenomics

Documents
how to encode

how to encode

Documents
Perspectives on ENCODE10.1038... · Supplementary Information Perspectives on ENCODE ENCODE Perspective authors The ENCODE Project Consortium#, 4Michael P. Snyder1,2, Thomas R. Gingeras3,

Perspectives on ENCODE10.1038... · Supplementary Information Perspectives on ENCODE ENCODE Perspective authors The ENCODE Project Consortium#, 4Michael P. Snyder1,2, Thomas R. Gingeras3,

Documents
Encode™ Restorative System · Indications • Cement-retained ... With the introduction of the Encode Restorative System, ... Pindex the first pour Encode Cast. The pinned sections

Encode™ Restorative System · Indications • Cement-retained ... With the introduction of the Encode Restorative System, ... Pindex the first pour Encode Cast. The pinned sections

Documents
ENCODE ANALYSIS PIPELINES - Genome · Workshop Goals J. Seth Strattan, PhD ENCODE DCC 3 • Introduce the ENCODE Analysis Pipelines. • Run the transcription factor ChIP-seq pipeline

ENCODE ANALYSIS PIPELINES - Genome · Workshop Goals J. Seth Strattan, PhD ENCODE DCC 3 • Introduce the ENCODE Analysis Pipelines. • Run the transcription factor ChIP-seq pipeline

Documents
Gene regulaon: ENCODE project and the Human Epigenome Atlas€¦ · The ENCODE Project hLps:// The Encyclopedia of DNA Elements (ENCODE) Consor.um is an internaonal collaboraon of

Gene regulaon: ENCODE project and the Human Epigenome Atlas€¦ · The ENCODE Project hLps:// The Encyclopedia of DNA Elements (ENCODE) Consor.um is an internaonal collaboraon of

Documents
ENCODE Discussion - Schatzlabschatzlab.cshl.edu/teaching/2014/Genomics Lecture 2. ENCODE Disc… · 26/09/2014  · “Given that the ENCODE project did not assay all cell types,

ENCODE Discussion - Schatzlabschatzlab.cshl.edu/teaching/2014/Genomics Lecture 2. ENCODE Disc… · 26/09/2014  · “Given that the ENCODE project did not assay all cell types,

Documents
The ENCODE Project

The ENCODE Project

Documents
Philosophy · Philosophy - Allow all input - Encode all output Do not filter or encode input that gets stored but always protect the user on output. - Encode at the very end Encode

Philosophy · Philosophy - Allow all input - Encode all output Do not filter or encode input that gets stored but always protect the user on output. - Encode at the very end Encode

Documents
Scientists Encode First Women Genome

Scientists Encode First Women Genome

Technology
php url encode

php url encode

Documents
Encode AHD

Encode AHD

Documents
ENCODE Pseudogene Call Summary

ENCODE Pseudogene Call Summary

Documents
The ENCODE Portal REST API

The ENCODE Portal REST API

Science