Ntino Cloud BioLinux Barcelona Spain 2012

Cloud BioLinux: Pre-configured Bioinformatics Computing for the Genomics Community

Ntino KrampisAsst. Professor - Informatics

J. Craig Venter Institute

[email protected]://www.jcvi.org/cms/about/bios/kkrampis/

Tuesday, November 6, 12

mailto:[email protected]


http://www.jcvi.org/cms/about/bios/kkrampis/

http://www.jcvi.org/cms/about/bios/kkrampis/

J. Craig Venter Institute ( JCVI )

• Human Microbiome Project (Nelson et al. Science 2010; 328: 994–99)

• NIH funded, launched in 2008, $115 million

• metagenomic sequencing of microbial genomes from the human body

• sequence everything in sample, use informatics to separate genomes


J. Craig Venter Institute

• Global Ocean Survey (first publication, Venter et al. Science 2004; 304: 66-74)

• metagenomic sequencing of microbes from oceans around the world

• Darwin’s route ?

• Numbers: HMP > 2 mil. new proteins, GOS > 1.2


Big Data and sequencing

• JCVI sequencing facility: 454, Solexa, HiSeq, and IonTorrent on the way

• Processed data: size information content

• But... look at SOLiD 3

Source: http://www.politigenomics.com/next-generation-

sequencing-informatics


http://www.politigenomics.com/next-generation-sequencing-informatics




JCVI: sequencing and computing infrastructure

• “big” sequencing needs large-scale informatics

• ~1000 node Grid Engine cluster

• research with Hadoop / MapRecuce, and a small private cloud

• 50+ bioinformaticians and software developers


A new paradigm:Low-cost, bench-top sequencers

• GS Junior - 454, MiSeq -Illumina

• complete sequencing of bacterial, viral, fungal genomes

• RNAseq (gene expression), ChiPseq (protein interactions), gene variant discovery

• sequencing as a standard technique in basic genetics research - like PCR ?


Will smaller academic labs become the long tail of sequencing ?

“sequencing factories” :JCVI, Broad Inst. Washington Univ.

Inst. of Genome Sciences

small academic labs withbench-top sequencers

Amountof

sequencing

Number of labs


Sequencers shipped without clusters

• Problem A : sequence analysis requires computational capacity

• genome assembly, BLAST, gene finders - annotation

• Problem B: bioinformatics tools need software engineering expertise

• unix/linux operating systems, maintaining software libraries, compiling source code

???


Each lab builds a cluster ?

• need additional funds to buy the hardware

• funds for personnel to maintain the cluster and software

• duplication of effort across labs

• sub-optimal utilization of the hardware


Centralized bioinformatics services

• Bioinformatic Resource Centers ex. GSCID

• bioinformatic services usually coupled with sequencing of a genome

• provide mostly data access to external PIs

• cannot support to every lab with a sequencer


Problem A : sequence analysis requires computational capacity

• Amazon Elastic Compute Cloud (EC2), pay-by-the-hour computing

• cloud servers cost $0.085 - $2 per hour

• max capacity 64GB RAM / 8 CPU (can boot hundreds of servers)

750 hours free for new users: aws.amazon.com/free/

free compute for teaching: aws.amazon.com/grants/

World-wide data centers


http://aws.amazon.com/free/

http://aws.amazon.com/free/

http://aws.amazon.com/grants/

http://aws.amazon.com/grants/

Cloud Computing and Virtualization

• OS, software and data, pre-installed in Virtual Machine (VM)

• cloud provider: hardware and virtualization layer

• VM is a full-featured server in a single file

• VM transfer on private cloud

Credit: VMware Inc.


Problem B: bioinformatics tools need software engineering expertise

• VM with pre-installed software on the cloud

• avoid compiling source code, or other software dependencies

• rent computational capacity, on a pay as you go basis

• run the VM on the closest Amazon data center


Solving Problems A & B : Cloud BioLinux

• Cloud BioLinux: publicly accessible VM on EC2

• 100+ pre-installed bioinformatics tools

• remote desktop for non-command line experts

• you can create a cluster with Cloud BioLinux - CloudMan Krampis K, Booth T, Chapman B, Tiwari B, Bicak M,

Field D, Nelson K

Cloud BioLinux: pre-configured and on-demand bioinformatics computing for the genomics community.

BMC Bioinformatics. 2012 Mar 19; 13: 42.


http://www.jcvi.org/cms/about/bios/kkrampis


http://www.jcvi.org/cms/about/bios/knelson


Accessing Cloud BioLinux

http://aws.amazon.com/console




Launch through the EC2 cloud console


Amazon EC2 VM launch wizard

cloudbiolinux.org



Cloud BioLinux desktop remote connection

tinyurl.com/bootcloud1 tinyurl.com/bootcloud2


http://tinyurl.com/bootcloud1




Cloud BioLinux desktop


Cloud BioLinux desktop


Data exchange on the cloudVM snapshots


Cloud computing research at JCVI

• open-source cloud platforms, fully compatible with Amazon EC2

• active funding, NIAID viral genomics pipeline on cloud

• end-to-end, sequence to assembly, annotation, visualization via Galaxy

• run on Amazon, private cloud, or desktop


Scriptable Cloud Infrastructures

• Cloud BioLinux VM configuration in plain text

• high-level configuration, software groups

• each group individual bioinformatics tools

Fabricframework


• Python Fabric leverages Linux packages (APTitude repositories)

• mix and match software from repositories

• share VM configuration as source code

• clone across clouds

Krampis K, Booth T, Chapman B, Tiwari B, Bicak M, Field D, Nelson KCloud BioLinux: pre-configured and on-demand bioinformatics computing for the genomics community.

BMC Bioinformatics. 2012 Mar 19; 13: 42.

Scriptable Cloud Infrastructures






Scalable Data Analysis

• Cloud BioLinux + Cloudman

• dual role : Master / Worker

• Cloud BioLinux VM, has Cloudman scripts that start more copies of itself

• Grid Engine (SGE) cluster

• http://usecloudman.org/Afgan, E., Chapman, B. et al. (2012). Using Cloud Computing Infrastructure with CloudBioLinux, CloudMan, and Galaxy.Current Protocols in Bioinformatics, 11-9.


http://usecloudman.org/

http://usecloudman.org/

Goodies with Cloud BioLinux


Goodies with Cloud BioLinux


From sequencer to the cloud

credit:basespace.illumina.com


Acknowledgments

• Cloud BioLinux community: Brad Chapman, Enis Afgan,Tim Booth, Mesude Bicak, Dawn Field

• JCVI collaborators: Alex Richter, Ravi Sanka, Andrey Tovichgrechko, Johannes Goll, Karen Nelson, Bill Nierman, JCVI IT support.

• NIAID and for funding: Maria Giovani, Punam Mathur

cloudbiolinux.org

groups.google.com/group/cloudbiolinux

tinyurl.com/cloudboot1

tinyurl.com/cloudboot2

[email protected]

slideshare.com/agbiotec

Thank you !Tuesday, November 6, 12



Technology

Ntino Cloud BioLinux Barcelona Spain 2012