NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Scalable WES Processing And Variant InterpretationWith Provenance Recording

Using Workflow On The Cloud

Paolo Missier, Jacek Cała, Yaobo Xu,

Eldarina Wijaya, Ryan Kirby

School of Computing Science and Institute of Genetic MedicineNewcastle University, Newcastle upon Tyne, UK

NGS Data Congress

London, June 15th, 2015

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

The Cloud-e-Genome project at Newcastle

1. NGS data processing:

• Implement a flexible WES/WGS pipeline

• Scalable deployment over a public cloud

• Cost control• Scalability• Flexibility

• Of design• Of maintenance

• Ensure accountability through traceability

• Enable analytics over past patient cases

2. Traceable variant interpretation:

• Design a simple-to-use tool to facilitate clinical diagnosis by clinicians

• Maintain history of past investigations for analytical purposes

Objectives: With an aim to:

• 2 year pilot project: 2013-2015• Funded by UK’s National Institute for Health Research (NIHR)• Cloud resources from Azure for Research Award

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Part I: data processing

Objectives:• Design and Implement a flexible WES/WGS pipeline

• Using workflow technology high level programming

• Providing scalable deployment over a public cloud

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Scripted NGS data processing pipeline

RecalibrationCorrects for system bias on quality scores assigned by sequencerGATK

Computes coverage of each read.

VCF Subsetting by filtering, eg non-exomic variants

Annovar functional annotations (eg MAF, synonimity, SNPs…)followed by in house annotations

Aligns sample sequence to HG19 reference genomeusing BWA aligner

Cleaning, duplicate elimination

Picard tools

Variant calling operates on multiple samples simultaneouslySplits samples into chunks.Haplotype caller detects both SNV as well as longer indels

Variant recalibration attempts to reduce false positive rate from caller

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Scripts to workflow - Design

Design Cloud Deployment Execution Analysis

• Better abstraction

• Easier to understand, share, maintain

• Better exploit data parallelism

• Extensible by wrapping new tools

Theoretical advantages of using a workflow programming model

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Workflow Design

echo Preparing directories $PICARD_OUTDIR and $PICARD_TEMPmkdir -p $PICARD_OUTDIRmkdir -p $PICARD_TEMP

echo Starting PICARD to clean BAM files...$Picard_CleanSam INPUT=$SORTED_BAM_FILE OUTPUT=$SORTED_BAM_FILE_CLEANED

echo Starting PICARD to remove duplicates...$Picard_NoDups INPUT=$SORTED_BAM_FILE_CLEANED OUTPUT = \$SORTED_BAM_FILE_NODUPS_NO_RG METRICS_FILE=$PICARD_LOG REMOVE_DUPLICATES=true ASSUME_SORTED=true

echo Adding read group information to bam file...$Picard_AddRG INPUT=$SORTED_BAM_FILE_NODUPS_NO_RG OUTPUT=$SORTED_BAM_FILE_NODUPS RGID=$READ_GROUP_ID RGPL=illumina RGSM=$SAMPLE_ID \RGLB="${SAMPLE_ID}_${READ_GROUP_ID}” RGPU="platform_Unit_${SAMPLE_ID}_${READ_GROUP_ID}”

echo Indexing bam files...samtools index $SORTED_BAM_FILE_NODUPS

“Wrapper”blocksUtility

blocks

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Workflow design

Conceptual:

Actual:

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Anatomy of a complex parallel dataflow

eScience Central: simple dataflow model…

Sample-split:Parallel processing of samples in a batch

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Anatomy of a complex parallel dataflow

… with hierarchical structure

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Phase II, top level

Chromosome-split:Parallel processing of each chromosome across all samples

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Phase III

Sample-split:Parallel processing of samples

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Implicit parallelism in the pipeline

align-clean-recalibrate-coverage

…

align-clean-recalibrate-coverage

Sample1

Samplen

Variant callingrecalibration

Variant callingrecalibration

Variant filtering annotation

Variant filtering annotation

……

Chromosomesplit

Per-sample Parallelprocessing

Per-chromosomeParallelprocessing

Stage I Stage II Stage III

How does the workflow design exploit this parallelism?

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Parallel processing over a batch of exomes

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Cloud Deployment

Design Cloud Deployment Execution Analysis

• Scalability• Fewer installation/deployment requirements, staff hours required

• Automated dependency management, packaging

• Configurable to make most efficient use of a cluster

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Workflow on Azure Cloud – modular configuration

<<Azure VM>>Azure Blob

store

e-SC db backend

<<Azure VM>>

e-Science Central

main server JMS queue

REST APIWeb UI

web browser

rich client app

workflow invocations

e-SC control data

workflow data

<<worker role>>Workflow

engine

<<worker role>>Workflow

engine

e-SC blob store

<<worker role>>Workflow

engine

Workflow engines Module configuration:3 nodes, 24 cores

Modular architecture indefinitely scalable!

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Scripts to workflow

Design Cloud Deployment Execution Analysis

3. Execution

• Runtime monitoring

• provenance collection

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Performance

3 workflow engines perform better than our HPC benchmark on larger sample sizes

Technical configurations for 3VMs experiments:

HPC cluster (dedicated nodes): used 3x8-core compute nodes Intel Xeon E5640, 2.67GHz CPU, 48 GiB RAM, 160 GB scratch space

Azure workflow engines: D13 VMs with 8-core CPU, 56 GiB of memory and 400 GB SSD, Ubuntu 14.04.

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Scalability

There is little incentive to grow the VM pool beyond 6 engines

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Cost

Again, a 6 engine configuration achieves near-optimal cost/sample

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Lessons learnt

Design Cloud Deployment Execution Analysis

Better abstraction

• Easier to understand, share, maintain

Better exploit data parallelismExtensible by wrapping new tools

• Scalability Fewer installation/deployment

requirements, staff hours required Automated dependency management,

packaging Configurable to make most efficient

use of a cluster

Runtime monitoring Provenance collection

Reproducibility Accountability

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Part II: SVI- Simple, traceable variant interpretation

Objectives:

• Design a simple-to-use tool to facilitate clinical diagnosis by clinicians

• Maintain history of past investigations for analytical purposes

• Ensure accountability through traceability

• Enable analytics over past patient cases

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

A database of patient cases and investigations

Cases:

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Investigations

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Provenance of variant identification

• A provenance graph is generated for each investigation

It accounts for the filtering process for each variant listed in the result

Enables analytics over provenance graphs across many investigations

- “which variants where identified independently on different cases, and how do they correlate with phenotypes?”

Work in progress!

NG

S D

ata

Con

gres

sLo

ndon

, Jun

e 20

15P.

Mis

iser

Summary

1. WES/WGS data processing to annotated variants

• Scalable, Cloud-based

• High level

• Low cost / sample

2.Variant interpretation:• Simple• Targeted at clinicians• Built-in accountability of genetic diagnosis• Analytics over a database of past

investigations

What we are delivering to NIHR: