Cuneiform - A Functional Workflow Language … · Cuneiform A Functional Workflow Language Implementation in Erlang Jörgen Brandt Humboldt-Universität zu Berlin 2015-12-01 Jörgen

CuneiformA Functional Workflow Language Implementation in Erlang

Jorgen Brandt

Humboldt-Universitat zu Berlin

2015-12-01

Jorgen Brandt (HU Berlin) Cuneiform 2015-12-01 1 / 27

Cuneiform

CuneiformA Functional Language for Large Scale Scientific Data Analysis

Black-box operator modelPro: Operators can be any piece of software

Black-box data modelPro: Input and Output data can be anything

Features of advanced workflow languagesAbstractions, lists, operations on lists, conditions

Light-weight Foreign Function Interface (FFI)Wrapping in R, Matlab, Octave, Python, Lisp, Perl, Bash

Automatic parallelizationScalability with large data sets


Cuneiform








Cuneiform








Cuneiform








Cuneiform








Cuneiform








Motivation

“New hardware is increasingly parallel, so new programming languagesmust support concurrency or they will die.”

Joe Armstrong


“New Hardware”


DNA Sequencing is becoming cheap


Decentralized software development


Scientific Workflow Systems



Workflows as DAGs

Scientific Workflows areDAGs

Nodes are tasksEdges are data dependencies



Workflows as DAGs

Scientific Workflows areDAGsNodes are tasks

Edges are data dependencies



Workflows as DAGs

Scientific Workflows areDAGsNodes are tasksEdges are data dependencies


Example: Galaxy Workflow System

Focus onUsability

Integration of tools/libraries

Systematic documentationReproducibility



Focus onUsabilityIntegration of tools/libraries





Systematic documentation

Reproducibility






The Next Generation Sequencing use case










Desired Features in a Language

Is there a language that is . . .

Like a workflow languageSo we can integrate all the tools

Like MapReduceSo we can derive parallelism and distribute the work

Like a functional programming languageSo we can write arbitrary programs using lists and operations on lists



Is there a language that is . . .Like a workflow languageSo we can integrate all the tools














Cuneiform example

deftask gunzip( out( File ) : gz( File ) )in bash *{ gzip -c -d $gz > $out}*


Cuneiform example


gunzip( gz: 'myarchive1.gz');

gunzip


Cuneiform example


gunzip( gz: 'myarchive1.gz' 'myarchive2.gz');

gunzip


Cuneiform example


gunzip( gz: 'myarchive1.gz' 'myarchive2.gz' 'myarchive3.gz');

gunzip


Workflow Implementations Available

Available example workflows:Variant callinghttps://www.github.com/joergen7/variant-call

Methylationhttps://www.github.com/joergen7/methylation

RNA-Seq https://www.github.com/joergen7/rna-seq

etc (ChIP-Seq, miRNA detection,consensus prediction, . . . )

samtools-faidx

fastq-dump

cufflinks

cummerbund

tophat-single

cuffdiff

cuffmerge


https://www.github.com/joergen7/variant-call


https://www.github.com/joergen7/methylation


https://www.github.com/joergen7/rna-seq







samtools-faidx

fastq-dump

cufflinks

cummerbund

tophat-single

cuffdiff

cuffmerge













samtools-faidx

fastq-dump

cufflinks

cummerbund

tophat-single

cuffdiff

cuffmerge













samtools-faidx

fastq-dump

cufflinks

cummerbund

tophat-single

cuffdiff

cuffmerge








Cuneiform Operational Semantics


Program Interpretation in Cuneiform

Current design in Java implementation:

Parser Generator(ANTLR)

EBNF

Scanner/Parser

Transcrip�onVisitor

Program Interpreter ResultParse treeAbstractProgram



Designated design but never implemented:


EBNF

Scanner/Parser


Program Interpreter Result

Program Extrac�on(?)

Opera�onalSeman�cs

Parse treeAbstractProgram



Designated design in Erlang:


EBNF

Scanner/Parser


Program Interpreter Result

Program Extrac�on(?)

Opera�onalSeman�cs

Parse treeAbstractProgram


The eval Function

fun eval(Expr , ρ,GetFuture,Global ,Fin) → Result whenExpr :: exprρ :: string => exprGetFuture :: funGlobal :: string => lamFin :: id => exprResult :: expr

Expr The expression to be evaluatedρ Current scope

GetFuture A function returning a future for a foreign task applicationGlobal Task definitions

Fin Results of foreign task applicationsResult The result of evaluation (may contain futures)


The eval Function


Expr The expression to be evaluated

ρ Current scopeGetFuture A function returning a future for a foreign task application

Global Task definitionsFin Results of foreign task applications

Result The result of evaluation (may contain futures)


The eval Function






The eval Function



GetFuture A function returning a future for a foreign task application

Global Task definitionsFin Results of foreign task applications



The eval Function






The eval Function




Fin Results of foreign task applications



The eval Function






Computational Semantics

eval(Expr , ρ,GetFuture,Global ,Fin) →Next = step(Expr , ρ,GetFuture,Global ,Fin)case Next of

Expr → Expr→ eval(Next, ρ,GetFuture,Global ,Fin)

end





end

single step is computed





end

single step is computedIf the step has no effect evaluation terminates



fun eval(Expr , ρ,GetFuture,Global ,Fin) →Next = step(Expr , ρ,GetFuture,Global ,Fin)case Next of


end

single step is computedIf the step has no effect evaluation terminatesOtherwise eval is called recursively


Languages Suitable for Operational Semantics

Choosing a languageFunctional Language

Common Lisp

Scala

ML

Erlang

Haskell


Formalisms Suitable for Operational Semantics

Choosing a languageFunctional LanguageWith Pattern Matching

Common Lisp

Scala

ML

Erlang

Haskell


Formalisms Suitable for Operational Semantics

Choosing a languageFunctional LanguageWith Pattern MatchingConcurrency Orientation

Common Lisp

Scala

ML

Erlang

Haskell



New design:

Parser Generator

(Leex/Yecc)

EBNF

Scanner/

ParserProgram Interpreter Result

Compiler

(Erlang)

Opera�onal

Seman�cs

Abstract

Program



New design:

Parser Generator

(Leex/Yecc)

EBNF

Scanner/

ParserProgram Interpreter Result

Compiler

(Erlang)

Opera�onal

Seman�cs

Abstract

Program


Fault Tolerance


Two types of complexity

n Processes

n Tasks

Distributed applications can becomplex in two independent ways:

in the number of processesinvolved to compute one taskthe more system components the morelikely one component fails

in the number of taskscontributing to a workflowthe more tasks the more likely one taskfails



n Processes

n Tasks






n Processes

n Tasks






n Processes

n Tasks






n Processes

n Tasks





Distributed Application: Workflow System

Query Query Query



Cache

Query Query Query



Scheduler

Cache

Query Query Query



Scheduler

FS Work FS Work FS Work

Cache

Query Query Query


How to achieve fault tolerance when

Workflow systems are complex in bothNumber of processes involved in computing one taskNumber of tasks in one workflow

so failures are likely

All components need to maintain stateso plain restarting of components is not enough

Restarting of workflow helps only if workflows are small and systemhas few components


How to achieve fault tolerance whenWorkflow systems are complex in both

Number of processes involved in computing one taskNumber of tasks in one workflow

















Generic process behaviour

Golden path:P1 sends request to P2

requestP1 P2



Golden path:P1 sends request to P2

Asynchronously P1 receives reply requestP1 P2

reply



P2 may fail:P1 sends request to P2

P2 fails requestP1 P2




P1 creates monitor on P2 requestP1 P2

monitor




P1 creates monitor on P2

P1 memorizes request requestP1 P2

monitor

{ }





P1 memorizes requestP2 fails

requestP1 P2

monitor

{ }





P1 memorizes requestP2 failsP2 supervisor restarts P2

P1 P2

{ }





P1 memorizes requestP2 failsP2 supervisor restarts P2

request is replayed to P2

monitor is recreated

requestP1 P2

monitor

{ }








P2 creates monitor on P1 requestP1 P2

monitor






monitor





P1 failsrequest is canceledsupervisor restarts P1

P1 P2


Conclusion

Cuneiform:

FunctionalIntegrate anythingParallelism

Runs on HadoopImplementation in Erlang:

Concise statelesssemanticsFine-grained faulttolerance

samtools-sort samtools-faidx

varscan

gunzip bowtie2-build

samtools-view

untar

annovar

bowtie2-align

samtools-mpileup

https://github.com/joergen7/cre



Conclusion

Cuneiform:Functional

Integrate anythingParallelism




varscan


samtools-view

untar

annovar

bowtie2-align

samtools-mpileup




Conclusion

Cuneiform:FunctionalIntegrate anything

ParallelismRuns on HadoopImplementation in Erlang:



varscan


samtools-view

untar

annovar

bowtie2-align

samtools-mpileup




Conclusion

Cuneiform:FunctionalIntegrate anythingParallelism




varscan


samtools-view

untar

annovar

bowtie2-align

samtools-mpileup




Conclusion


Runs on Hadoop

Implementation in Erlang:



varscan


samtools-view

untar

annovar

bowtie2-align

samtools-mpileup




Conclusion





varscan


samtools-view

untar

annovar

bowtie2-align

samtools-mpileup




Conclusion



Concise statelesssemantics

Fine-grained faulttolerance


varscan


samtools-view

untar

annovar

bowtie2-align

samtools-mpileup




Conclusion





varscan


samtools-view

untar

annovar

bowtie2-align

samtools-mpileup




Documents

Cuneiform - A Functional Workflow Language … · Cuneiform A Functional Workflow Language Implementation in Erlang Jörgen Brandt Humboldt-Universität zu Berlin 2015-12-01 Jörgen