Building pipelines with Make

Melvin ZhangNational University of Singapore

April 28, 2011http://www.slideshare.net/melvinzhang/

building-pipelines-with-make

1 / 15

Oh no! I found an error in raw data . . .

Do youI manually redo each step?I run ./gen trees.sh human-mouse-rat.mgr?I run make human-mouse-rat trees.tree?

2 / 15

Naı̈ve solution: write a shell script

3 / 15

Naı̈ve solution: write a shell script

4 / 15

maximilianh’s blog

It’s a huge mess and after a while I’ve completelyforgotten which part depends on which others.Then, at a visit to the Vingron group I’ve seenpeople typing in ”make” all the time. It’s sosimple, I don’t know why I’ve never thought ofthis.

– http://archive.nodalpoint.org/2007/03/18/a pipeline is a makefile

5 / 15

Makefile for compilation

all: helloworld

clean:-rm main.o hello

hello:g++ -g -o hello main.o

main.o: main.cppg++ -g -c main.cpp

6 / 15

Makefile for bioinformatics

all: HMR_trees.tree

clean:-rm *.tree

HMR_trees.tree: HMR_mgr.tree HMR_grappa.treecat HMR_mgr.tree HMR_grappa.tree > HMR_trees

HMR_mgr.tree: HMR.mgrbin/mgr HMR.mgr > HMR_mgr.tree

HMR_grappa.tree: HMR.mgrbin/grappa HMR.mgr > HMR_mgr.tree

7 / 15

Make is declarative

Light weight syntax to document dependency amongresults. Encourages small well defined steps.

# commentstarget1: dep_1 dep_2 dep_3<tab><commands><tab><commands><tab>...

dep_1: input...

8 / 15

Make is smartall: HMR_trees.treeclean:

-rm *.treeHMR_trees.tree: HMR_mgr.tree HMR_grappa.tree

cat HMR_mgr.tree HMR_grappa.tree > HMR_treesHMR_mgr.tree: HMR.mgr

bin/mgr HMR.mgr > HMR_mgr.treeHMR_grappa.tree: HMR.mgr

bin/grappa HMR.mgr > HMR_mgr.tree

$ make HMR_trees.treebin/mgr HMR.mgr > HMR_mgr.treebin/grappa HMR.mgr > HMR_grappa.treecat HMR_mgr.tree HMR_grappa.tree > HMR_trees

Make only runs command if target is older thandependents

9 / 15

Make is smartall: HMR_trees.treeclean:

-rm *.treeHMR_trees.tree: HMR_mgr.tree HMR_grappa.tree

cat HMR_mgr.tree HMR_grappa.tree > HMR_treesHMR_mgr.tree: HMR.mgr

bin/mgr HMR.mgr > HMR_mgr.treeHMR_grappa.tree: HMR.mgr

bin/grappa HMR.mgr > HMR_mgr.tree

$ make HMR_trees.treebin/mgr HMR.mgr > HMR_mgr.treebin/grappa HMR.mgr > HMR_grappa.treecat HMR_mgr.tree HMR_grappa.tree > HMR_trees

Make only runs command if target is older thandependents

9 / 15

Make is older than you

Figure: VP Engineering @ Google

Created by Stuart Feldman in 1977 at Bell Labs. In 2003Dr. Feldman received the ACM Software System Awardfor the authoring of this widespread tool.

10 / 15

Basics of a Make rule# commentstarget1: dep_1 dep_2 dep_3<tab><commands><tab><commands><tab>...

dep_1: input...

Example

# combine result of MGR and GRAPPA for# human-mouse-rat into a single fileHMR_trees.tree: HMR_mgr.tree HMR_grappa.tree

cat HMR_mgr.tree HMR_grappa.tree > HMR_trees

11 / 15

Basics of a Make rule# commentstarget1: dep_1 dep_2 dep_3<tab><commands><tab><commands><tab>...

dep_1: input...

Example

# combine result of MGR and GRAPPA for# human-mouse-rat into a single fileHMR_trees.tree: HMR_mgr.tree HMR_grappa.tree

cat HMR_mgr.tree HMR_grappa.tree > HMR_trees

11 / 15

Don’t repeat yourself (DRY)

Automatic variables

output.png: output.pdfconvert -trim -density 300 $ˆ $@

# $ˆ is the list of dependents# $@ is the name of the target

User defined variables

CC = gcctest.out: test.cpp

$(CC) -o test.out test.cpp

12 / 15

Don’t repeat yourself (DRY)

Automatic variables

output.png: output.pdfconvert -trim -density 300 $ˆ $@

# $ˆ is the list of dependents# $@ is the name of the target

User defined variables

CC = gcctest.out: test.cpp

$(CC) -o test.out test.cpp

12 / 15

Generalize with Pattern rules

%.png: %.pdfconvert -trim -density 300 $ˆ $@

# % is like * in shell

13 / 15

Use descriptive filenames

data alg1 alg2 ... algn.format

HMR: HMR_mgr_prune.tree

%_mgr.tree: %.mgrbin/mgr $ˆ > $@

%_prune.tree: %.treebin/prune $ˆ > $@

$ make HMRbin/mgr HMR.mgr > HMR_mgr.treebin/prune HMR_mgr.tree > HMR_mgr_prune.tree

14 / 15

Use descriptive filenames

data alg1 alg2 ... algn.format

HMR: HMR_mgr_prune.tree

%_mgr.tree: %.mgrbin/mgr $ˆ > $@

%_prune.tree: %.treebin/prune $ˆ > $@

$ make HMRbin/mgr HMR.mgr > HMR_mgr.treebin/prune HMR_mgr.tree > HMR_mgr_prune.tree

14 / 15

Further readings

I Parker DS, Gorlick MM, Lee CJ. Evolving frombioinformatics in-the-small to bioinformaticsin-the-large. OMICS. 2003 Spring;7(1):37-48.http://www.ncbi.nlm.nih.gov/pubmed/12831555

I http://www.gnu.org/software/make/I http://biowiki.org/MakefileManifestoI http://www.slideshare.net/giovanni/makefiles-bioinfo

15 / 15