Build System for Large Codebases

Angad SinghPyCon SG 2016

@angadsg

NUS Computer Engineering (2009-13)

SRE at Twitter (2013-14)

DevOps at Viki (2014-..)

Contact: t.co/as

Twitter: @angadsg

About me

Agenda

Code Organization

Pain Points

Pants as a build tool

Python PEX format

Code Examples

Service A Service B Service C

Project Repo A

Project Repo B

Project Repo C

Code Organization

Shared code

Does not scale well for a large number of microservices

Complex method of sharing libraries (publishing artifacts, versioning hell)

Code Organization

Libraries repository

Service A

Service B

Service C

Libraries as Code Units

Single Lint, Build, Test and Release process

Easy to coordinate changes across modules

Easier to setup development environment

Tests run across modules are run together

Promote the idea of writing shareable code

Monorepo- A repository with a defined structure for organizing reusable components of code

Pain Points

Virtualenv to manage dependencies for python projects is painful. Need something simpler.

Need easier code sharing amongst projects. Fixing a bug in a function should not require changing versions of other downstream projects.

Need standardization in testing and building process

PantsBuild system for managing targets sharing

a single repository

Dependencies are managed in BUILD files that live alongside the code.

History - Used to be a python wrapper around Ant build tool which generated build.xml files and handed the build files to ant. (Python + Ant = Pants)

Later, rewritten to be an independent build tool with main support for JVM languages and Python.

Pants

Define source tree - src/<lang> e.g. src/python/

BUILD files define targets at each leaf node in the source tree.

DSL, which invokes python constructors in the background

Targets can be either a binary (e.g PEX for python, JAR for Java) or a library which can be referenced by other targets.

PEX

PEX files - Python Executables, similar in idea to a virtual environment.

Generate Immutable artifacts, that will run on any server

Run targets locally, without maintaining complex virtual environments

Easier debugging through standardized versioning of 3rdparty dependencies

BUILD file

python_binary(name=’cli’, dependencies = [ “src/python/3rdparty/python:requests”, “:shared_lib” ] source=’cli.py’)

python_library(name=’shared_lib’, dependencies = [ “src/python/3rdparty/python:fabric” ] source=’lib.py’)

Examples

A simple python flask application

https://github.com/angad/pants-flask

Twitter Commons

https://github.com/twitter/commons

Thank you