Build and deploy scientific Python Applications

Build and Deploy Scientific Python

ApplicationsSciPy India 2014|IIT Bombay

Ramakrishna Reddy Yekullahttp://ramkrsna.net/

@ramkrsna

Prerequisites

● Have written python or any applications● Basic Linux Commands

How is this talk going to go about

● Binary Interfaces and Stability ● Python packaging for Application

Developer● Packaging your Scipy app for *nix based OS● Deployments | Docker

Application Binary Interfaces● An application binary interface (ABI) is the interface between two program

modules, one of which is often a library or operating system, at the level of machine code. An ABI determines such details as how functions are called and in which binary format information should be passed from one program component to the next, or to the operating system in the case of a system call.

● Adhering to ABIs (which may or may not be officially standardized) is usually the job of the compiler, OS or library writer, but application programmers may have to deal with ABIs directly when writing programs in a mix of programming languages, using foreign function call interfaces between them.

● ABIs differ from application programming interfaces (APIs), which similarly define interfaces between program components, but at the source code level.

Operating System Stability

● Filesystem Reliability● Glibc Interfaces ~9000 Stable Interfaces● Robust Tested Software both by the

communityand Enterprise Software Industry

Operating System Stability

● Separation of concerns● Reliability ● Repeatability. ● Ease of use.● Built it, Run it.● Tools of the trade ::

Binutils – readelf, objdump, nm, c++filt

Elfutils – eu readelf, eu objdump, eu nm

System Packaging Solutions

● GNU/Linux :: apt-get | dpkg, yum | RPM● Windows :: npackd | chocolatey● Mac OS X :: homebrew | macports

What Every Developer Needs to Know about Memory

● Ulrich Drepper

What Every Python Developer should be aware of..

● distutils● buildout● virtualenv● pip● setup.py develop● **conda

VirtualenvSetting up the virtualenv can take a significant amount of time because it pulls down packages from PyPI. I was surprised that it does not try to use the locally installed packages. One way to speed up the rebuild is to use a local caching mirror of PyPI. This can be accomplished by using devpi.

● Step 1:

Install devpi package

sudo pip install devpi

● Step 2: Add the following lines in your /root/.pip/pip.conf file

[global]

index-url = http://localhost:3141/rooint/pypi/+simple/

extra-index-url = https://pypi.python.org/simple/

Python Apps on Distributions

● RPM's Based Deployments

● Software Collections

● Context based Security Benefits

● PACKER | Virt-Builder

General Rule Of Packaging

● MAJOR version when you make incompatible API changes

● MINOR version when you add functionality in a backwards-compatible manner, and

● PATCH version when you make backwards-compatible bug fixes.

● Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

http://semver.org

https://access.redhat.com/documentation/en-US/Red_Hat_Software_Collections/

http://nvie.com/posts/pin-your-packages/

Fedora

● In Fedora we have multiple python runtimes, one for each supported major release. At this point that's one for python2.x and one for python3.x

● Each runtime corresponds to a binary of the form● /usr/bin/python$MAJOR.$MINOR● One of these python runtimes is the "system runtime"

which is what we run when invoking● All python runtimes have a virtual provide for● python(abi) = $MAJOR-$MINOR .● /usr/bin/python .● On Fedora 20 this is● /usr/bin/python2.7

Package Versioning

● MAJOR version when you make incompatible API changes

● MINOR version when you add functionality in a backwards-compatible manner, and

● PATCH version when you make backwards-compatible bug fixes.

● Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

http://semver.org

https://access.redhat.com/documentation/en-US/Red_Hat_Software_Collections/http://nvie.com/posts/pin-your-packages/

Typical Dev Setup

● Vagrant, virtualenv, virtualenvwrapper

● Zero configuration Python/Unix web based teaching env ::

https://github.com/ptone/jiffylab

● Another Approach for Dev Environment ::

http://collaboratool.berkeley.edu

Typical Deploy Setup

● BCFG2, Python-based but XML-infested.

● CFEngine

● Ansible, Python-based, gaining a lot of traction lately

● Chef, Puppet ( Ruby Based)

● Salt, Python

● Juju, Ubuntu

● Docker

Thank YouQuestions ?!

42@ramkrsna.net