Upload
hakka-labs
View
5.583
Download
1
Embed Size (px)
DESCRIPTION
Deployment issues surrounding the use of virtualenv within Debian packages at Spotify
Citation preview
1
Spotify Tech
Web/iOS/Android/Native clients Service oriented architecture > 50 services in Java and Python Primarily using Cassandra for persistent storage 7,000 Servers in production ~500 in Engineers worldwide
2
Lost in the forest of dependency building
What have we been doing?
3
Deploy via Debian Packages
Use Debian Stable repositories Package/Backport needed libraries as needed All Dependencies defined in Debian package Deployed via Puppet/Aptitude
4
Package
Mirroring
Apps
PyPi
5
Spending too much time managing dependencies and not enough building cool shit
What’s wrong with that?
6
Using Debian Stable does not included latest and greatest
7
Requires “Backporting” for newer libraries
8
Does not used normal python packaging
9
Limited use on developer systems
(or bad/duplicated information)
10
Full rollback non-existent
11
What you test is NOT what you deploy
12
Dependency lock in due to sharing a common python environment
13
Avoid duplication of effort
There MUST be a better way
14
Increase stability and repeatability
There MUST be a better way
15
Offer easy rollback
There MUST be a better way
16
Scale
There MUST be a better way
17
Deploying Python @ Scale
What about now?
18
Enter Dh-viurtualenv
19
Dh == Debian Helper
• Registers itself as a Debian Helper • Bundles the requirements in requirements.txt into virtual environment • Deploys virtualenv into /usr/share/python/<package-name> • Still requires native dependencies to be installed via packaging • Just include the following in the debian/rules:
%: dh $@ -‐-‐with python-‐virtualenv
20
Virtualenv
Mirroring
Apps
Pip
21
IT Works!
22
Example: Using Sentry
Define Dependencies in requirements.txt
sentry[postgres]==6.2.0 eventlet==0.13.0 hiredis==0.1.1 django-‐auth-‐ldap==1.1.4
23
Example: Using Sentry
Define Debian dependencies in control file
Build-Depends: python (>= 2.6.6-3~), debhelper (>= 8), dh-virtualenv, python-dev, libpq-dev, libldap2-dev, libsasl2-dev, Standards-Version: 3.9.3 X-Python-Version: >= 2.6
24
Example: Using Sentry
Use dh_virtualenv in Debian rules file
%: dh $@ --with python-virtualenv
override_dh_virtualenv:
dh_virtualenv --index-url='http://localhost/simple'
25
Have done this with RPMs as well (@ Amplify)
Within RPM build virtaulenv and mark it as relocatable:
/usr/local/bin/virtualenv --no-site-packages build_venv source build_venv/bin/activate env CFLAGS="$RPM_OPT_FLAGS" easy_install -Z . deactivate /usr/local/bin/virtualenv --relocatable build_venv
26
Benefits
No longer stuck on Debian stable
27
Benefits
No more backporting of new libraries or versions
28
Benefits
Code isolation/use own dependencies
29
Benefits
Single package rollback option
30
Benefits
Development and Deployed systems using same cross-platform setup
31
What’s Next?
32
Not Sure…
33
Dh_virtualenv now available (ironically) in Debian Unstable repositories
34
Still requires managing native dependencies locally
35
But experimenting with containers!
36
Stay Tuned!
37
Whitepaper: http://labs.spotify.com/2013/10/10/packaging-in-your-packaging-dh-virtualenv/ Code: https://github.com/spotify/dh-virtualenv
Thanks to Jyrki Pulliainen and Jim Whitehead @ Spotify Stockholm!
38
Chris Angove, Chapter Lead - Backend Email: [email protected] Blog: http://labs.spotify.com/ Listen: https://play.spotify.com/
Questions?
January 24, 2014
Check out spotify.com/jobs or @Spotifyjobs for more information.
Want to join the band?