Continuous Integration Testing in Django

Continuous Integration TestingDjango Boston Meetup

24 April 2014

Kevin Harvey@kevinharveykevin@storyandstructure.com

Who is this guy?

Who is this guy?

Fast Slow

Unit Functional• Chief Technologist at Story+Structure

• Djangonaut since 2007 (0.96)

Here’s the best $24 I ever spent.

What are we doing?

Agenda

• What is Continuous Integration?

• Why use Continuous Integration?

• CI in Practice: Travis

• Coverage Thresholds with coverage.py

• CI in Practice: Jenkins

• Best Practices & Further Exploration

What is CI?

Continuous Integration is like having a tireless robot that cleans up after you.

What is CI?

1. Checkout out the latest copy of your code

2. Install the dependencies

3. Run the tests

4. Report the outcome

Some server somewhere checks out your code, installs all your dependencies, runs your tests, and let’s you know if it worked or not.

Testing in Django# jmad/tunes/tests/test_views.py

from django.test import TestCase

class TunesViewTestCase(TestCase):

def test_index_view(self): response = self.client.get('/') self.assertEqual(response.status_code, 200) self.assertTemplateUsed(response, 'index.html')

Here’s a quick primer on Django tests, for the uninitiated.

Testing in Django# jmad/tunes/views/__init__.py

from django.views.generic import TemplateView

class IndexView(TemplateView): template_name = "index.html"

# jmad/jmad/urls.py

from django.conf.urls import patterns, url

from tunes.views import IndexView

urlpatterns = patterns('', url(r'^$', IndexView.as_view(), name='index'),)

Here’s the code to make that test pass (assuming there’s an index.html in a template directory somewhere).

Testing in Django

$ python manage.py test.----------------------------------------------------------------------Ran 1 test in 0.029s

OKDestroying test database for alias 'default'...$

And here’s how you’d run the test suite for the project.

Why CI?

Why CI?

Drink more beer.

How can I drink more beer if the client calls me on Saturday night because the site is down?Bugs should show up fast, and you should fix them before your code is deployed.

Why CI?

Your test suite is only useful if you run it.

You will forget to run your tests. Your collaborators will forget to run your tests. If you don’t run the tests, you won’t know your code is busted when it’s time to deploy.

Why CI?

I run tests with SQLite, d@&$#t.

It’s fast. It’s there. If I want to build my project on my mom’s Mac Mini I don’t have to install homebrew. Plus, running a test suite creates a new database everytime. Let the m1.small sitting in Northern Virginia wait around for TestCase to create the PostgreSQL database. It doesn’t even like beer.

Why CI?

Your project is not getting deployed to a Macbook.

Having said that, you need to run your tests in an environment like your production environment. Your project may have wacky dependencies that you installed a long time ago on your Macbook.

Why CI?

Know immediately if you can’t install a package

Why CI?

# requirements.txt

...

PIL=1.1.7 # need to redeploy?

The time to learn about a neat new package replacing an old clunky one is NOT when you’re trying to do an emergency redeployment.

Our example project

JMAD

The Jazz Musicianship Archive and Database

http://jmad.us

code: https://github.com/kcharvey/jmad

The Jazz Musicianship Archive and Database

http://jmad.usI’m a bass player, I play some jazz.

http://www.scu.edu/profiles/?p=5184

More importantly, I know Bill Stevens. He’s jazz faculty at Santa Clara University. He wrote a book called “Jazz Musicianship”. It’s a guide for jazz improvisation based on patterns that exists in many different jazz tunes.

While we read headings like “Adding Callables to ModelAdmin Classes”...

... his headings are like “Engaging the Bebop Principle”.

Screenshot

JMAD is an archive of jazz music that’s been tagged with these different musical concepts. Basically you search by concept, instrument, difficulty, etc., and you get back solos with those parameters

CI in Practice: Travis

Travis• http://travis-ci.org

• It’s where all these come from:

Travis: Setup1. Sign in with your Github account

2. Select the repos you want Travis to work on

3. Add a .travis.yml file to the root of the repo

4. Commit

Travis: Basic .travis.ymllanguage: python

python:  - "3.3"  - "2.7"# command to install dependenciesinstall:  - "pip install -r requirements/ci.txt"# command to run testsscript: "python manage.py test"

Travis: Egg .travis.ymllanguage: pythonpython:  - "3.3"  - "2.7"# command to install dependenciesinstall:  - "pip install -r requirements/ci.txt"  - "pip install ."# command to run testsscript: "cd testproject;python manage.py test myapp"

Travis: Pros

SaaS

No server to set up, no apt-getting anything ever. It’s just there.

Travis: Pros

Free as in “free beer” (for public repos)

Travis: Pros

Config is maintained very obviously in Git

Explicit, no implicit.

Travis: Pros

Cool little badgy thingy.

Impress your hacker friends with your TDD prowess.

Travis: Pros

Unbelievably, stupidly easy.

Travis: Pros

Just do it, even if you don’t have any tests.

It will still check your requirements and build your app.

Travis: Cons

Hope you’re using GitHub.

Maybe there’s a way to use non-GitHub repos, but I’ll be damned if it’s documented anywhere.

Travis: Cons

You’re stuck with their architecture.

For instance, when I was putting this app together they didn’t have Python 3.4 yet.

coverage.py

coverage.py is a utility created by Boston’s own Ned Batchelder.

coverage.py

How much of my code is not covered by tests?

It answers this question.

coverage.py$ coverage run --source='.' manage.py test --settings=jmad.settings.baseCreating test database for alias 'default'......E...----------------------------------------------------------Ran 4 tests in 3.353s

FAILED (errors=1)Destroying test database for alias 'default'...

Run your tests with coverage...

coverage.py$ coverage report --omit=*test*,*settings*Name Stmts Miss Cover-------------------------------------------------jmad/__init__ 0 0 100%jmad/urls 6 0 100%jmad/wsgi 4 4 0%manage 6 0 100%people/__init__ 0 0 100%people/admin 1 0 100%people/models 1 0 100%people/views/__init__ 4 0 100%tunes/__init__ 0 0 100%tunes/admin 1 0 100%tunes/models 1 0 100%tunes/templatetags/__init__ 0 0 100%tunes/templatetags/add_css 5 0 100%tunes/urls 3 0 100%tunes/views/__init__ 5 0 100%-------------------------------------------------TOTAL 37 4 89%

... then generate a report to see what’s not being tested.

coverage.py

$ coverage report --omit=*test*,*settings*

So let’s take a closer look at the report command

coverage.py

$ coverage report --omit=*test*,*settings*

--omit tells coverage to not report on a few things. Since we have multiple settings files, and we don’t test our tests, we omit them both.

coverage.py

$ coverage report --omit=*test*,*settings* --fail-under=85

--fail-under take an integer, and compares that to the total percentage of coverage, and makes this command return a 2 (anything but 0 is failure).

Travis and coverage.pylanguage: pythonpython:  - "3.3"  - "2.7"# command to install dependenciesinstall:  - "pip install -r requirements/ci.txt"# command to run testsscript:  - "coverage run source=’.’ manage.py test"  - "coverage report --omit=*settings*,*test* --fail-under=85" # 85% coverage minimum

Swap out your script with these two lines: one to run the tests with coverage, and the other to run a report that can fail. Now if we’re at 85% coverage and someone pushes new code without test coverage, we’ll drop below 85% and get a failed build.

coverage.py

“Code without tests is broken by design.”

-- Jacob Kaplan-Moss

This is a good thing.

CI in Practice: Jenkins

Jenkins• http://jenkins-ci.org/

• Django’s Jenkins: http://ci.djangoproject.com/

Installing Jenkins$ wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war$ java -jar jenkins.war

http://localhost:8080

Just get the .war file, run it, and hit port 8080 on your machine. You could deploy it behind Tomcat or the like.

Or grab a VM

Installing Jenkins

I used a TurnKey Linux prebuilt VM.

Configuring JenkinsYou’ll need some plugins:

- Jenkins Violations Plugin

- Jenkins Git Plugin

- Jenkins Cobertura Plugin

Install a few plugins.

Configuring Jenkins

Got Selenium tests?

So, if you’re like me, you’ve got Jenkins on a headless version of Linux. We’ll have to do some magic to get our Selenium tests to run.

Configuring Jenkins# on the Jenkins box

$ sudo apt-get install iceweasel # install firefox

$ sudo apt-get install xvfb # frame buffer emulator

Install firefox (the package is called iceweasel for some reason).

Configuring Jenkins# /etc/init.d/xvfb

#!/bin/bash

if [ -z "$1" ]; thenecho "`basename $0` {start|stop}"exitfi

case "$1" instart)/usr/bin/Xvfb :99 -ac -screen 0 1024x768x8 &;;

stop)killall Xvfb;;esac

Configure xvfb to run when the server starts

Configuring Jenkins$ sudo chmod 755 /etc/init.d/xvfb

$ sudo shutdown -r now

make that file executable, and restart the server

Configuring Jenkins$ pip install django-jenkins

INSTALLED_APPS = ( ... 'django_jenkins',)...

JENKINS_TASKS = ( 'django_jenkins.tasks.run_pylint', 'django_jenkins.tasks.with_coverage', 'django_jenkins.tasks.run_pep8', # there are more of these)

$ python manage.py jenkins # Jenkins will run this command

django-jenkins is a plugin that runs out tests and outputs the files Jenkins needs to show our build stats. pip install and add some stuff to your settings.py

Configuring Jenkins

1. Configure a new test (name, description)

2. Give it your repo URL

3. Tell it how often to build

4. Tell it the commands to run

5. Configure where to save the reports

6. Click “Build Now”

Check out the tutorials in the “Resources” section of this slide deck for more on configuring your repo. It’ll take about 15 minutes the first time.

Configuring Jenkins#!/bin/bash

virtualenv -p python3.4 env

env/bin/pip install -r requirements/ci.txt

export SECRET_KEY='dnqsj22jdv9wjsldfub9'

export DISPLAY=:99

env/bin/python manage.py jenkins --settings=jmad.settings.ci

here’s what that command really should be

So what did we get?

Jenkins

I used a TurnKey Linux prebuilt VM.

Here’s a list of all the projects Jenkins knows to build

The project page for JMAD. Note the historic list of builds at the bottom left.

An individual build. That’s a list of commit messages under ‘changes’. And the link to the console output.

Logged output

The workspace it built.

Latest test result

Free as in beer and speech

Jenkins: Pros

Open. Extensible. Good for the spirit.

Plugins install like Wordpress

Jenkins: Pros

And by that I mean, it’s almost *too* easy. Just search for them from your installation and click “install”.

Distributed builds

Jenkins: Pros

I haven’t done any work with this at all, but Jenkins supports a ‘master/slave’ mode, allowing a single Jenkins instance to control many others. This would allow you to test a project on a bunch of different platforms simultaneously. You can see how that would benefit the Django project itself, or other large Python packages.

It’s your architecture.

Jenkins: Pros

Need to run Python compiled with some magical incantation? Need a special server utility installed? Jenkins runs on an OS you control, so do what you gotta do.

It’s nobody else’s architecture.

Jenkins: Cons

That, of course, leads to our first con.

To paraphrase Uncle Ben, “With great power can come a whole lot of bullshit.”

15 apt-get update 16 sudo apt-get install build-essential 17 apt-get install build-essential 18 apt-get install libsqlite3-dev 19 apt-get install sqlite3 20 apt-get install bzip2 libbz2-dev 21 ls 22 ls .. 23 mkdir src 24 cd src/ 25 wget http://www.python.org/ftp/python/3.4.0/Python-3.4.0.tar.xz 26 tar xJf ./Python-3.4.0.tar.xz 27 cd Python-3.4.0/ 28 ./configure --prefix=/opt/python3.4 29 make && make install 30 python3.4 31 ln -s /opt/python3.4/bin/python3.4 ~/bin/python3.4 32 ls ~ 33 mkdir bin 34 ln -s /opt/python3.4/bin/python3.4 ~/bin/python3.4 35 ls

Jenkins: Cons

Here’s the first twenty lines of me fumbling through installing Python 3.4, pip, and virtualenv

36 ls bin 37 rm -rf bin 38 mkdir ~/bin 39 ln -s /opt/python3.4/bin/python3.4 ~/bin/python3.4 40 python2.4 41 python3.4 42 virtualenv 43 ls /opt/python3.4/bin/ 44 cd 45 ls 46 ls bin/ 47 bin/python3.4 48 python3.4 49 ls /usr/bin/ 50 ln -s /opt/python3.4/bin/python3.4 /usr/bin/python3.4 51 python3.4 52 rm -rf bin/ 53 wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py 54 python3.4 get-pip.py 55 apt-get install zlib 56 apt-get install zlib1g

Jenkins: Cons

... and here’s the next twenty lines...

57 python3.4 get-pip.py 58 apt-get install zlib1g 59 apt-get install zlib-dev 60 apt-get install zlibc 61 python3.4 get-pip.py 62 apt-get install zlib1g-dev 63 python3.4 get-pip.py 64 apt-get install zlib-bin 65 python3.4 get-pip. 66 python3.4 get-pip.py 67 cd src/ 68 ls 69 cd Python-3.4.0/ 70 history 71 ./configure --prefix=/opt/python3.4 72 make && make install 73 apt-get install libssl-dev openssl 74 make && make install 75 which pip 76 cd ~ 77 ln -s /opt/python3.4/bin/pip3.4 /usr/bin/pip3.4 78 pip3.4 install virtualenv

Jenkins: Cons

... and the next. So you’ll need some sysadmin chops.

And I still need to set up a mail server.

Jenkins: Cons

Git is a plugin

Jenkins: Cons

Again, it’s not hard to install plugins, but other tools are Git-centric, so it’s worth mentioning.

Best Practices

Best Practices

Use multiple settings.py and requirements.txt files

Best Practices# jmad/settings/ci.py

INSTALLED_APPS = (...‘django_jenkins’

)

...

JENKINS_TASKS = ( ‘django_jenkins.tasks.run_pylint’, ‘django_jenkins.tasks.with_coverage’, ‘django_jenkins.tasks.run_pep8’)

Keep this stuff out of your production app (and your dev environment, for that matter).

Best Practices

# requirements/ci.txt

...coverage==3.7.1 # duped in dev.txtdjango_jenkins==0.15.0pep8==1.5.6

(and your dev environment, for that matter).

Best Practices

Rebuild your env in Jenkins

Best Practices

#!/bin/bashrm -rf envvirtualenv -p python2.7 envenv/bin/pip install -r requirements/ci.txtexport SECRET_KEY='dnqsj22jdv9wjsldfub9'env/bin/python manage.py jenkins --settings=jmad.settings.ci

Toss the old env before you recreate it.

Further Exploration

“Just-in-time Jenkins”

a.k.a.

“The AWS Miser”

a.k.a

“Big Testing”

Further Exploration

You could imagine a scenario in which, using one of the many DevOps tools available, you spin up and AWS instance, install and configure Jenkins, set up your tests, report and then tear down the box. This would be particularly useful if you wanted to test on a multidtude of platforms but didn’t want to pay to keep them all up all the time.

Alternatives to Travis

• https://circleci.com/

• https://drone.io/

Further Exploration

Travis is not the only SaaS CI game in town. Drone.io works with BitBucket.

tox

Further Exploration

It’s a tool to test your project in multiple version of Python in one go, and can act as a front end to a CI server. Has anyone here used tox?

django-jenkins Tasks

There are a number of tasks available in django-jenkins that I haven’t showed you yet.

django_jenkins.tasks.run_jshint

django_jenkins.tasks.run_csslint

django-jenkins Tasks

Runs jshint or csshint tools of your static directory, and produces reports that work in Jenkins. You’ll need to install jshint/csslint on the box that Jenkins is running on.

django_jenkins.tasks.run_pyflakes

django-jenkins Tasks

If you’re using Pyflakes, django-jenkins has you covered. Add Pyflakes to you requirements/ci.txt file.

django_jenkins.tasks.run_flake8

django-jenkins Tasks

Same deal with flake8. You’re beginning to see how jenkins is trying to fit in with whatever testing tools you’re already using.

django_jenkins.tasks.run_sloccount

django-jenkins Tasks

SLOCCount is a utility for counting lines of code. The developer missed a big opportunity to call his project SLOCConut, IMHO. Install SLOCCount on the server to get this one going.

django_jenkins.tasks.run_sloccount

django-jenkins Tasks

SLOCCount is a utility for counting lines of code. The developer missed a big opportunity to call his project SLOCConut, IMHO. Install SLOCCount on the server to get this one going.

django_jenkins.tasks.run_graphmodels

django-jenkins Tasks

You can also let Jenkins graph your models for you.

You need django-extensions and pygraphviz in your CI requirements for this one to work.

django_jenkins.tasks.with_local_celery

django-jenkins Tasks

Django Jenkins also provides a test runner to run your tests with Celery, if you’re in to that sort of thing.

Thanks!

@kevinharvey

github.com/kcharvey

Questions?

@kevinharvey

github.com/kcharvey

Show of hands: if your boss/client/conscience dictated that you had to implement CI tomorrow, would you use one of the hosted/SaaS tools or go with Jenkins?

ResourcesSetting up Jenkins for Selenium tests

• http://www.labelmedia.co.uk/blog/setting-up-selenium-server-on-a-headless-jenkins-ci-build-machine.html

• http://www.installationpage.com/selenium/how-to-run-selenium-headless-firefox-in-ubuntu/

• Just Google it