Django Bootstrapping with Ease

BootstrappingDjango with easeOSCON 2012www.concentricsky.com@concentricsky // biglan / wiggins / bothun

Thursday, July 19, 12

http://www.concentricsky.com


Introductions

Mike BiglanCTO

WigginsArchitect

@coffindragger

Josh BothunSenior Developer


Areas of Concentric Sky

Mobile EnterpriseWeb/Django


(Some) Technologies We Use

• Chef

• Jenkins

• AWS

• Sphinx

• Memcached

• South

• git

• Backbone.js

• Jinja2

• Celery

• DebugToolbar

• jQuery / jQueryUI

• Compass / Sass

• Mongo & MySQL


Released CSky Creations

• djenesis – Easy Django Bootstrapping

• breakdown - Lightweight jinja2 template prototyping server

• django-cachemodel – Automatic caching for django models


http://pypi.python.org/pypi/djenesis

http://pypi.python.org/pypi/djenesis

http://pypi.python.org/pypi/breakdown

http://pypi.python.org/pypi/breakdown

https://github.com/concentricsky/django-cachemodel

https://github.com/concentricsky/django-cachemodel

CSky Creations in the Works

• skycloud – Tool for AWS auto-scaling

• django-skycms – CMS as a library

• django-sky-visitor – Invitations, email confirmation, etc

• django-client-admin – Flexible admin for client


CSky Teams

STAFF

50+10 Django Team

5 Design Team

9 PM Team

3 System Admins


Team Interfacing

• Each project is a mix from each team

• Members might switch amongst projects

• Interfaces between teams

• Clean separation of handoffs and responsibilities

• Best-practices shared within teams


“Bootstrapping”

• Pulling one up by one’s bootstraps

• Computers do this much better than humans

• Django – starting a project, new or existing


Bootstrapping with Ease

Making booting (and rebooting) a django project as easy as possible

• Ease is a continually moving target

• Shouldn’t sacrifice other goals for ease (e.g. security, modularity, be PHP)


Moving Targets

Moving targets create new problems. And new solutions.

• CGI – Solving static HTML

• JavaScript – Solving non-interactivity

• CSS – Solving 100+ HTML pages

• Puppet/Chef – Solving 100+ servers


Problem/Solution Lifecycle

1. Problem grows

2. Many create their own solution; some polish; some release

3. Community flocks to one (or a few) solutions

4. Solution(s) gets anointed and others die off


Goals of Easy Bootstrapping

1. More people using Django

2. More people using Django better


More People Using Django

• As few commands as possible to try Django out

• And to try someone else’s project out.

• Must be highly likely to just work

• All especially critical for beginners

• Django’s runserver & sqlite...great!

• It just works...


Making Mistakes should be Hard

• Checklists are great

• The best checklists are invisible

• To automate lists of commands, must have consistency


Consistency Makes things Easier

• Ease of replication across servers

• Ease of sharing

• Both templates and existing projects

• Easier to build upon


Consistently Standardized

• Standardization: a tool to get to consistency

• This can be in an organization. Or the full community

• Oh yeah, and...


Standard had Better be Right


Fragmentation

• Poor standards lead to fragmentation

• Hard to build things on top of fragmented environments

• Imagine building for single browser. Not Mosaic.


Android Fragmentation

http://www.businessinsider.com/android-fragmentation-chart-from-opensignalmaps-2012-5




Flexibility vs Standardization

• Standardization can be rigid; can kill flexibility

• Especially as the target moves

• E.g. contrib.auth

• Flexibility allows preference, innovation

• Best of both worlds is standardization with structured options of flexibility

• Django does this well


Django

• Django is a set of higher-order building blocks

• If blocks are too large, sacrifices what can be made (i.e. not flexibile)

• If blocks are too small, becomes inefficient and unmanageable

• If block types are wrong, not loosely coupled (i.e. non-orthogonal)


3 Stages of Project Structure Grief• Denial

• Bargaining

• Acceptance


Denial

• Stuff required libraries in a ‘lib’ directory

• Stick django there too

• Name everything MyProjectSomething

I’ll just take the easy way out...


Denial

• project starts to get bloated

• version management becomes a headache

• upgrade paths become tangled

I’ll just take the easy way out...

Why this doesn’t work


Bargaining

1. Have the team argue about which way to do things

2. Assign someone to write a document about it

3. ???

4. Ignore it when you start a new project

We’ll just adopt some standard practices...


Bargaining

• Projects become inconsistent

• It is not in source control

• No way to manage changes across projects

• There is no history of changes or discussion

We’ll just adopt some standard practices...

Why this doesn’t work


Acceptance

• requirements.txt, version management

• also solves the ‘git submodule’ problem

• virtualenv is great, always use it

Embrace the community


Acceptance

• dependency management

• concurrent project management

• sand-boxing

• solves the “chicken-n-egg” problem

• replication

No, really, use virtualenv


What’s in a template?

• For a django project, there are two styles:

1. toplevel directory is python code

2. toplevel directory is a website project


Toplevel is python code

• Pros

• More pythonic; Everything is python code

• It just works

• Cons

• Everything is python code

• Where do non public files go?

Django’s default template

./manage.py

./my_project/__init__.py

./my_project/settings.py

./my_project/urls.py

./my_project/wsgi.py


Default Django Project./manage.py./my_project/__init__.py./my_project/settings.py./my_project/urls.py./my_project/wsgi.py./my_project/my_app/__init__.py./my_project/my_app/models.py

• How about ./my_project/my_app/?

• Name then becomes ‘my_project.my_app’

• What about reusability?

• What if I want to rename my_project?

•Where do apps go?


Default Django Project

• How about ./my_app/?

• Better for reusability, but...

• Now I have to add ./ to the PYTHON_PATH

•Where do apps go?./manage.py./my_project/__init__.py./my_project/settings.py./my_project/urls.py./my_project/wsgi.py./my_app/__init__.py./my_app/models.py


Default Django Project

• Was it ‘myapp1’ or ‘myapp2’ that contains the main urls and settings...

•What was the name again?

./manage.py

./myapp1/models.py

./myapp1/urls.py

./myapp2/models.py

./myapp2/urls.py

./myapp3/models.py

./myapp3/urls.py


Toplevel is website project

• Pros

• Closer to real world environments

• Consistency across projects

• Cons

• ??? None I guess!

http://github.com/concentricsky/djenesis-template.git

./manage.py

./wsgi.py

./requirements.txt

./apps/mainsite/__init__.py

./apps/mainsite/settings.py

./apps/mainsite/settings_local.py

./apps/mainsite/urls.py

./apps/my_app/models.py

./apps/my_app/urls.py




• only have to add ./apps to PYTHON_PATH

• everything else is ignored by WSGI server

./manage.py

./wsgi.py

./requirements.txt



./apps/mainsite/settings_local.py.example




Python code is sandboxed



• I always know where to look first: ./apps/mainsite/urls.py

• More important than it may seem at first blush

./manage.py

./wsgi.py

./requirements.txt







Consistent Entry Point Name



• Loaded after settings.py as overrides.

• Easily set up a local development environment.

• Production credentials are at no risk.

• Easily deploy devel, staging, production stacks.

./manage.py

./wsgi.py

./requirements.txt







settings_local



• Library dependencies and versioning is simple.

• New developers can start working quicker.

• Simple virtualenv integration

./manage.py

./wsgi.py

./requirements.txt







I have requirements.txt



Okay, but now what?

1. Install and use virtualenv

2. Design and use a template


How hard can it be?


Taking a closer look

$ git clone http://github.com/my/template.git /tmp/mytemplate

$ pip install django>=1.4

$ django-admin.py startproject --template=/tmp/mytemplate ~/myproject/code

$ rm -rf /tmp/mytemplate

$ mkvirtualenv myproject && workon myprojectOR

$ virtualenv ~/myproject/env && source ~/myproject/env/bin/activate

THEN


http://github.com/my/template.git

http://github.com/my/template.git

Our approach

Djenesis: our solution to the bootstrapping problem


djenesis

• What is it?

• A tool that facilitates the process of creating new and setting up existing Django projects.

• Available on github & PyPi (apache license)

• github.com/concentricsky/djenesis


djenesis - Benefits

• Simplifies routine workflows for:

• Starting a new project

• Getting an existing project up and running with ease

• Encourages developers to use a standard project structure and best python practices (virtualenv)

• Allows external project templates

• Supports git, svn, hg URLs

• Supports Django 1.4 project templates


djenesis - History

1. Single zip file of a stub project

2. Script that wrapped stub

3. Decouples templates from script

4. Can use external templates, git URL

5. Now: supports Django 1.4 templates


djenesis - External Templates

• Inflating a project from a git/svn/hg source means you can (and should) version control your project templates

• Collaborate with your internal teams on an ideal project structure


djenesis Stack


djenesis - Setup

• Requires virtualenv

• Install virtualenv through pip or easy_install

• Install djenesis through pip or easy_install


djenesis – New Project

Before$ virtualenv ~/myproject/env$ source ~/myproject/env/bin/activate$ pip install django$ git clone github.com/my/template.git /tmp/tmpl$ django-admin.py startproject --template=/tmp/tmpl ~/myproject/code$ rm -rf /tmp/tmpl

With djenesis$ djenesis myproject git+https://github.com/my/template.git$ source ~/myproject/env/bin/activate


djenesis – Existing Project

Before$ virtualenv ~/myproject/env$ source ~/myproject/env/bin/activate$ git clone github.com/my/project.git ~/myproject/code$ pip install -r ~/myproject/code/requirements.txt

With djenesis$ djenesis myproject -i git+https://github.com/my/project.git$ source ~/myproject/env/bin/activate


djenesis - Configuration

• Use a default template DJENESIS_DEFAULT_TEMPLATE=/path/to/template

• Always use virtualenvwrapper (if installed) DJENESIS_VIRTUALENVWRAPPER=1

• Set these in your .bashrc


djenesis – Next?

1. startapp template in project template

2. setup.py Support

3. djenesis post-create hooks for common tasks (copy local settings, syncdb, import fixtures)

4. Activate virtualenv when djenesis is called


BootstrappingDjango with easeOSCON 2012www.concentricsky.com@concentricsky // biglan / wiggins / bothun




Technology

Django Bootstrapping with Ease