Download pdf - Large problems, Mostly Solved

Large ProblemsMostly Solved

Eric Holscherhttp://ericholscher.com

Djangocon 2010

Wednesday, September 8, 2010

http://ericholscher.com

http://ericholscher.com

What this talk isn’t

» In depth

» Overly technical

» Conclusive


What this talk is

» Providing current best-of-class solutions to common problems

» Identifying problems that aren’t currently solved well


Overview

» Patterns among good apps

» The problems that are mostly solved

» Unsolved Problems


A good application


Leverages previous knowledge/APIs


Easy Setup


Good upgrade path


Good documentation


Well Tested


Solves a real problem


Mostly Solved Problems


Search


Haystack

» 9782 lines of docs

» 8882 lines of tests

» http://haystacksearch.org/

» Written by Daniel Lindsley


http://haystacksearch.org

http://haystacksearch.org

Line counts

find . |egrep "rst|txt" |xargs wc -l |cut -c1-8 | awk '{s+=$1} END {print s}'

find . |egrep "py" |xargs wc -l |cut -c1-8 | awk '{s+=$1} END {print s}'

#FORE!


Setup» pip install django-haystack

» Add ‘haystack’ to Installed Apps

» Settings for siteconf and backend

» Create Search Site

» Create Search Indexes

» Add views

» Reindex


Upgrade Path

» Simple backend

» Whoosh

» Solr


Beautiful APIs

unfriendly_results = SearchQuerySet().exclude(content='hello').filter(content='world')

unfriendly_results.order_by('-pub_date')[:5]


Beautiful APIs

class NoteIndex(SearchIndex): text = CharField(document=True, use_template=True) author = CharField(model_attr='user') pub_date = DateTimeField(model_attr='pub_date')

def get_queryset(self): return Note.objects.filter(pub_date__lte=datetime.datetime.now())


Awesome Features

» Faceting

» Highlighting

» More Like This

» Easy Customization



Amazing full-text search


In an hour.


Documentation


Sphinx



» http://sphinx.pocoo.org/

» Written by Georg Brandl


http://sphinx.pocoo.org

http://sphinx.pocoo.org

Setup

» pip install Sphinx

» sphinx-quickstart

» Edit/Create Restructured Text files

» make html


Awesome features

» Uses ReStructuredText

» Easily generate PDFs

» Link between your own and other docs

» Themes


Shameless Self Promotion

» Use Read The Docs

» Host Sphinx docs, or create them there.

» http://rtfd.org

» http://media.readthedocs.org/django.pdf


http://rtfd.org

http://rtfd.org

http://media.readthedocs.org/django.pdf





Make Beautiful Documentation With Tools

You Know


Database Migrations


South



» http://south.aeracode.org/

» Written by Andrew Godwin


http://south.aeracode.org

http://south.aeracode.org

Setup

» pip install south

» south in Installed Apps

» manage.py schemamigration app --initial

» manage.py schemamigratoin app name --auto

» manage.py migrate


Beautiful API

def forwards(self, orm): for adopter in orm.Adopter.objects.all(): try: adopter.first_name, adopter.last_name = adopter.name.split() except ValueError: adopter.first_name, adopter.last_name = adopter.name, "" adopter.save()


Boring


Safe, Painless Data Migration


Delayed Execution


Run commands out of process


Celery



» http://celeryproject.org/

» Written by Ask Solem


http://celeryproject.org

http://celeryproject.org

Setup

» pip install django-celery

» Add ‘djcelery’ to Installed Apps

» Syncdb

» Configure your settings backend

» Add @task decorator to function


When and Why

» Makes your site fast

» Use for tasks that users don’t immediately care about

» Even if they do, return fast to a waiting page


Examples

» Activity

» Sending Email

» Complex DB queries



Path

» Start with ALWAYS_EAGER

» Use a ghetto queue

» Rabbit MQ


Awesome features

» Run on multiple machines

» Cron replacement

» Good error reporting


Magically make your code async


Remote Execution


Run commands on remote servers (for deployment)


Fabric



» http://fabfile.org

» Written by Christian Vest Hansen & Jeff Forcier


http://fabfile.org

http://fabfile.org

Setup

» pip install fabric

» Add commands to a fabfile.py

» fab <command>


Simple example

#Fabfilefrom fabric.api import run

def host_type(): run('uname -s')

# Command$ fab -H example.com host_type


Write python to deploy your code


Deployment


Gunicorn



» http://gunicorn.org/

» Created by Benoit Chesneau


http://gunicorn.org

http://gunicorn.org

Setup

» pip install gunicorn

» add gunicorn to your Installed Apps

» manage.py run_gunicorn


Need to install Nginx in front


Nginx setup

server { listen 80; server_name example.com; access_log /var/log/nginx/example.log;

location / { proxy_pass http://127.0.0.1:8080; }}


http://127.0.0.1:1337

http://127.0.0.1:1337

Awesome Features

» Async backend workers

» Simple configuration

» Python


Stupidly Simple, Production Ready

Deployment


Packaging


Pip & Virtualenv

» Written by Ian Bicking

» http://pip-installer.org

» http://virtualenv.openplans.org


http://pip-installer.org

http://pip-installer.org

http://virtualenv.openplans.org

http://virtualenv.openplans.org

Set up

» sudo easy_install pip # :D

» sudo pip install virtualenv

» virtualenv awesome_env

» source awesome_env/bin/activate

» pip install Django


Awesome Features

» Separate environments for each application or deployment

» Install code from repos

» Doesn’t use eggs

» User-owned environment


Adds community standards to something everyone was

doing their own way


No more PYTHONPATH hacking


Competing Applications


APIs


APIs

» TastyPie

» Piston


Tagging


Tagging

» Django Taggit *

» Django Tagging


Continuous Integration


Continuous Integration

» Hudson *

» Buildbot


Smaller Problems


Debugging in Development


django-debug-toolbar

» http://github.com/robhudson/django-debug-toolbar/

» Written by Rob Hudson


http://github.com/robhudson/django-debug-toolbar/





Profiles & Registration


django-[profiles|registration]

» http://bitbucket.org/ubernostrum/django-profiles/

» http://bitbucket.org/ubernostrum/django-registration/

» Written by James Bennett


http://bitbucket.org/ubernostrum/django-profiles/




http://bitbucket.org/ubernostrum/django-registration/




Filtering


django-filter

» http://github.com/alex/django-filter

» Written by Alex Gaynor


http://github.com/alex/django-filter

http://github.com/alex/django-filter


Finding packages


Djangopackages.com


Unsolved Problems


Template Tags


Logging


Model Introspection


Class Based Views


Source code arrangement


Notifications/Email


Debugging in Production


OpenID/OAuth


VCS Abstraction


Positive Note


Cal’s list of problems ’08» Multiple Databases

» Multiple multi-keyed cache requests

» Slow HTTP requests while we're doing other stuff

» ORM creates dumb SQL

» Verbose template syntax

» No query debugger

» Ugly SQL syntax

» Commit all fields at once

» No mascot

» No deployment system

» Migrating ModelsWednesday, September 8, 2010

Cal’s list of problems ’08» Multiple Databases

» Multiple multi-keyed cache requests

» Slow HTTP requests while we're doing other stuff

» ORM creates dumb SQL

» Verbose template syntax

» No query debugger

» Ugly SQL syntax (Table aliases)

» Commit all fields at once

» No mascot

» No deployment system

» Extending modelsWednesday, September 8, 2010

Remember

» Leverage good applications in the community

» Take previous knowledge and apply it to new situations

» You can create an awesome application used by other people


Questions?

» [email protected]

» twitter.com/ericholscher


mailto:[email protected]

mailto:[email protected]