Django SF Meetup, January 2010files.meetup.com/1333474/DjangoMeetupJanuary2010.pdf · Django SF Meetup, January 2010 At Google and with Pizza and with Andy Smith thereÕs soda too

Date

Django SF Meetup, January 2010At Google and with Pizza and with Andy Smith

there’s soda too

and wow, that’s quite a lot of people -schmooze-,

I hope I can get all this AV stuff working nicely

too bad they don’t have one of those cool remote mics like at ted, i could maybe do a little song and dance up here for y’all, waving my arms around and everything

1Thursday, January 28, 2010

Date

I Had Trouble With The TitleThe Following Are Some Failed (?) Attempts

There were a few different topics I had that all sort of fit together but didn’t have a really easy theme that also sounded good


Date

Of Interest To Various PartiesMusings On Django, App Engine and The Future


Date

Djevelopers, Djevelopers, Djevelopers, DjevelopersDjevelopers, Djevelopers, Djevelopers, Djevelopers

it’s a steve ballmer thing


Date

Django and App EngineSomething Pretentious Sounding


Date

How About That Apple Tablet?!Friggin Awesome!

actually, what a let down, it’s a big ipod touch, even at best it feels like using that “navigator” app that we were forced into in 2nd grade.

I had such high hopes, too.


Date

EditorialA Keynote Theme By Apple


Date

Everything Looks GreatWhen Written In A Keynote Theme


Date

Don’t You Think?Of Course You Do


I’m going to talk about three thingsWeb Development on App EngineDjango on App Enginedjango-nonrel

that’s me with a beard, I cut it off around the beginning of the year, i sort of miss it. -single tear-


Date

Web Development on App EngineIs Awesome.

Always Write Everything So Ominous Messages are Expected.

Just kidding.

Well, I mean you should anyway, it’s just not an App Engine specific thing.


But Andy, tell us whyApp Engine Is Awesome to develop for because of all the things it lets you do and how many painful places it frees you from having to go

No. Okay, fine.


But why should we believe you?Because I speak confidently and persuasively.

I come from a web development background, been making web pages since most of you were in diapers, etc etc.

That’s not true but it gets closer every year.

It’s weird working in an industry where the average age of your coworkers seems to go down


What’s web development anyway?Is it Apache, mod_python, nginx, memcache, Twisted, cron jobs, logging utils, mail servers, MySQL servers, nagios, ganglia, capistrano, daemon tools and CPAN?

I’ve configured many instances of:Apache,mod_python,nginx,memcache,Twisted services,cron jobs,logging utilities,mail servers,MySQL servers,nagios or other monitoring systems,ganglia (I don’t even remember what this for anymore),capistrano,daemon toolsand far more CPAN (that’s perl for you lucky people who haven’t heard of it... everything has a perl dependency if you dig deep enough


I hope not


App Engine takes that all awayAll of it.


Write your code, hit deploy


A long-winded anecdote(about launching things on App Engine) with pictures!

One of my favorite stories to tell is an anecdote about launching Jaiku on App Engine and how anti-climactic it can feel (but it is a good anti-climactic)



You write a bunch of awesome code

Writing



You test the hell out of it

Writing Testing



Writing Testing Deploying

Finally you deploy it




Some big blogger writes about it

Blogging


The Old Way


You start watching the load on your database

Blogging

Load


The Old Way


Hoping that memcache fills up fast enough

Blogging

Load Memcache


The Old Way


As http instances start to stack up

Blogging

Load Memcache Apache


The Old Way


Fearing the dreaded connections exceeded message

Blogging



502 Gateway Error


Or worse, 502 Gateway Error

Blogging



The App Engine Way


But not on App Engine, you watch the traffic go up

Blogging

Traffic


The App Engine Way


The error rate stays low (you did test, right?)

Blogging

Traffic Errors


The App Engine Way


Then you grab yourself some beer

Blogging

Traffic Errors Beer


The App Engine Way


Some pizza

Blogging

Traffic Errors Beer Pizza


The App Engine Way


And watch some Star Trek

Blogging

Traffic Errors Beer Pizza Star Trek


App Engine has limits (for now)But mostly stuff you want too to make a good website.

Any App Engine developers in the room are welcome to shout something out if any of things I say you can’t do you _can_ actually do, though it’d be swell if you kept any claims about what it can’t do until question time ;)


I did another presentation on thisI’ll go quickly, you can read: http://term.ie/data/MigratingFromLampToAppEngine.pdf

Apologies to those of you saw it in Prague or read it online (though, sweet!) this is updated a bit and quick and is mostly geared as an introductory overview of how developing on app engine will be different


http://term.ie/data/MigratingFromLampToAppEngine

http://term.ie/data/MigratingFromLampToAppEngine

Say No to JOINsAnd GROUP BY and nested queries and a variety of other SQLisms.

joins have a tendency to suddenly explode when part of the dataset trigger certain conditions, usually running out of sorting memory or missing an index

it’s not as scary as it sounds!

most of these things are just your habitual way of doing things and in many cases the alternate approach is just as, if not more elegant, plus you’re really going to love thinking about things differently, it makes you feel invincible


But it is really fastYou can usually just make two calls instead of a JOIN, primary key lookups can be cached.

doing a query for your main data set then another to get the associated data is usually fast enough that you’ll barely notice

pretty much all basic types of queries hit an index directly meaning the lookup time for things is fast fast fast, and since you’re already thinking in terms of useful primary keys for direct Gets instead of Queries where possible, individual lookups are perfect for caching


DenormalizationDon’t be afraid to store blobs: if you aren’t indexing it don’t make it a property.

denormalization usually means that you aren’t going to use join tables as often to link data together but rather keep data that is fetched together in one spot as well as sometimes keep copies of data in multiple places where they will be accessed frequently


Aim for one Query() per pageAt scale your data starts looking a lot like your presentation, shape it how you use it.

On that same note, basically your goal would be to only make one slow call per page, that’s not always possible but you may be surprised at how well you can optimize your data once you start thinking of it this way


Avoid countingYou probably don’t really need to very often, but if you do compute it at write-time.

And the bigger the number the less accurate it really needs to be.

App Engine, or if you’ve used InnoDB on MySQL, isn’t really keeping track of counts for, so running a Count() call is quite a lot slower than you might expect it to be, and if you have a really big dataset it is pretty much useless.

If you need the number of something, you need to compute when you add new things to that list.

And if I can save anybody some stress by convincing them that the number of significant digits goes down as the orders of magnitude increase, my job here is done.


Say No to Long RequestsYou don’t want your users waiting that long anyway.

You actually have quite a lot of time, like 30 seconds, to do stuff, much longer than I’d be happy waiting for your site to load


Task queues to handle big jobsSet them off in the background and get it done on your time, not your users’.

Task queues are really great, if you are familiar with web hooks the concept will come very naturally to you, basically you register a “task” that is just a url to call and payload, the system handles the rest, handles duplicates for you as long as you name your tasks and plenty more


Say No to Checked Foreign KeysBut there are libraries for everything.

While easing the burden of data integrity checked foreign keys can require a level of transactional support that is not free on App Engine’s architecture. User-land libraries are able to accomplish many of these types of goals.

Every so often I complain to one of the other App Engine devs, “why don’t we do this thing I want” and it seems like the answer is almost always, “actually, we do, so and so wrote a library for that months ago”


Be idempotentYou need to prevent duplicates, any unique properties should be in your primary key.

Idempotency is a good strategy for data integrity.

Idempotent generally means that repeating an action shouldn’t result in any bad side effects.


Roll forwardIf something broke, trying it again should fix it.

Along those lines, a good habit to be in is to, should you find yourself repeating something, ensure that everything is order before you decide to skip doing any work.

Example: maybe the reason this is being repeated is because it didn’t complete successfully last time.


Clean up your messIf you get some data that looks broken, try to fix it or make sure your code doesn’t need it.

This goes along with rolling forward, if you fetch something but it doesn’t seem right, it is probably a good idea to try to fix it.

Example: When a user is logging in is a good time to ensure that it has all the required initial setup.


Case sensitivityThere’s no utf8_unicode_ci here, normalize and separate display from lookup.

Quite a few mysql developers take advantage of case insensitive string matching, so watch out.


Entity Groups are complicatedThey certainly don’t work the way I think they do, so do your reading before using them.

Quite a lot of people guess wrong when they start using them, look before you leap


Be ready for timeoutsThey aren’t the end of the world on App Engine.

In a typical LAMP stack a timeout usually means your whole app timed out in responding to the load balancer, in app engine individual calls can time out and it isn’t the end of the world.

Various types of calls can timeout long before your request does, you may want to retry the call or take the user to another page that helps them figure out what else to do.


The list keeps getting shorterEvery time I talk about this it is harder to think of things you can’t do.

App Engine actually supports a ridiculous amount of stuff, beyond normal things you expect in a web server it also has

quick access to XMPP (something that is really quite a pain to set up on your own)

and more recently the Blobstore that is pretty close to having something like S3 built in


Date

Django on App EngineThis is where you come in!


There are no code examples hereBut there are plenty of out there, I will put some up at the end of the show.

These are mostly high-level concepts related to working with Django


WSGI is goodApp Engine provides a basic WSGI interface, Django uses that to hook in.

If you haven’t used it before, WSGI (Whiskey) is pretty much the standard interface most web stacks try to provide or write to, it means something like... Web Services Gateway Interface.


google-app-engine-djangoIs generally the library I talk about when talking on this subject, but there are others.

I worked on the creation of the Google App Engine Helper for Django (let too many lawyers get into that naming kitchen).

In general it is less interested in supporting Django’s various accoutrement than some of the other projects,


0 to appspot.com in secondspython manage.py update

Django is definitely the awesomest web framework and definitely the awesomest web framework to use on App Engine.


You aren’t using SQLA lot of Django “apps” assume you are and have literal SQL statements in them.

Django “apps” break all forms of perceived encapsulation boundaries, expect the unexpected.


A word on reusable “apps”Many of the problems efforts to support Django on App Engine have are based on this.

The API defined for an “app” is huge, and for the most part I think the word should be extinguished in its use for things that are libraries, and that a plugin/hook system with defined apis would help the platform forward.

Can I get a hell yeah?

A component designed for reusability should have a well defined and small API it is written to, otherwise the domain it is reusable under is rather limited.


Port your modelsYou’re going to have to use App Engine’s models or a wrapper to get at the datastore.

I’ve never been a fan of most ORM systems to begin with so I almost always use my preferred db layer with any framework I am in.

App Engine’s isn’t very different from Django’s all in all, so you shouldn’t be afraid.

The helper provides a nice wrapper for the GAE models that make them look enough like Django models that things like fixtures and tests work reasonably.


Package your dependenciesIf you use a lot of external libraries you’re going to need to zip them and upload them.

This is generally pretty easy and really is just running “zip” on the python directory.


Date

There is a built-in admin interfaceAnd it isn’t Django’s but it is good. http://localhost:8080/_ah/admin

It works quite nicely.

App-engine-patch has some support for Django’s admin interface but it is difficult to emulate on a system with no SQL.


http://localhost:8080/_ah/admin

http://localhost:8080/_ah/admin

A lot is supported but a lot isn’tBut most of it could be.

There are plenty of things that are tricky to get right as far as getting Django working on App Engine. All the main parts are there for certain definitions of “main” and most of the rest is just due to stray SQL statements.


Dramatic lead-in... could be ... could be ... could be ...


Date

django-nonrelThe future of Django.


multi-db is the beginningIt starts with master-slave, but what about other types of databases.

People are starting to build higher quality websites that want to get the performance benefits of multiple databases.


NoSQL is gaining popularityDifferent database systems are much better for different kinds of problems.

Many people have been burned by MySQL and relational databases in general, they are looking to other databases for help.


Django should helpOtherwise where else can those users go?

Moving forward, more and more people will be turning to alternate databases for their websites, more of Django needs to not barf when SQL is not available.


django-nonrel is taking it onAnd we’re not actually too far from getting the basics.

The project has been around for a little while and grew out of frustrations with trying to write these libraries to monkey patch django to support app engine


SQLCompiler helped a bunchIt gives a direct entry point into the database backend that can be overloaded.

Prior to SQLCompiler django-nonrel was faced with porting the existing SQL backend over to some new intermediate format


It’s still an uphill battledjango-nonrel has to try to emulate SQLisms because Django uses them profusely.

Django encourages many practices that only work on a standard SQL system, this forces other backends to attempt to emulate features that are not performant.


Get on the NoSQL trainDjango needs small changes all over to make it more reasonable in a NoSQL environment.

There are lots of places to change, those of us in support of django-nonrel’s goals need to start finding alternatives to the SQL-centric Django implementations and patching them so that we aren’t left behind when this train leaves the station.


Date

Big Finish!Hooray!

There are some App Engine developers in the room, go tell them how awesome app engine sounds and how much you want to use django on it.


Useful Information

✤ Email me: [email protected]

✤ Django code on App Engine

✤ http://code.google.com/p/rietveld/

✤ http://code.google.com/p/google-app-engine-django/

✤ http://code.google.com/p/jaikuengine

✤ This: http://term.ie/data/DjangoMeetupJanuary2010.pdf

✤ Other: http://term.ie/data/MigratingFromLampToAppEngine.pdf


mailto:[email protected]

mailto:[email protected]

http://code.google.com/p/rietveld

http://code.google.com/p/rietveld

http://code.google.com/p/google-app-engine-django/

http://code.google.com/p/google-app-engine-django/

http://code.google.com/p/jaikuengine

http://code.google.com/p/jaikuengine

http://term.ie/data/DjangoMeetupJanuary2010.pdf

http://term.ie/data/DjangoMeetupJanuary2010.pdf

Documents

Django SF Meetup, January 2010files.meetup.com/1333474/DjangoMeetupJanuary2010.pdf · Django SF Meetup, January 2010 At Google and with Pizza and with Andy Smith thereÕs soda too