View
223
Download
0
Category
Preview:
Citation preview
8/8/2019 Pick n Mix DJUGL Sep 09
1/56
Ben Firshman
8/8/2019 Pick n Mix DJUGL Sep 09
2/56
8/8/2019 Pick n Mix DJUGL Sep 09
3/56
Performance:* Incredibly important, especially now web applications doing increasingly complex things.
* Amazon found that 100ms of latency caused 1% loss in sales* Extra half a second on Google search results caused 20% drop in trafc
* Especially when altering data, it still needs to be responsive:* Posting a tweet to followers* Uploading images* Importing API data
* Running stuasynchronously
8/8/2019 Pick n Mix DJUGL Sep 09
4/56
* Running TASKS asynchronously
8/8/2019 Pick n Mix DJUGL Sep 09
5/56
Tasks
Independent components
Loosely coupled
Easily scalable
* Independent components that can run concurrently* Loosely coupled: only depends on initial input* Thanks to which, theyre easily scalable.
* Essentially:* Stu you can send o to run in the background.* Stu the user doesnt immediately care about.
8/8/2019 Pick n Mix DJUGL Sep 09
6/56
Application
* Say we have our application* and a bunch of tasks that need running* What runs these tasks?
8/8/2019 Pick n Mix DJUGL Sep 09
7/56
Application
* Say we have our application* and a bunch of tasks that need running* What runs these tasks?
8/8/2019 Pick n Mix DJUGL Sep 09
8/56
Application
??
?
?
* Say we have our application* and a bunch of tasks that need running* What runs these tasks?
8/8/2019 Pick n Mix DJUGL Sep 09
9/56
Application
Worker Worker
* Workers process the tasks.* Unlike ghetto queues done with cron jobs and databases, these are processes that havetasks pushed to them* These could either be on the same server as the application, or, thanks to independence,spread out onto many* Some way for application to publish tasks that need to be completed and distributing toworkers
8/8/2019 Pick n Mix DJUGL Sep 09
10/56
Application
Message Queue
* Message Queues are useful for this. (explain)* The messages can just be pickled tasks
8/8/2019 Pick n Mix DJUGL Sep 09
11/56
Application
Message Queue
* Message Queues are useful for this. (explain)* The messages can just be pickled tasks
8/8/2019 Pick n Mix DJUGL Sep 09
12/56
Application
Message Queue
8/8/2019 Pick n Mix DJUGL Sep 09
13/56
Application
Message Queue
Application
* Nice thing with message queues: we can have multiple applications push things to thequeue.* Means the frontend servers can be scaled horizontally without any of the taskinfrastructure caring
8/8/2019 Pick n Mix DJUGL Sep 09
14/56
Application
Message Queue
Application
* Nice thing with message queues: we can have multiple applications push things to thequeue.* Means the frontend servers can be scaled horizontally without any of the taskinfrastructure caring
8/8/2019 Pick n Mix DJUGL Sep 09
15/56
Application
Message Queue
Application
Worker Worker
8/8/2019 Pick n Mix DJUGL Sep 09
16/56
Application
Message Queue
Application
Worker Worker
8/8/2019 Pick n Mix DJUGL Sep 09
17/56
Application
Message Queue
Application
Worker Worker
8/8/2019 Pick n Mix DJUGL Sep 09
18/56
Application
Message Queue
Application
Worker Worker
8/8/2019 Pick n Mix DJUGL Sep 09
19/56
Application
Message Queue
Application
Worker Worker
8/8/2019 Pick n Mix DJUGL Sep 09
20/56
Application
Message Queue
Application
Worker Worker
8/8/2019 Pick n Mix DJUGL Sep 09
21/56
* This is where celery comes in* Ask Solem Hoel has written an excellent piece of software that provides everything youneed for a task queue
8/8/2019 Pick n Mix DJUGL Sep 09
22/56
Celery
Worker daemon
8/8/2019 Pick n Mix DJUGL Sep 09
23/56
Celery
Worker daemon API for defining and executing tasks
8/8/2019 Pick n Mix DJUGL Sep 09
24/56
Celery
Worker daemon API for defining and executing tasks
Handles communication with message queue (AMPQ or
STOMP)
8/8/2019 Pick n Mix DJUGL Sep 09
25/56
8/8/2019 Pick n Mix DJUGL Sep 09
26/56
Code!
>>> from myapp.tasks import ImportTweetsTask
>>> ImportTweetsTask.delay(username=londonpython)
* This is the way a simple task is run.* If youre just writing something that writes the result to the database, this is fine
8/8/2019 Pick n Mix DJUGL Sep 09
27/56
Code!
>>> result = ImportTweetsTask.delay(username=londonpython)
>>> result.ready()False
>>> result.ready()True
>>> [t.text for t in result.result][u'I will also be looking into paying for pizza forDJUGL attendees next Thursday at Global Radio officesin Leicester Square.', ...]
8/8/2019 Pick n Mix DJUGL Sep 09
28/56
map
>>> import operator
>>> from celery.task import dmap
>>> dmap(operator.add, [[1, 2], [4, 2], [3, 8]])[3, 6, 11]
* Callable is pickled, so it must be available on the workers path
8/8/2019 Pick n Mix DJUGL Sep 09
29/56
More goodies
Periodic tasks as a distributed replacement for cron
View for Ajax to check status
Statistics
Task sets for progress bars
Retry tasks on failure
* Stats: how many tasks have been executed by type, time taken etc
8/8/2019 Pick n Mix DJUGL Sep 09
30/56
http://www.celeryq.org/
http://www.celeryq.org/http://www.celeryq.org/8/8/2019 Pick n Mix DJUGL Sep 09
31/56
8/8/2019 Pick n Mix DJUGL Sep 09
32/56
py.test
No boilerplate tests
Useful test failure information
Extensibility
* No boilerplate: Automatic detection of tests and use of built in Python features like assert
8/8/2019 Pick n Mix DJUGL Sep 09
33/56
Code!
def test_equality():x = 5assert x == 5
def test_not_equal():y = 9assert y != 2
def test_articles_exist(articles):assert articles.objects.count() > 0
* funcargs provide fixtures
8/8/2019 Pick n Mix DJUGL Sep 09
34/56
8/8/2019 Pick n Mix DJUGL Sep 09
35/56
Funcargs
def test_something(client):assert 'Success!' in client.get('/path/').content
def test_article_view(rf):request = rf.get('/article/4/')response = article_view(request, 4)assert 'Test title' in response.content
8/8/2019 Pick n Mix DJUGL Sep 09
36/56
Funcargs cont.
def test_middleware(settings, client):settings.MIDDLEWARE_CLASSES = (
'app.middleware.SomeMiddleware',)assert 'Middleware works!' in client.get('/')
8/8/2019 Pick n Mix DJUGL Sep 09
37/56
Decorators
@py.test.params([dict(a=1, b=2), dict(a=3, b=3)])
def test_equals(a, b):assert a == b
@py.test.urls('myapp.test_urls')def test_something(client):
assert 'Success!' in client.get('/some_path/')
8/8/2019 Pick n Mix DJUGL Sep 09
38/56
8/8/2019 Pick n Mix DJUGL Sep 09
39/56
8/8/2019 Pick n Mix DJUGL Sep 09
40/56
syndication-view
An attempt to clean up django.contrib.syndication
8/8/2019 Pick n Mix DJUGL Sep 09
41/56
syndication-view
An attempt to clean up django.contrib.syndication
Feeds as class based views
d
8/8/2019 Pick n Mix DJUGL Sep 09
42/56
syndication-view
An attempt to clean up django.contrib.syndication
Feeds as class based views
Backwards compatible
d
8/8/2019 Pick n Mix DJUGL Sep 09
43/56
syndication-view
An attempt to clean up django.contrib.syndication
Feeds as class based views
Backwards compatible
Eventually, refactoring generation of feeds including full
Atom support
C d !
8/8/2019 Pick n Mix DJUGL Sep 09
44/56
Code!
from django.contrib.syndication.feeds import Feedfrom articles.models import Article
class ArticleFeed(Feed):title = "example.com news"link = "/sitenews/"description = "Latest news from example.com."
def items(self):return Article.objects.all()[:15]
Y k
8/8/2019 Pick n Mix DJUGL Sep 09
45/56
Yuck.
from articles.feeds import ArticleFeed
feeds = {'articles': ArticleFeed,
}
urlpatterns = patterns('',# ...(r'^feeds/(?P.*)/$',
'django.contrib.syndication.views.feed',
{'feed_dict': feeds}),# ...
)
C d !
8/8/2019 Pick n Mix DJUGL Sep 09
46/56
Code!
from syndication.views import Feedfrom articles.models import Article
class ArticleFeed(Feed):title = "example.com news"link = "/sitenews/"description = "Latest news from example.com."
def items(self):return Article.objects.all()[:15]
M h b
8/8/2019 Pick n Mix DJUGL Sep 09
47/56
Much better.
from articles.feeds import ArticleFeed
urlpatterns = patterns('',# ...(r'^articles/feed/$', ArticleFeed()),# ...
)
8/8/2019 Pick n Mix DJUGL Sep 09
48/56
http://github.com/bfirsh/syndication-view/
If you have any pet hates of contrib.syndication, let me know!
http://github.com/bfirsh/syndication-view/http://github.com/bfirsh/syndication-view/8/8/2019 Pick n Mix DJUGL Sep 09
49/56
django mptt
8/8/2019 Pick n Mix DJUGL Sep 09
50/56
django-mptt
Making trees out of Django models.
Efficient retrieval.
Methods added to models for traversing trees(get_children(), get_root(), get_decendants()etc)
* Efcient retrieval as opposed to a simple parent field and retrieving children with relatedname
Code!
8/8/2019 Pick n Mix DJUGL Sep 09
51/56
Code!
from django.db import models
import mptt
class Category(models.Model): name = models.CharField(max_length=50) parent = models.ForeignKey('self', null=True,blank=True, related_name='children')
mptt.register(Category)
8/8/2019 Pick n Mix DJUGL Sep 09
52/56
Code!
8/8/2019 Pick n Mix DJUGL Sep 09
53/56
Code!
from django.db import modelsimport mptt
class Category(mptt.Model): name = models.CharField(max_length=50) parent = models.ForeignKey('self', null=True,blank=True, related_name='children')
8/8/2019 Pick n Mix DJUGL Sep 09
54/56
http://github.com/bfirsh/django-mptt/
http://github.com/bfirsh/django-mptt/http://github.com/bfirsh/django-mptt/8/8/2019 Pick n Mix DJUGL Sep 09
55/56
http://benfirshman.com/@bfirsh
http://twitter.com/bfirshhttp://twitter.com/bfirshhttp://benfirshman.com/http://benfirshman.com/8/8/2019 Pick n Mix DJUGL Sep 09
56/56
Credits
http://www.flickr.com/photos/ilovetechnology/3048585444/http://www.flickr.com/photos/tomhakase/312785554/http://www.flickr.com/photos/pensive-reflections/3829587788/http://www.flickr.com/photos/purpleslog/183842413/
http://www.flickr.com/photos/chrissie64/1425138918/http://www.flickr.com/photos/patlejch/2951799028/http://www.flickr.com/photos/mckenzieo/1601233059/http://www.flickr.com/photos/webel/2429902979/
http://www.flickr.com/photos/webel/2429902979/http://www.flickr.com/photos/webel/2429902979/http://www.flickr.com/photos/mckenzieo/1601233059/http://www.flickr.com/photos/mckenzieo/1601233059/http://www.flickr.com/photos/patlejch/2951799028/http://www.flickr.com/photos/patlejch/2951799028/http://www.flickr.com/photos/chrissie64/1425138918/http://www.flickr.com/photos/chrissie64/1425138918/http://www.flickr.com/photos/purpleslog/183842413/http://www.flickr.com/photos/purpleslog/183842413/http://www.flickr.com/photos/pensive-reflections/3829587788/http://www.flickr.com/photos/pensive-reflections/3829587788/http://www.flickr.com/photos/tomhakase/312785554/http://www.flickr.com/photos/tomhakase/312785554/http://www.flickr.com/photos/ilovetechnology/3048585444/http://www.flickr.com/photos/ilovetechnology/3048585444/Recommended