Upload
emptysquare
View
16.707
Download
5
Tags:
Embed Size (px)
DESCRIPTION
A talk on Python coroutines for asynchronous programming, both *now* in Tornado and Toro, and in the future in Tulip and Python 3.4.
Citation preview
Python CoroutinesPresent and Future
A. Jesse Jiryu Davishttp://emptysquare.net
10gen
Python CoroutinesPresent and Future
Agenda:
• Coroutines are wonderful for async• …but weird.• Understanding the weirdness.• Coroutine Kung Fu.• The Future!
Coroutines arewonderful for async
Async with callbackfrom tornado.web import RequestHandler
class AsyncHandler(RequestHandler): @asynchronous def get(self): http_client = AsyncHTTPClient() http_client.fetch( "http://example.com", callback=self.on_fetch)
def on_fetch(self, response): do_something_with_response(response) self.render("template.html")
Async with coroutine
from tornado.web import RequestHandler
class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() response = yield http_client.fetch( "http://example.com") do_something_with_response(response) self.render("template.html")
Async with coroutine
from tornado.web import RequestHandler
class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
Agenda:
• Coroutines are wonderful for Async I/O• …but weird.
• yield • future • coroutine
yield
def make_generator(): yield 0 yield 1
g = make_generator()print(g)# <generator object make_generator at 0x1006b4b40>
def make_generator(): yield 0 yield 1
g = make_generator()for i in g: print(i)
def make_generator(): yield 0 yield 1
g = make_generator()while True: try: i = g.__next__() print(i) except StopIteration: break
def make_generator(): yield 0 yield 1
g = make_generator()while True: try: i = g.send(None) print(i) except StopIteration: break
Prints: 0 got 10 1 got 11
def make_generator(): return_value = yield 0 print 'got', return_value return_value = yield 1 print 'got', return_value
g = make_generator()i = g.send(None) # Start gwhile True: try: print(i) i = g.send(i + 10) except StopIteration: break
Agenda:
• Coroutines are wonderful for Async I/O• …but weird.
• yield • future • coroutine
Future
class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
future = Future()
future.done() # False
future.set_result('foo')
future.set_exception( SomeException('error message'))
future.add_done_callback(callback)
# Return result or raise errorfuture.result()
class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
Agenda:
• Coroutines are wonderful for Async I/O• …but weird.
• yield • future • coroutine
# part of the Tornado framework
class Runner(object): def __init__(self, make_generator): self.gen = make_generator() # Starts done, with result None self.future = NullFuture()
class Runner(object): # ... def run(self): while True: if not self.future.done(): self.future.add_done_callback(self.run) return
value = self.future.result()
try: self.future = self.gen.send(value) except (StopIteration, Return) as e: return
"recurse"
class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
Agenda:
• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.
@gen.coroutinedef print_code(): response = yield get('http://example.com') print response.code
@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)
print_code()
weird
@gen.coroutinedef print_code(): future = get('http://example.com') response = yield future print response.code
@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)
print_code()weird
@gen.coroutinedef print_code(): future = get('http://example.com') response = yield future print response.code
@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) return response
print_code()
Python 3.3
normal
@gen.coroutinedef print_code(): try: code = yield get('http://example.com') print code except HTTPError, e: print e
@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response.code)
print_code()
normal
Agenda:
• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.• Coroutine Kung Fu.
Fan-out
@gen.coroutinedef f(): client = AsyncHTTPClient() responses = yield [ client.fetch('http://mongodb.org'), client.fetch('http://10gen.com')]
print responses
f()
Fan-out
@gen.coroutinedef f(): client = AsyncHTTPClient() future0 = client.fetch('http://mongodb.org') future1 = client.fetch('http://10gen.com') responses = yield [future0, future1] print responses
f()
Toro
Synchronization Primitives forTornado Coroutines
event = toro.Event()
@gen.coroutinedef waiter(): print "I'll wait right here" yield event.wait() # Yield a Future print "I'm done waiting"
@gen.coroutinedef setter(): print "About to set" event.set() print "Done setting"
waiter()setter()
q = toro.Queue(maxsize=3)
@gen.coroutinedef producer(): for item in range(5): print 'Sending', item yield q.put(item)
@gen.coroutinedef consumer(): while True: item = yield q.get() print '\t\t', 'Got', item
consumer()producer()
$ python producer_consumer.pySending 0
! ! Got 0Sending 1
! ! Got 1Sending 2
! ! Got 2Sending 3
! ! Got 3Sending 4
! ! Got 4
Agenda:
• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.• Kung Fu.• The Future!
Tulip
• A standard event loop
• A standard coroutine library
• For inclusion in Python 3.4
yield from
@tulip.coroutinedef print_code(): response = yield from get('http://example.com') print(response.status)
@tulip.coroutinedef get(url): request = tulip.http.request('GET', url) response = yield from request return response
Tornado
Tulip
normal return
"yield from"
@gen.coroutinedef print_code(): response = yield get('http://example.com') print response.code
@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)
Guidoon
"The Difference Between yield and yield-from":
http://bit.ly/yieldfrom
q = tulip.queues.Queue(maxsize=3)
@tulip.coroutinedef producer(): for item in range(5): print('Sending', item) yield from q.put(item)
@tulip.coroutinedef consumer(): while True: item = yield from q.get() print('\t\t', 'Got', item)
Task(consumer())Task(producer())
StopIteration
A. Jesse Jiryu Davishttp://emptysquare.net
10gen