Marrow: A Meta-Framework for Python 2.6+ and 3.1+

MarrowMeta–Framework for Python 2.6+ and 3.1+

Alice Zoë Bevan–McGregor

Overview

ConfigurationYAML-Based Application Configuration

Introspective ScriptingNon-Imperative Command-Line Parsing

BlueprintTemplate–Derived Directory Trees

Interactive & Command–Line Interrogation

Streaming TemplatesA Python Micro–Language

Server InterfaceModified Tornado IOLoop and IOStream

Server and Protocol Wrappers

HTTP/1.1 WSGI 2 ServerHighly Performant Pure Python HTTP/1.1 Implementation

Object WrappersPEP 444 Request / Response Objects

HTTP Status Code Exception Applications

Middleware / FiltersCompression, Sessions, etc.

Performance & OptimizationsTime for Timeit

CompatibilityPython 2.6+ and 3.1+

Configuration

marrow.config

Unfortunately…

Least Developed(So far.)

Paste Deploy

1 [server] 2 use = marrow.server.http:HTTPServer 3 host = 127.0.0.1, ::1 4 port = 8080, 8088 5 6 [mapping] 7 / = root 8 9 [app:root] 10 use = marrow.server.http.testing:hello 11 name = ConFoo

INI = Evil

Typecasting

1 [server] 2 use = marrow.server.http:HTTPServer 3 host = 127.0.0.1, ::1 4 port = 8080, 8088 5 6 [mapping] 7 / = root 8 9 [app:root] 10 use = marrow.server.http.testing:hello 11 name = ConFoo

String to List

String to Integer

YAML to the Rescue

1 version: 1 2 3 server: 4 use: marrow.server.http:HTTPServer 5 host: ["127.0.0.1", "::1"] 6 port: [8080, 8088] 7 8 mapping: 9 /: *root 10 11 root: &root 12 use: marrow.server.http.testing:hello 13 name: ConFoo

References

Direct Object Access(Entry points are for chumps.)

Logging

16 logging: 17 formatters: 18 brief: 19 format: '%(levelname)-8s: %(name)-15s: %(message)s' 20 handlers: 21 console: 22 class: logging.StreamHandler 23 formatter: brief 24 level: INFO 25 stream: ext://sys.stdout 26 loggers: 27 foo: 28 level: ERROR 29 handlers: [console] 30 root: 31 level: DEBUG 32 handlers: [console]

Scripting

marrow.script

Existing Solutions

sys.argv(painful)

sys.argv(inconsistent)

getopt(old–school)

optparse(old–school)

optparse(deprecated)

argparse(new old–school)

Paste Script(fancy)

Paste Script(entry point magic)

Paste Script(paster <name> […])

Paste Script(context–aware)

Commonality?

Commonality?(un–Pythonic…)

Commonality?(…hideous, hideous, imperative parser objects…)

1 import optparse 2 3 if __name__=="__main__": 4 parser = optparse.OptionParser("usage: %prog [options]

arg1 arg2") 5 parser.add_option("-H", "--host", dest="hostname", 6 default="127.0.0.1", type="string", 7 help="specify hostname to run on") 8 parser.add_option("-p", "--port", dest="portnum", 9 default=80, type="int", 10 help="port number to run on") 11 (options, args) = parser.parse_args() 12 if len(args) != 2: 13 parser.error("incorrect number of arguments") 14 hostname = options.hostname 15 portnum = options.portnum

1 import argparse 2 3 parser = argparse.ArgumentParser(description='Process some integers.') 4 parser.add_argument('integers', metavar='N', type=int, nargs='+', 5 help='an integer for the accumulator') 6 parser.add_argument('--sum', dest='accumulate', action='store_const', 7 const=sum, default=max, 8 help='sum the integers (default: find the max)') 9 10 args = parser.parse_args() 11 print(args.accumulate(args.integers))

Most needed?

Simplicity

Arguments ➢ Variables

1 # encoding: utf-8 2 3 def ultima(required, value=None, name="world",

switch=False, age=18, *args, **kw): 4 print "Hello %s!" % (name, ) 5 6 7 if __name__ == '__main__': 8 __import__('marrow.script').script.execute(ultima)

Usage: ultima.py [OPTIONS] [--name=value...] <required> [value...]

OPTIONS may be one or more of:

-a, --age=VAL Override this value. Default: 18 -h, --help Display this help and exit. -n, --name=VAL Override this value. Default: 'world' -s, --switch Toggle this value. Default: False -v, --value=VAL Set this value.

Additional Detail

__doc__ = Help Text

Decorators

@annotateArgument Typecasting & Validation Callbacks

@describeHelp Text

@shortArgument Abbreviations

@callbacksSimple Callbacks

optparse Example

1 # encoding: utf-8 2 3 import marrow.script 4 5 6 @marrow.script.describe( 7 host = "specify a hostname to run on", 8 port = "port number to run on" 9 ) 10 def serve(arg1, arg2, host="127.0.0.1", port=80): 11 pass 12 13 14 if __name__ == '__main__': 15 marrow.script.execute(serve)

Sub–Commands

Method = Command

__init__ + method

Blueprint

marrow.blueprint

Paste Script

Á La Carte Templates

Best way to describe it…

1 class PackageBlueprint(Blueprint): 2 """Create an installable Python package.""" 3 4 base = 'marrow.blueprint.package/files' 5 engine = 'mako' 6 7 settings = [ 8 Setting( 9 'name', 10 "Project Name", 11 "The name to appear on the Python Package Index, e.g. CluComp.", 12 required=True 13 ), 14 Setting( 15 'package', 16 "Package Name", 17 "The name of the Python package, periods indicating namespaces, e.g. clueless.compiler.", 18 required=True 19 ), 20 # ... 21 ] 22 23 manifest = [ 24 # ... 25 File('setup.py'), 26 File('setup.cfg'), 27 Folder('tests', children=[ 28 File('.keep', 'keep') 29 ]), 30 package 31 ]

1 def package(settings): 2 def recurse(name): 3 head, _, tail = name.partition('.') 4 5 return [ 6 Folder(head, children=[ 7 File('__init__.py', 'namespace.py' if tail else 'init.py') 8 ] + (recurse(tail) if tail else [])) 9 ] 10 11 return recurse(settings.package)

class Setting

target

required

validator

condition

values

default

hidden

class File

target

source

condition

class Folder

≈ File- data

Class Inheritance

pre/post Callbacks

Interactive Questioning

Command–Line Answers(marrow.script + **kw ;)

Templating

marrow.tags

Streaming

Enter / Exit

Text / Flush

High–Level

Widgets

Python ±

1 #!/usr/bin/env python 2 # encoding: utf-8 3 4 from __future__ import unicode_literals 5 6 from marrow.tags.html5 import * 7 8 from master import SITE_NAME, site_header, site_footer 9 10 11 def welcome(): 12 return html [ 13 head [ title [ 'Welcome!', ' — ', SITE_NAME ] ], 14 flush, # allow the browser to start downloading static resources early 15 body ( class_ = "nav-welcome" ) [ 16 site_header, 17 p [ 18 'Lorem ipsum dolor sit amet, consectetur adipisicing elit…' 19 ], 20 site_footer 21 ] 22 ] 23 24 25 if __name__ == '__main__': 26 with open('welcome.html', 'w') as fh: 27 for i in welcome().render('utf8'): 28 fh.write(i)

1 login = Form('sign-in', class_="tabbed", action='/users/action:authenticate', children=[ 2 HiddenField('referrer'), 3 FieldSet('local', "Local Users", TableLayout, [ 4 TextField('identity', "User Name", autofocus=True), 5 PasswordField('password', "Password") 6 ]), 7 FieldSet('yubikey', "Yubikey Users", TableLayout, [ 8 TextField('identity', "User Name"), 9 PasswordField('password', "Password"), 10 PasswordField('yubikey', "Yubikey") 11 ]), 12 FieldSet('openid', "OpenID Users", TableLayout, [ 13 URLField('url', "OpenID URL") 14 ]) 15 ], footer=SubmitFooter('form', "Sign In"))

1 class Tag(Fragment): 2 def __call__(self, data_=None, strip=NoDefault, *args, **kw): 3 self = deepcopy(self) 4 5 self.data = data_ 6 if strip is not NoDefault: self.strip = strip 7 self.args.extend(list(args)) 8 self.attrs.update(kw) 9 10 return self

12 def __getitem__(self, k): 13 if not k: return self 14 15 self = deepcopy(self) 16 17 if not isinstance(k, (tuple, list)): 18 k = [k] 19 20 for fragment in k: 21 if isinstance(fragment, basestring): 22 self.children.append(escape(fragment)) 23 continue 24 25 self.children.append(fragment) 26 27 return self

29 def __unicode__(self): 30 """Return a serialized version of this tree/branch.""" 31 return ''.join(self.render('utf8')).decode('utf8') 32 33 def enter(self): 34 if self.strip: 35 raise StopIteration() 36 37 if self.prefix: 38 yield self.prefix 39 40 yield u'<' + self.name + u''.join([attr for attr in quoteattrs(self, self.attrs)]) + u'>' 41 42 def exit(self): 43 if self.simple or self.strip: 44 raise StopIteration() 45 46 yield u'</' + self.name + u'>'

48 def render(self, encoding='ascii'): 49 # ... 50 51 for k, t in self: 52 if k == 'enter': 53 # ... 54 continue 55 56 if k == 'exit': 57 # ... 58 continue 59 60 if k == 'text': 61 # ... 62 continue 63 64 if k == 'flush': 65 yield buf.getvalue() 66 del buf 67 buf = IO() 68 69 yield buf.getvalue()

71 def __iter__(self): 72 yield 'enter', self 73 74 for child in self.children: 75 if isinstance(child, Fragment): 76 for element in child: 77 yield element 78 continue 79 80 if hasattr(child, '__call__'): 81 value = child(self) 82 83 if isinstance(value, basestring): 84 yield 'text', unicode(value) 85 continue 86 87 for element in child(self): 88 yield element 89 90 continue 91 92 yield 'text', unicode(child) 93 94 yield 'exit', self

29 def __unicode__(self): 30 """Return a serialized version of this tree/branch.""" 31 return ''.join(self.render('utf8')).decode('utf8')

96 def clear(self): 97 self.children = list() 98 self.args = list() 99 self.attrs = dict() 100 100 def empty(self): 101 self.children = list()

Server Interface

Asynchronous IO

Callbacks

Low–Level

marrow.io

Py3K Tornado + PatchesIOLoop + IOStream

Apache License

1 # encoding: utf-8 2 3 """An example raw IOLoop/IOStream example. 4 5 Taken from http://nichol.as/asynchronous-servers-in-python by Nicholas Piël. 6 """ 7 8 import errno, functools, socket 9 10 from marrow.util.compat import exception 11 from marrow.io import ioloop, iostream 12 13 def connection_ready(sock, fd, events): 14 while True: 15 connection, address = sock.accept() 16 17 connection.setblocking(0) 18 stream = iostream.IOStream(connection) 19 stream.write(b"HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n", stream.close) 20 21 if __name__ == '__main__': 22 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) 23 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 24 sock.setblocking(0) 25 sock.bind(("", 8010)) 26 sock.listen(5000) 27 28 io_loop = ioloop.IOLoop.instance() 29 io_loop.set_blocking_log_threshold(2) 30 callback = functools.partial(connection_ready, sock) 31 io_loop.add_handler(sock.fileno(), callback, io_loop.READ) 32 33 try: 34 io_loop.start() 35 except KeyboardInterrupt: 36 io_loop.stop()

Not fun!

High–Level

marrow.server

1 # encoding: utf-8 2 3 """A simplified version of the raw example.""" 4 5 6 from marrow.server.base import Server 7 from marrow.server.protocol import Protocol 8 9 10 11 class HTTPResponse(Protocol): 12 def accept(self, client): 13 client.write( b"HTTP/1.0 200 OK\r\nContent-Length: 7\r\n\r\nPong!\r\n", client.close) 14 15 16 if __name__ == '__main__': 17 Server(None, 8010, HTTPResponse).start()

functools.partial(Pass the client object to your callbacks.)

Your BFF

Single–Thread Async

Futures–Based Threading

Multi–Process(incl. processor detection)

PEP 444

Warning!

PEP 444Hurts babies!

PEP 444Is highly addictive?

PEP 444Is a draft of one possible WSGI 2 solution.

Complete Rewrite

Simplified

Consistent

1 def hello(environ): 2 return b'200 OK', [ 3 (b'Content-Type', b'text/plain')], 4 (b'Content-Length', b'12') 5 ], b"Hello world!"

Distinctions

Byte String

Unicode String

Native String

RFC–Style

Applications…

An application is any function, method, or instance with a __call__ method. Applications must:

1. Be able to be invoked more than once. If this can not be guaranteed by the application implementation, it must be wrapped in a function that creates a new instance on each call.

2. Accept a single positional argument which must be an instance of a base Python dictionary containing what is referred to as the WSGI environment. The contents of this dictionary are fully described in the WSGI Environment section.

3. Return a 3-tuple of (status, headers, body) where:1.status must contain the HTTP status code and reason phrase of the response. The status code and reason must be present, in that

order, separated by a single space. (See RFC 2616, Section 6.1.1 for more information.)2.headers must be a standard Python list containing 2-tuples of (name, value) pairs representing the HTTP headers of the response. Each

header name must represent a valid HTTP header field name (as defined by RFC 2616, Section 4.2) without trailing colon or other punctuation.

3.body must be an iterable representing the HTTP response body.4.status and the name of each header present in headers must not have leading or trailing whitespace.5.status, and the contents of headers (both name and value) must not contain control characters including carriage returns or linefeeds.6.status, headers, and the chunks yielded by the body iterator should be returned as byte strings, though for implementations where

native strings are unicode, native strings may be returned. The server must encode unicode values using ISO-8859-1.7. The amount of data yielded by the body iterable must not exceed the length specified by the Content-Length response header, if

defined.8. The body iterable may be a native string, instead of a native string wrapped in a list for the simple case, but this is not recommended.

Additionally, applications and middleware must not alter HTTP 1.1 "hop-by-hop" features or headers, any equivalent features in HTTP 1.0, or any headers that would affect the persistence of the client's connection to the web server. Applications and middleware may, however, interrogate the environment for their presence and value. These features are the exclusive province of the server, and a server should consider it a fatal error for an application to attempt sending them, and raise an error if they are supplied as return values from an application in the headers structure.

Servers…

A WSGI 2 server must:

1. Invoke the application callable once for each request it receives from an HTTP client that is directed at the application.2. Pass a single positional value to the application callable representing the request environment, described in detail in the WSGI Environment

section.3. Ensure that correct response headers are sent to the client. If the application omits a header required by the HTTP standard (or other relevant

specifications that are in effect), the server must add it. E.g. the Date and Server headers.1. The server must not override values with the same name if they are emitted by the application.

4. Raise an exception if the application attempts to set HTTP 1.1 "hop-by-hop" or persistence headers, or equivalent headers in HTTP 1.0, as described above.

5. Encode unicode data (where returned by the application) using ISO-8859-1.6. Ensure that line endings within the body are not altered.7. Transmit body chunks to the client in an unbuffered fashion, completing the transmission of each set of bytes before requesting another one.

(Applications should perform their own buffering.)8. Call the close() method of the body returned by the application, if present, upon completion of the current request. This should be called

regardless of the termination status of the request. This is to support resource release by the application and is intended to complement PEP 325's generator support, and other common iterables with close() methods.

9. Support the HTTP 1.1 specification where such support is made possible by the underlying transport channel, including full URL REQUEST_URI, pipelining of requests, chunked transfer, and any other HTTP 1.1 features mandated in the relevant RFC.

Additionally,

1. HTTP header names are case-insensitive, so be sure to take that into consideration when examining application-supplied headers.2. The server may apply HTTP transfer encodings or perform other transformations for the purpose of implementing HTTP features such as

chunked transfer.3. The server must not attempt to handle byte range requests; the application can optimize this use case far more easily than a server. (For example

an application can generate the correct body length vs. generating the whole body and having the server buffer and slice it.)4. Servers must not directly use any other attributes of the body iterable returned by the application.

More Demanding

(Optional = Never)

HTTP/1.1

Chunked Encoding

(Request)

(Response)

Expect/Continue

Pipelining / Keep–Alive

HTTP/1.1 Server

4.5KR/sec(Single process, single thread.)

C10K(4 processes, single thread.)

10KR/sec(4 process, single thread, lower concurrency.)

Pure Python!

(171 Opcodes)

PEP 444

Threading

Futures!

Multi–Process

Explicit

Processor Detection

Request Cycle

Socket Accept

Protocol .accept()

Read Headers

Pre–Buffer Body

Dispatch to Worker

Emit Status & Headers

Stream Body

Keep–Alive

wsgi.errors ➢ logging

Object Wrappers

marrow.wsgi.objects

Request / Response

Exceptions

1 #!/usr/bin/env python 2 # encoding: utf-8 3 4 from __future__ import unicode_literals 5 6 from pprint import pformat 7 8 from marrow.server.http import HTTPServer 9 from marrow.wsgi.objects.decorator import wsgify 10 11 12 @wsgify 13 def hello(request): 14 resp = request.response 15 resp.mime = "text/plain" 16 resp.body = "%r\n\n%s\n\n%s" % (request, request, pformat(request.__dict__)) 17 18 19 if __name__ == '__main__': 20 import logging 21 logging.basicConfig(level=logging.DEBUG) 22 23 HTTPServer(None, 8080, application=hello).start()

Request

Dict Proxy

WSGI Environment

Singleton

Accessor Objects

… 24 class Request(object): … 28 body = RequestBody('wsgi.input') 29 length = Int('CONTENT_LENGTH', None, rfc='14.13') 30 mime = ContentType('CONTENT_TYPE', None) 31 charset = Charset('CONTENT_TYPE') …

… 102 def __getitem__(self, name): 103 return self.environ[name] 104 105 def __setitem__(self, name, value): 106 self.environ[name] = value 107 108 def __delitem__(self, name): 109 del self.environ[name] …

1 class ReaderWriter(object): 2 default = NoDefault 3 rw = True 4 5 def __init__(self, header, default=NoDefault, rw=NoDefault, rfc=None): 6 pass # save arguments 7 8 def __get__(self, obj, cls): 9 try: 10 return obj[self.header] 11 12 except KeyError: 13 pass 14 15 if self.default is not NoDefault: 16 if hasattr(self.default, '__call__'): 17 return self.default(obj) 18 19 return self.default 20 21 raise AttributeError('WSGI environment does not contain %s key.' % (self.header, )) 22 23 def __set__(self, obj, value): 24 if not self.rw: 25 raise AttributeError('%s is a read-only value.' % (self.header, )) 26 27 if value is None: 28 del obj[self.header] 29 return 30 31 obj[self.header] = value 32 33 def __delete__(self, obj): 34 if not self.rw: 35 raise AttributeError('%s is a read-only value.' % (self.header, )) 36 37 del obj[self.header]

Filtering

Ingress

Egress

“Light-Weight Middleware”

38 class CompressionFilter(object): 39 def __init__(self, level=6): 40 self.level = level 41 42 super(CompressionFilter, self).__init__() 43 44 def __call__(self, request, status, headers, body): 45 """Compress, if able, the response. 46 47 This has the side effect that if your application does not declare a content-length, this filter will. 48 """ 49 50 # TODO: Remove some of this debug logging; it'll slow things down and isn't really needed. 51 52 if request.get('wsgi.compression', True) is False: 53 log.debug("Bypassing compression at application's request.") 54 return status, headers, body 55 56 if request.get('wsgi.async') and hasattr(body, '__call__'): 57 log.debug("Can not compress async responses, returning original response.") 58 return status, headers, body 59 60 if b'gzip' not in request.get('HTTP_ACCEPT_ENCODING', b''): 61 log.debug("Browser support for GZip encoding not found, returning original response.") 62 return status, headers, body

Exit Early

Stream Process

Flat Stack

Performance & Optimization

timeit FTW

s="Content-Type: text/html\r\n"^

Split or Partition?

a,b = s.split(":", 1)

a,b = s.split(":", 1)• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":", 1)

a,b = s.split(":")• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":", 1)

• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":", 1)

• Python 2.7: 0.665 • Python 3.1: 0.909

a,c = s.partition(":")[::2]

a,b = s.split(":", 1)

• Python 2.7: 0.665 • Python 3.1: 0.909

a,c = s.partition(":")[::2]• Python 2.7: 0.642 • Python 3.1: 0.690

a,b = s.split(":", 1)

• Python 2.7: 0.665 • Python 3.1: 0.909

a,b,c = s.partition(":")

a,b = s.split(":", 1)

• Python 2.7: 0.665 • Python 3.1: 0.909

a,b,c = s.partition(":")• Python 2.7: 0.407 • Python 3.1: 0.429

s="Content-Type: text/html\r\n"

.upper() or .lower()?

"Content-Type: text/html\r\n".upper()

"Content-Type: text/html\r\n".upper()• Python 2.7: 0.479 • Python 3.1: 0.469

"CONTENT-TYPE: text/html\r\n".upper()