OpenERP 6.1 Framework Changes

@odony – 6.1 Framework Changes – 05.2012

OpenERP 6.1Technical Changes atAPI/Framework level


OpenERP 6.1

● Released February 2011● Normal release – non LTS = limited support● Release Notes

● http://bit.ly/openerp61RN

● Focus● Usability

● Social features

● New web client

● Cleanup

http://bit.ly/openerp61RN


Agenda

● Framework & API● Architecture changes

● API evolution

● New views and UI features

● Other important bits

● Exercises, Q&A

Agenda


Evolution: Architecture changes

● Unified server-side deployment

● Scalability through multi-processing● Use all those cores on modern hardware

● Bypass Python GIL issues

● Reliability (cpu/memory/crashes)

● WSGI● Standard Python solution – OpenERP 6.1 is compliant

● Gunicorn - a WSGI compliant HTTP server

● Statelessness

● Distributed scheduled tasks (cron)


Unified server-side deployment

● No more need to deploy web client● Web client is now an OpenERP module● Only one package to install● Pre-loaded server-wide via –-load=web● All modules in same place (addons_path)● openerp-web still available stand-alone

● For testing/debugging purposes

● Needs to have copy of normal modules too!


Scalability through multi-processing

● Use all those cores on modern hardware● Bypass Python GIL limitations● Improve reliability (cpu/memory/crashes)

→ What does this all mean??


Multi-processing and GIL (1)

● CPython has a Global Interpreter Lock/GIL● Simplifies low-level memory management,

C calls, etc.● 1 Python thread = 1 OS thread● Parallel execution of threads forbidden● Cooperative multi-tasking on 1 CPU● Worse performance on multi-CPU/core!

● Battle for GIL on the different virtual CPUs!


Multi-processing and GIL (2)

● Cooperative multi-tasking● Release on I/O or every 100 ticks● 1 tick ≈ 1 interpreter instruction


Scalability through multi-processing

● multiprocessing module was an option● Using Unix tools and real processes was

more flexible and easier to scale up● Managing the processes was not

necessary, many tools exist to do that (e.g. Gunicorn)

● Just had to make OpenERP WSGI-compliant


WSGI (1)

● Web Server Gateway Interface for Python● WSGI applications can be embedded in

WSGI-compliant web servers● Many web frameworks support WSGI

● Django, CherryPy, …

● Many HTTP servers support WSGI● Apache(+mod), nginx(+mod), tornado, gunicorn, …


WSGI (2)

● Very simple API: one method

● environ = HTTP environment map● HTTP_HOST, REQUEST_METHOD, etc.

● start_response = callback(status,headers)

● Returns response as unicode string

● OpenERP's entry point: wsgi.core.application


Gunicorn (1)

● WGSI compliant HTTP Server● Python port of Ruby Unicorn● Pre-fork multi-process model● Flexible and easily configurable● UNIX only (fork support needed)● See also gunicorn.org


Gunicorn (2)

● Master “Arbiter” process● Starts and monitors workers (crash/timeout/...)

● Restarts killed workers fork→

● Increment/Decrement worker count (TTIN/TTOU signals)

● Doesn't speak HTTP naive dispatch→

● Forked worker processes● Speak HTTP Perform WSGI translation→

● Dispatch to application entry point

● Can restart after given number of requests


Gunicorn (3)

● OpenERP can't use normal config file or

command-line options● Config has to be provided via Gunicorn● gunicorn.conf, written in Python!

● Contains Gunicorn configuration (workers, limits, etc.)

● Contains OpenERP configuration

● Sets up requests/application hooks (pre/post request...)

● Arbiter controls max-requests limit● Workers control memory/cpu limits


Gunicorn (4)

$ gunicorn openerp:wsgi.core.application -c gunicorn.conf.pyRUN

CONF


Good to know (1)

● Gunicorn must be used explicitly→● Linear speed-up for CPU-bound tasks, less for I/O or DB

● Werkzeug built-in WSGI-compliant server→● openerp-server script

● multi-threaded mono-process

● Proxy mode specific handling (X-Forwarded..)● When deploying behind reverse proxy (e.g. with ProxyPass)

● Gunicorn: wsgi.proxied.application entry point instead

● Werkzeug: --proxy_mode


Good to know (2)

● No cron job with Gunicorn● Could be killed at any point!

● Start separate openerp-cron-worker job● Anywhere!

More about cron jobs in a minute...


OpenERP 6.0 deployment model

OpenERP Server PostgreSQLRequests OpenERP

Web Client


OpenERP WorkerOpenERP Worker

OpenERP 6.1 deployment model (1)


PostgreSQLRequests



OpenERP 6.1 deployment model (2)


PostgreSQLRequests

OpenERP WorkerOpenERP WorkerOpenERP WorkerOpenERP Worker

LoadBalancer


Statelessness (1)

● OpenERP 6.0 was still partially stateful:

1) Model registry (pool)

2) Wizards (osv_memory) kept in-memory

3) Application caches – menus, perms, … !

4) Background task processing: cron

● Not suitable for multi-processing


Caches and Model Registries

● Caches and Registries have state● Must be cleared on all workers when

invalidated – across multiple machines!● Signaling works with 2 DB sequences

● base_registry_signaling

● base_cache_signaling

● Before each request, both are checked● After each request, increment if changed


TransientModel

● osv_memory → TransientModel● Real database-backed model

● Automatically vacuum-cleaned

● Based on size limit or time limit

● Default: 2 hours

● Has special and implicit access rights● Everyone can perform CRUD

● But only access on your own records

● Can be used indifferently by multiple workers


Background Tasks (cron) (1)

● Problem:● Each worker possibly runs a cron processor

● Must prevent duplicate run of same task

● Must benefit from multi-processing when multiple cron processors

are running (each runs different tasks)

● Solution:● Database row-level mutex using LOCK

● Cron processor can be run on any number of servers

● Disabled on Gunicorn workers by default

● Can be run separately with openerp-cron-worker script added in 6.1

● Cron tasks can't be changed while running (locked)


Background Tasks (cron) (2)

● And by the way, cron tasks are executed in parallel in OpenERP 6.1

● Also if running mono-process, up to number or thread specified with –-max-cron-threads (default: 4)

● Allows multiple tasks to be executed at the same time


Agenda


● API evolution



● Exercises, Q&A

Agenda


Date Storage and Computation (1)

● 6.0 stored timestamps in “server timezone”● Worked well for small teams with only one TZ

● Database not portable, Reports incorrect for cross-TZ teams, DST issues!

● 6.1 stores timestamps as UTC always!

● Big change, but Right Thing to Do™● Database becomes portable

● Dates monotonically incremented in DB

● Simplifies conversion for display UTC Client TZ↔

● Timestamps computation works as before on server-side

● Watch out!● “Dates” have no TZ

● Server-side dates must be initialized in client TZ



● Server-side● All computation in UTC

● PDF reports use client TZ from context/user

● Database● All timestamps stored as TIMESTAMP WITHOUT TZ

● SELECT now() at time zone ‘UTC’ (should not need it)

● Client● Converted back and forth between UTC and client TZ

● Web uses browser TZ for display (JS limitation)





● Be very careful with date (that)!● Pure dates should be taken in user TZ● It's very tempting to use datetime.date.today()

● But may be wrong for half of the planet!● Use fields.date.context_today() instead


Unified Mail Subsystem

● 6.0 had 3+ independent mail stacks

● 6.1 has one unified mail stack (mail)● Per-database SMTP settings (SSL, Priority, Debug)

● Single point of extension

● Single mail queue/archive

● Common features● Scheduler

● Template (email_template)

● Composition wizard

ir.mail_server

mail.messagemail.thread

compositionscheduler

email_template


Many2Many auto-naming and inheritance

● What happens when you _inherit a model containing m2m field, w/ different _name?

● Problem more apparent with abstract osv_memory models

● In 6.1 many2many have auto names● rel_table, rel_from, rel_to are automatically computed

● Simpler/Partial declaration, clean inherit

● rel_table override needed for multiple relationships


YAML improvements (1)

● 6.1 YAML tests:● 65% coverage, 150 scenarios, 1800 test steps

● Simulating workflows works well

● But what about UI: views, on_change, … ?

● Done! YAML will now automatically use form

views and on_change methods● Better coverage

● Less code: on_change can fill in implicit values

● Can be overridden with view attribute


YAML improvements (2)


Simplified Configuration (1)

● Setup process streamlined – 1-click install● Kanban module view – ordered by sequence

● Configuration Overview Dashboard● Based on ir.action.todo and ir.actions.todo.category

● New ir.actions.todo● Reminder: group_ids, action_id, and note

● States: Todo or Done

● Types: Manual, Manual Once, Automatic

● Most are Manual or Manual Once now

● New category_id for Configuration Overview


Simplified Configuration (2)


Logging

● Python has a decent logging system● netsvc.notifyChannel() deprecated in 6.0● Convenience hierarchical loggers● One _logger per file recommended

_logger = logging.getLogger(__name__)

● New –-log-handler config parameter● --log-handler=PREFIX:LEVEL

● --log-handler=openerp.netsvc.rpc.request:DEBUG

● --log-handler=werkzeug:CRITICAL


Agenda


● API evolution



● Exercises, Q&A

Agenda


Kanban (1)

● Kanban (カンバン ): Japanese billboard● Toyota's Lean/JIT production: “pull” or

“demand-driven” SCM● Kanban cards signal need to produce


Kanban (2)

● Kanban method, applied to Management● Signboard shows whole workflow at a glance

● Limit WIP (work-in-progress) for each state (Pull!)

● Monitor flow


Kanban (3)

● kanban views are written in Qweb● Automatic column layout with Group By● Drag & Drop reassigns grouped column● One mandatory template “kanban-box”● Can use normal OpenERP fields● Any CSS/HTML you like


Kanban (4)

● Typical structure


Kanban (5)

● Drag&Drop is nice, but group_by only shows existing data

● How about missing columns?● Solution: add a _group_by_full map● Return type must be like read(),

depending on the column type


Kanban (6)

● group_by_full, complex example


Kanban (7)

Exercise: add test kanban view for res.partner


Kanban (8)

● More details about Kanban API:

http://pad.openerp.com/kanban-api

(temporary location)

http://pad.openerp.com/kanban-api


Customizable dashboards

● Dashboards now feature multiple layouts● Saved in ir.ui.view_custom, per user● Filters menu now has “Add to dashboard”● Works with kanban views too


New many2one widget

● Many2One fields = eternal UI problem● 6.1 widget:

● Auto-complete selection as you type

● Intuitive quick creation

● New ORM method: name_create(cr, uid, name, context)

● If fails, default to normal form creation

● Disabled with options='{"quick_create": false}'


New statusbar widget

● Old state selection was not user-friendly● statusbar widget shows workflow steps

● Only for selection fields● Options: list of visible values, colors


Agenda


● API evolution



● Exercises, Q&A

Agenda


Module Manifest changes

● Web-related keys: js, css, qweb● active auto_install → (backwards-compat): 'depends' triggers!

● complexity (easy|normal|expert)

● application (bool): Application or Extra

● sequence (integer): Application install screen order

● New category: 'Hidden' – Technical, hidden by default

● Module icon for install screen/static/src/img/icon.png

● OpenERP Apps● Description in RST, one main section

● 'images': ['/path/to/screenshot.png', '/path/to/screenshot2.png']


EDI Subsystem (1)

● Goal: bring companies up to par with people and social network tools

● EDI module installed with Sales, Account● EDI Setup

● Company Addres & Logo

● Outgoing Email Server (SMTP)

● User emails

● Customers/Contacts emails


EDI Subsystem (2)


EDI Subsystem (3)

● Fine-tuning● Payment options

● Paypal account on company

● Bank accounts on company

● Email templates

● EDI Preview templates: Qweb

● Opt-out● Per-customer

● Global – workflow level

● Global – template


EDI Subsystem (4)

● EDI Document is simple JSON

● EDI engine is generic


Agenda


● API evolution



● Exercises, Q&A

Agenda


Exercises

● Gunicorn:● Install Gunicorn and start OpenERP w/ it

● Kanban● Add a kanban view to the model of your choice

● Set a default group_by on the menu action

● Implement a working _group_by_full

● Add the Kanban view to a dashboard

● EDI● Modify an EDI Preview template to show Quotations

Technology

OpenERP 6.1 Framework Changes