Diving into byte code optimization in python

  • View

  • Download

Embed Size (px)


Diving into byte code optimization in python

Text of Diving into byte code optimization in python

  • 1.Diving into Byte-codeoptimization in PythonSciPy India,IIT Bombay Dec 05th 2011Chetan Giridhar and Vishal Kanaujia

2. Fundamentals of Bytecode Python source code compiled into Pythonbyte code by the CPython interpreter .pyc? Automatic compilation : importing a module Explicit compilation :py_compile.compile(module.py) generatesmodule.pyc The module compileall{} Attribution-NonCommercial CC BY-NC 3. Fundamentals | more A program doesnt run any faster when it isread from a .pyc file. But, why? .pyc for a script executed on the commandline .pyc files good enough to distribute yourcode, but with a caveat! Attribution-NonCommercial CC BY-NC 4. Compilation phases Uses a lexer to prepare tokens of Pythoncode A parser arranges token according to thelanguage grammar and prepares concretesyntax tree Concrete syntax tree is transformed in to AST AST is compiled to produce Byte-codes Attribution-NonCommercial CC BY-NC 5. Python AbstractParse TreesourceSyntaxgenerated codeTreePgen.c ast.c compile.cExecutedOptimizedBytecodeby Pythonbytecode generated VMceval.cpeephole.c 6. Python ast module$ cat myast.py$python myast.pyimport astModule(body=[Expr(value=BinOpnod = ast.parse(a +2) (left=Name(id=a, ctx=Load()),print ast.walk(nod) op=Add(), right=Num(n=2)))])print ast.dump(nod) Convenient for analysis, code transformations and generation ASTs are compiled to code objects Attribution-NonCommercial CC BY-NC 7. A peek into Bytecodes$ cat scipy.py $ python scipy.py 1 import dis 40 LOAD_CONST 1 (10) 3 STORE_FAST 0 (i) 2 3 def foo(): 5 6 LOAD_FAST 0 (i) 9 PRINT_ITEM 4 i = 1010 PRINT_NEWLINE 5 print i 11 LOAD_CONST0 (None) 14 RETURN_VALUE 6 7 print dis.dis(foo) Bytecode stream: An opcode mix Defined in Python-2.7.2/Include/opcode.hAttribution-NonCommercial CC BY-NC 8. Python VM Engine to execute Bytecodes CPython VM is a C implementation Stack based process VM PUSH/ POP operations Implementation of Python VM?Attribution-NonCommercial CC BY-NC 9. Python VM: ImplementationPython/ceval.c--> PyEval_EvalFrameEx()for(; ;) { /* Extract opcode and argument */opcode = NEXTOP();if (HAS_ARG(opcode))oparg = NEXTARG();switch(opcode) { case LOAD_CONST: }} Attribution-NonCommercial CC BY-NC 10. OptimizationsToolsGetting in to the Problem Space!! Attribution-NonCommercial CC BY-NC 11. Pyrex Python like language to create C module forPython Create your pyx files and compile them in .cfiles Import them as modules in your applications Pyrex used as: speed up the execution of Python code Python interface to existing C modules/libraries Lot of work for developer .py to .pyx? thinking in CAttribution-NonCommercial CC BY-NC 12. Psyco An intelligent option JIT compilation Profiles dynamically an application for hot-spots in-memory prepares C extension and hookthem appropriately Solves duck typing Memory footprint? Support till CPython 2.5Attribution-NonCommercial CC BY-NC 13. Psyco| Intelligent useIterationsWithoutWith Pysco(ms) Pysco(ms) 1000125 151 100000 12900 12570 from psyco.classes import * pysco.bind(func)Attribution-NonCommercial CC BY-NC 14. OptimizationsBytecode levelGetting in to the Problem Space!!Attribution-NonCommercial CC BY-NC 15. Why optimize Bytecode?def foo(): Python optimization i=0i =i+1are ineffective,print isometime40 LOAD_CONST1 (0) Duck typing3 STORE_FAST0 (i)56 LOAD_FAST0 (i) Run-time types9 LOAD_CONST2 (1)12 BINARY_ADD Optimizer misses13 STORE_FAST 0 (i)many opportunities6 16 LOAD_FAST0 (i)19 PRINT_ITEM TCO 20 PRINT_NEWLINE21 LOAD_CONST0 (None) Attribution-NonCommercial CC BY-NC 16. Optimizations: Tail Recursive Calls$ cat runbytecode3.py $ python runbytecode3.pyimport dis 51 LOAD_CONST 102 STORE_FAST x 6 4 LOAD_FASTxdef foo():5 LOAD_CONST 10x = 106 COMPARE_OP ==7 POP_JUMP_IF_FALSEto 17if x == 10: 79 LOAD_CONST 0 x=0 10 STORE_FAST x foo() 8 12 LOAD_GLOBALfoo 13 CALL_FUNCTION 0 14 POP_TOPprint dis.dis(foo) 15 JUMP_FORWARDto 17 >> 17 LOAD_CONSTNone 18 RETURN_VALUENoneAttribution-NonCommercial CC BY-NC 17. BytePlay: Do it yourself! Experiment with Bytecodes Generates, recompiles and runs code on-the-fly Very useful to evaluate different codeoptimizations ExampleAttribution-NonCommercial CC BY-NC 18. Playing with Bytecode$cat runbytecode2.py $ python runbytecode2.py reassembled byteplay.py imported.from byteplay import *from pprint import pprint 5 1 LOAD_CONST10