Python Performance Benchmark Suite Documentation

Python Performance Benchmark SuiteDocumentation

Release 1.0.3

Victor Stinner

Dec 02, 2021

Contents

1 Usage 31.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Run benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Compile Python to run benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4 How to get stable benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.5 pyperformance virtual environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.6 What is the goal of pyperformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 Benchmarks 112.1 Available Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 Available Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 CPython results, 2017 253.1 Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.2 Slowdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.3 Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4 Changelog 314.1 Version 1.0.2 (2021-05-11) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.2 Version 1.0.1 (2020-03-26) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.3 Version 1.0.0 (2019-12-17) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.4 Version 0.9.1 (2019-07-29) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.5 Version 0.9.0 (2019-05-29) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.6 Version 0.8.0 (2019-05-10) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.7 Version 0.7.0 (2018-10-16) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344.8 Version 0.6.1 (2018-01-11) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344.9 Version 0.6.0 (2017-07-06) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.10 Version 0.5.5 (2017-05-29) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.11 Version 0.5.4 (2017-04-10) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.12 Version 0.5.3 (2017-03-27) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.13 Version 0.5.2 (2017-03-17) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.14 Version 0.5.1 (2017-01-16) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.15 Version 0.5.0 (2016-11-16) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.16 Version 0.4.0 (2016-11-07) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.17 Version 0.3.2 (2016-10-19) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.18 Version 0.3.1 (2016-10-19) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

i

4.19 Version 0.3.0 (2016-10-11) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.20 Version 0.2.2 (2016-09-19) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.21 Version 0.2.1 (2016-09-10) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.22 Version 0.2 (2016-09-01) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.23 Version 0.1.2 (2016-08-27) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.24 Version 0.1.1 (2016-08-24) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.25 Version 0.1 (2016-08-24) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.26 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

ii

Python Performance Benchmark Suite Documentation, Release 1.0.3

The pyperformance project is intended to be an authoritative source of benchmarks for all Python implementations.The focus is on real-world benchmarks, rather than synthetic benchmarks, using whole applications when possible.

• pyperformance documentation

• pyperformance GitHub project (source code, issues)

• Download pyperformance on PyPI

pyperformance is distributed under the MIT license.

Documenation:

Contents 1

http://pyperformance.readthedocs.io/

https://github.com/python/pyperformance

https://pypi.python.org/pypi/pyperformance


2 Contents

CHAPTER 1

Usage

1.1 Installation

Command to install pyperformance:

python3 -m pip install pyperformance

The command installs a new pyperformance program.

If needed, pyperf and six dependencies are installed automatically.

pyperformance works on Python 3.6 and newer, but it may work on Python 3.4 and 3.5.

Mercurial might need Python developement headers to build its C extensions. For example, on Fedora, use:

sudo dnf install pypy3-devel

At runtime, Python development files (header files) may be needed to install some dependencies like dulwich_logor psutil, to build their C extension. Commands on Fedora to install dependencies:

• Python 3: sudo dnf install python3-devel

• PyPy: sudo dnf install pypy-devel

1.2 Run benchmarks

Commands to compare Python 3.6 and Python 3.7 performances:

pyperformance run --python=python3.6 -o py36.jsonpyperformance run --python=python3.7 -o py38.jsonpyperformance compare py36.json py38.json

Note: python3 -m pyperformance ... syntax works as well (ex: python3 -m pyperformance run-o py38.json), but requires to install pyperformance on each tested Python version.

3


JSON files are produced by the pyperf module and so can be analyzed using pyperf commands:

python3 -m pyperf show py36.jsonpython3 -m pyperf check py36.jsonpython3 -m pyperf metadata py36.jsonpython3 -m pyperf stats py36.jsonpython3 -m pyperf hist py36.jsonpython3 -m pyperf dump py36.json(...)

It’s also possible to use pyperf to compare results of two JSON files:

python3 -m pyperf compare_to py36.json py38.json --table

pyperformance actions:

run Run benchmarks on the running pythonshow Display a benchmark filecompare Compare two benchmark fileslist List benchmarks which run command would runlist_groups List all benchmark groupsvenv Actions on the virtual environment

1.2.1 Common options

Options available to all commands:

-p PYTHON, --python PYTHONPython executable (default: use running Python)

--venv VENV Path to the virtual environment--inherit-environ VAR_LIST

Comma-separated list of environment variable namesthat are inherited from the parent environment whenrunning benchmarking subprocesses.

1.2.2 run

Options of the run command:

-r, --rigorous Spend longer running tests to get more accurateresults

-f, --fast Get rough answers quickly-m, --track-memory Track memory usage. This only works on Linux.-b BM_LIST, --benchmarks BM_LIST

Comma-separated list of benchmarks to run. Can containboth positive and negative arguments:--benchmarks=run_this,also_this,-not_this. If thereare no positive arguments, we'll run all benchmarksexcept the negative arguments. Otherwise we run onlythe positive arguments.

--affinity CPU_LIST Specify CPU affinity for benchmark runs. This way,benchmarks can be forced to run on a given CPU tominimize run to run variation. This uses the tasksetcommand.

(continues on next page)

4 Chapter 1. Usage


(continued from previous page)

-o FILENAME, --output FILENAMERun the benchmarks on only one interpreter and writebenchmark into FILENAME. Provide only baseline_python,not changed_python.

--append FILENAME Add runs to an existing file, or create it if itdoesn't exist

1.2.3 show

Usage:

show FILENAME

1.2.4 compare

Options of the compare command:

-v, --verbose Print more output-O STYLE, --output_style STYLE

What style the benchmark output should take. Validoptions are 'normal' and 'table'. Default is normal.

1.2.5 list

Options of the list command:

-b BM_LIST, --benchmarks BM_LISTComma-separated list of benchmarks to run. Can containboth positive and negative arguments:--benchmarks=run_this,also_this,-not_this. If thereare no positive arguments, we'll run all benchmarksexcept the negative arguments. Otherwise we run onlythe positive arguments.

Use python3 -m pyperformance list -b all to list all benchmarks.

1.2.6 venv

Options of the venv command:

-p PYTHON, --python PYTHONPython executable (default: use running Python)

--venv VENV Path to the virtual environment

Actions of the venv command:

show Display the path to the virtual environment and it's status (created or not)create Create the virtual environmentrecreate Force the recreation of the the virtual environmentremove Remove the virtual environment

1.2. Run benchmarks 5


1.3 Compile Python to run benchmarks

pyperformance actions:

compile Compile, install and benchmark CPythoncompile_all Compile, install and benchmark multiple branches and revisions of→˓CPythonupload Upload JSON file

All these commands require a configuration file.

Simple configuration usable for compile (but not for compile_all nor upload), doc/benchmark.conf:

[config]json_dir = ~/prog/python/bench_json

[scm]repo_dir = ~/prog/python/masterupdate = True

[compile]bench_dir = ~/prog/python/bench_tmpdir

[run_benchmark]system_tune = Trueaffinity = 2,3

Configuration file sample with comments, doc/benchmark.conf.sample:

[config]# Directory where JSON files are written.# - uploaded files are moved to json_dir/uploaded/# - results of patched Python are written into json_dir/patch/json_dir = ~/json

# If True, compile CPython is debug mode (LTO and PGO disabled),# run benchmarks with --debug-single-sample, and disable upload.## Use this option used to quickly test a configuration.debug = False

[scm]# Directory of CPython source code (Git repository)repo_dir = ~/cpython

# Update the Git repository (git fetch)?update = True

# Name of the Git remote, used to create revision of# the Git branch. For example, use revision 'remotes/origin/3.6'# for the branch '3.6'.git_remote = remotes/origin

[compile]# Create files into bench_dir:


6 Chapter 1. Usage



# - bench_dir/bench-xxx.log# - bench_dir/prefix/: where Python is installed# - bench_dir/venv/: Virtual environment used by pyperformancebench_dir = ~/bench_tmpdir

# Link Time Optimization (LTO)?lto = True

# Profiled Guided Optimization (PGO)?pgo = True

# The space-separated list of libraries that are package-only,# i.e., locally installed but not on header and library paths.# For each such library, determine the install path and add an# appropriate subpath to CFLAGS and LDFLAGS declarations passed# to configure. As an exception, the prefix for openssl, if that# library is present here, is passed via the --with-openssl# option. Currently, this only works with Homebrew on macOS.# If running on macOS with Homebrew, you probably want to use:# pkg_only = openssl readline sqlite3 xz zlib# The version of zlib shipping with macOS probably works as well,# as long as Apple's SDK headers are installed.pkg_only =

# Install Python? If false, run Python from the build directory## WARNING: Running Python from the build directory introduces subtle changes# compared to running an installed Python. Moreover, creating a virtual# environment using a Python run from the build directory fails in many cases,# especially on Python older than 3.4. Only disable installation if you# really understand what you are doing!install = True

[run_benchmark]# Run "sudo python3 -m pyperf system tune" before running benchmarks?system_tune = True

# --benchmarks option for 'pyperformance run'benchmarks =

# --affinity option for 'pyperf system tune' and 'pyperformance run'affinity =

# Upload generated JSON file?## Upload is disabled on patched Python, in debug mode or if install is# disabled.upload = False

# Configuration to upload results to a Codespeed website[upload]url =environment =executable =project =


1.3. Compile Python to run benchmarks 7



[compile_all]# List of CPython Git branchesbranches = default 3.6 3.5 2.7

# List of revisions to benchmark by compile_all[compile_all_revisions]# list of 'sha1=' (default branch: 'master') or 'sha1=branch'# used by the "pyperformance compile_all" command# e.g.:11159d2c9d6616497ef4cc62953a5c3cc8454afb =

1.3.1 compile

Usage:

pyperformance compile CONFIG_FILE REVISION [BRANCH][--patch=PATCH_FILE][-U/--no-update][-T/--no-tune]

Compile Python, install Python and run benchmarks on the installed Python.

Options:

• --no-update: Don’t update the Git repository.

• --no-tune: Don’t run pyperf system tune to tune the system for benchmarks.

If the branch argument is not specified:

• If REVISION is a branch name: use it as a the branch, and get the latest revision of this branch

• Otherwise, use master branch by default

Notes:

• –enable-optimizations doesn’t enable LTO because of compiler bugs: http://bugs.python.org/issue28032 (seealso: http://bugs.python.org/issue28605)

• PGO is broken on Ubuntu 14.04 LTS with GCC 4.8.4-2ubuntu1~14.04: Modules/socketmodule.c:7743:1: internal compiler error: in edge_badness, at ipa-inline.c:895

1.3.2 compile_all

Usage:

pyperformance compile_all CONFIG_FILE

Compile all branches and revisions of CONFIG_FILE.

1.3.3 upload

Usage:

8 Chapter 1. Usage

http://bugs.python.org/issue28032



pyperformance upload CONFIG_FILE JSON_FILE

Upload results from a JSON file to a Codespeed website.

1.4 How to get stable benchmarks

• Run python3 -m pyperf system tune command

• Compile Python using LTO (Link Time Optimization) and PGO (profile guided optimizations): use the pyper-formance compile command with uses LTO and PGO by default

• See advices of the pyperf documentation: How to get reproductible benchmark results.

1.5 pyperformance virtual environment

To run benchmarks, pyperformance first creates a virtual environment. It installs requirements with fixed versions toget a reproductible environment. The system Python has unknown module installed with unknown versions, and canhave .pth files run at Python startup which can modify Python behaviour or at least slow down Python startup.

1.6 What is the goal of pyperformance

A benchmark is always written for a specific purpose. Depending how the benchmark is written and how the bench-mark is run, the result can be different and so have a different meaning.

The pyperformance benchmark suite has multiple goals:

• Help to detect performance regression in a Python implementation

• Validate that an optimization change makes Python faster and don’t performance regressions, or only minorregressions

• Compare two implementations of Python, for example CPython and PyPy

• Showcase of Python performance which ideally would be representative of performances of applications runningon production

1.6.1 Don’t disable GC nor ASLR

The pyperf module and pyperformance benchmarks are designed to produce reproductible results, but not at the priceof running benchmarks in a special mode which would not be used to run applications in production. For thesereasons, the Python garbage collector, Python randomized hash function and system ASLR (Address Space LayoutRandomization) are not disabled. Benchmarks don’t call gc.collect() neither since CPython implements it withstop-the-world and so applications don’t call it to not kill performances.

1.6.2 Include outliers and spikes

Moreover, while the pyperf documentation explains how to reduce the random noise of the system and other applica-tions, some benchmarks use the system and so can get different timing depending on the system workload, dependingon I/O performances, etc. Outliers and temporary spikes in results are not automatically removed: values are sum-marized by computing the average (arithmetic mean) and standard deviation which “contains” these spikes, instead

1.4. How to get stable benchmarks 9

http://pyperf.readthedocs.io/en/latest/run_benchmark.html#how-to-get-reproductible-benchmark-results

https://en.wikipedia.org/wiki/Tracing_garbage_collection#Stop-the-world_vs._incremental_vs._concurrent


of using median and the median absolute deviation for example which to ignore outliers. It is deliberate choice sinceapplications running in production are impacted by such temporary slowdown caused by various things like a garbagecollection or a JIT compilation.

1.6.3 Warmups and steady state

A borderline issue are the benchmarks “warmups”. The first values of each worker process are always slower: 10%slower in the best case, it can be 1000% slower or more on PyPy. Right now (2017-04-14), pyperformance ignore firstvalues considered as warmup until a benchmark reachs its “steady state”. The “steady state” can include temporaryspikes every 5 values (ex: caused by the garbage collector), and it can still imply further JIT compiler optimizationsbut with a “low” impact on the average pyperformance.

To be clear “warmup” and “steady state” are a work-in-progress and a very complex topic, especially on PyPy and itsJIT compiler.

1.7 Notes

Tool for comparing the performance of two Python implementations.

pyperformance will run Student’s two-tailed T test on the benchmark results at the 95% confidence level to indicatewhether the observed difference is statistically significant.

Omitting the -b option will result in the default group of benchmarks being run Omitting -b is the same as specifying-b default.

To run every benchmark pyperformance knows about, use -b all. To see a full list of all available benchmarks, use–help.

Negative benchmarks specifications are also supported: -b -2to3 will run every benchmark in the default group exceptfor 2to3 (this is the same as -b default,-2to3). -b all,-django will run all benchmarks except the Django templatesbenchmark. Negative groups (e.g., -b -default) are not supported. Positive benchmarks are parsed before the negativebenchmarks are subtracted.

If --track_memory is passed, pyperformance will continuously sample the benchmark’s memory usage. Thiscurrently only works on Linux 2.6.16 and higher or Windows with PyWin32. Because --track_memory introducesperformance jitter while collecting memory measurements, only memory usage is reported in the final report.

10 Chapter 1. Usage

CHAPTER 2

Benchmarks

2.1 Available Groups

Like individual benchmarks (see “Available benchmarks” below), benchmarks group are allowed after the -b option.Use python3 -m pyperformance list_groups to list groups and their benchmarks.

Available benchmark groups:

• all: Group including all benchmarks

• apps: “High-level” applicative benchmarks (2to3, Chameleon, Tornado HTTP)

• default: Group of benchmarks run by default by the run command

• math: Float and integers

• regex: Collection of regular expression benchmarks

• serialize: Benchmarks on pickle and json modules

• startup: Collection of microbenchmarks focused on Python interpreter start-up time.

• template: Templating libraries

Use the python3 -m pyperformance list_groups command to list groups and their benchmarks.

2.2 Available Benchmarks

In pyperformance 0.5.5, the following microbenchmarks have been removed because they are too short, not represen-tative of real applications and are too unstable.

• call_method_slots

• call_method_unknown

• call_method

• call_simple

11


• pybench

2.2.1 2to3

Run the 2to3 tool on the pyperformance/benchmarks/data/2to3/ directory: copy of the django/core/*.py files of Django 1.1.4, 9 files.

Run the python -m lib2to3 -f all <files> command where python is sys.executable. So thetest does not only mesure the performance of Python itself, but also the performance of the lib2to3 module whichcan change depending on the Python version.

Note: Files are called .py.txt instead of .py to not run PEP 8 checks on them, and more generally to not modifythem.

2.2.2 chameleon

Render a template using the chameleon module to create an HTML table of 500 lignes and 10 columns.

See the chameleon.PageTemplate class.

2.2.3 chaos

Create chaosgame-like fractals. Command lines options:

--thickness THICKNESSThickness (default: 0.25)

--width WIDTH Image width (default: 256)--height HEIGHT Image height (default: 256)--iterations ITERATIONS

Number of iterations (default: 5000)--filename FILENAME.PPM

Output filename of the PPM picture--rng-seed RNG_SEED Random number generator seed (default: 1234)

When --filename option is used, the timing includes the time to create the PPM file.

Copyright (C) 2005 Carl Friedrich Bolz

12 Chapter 2. Benchmarks


Image generated by bm_chaos (took 3 sec on CPython 3.5) with the command:

python3 pyperformance/benchmarks/bm_chaos.py --worker -l1 -w0 -n1 --filename chaos.→˓ppm --width=512 --height=512 --iterations 50000

2.2.4 crypto_pyaes

benchmark a pure-Python implementation of the AES block-cipher in CTR mode using the pyaes module.

The benchmark is slower on CPython 3 compared to CPython 2.7, because CPython 3 has no more “small int” type(int). The CPython 3 int type now always has an arbitrary size, as CPython 2.7 long type.

See pyaes: A pure-Python implementation of the AES block cipher algorithm and the common modes of operation(CBC, CFB, CTR, ECB and OFB).

2.2. Available Benchmarks 13

https://github.com/ricmoo/pyaes


2.2.5 deltablue

DeltaBlue benchmark

Ported for the PyPy project. Contributed by Daniel Lindsley

This implementation of the DeltaBlue benchmark was directly ported from the V8’s source code, which was in turnderived from the Smalltalk implementation by John Maloney and Mario Wolczko. The original Javascript implemen-tation was licensed under the GPL.

It’s been updated in places to be more idiomatic to Python (for loops over collections, a couple magic methods,OrderedCollection being a list & things altering those collections changed to the builtin methods) but largelyretains the layout & logic from the original. (Ugh.)

2.2.6 django_template

Use the Django template system to build a 150x150-cell HTML table.

Use Context and Template classes of the django.template module.

2.2.7 dulwich_log

Iterate on commits of the asyncio Git repository using the Dulwich module. Use pyperformance/benchmarks/data/asyncio.git/ repository.

Pseudo-code of the benchmark:

repo = dulwich.repo.Repo(repo_path)head = repo.head()for entry in repo.get_walker(head):

pass

See the Dulwich project.

2.2.8 fannkuch

The Computer Language Benchmarks Game: http://benchmarksgame.alioth.debian.org/

Contributed by Sokolov Yura, modified by Tupteq.

2.2.9 float

Artificial, floating point-heavy benchmark originally used by Factor.

Create 100,000 point objects which compute math.cos(), math.sin() and math.sqrt()

Changed in version 0.5.5: Use __slots__ on the Point class to focus the benchmark on float rather than testingperformance of class attributes.

2.2.10 genshi

Render a template using Genshi (genshi.template module):

• genshi_text: Render a HTML template using the NewTextTemplate class


https://github.com/v8/v8/blob/master/benchmarks/deltablue.js

https://www.dulwich.io/

http://benchmarksgame.alioth.debian.org/


• genshi_xml: Render an XML template using the MarkupTemplate class

See the Genshi project.

2.2.11 go

Artificial intelligence playing the Go board game. Use Zobrist hashing.

2.2.12 hexiom

Solver of Hexiom board game (level 25 by default). Command line option:

--level {2,10,20,25,30,36} Hexiom board level (default: 25)

2.2.13 hg_startup

Get Mercurial’s help screen.

Measure the performance of the python path/to/hg help command using pyperf.Runner.bench_command(), where python is sys.executable and path/to/hg is the Mercurial programinstalled in a virtual environmnent.

The bench_command() redirects stdout and stderr into /dev/null.

See the Mercurial project.

2.2.14 html5lib

Parse the pyperformance/benchmarks/data/w3_tr_html5.html HTML file (132 KB) usinghtml5lib. The file is the HTML 5 specification, but truncated to parse the file in less than 1 second (around250 ms).

On CPython, after 3 warmups, the benchmarks enters a cycle of 5 values: every 5th value is 10% slower. Plot of 1 runof 50 values (the warmup is not rendered):


http://pythonhosted.org/Genshi/

https://en.wikipedia.org/wiki/Zobrist_hashing

https://www.mercurial-scm.org/


See the html5lib project.

2.2.15 json_dumps, json_loads

Benchmark dumps() and loads() functions of the json module.

bm_json_dumps.py command line option:

--cases CASES Comma separated list of cases. Available cases: EMPTY,SIMPLE, NESTED, HUGE. By default, run all cases.

2.2.16 logging

Benchmarks on the logging module:

• logging_format: Benchmark logger.warn(fmt, str)

• logging_simple: Benchmark logger.warn(msg)

• logging_silent: Benchmark logger.debug(msg) when the log is ignored

Script command line option:


https://html5lib.readthedocs.io/


formatsilentsimple

See the logging module.

2.2.17 mako

Use the Mako template system to build a 150x150-cell HTML table. Includes:

• two template inherences

• HTML escaping, XML escaping, URL escaping, whitespace trimming

• function defitions and calls

• forloops

See the Mako project.

2.2.18 mdp

Battle with damages and topological sorting of nodes in a graph.

See Topological sorting.

2.2.19 meteor_contest

Solver for Meteor Puzzle board.

Meteor Puzzle board: http://benchmarksgame.alioth.debian.org/u32/meteor-description.html#meteor

The Computer Language Benchmarks Game: http://benchmarksgame.alioth.debian.org/

Contributed by Daniel Nanz, 2008-08-21.

2.2.20 nbody

N-body benchmark from the Computer Language Benchmarks Game. Microbenchmark on floating point operations.

This is intended to support Unladen Swallow’s perf.py. Accordingly, it has been modified from the Shootout version:

• Accept standard Unladen Swallow benchmark options.

• Run report_energy()/advance() in a loop.

• Reimplement itertools.combinations() to work with older Python versions.

Pulled from: http://benchmarksgame.alioth.debian.org/u64q/program.php?test=nbody&lang=python3&id=1

Contributed by Kevin Carson. Modified by Tupteq, Fredrik Johansson, and Daniel Nanz.


https://docs.python.org/dev/library/logging.html

http://docs.makotemplates.org/

https://en.wikipedia.org/wiki/Topological_sorting

http://benchmarksgame.alioth.debian.org/u32/meteor-description.html#meteor


http://benchmarksgame.alioth.debian.org/u64q/program.php?test=nbody&lang=python3&id=1


2.2.21 python_startup, python_startup_nosite

• python_startup: Measure the Python startup time, run python -c pass where python is sys.executable

• python_startup_nosite: Measure the Python startup time without importing the site module, runpython -S -c pass where python is sys.executable

Run the benchmark with pyperf.Runner.bench_command().

2.2.22 nqueens

Simple, brute-force N-Queens solver.

See Eight queens puzzle.

2.2.23 pathlib

Test the performance of operations of the pathlib module of the standard library.

This benchmark stresses the creation of small objects, globbing, and system calls.

See the documentation of the pathlib module.

2.2.24 pickle

pickle benchmarks (serialize):

• pickle: use the cPickle module to pickle a variety of datasets.

• pickle_dict: microbenchmark; use the cPickle module to pickle a lot of dicts.

• pickle_list: microbenchmark; use the cPickle module to pickle a lot of lists.

• pickle_pure_python: use the pure-Python pickle module to pickle a variety of datasets.

unpickle benchmarks (deserialize):

• unpickle: use the cPickle module to unnpickle a variety of datasets.

• unpickle_list

• unpickle_pure_python: use the pure-Python pickle module to unpickle a variety of datasets.

2.2.25 pidigits

Calculating 2,000 digits of 𝜋. This benchmark stresses big integer arithmetic.

Command line option:

--digits DIGITS Number of computed pi digits (default: 2000)

Adapted from code on: http://benchmarksgame.alioth.debian.org/


https://en.wikipedia.org/wiki/Eight_queens_puzzle

https://docs.python.org/dev/library/pathlib.html



2.2.26 pyflate

Benchmark of a pure-Python bzip2 decompressor: decompress the pyperformance/benchmarks/data/interpreter.tar.bz2 file in memory.

Copyright 2006–2007-01-21 Paul Sladen: http://www.paul.sladen.org/projects/compression/

You may use and distribute this code under any DFSG-compatible license (eg. BSD, GNU GPLv2).

Stand-alone pure-Python DEFLATE (gzip) and bzip2 decoder/decompressor. This is probably most useful for researchpurposes/index building; there is certainly some room for improvement in the Huffman bit-matcher.

With the as-written implementation, there was a known bug in BWT decoding to do with repeated strings. This hasbeen worked around; see ‘bwt_reverse()’. Correct output is produced in all test cases but ideally the problem wouldbe found. . .

2.2.27 raytrace

Simple raytracer.

Command line options:

--width WIDTH Image width (default: 100)--height HEIGHT Image height (default: 100)--filename FILENAME.PPM Output filename of the PPM picture

This file contains definitions for a simple raytracer. Copyright Callum and Tony Garnock-Jones, 2008.

This file may be freely redistributed under the MIT license, http://www.opensource.org/licenses/mit-license.php

From http://www.lshift.net/blog/2008/10/29/toy-raytracer-in-python


http://www.paul.sladen.org/projects/compression/

http://www.opensource.org/licenses/mit-license.php

http://www.lshift.net/blog/2008/10/29/toy-raytracer-in-python


Image generated by the command (took 68.4 sec on CPython 3.5):

python3 pyperformance/benchmarks/bm_raytrace.py --worker --filename=raytrace.ppm -l1→˓-w0 -n1 -v --width=800 --height=600

2.2.28 regex_compile

Stress the performance of Python’s regex compiler, rather than the regex execution speed.

Benchmark how quickly Python’s regex implementation can compile regexes.

We bring in all the regexes used by the other regex benchmarks, capture them by stubbing out the re module, thencompile those regexes repeatedly. We muck with the re module’s caching to force it to recompile every regex we giveit.

2.2.29 regex_dna

regex DNA benchmark using “fasta” to generate the test case.

The Computer Language Benchmarks Game http://benchmarksgame.alioth.debian.org/

regex-dna Python 3 #5 program: contributed by Dominique Wahli 2to3 modified by Justin Peel

fasta Python 3 #3 program: modified by Ian Osgood modified again by Heinrich Acker modified by Justin PeelModified by Christopher Sean Forgeron




2.2.30 regex_effbot

Some of the original benchmarks used to tune mainline Python’s current regex engine.

2.2.31 regex_v8

Python port of V8’s regex benchmark.

Automatically generated on 2009-01-30.

This benchmark is generated by loading 50 of the most popular pages on the web and logging all regexp operationsperformed. Each operation is given a weight that is calculated from an estimate of the popularity of the pages where itoccurs and the number of times it is executed while loading each page. Finally the literal letters in the data are encodedusing ROT13 in a way that does not affect how the regexps match their input.

Ported to Python for Unladen Swallow. The original JS version can be found at https://github.com/v8/v8/blob/master/benchmarks/regexp.js, r1243.

2.2.32 richards

The classic Python Richards benchmark.

Based on a Java version.

Based on original version written in BCPL by Dr Martin Richards in 1981 at Cambridge University Computer Labo-ratory, England and a C++ version derived from a Smalltalk version written by L Peter Deutsch.

Java version: Copyright (C) 1995 Sun Microsystems, Inc. Translation from C++, Mario Wolczko Outer loop addedby Alex Jacoby

2.2.33 scimark

• scimark_sor: Successive over-relaxation (SOR) benchmark

• scimark_sparse_mat_mult: sparse matrix multiplication benchmark

• scimark_monte_carlo: benchmark on the Monte Carlo algorithm to compute the area of a disc

• scimark_lu: LU decomposition benchmark

• scimark_fft: Fast Fourier transform (FFT) benchmark

2.2.34 spectral_norm

MathWorld: “Hundred-Dollar, Hundred-Digit Challenge Problems”, Challenge #3. http://mathworld.wolfram.com/Hundred-DollarHundred-DigitChallengeProblems.html

The Computer Language Benchmarks Game http://benchmarksgame.alioth.debian.org/u64q/spectralnorm-description.html#spectralnorm

Contributed by Sebastien Loisel. Fixed by Isaac Gouy. Sped up by Josh Goldfoot. Dirtily sped up by Simon Descar-pentries. Concurrency by Jason Stitt.


https://github.com/v8/v8/blob/master/benchmarks/regexp.js

https://github.com/v8/v8/blob/master/benchmarks/regexp.js

https://en.wikipedia.org/wiki/Successive_over-relaxation

https://en.wikipedia.org/wiki/Sparse_matrix

https://en.wikipedia.org/wiki/Matrix_multiplication_algorithm

https://en.wikipedia.org/wiki/Monte_Carlo_algorithm

https://en.wikipedia.org/wiki/LU_decomposition

https://en.wikipedia.org/wiki/Fast_Fourier_transform

http://mathworld.wolfram.com/Hundred-DollarHundred-DigitChallengeProblems.html

http://mathworld.wolfram.com/Hundred-DollarHundred-DigitChallengeProblems.html

http://benchmarksgame.alioth.debian.org/u64q/spectralnorm-description.html#spectralnorm

http://benchmarksgame.alioth.debian.org/u64q/spectralnorm-description.html#spectralnorm


2.2.35 sqlalchemy_declarative, sqlalchemy_imperative

• sqlalchemy_declarative: SQLAlchemy Declarative benchmark using SQLite

• sqlalchemy_imperative: SQLAlchemy Imperative benchmark using SQLite

See the SQLAlchemy project.

2.2.36 sqlite_synth

Benchmark Python aggregate for SQLite.

The goal of the benchmark (written for PyPy) is to test CFFI performance and going back and forth between SQLiteand Python a lot. Therefore the queries themselves are really simple.

See the SQLite project and the Python sqlite3 module (stdlib).

2.2.37 sympy

Benchmark on the sympy module:

• sympy_expand: Benchmark sympy.expand()

• sympy_integrate: Benchmark sympy.integrate()

• sympy_str: Benchmark str(sympy.expand())

• sympy_sum: Benchmark sympy.summation()

On CPython, some sympy_sum values are 5%-10% slower:

$ python3 -m pyperf dump sympy_sum.jsonRun 1: 1 warmup, 50 values, 1 loop- warmup 1: 404 ms (+63%)- value 1: 244 ms- value 2: 245 ms- value 3: 258 ms <----- value 4: 245 ms- value 5: 245 ms- value 6: 279 ms (+12%) <----- value 7: 246 ms- value 8: 244 ms- value 9: 245 ms- value 10: 255 ms <----- value 11: 245 ms- value 12: 245 ms- value 13: 256 ms <----- value 14: 248 ms- value 15: 245 ms- value 16: 245 ms...

Plot of 1 run of 50 values (the warmup is not rendered):


https://www.sqlalchemy.org/

https://www.sqlite.org/

https://docs.python.org/dev/library/sqlite3.html


See the sympy project.

2.2.38 telco

Telco Benchmark for measuring the performance of decimal calculations:

• http://speleotrove.com/decimal/telco.html

• http://speleotrove.com/decimal/telcoSpec.html

• A call type indicator, c, is set from the bottom (least significant) bit of the duration (hence c is 0 or 1).

• A rate, r, is determined from the call type. Those calls with c=0 have a low r: 0.0013; the remainder(‘distance calls’) have a ‘premium’ r: 0.00894. (The rates are, very roughly, in Euros or dollarates persecond.)

• A price, p, for the call is then calculated (p=r*n). This is rounded to exactly 2 fractional digits using round-half-even (Banker’s round to nearest).

• A basic tax, b, is calculated: b=p*0.0675 (6.75%). This is truncated to exactly 2 fractional digits (round-down), and the total basic tax variable is then incremented (sumB=sumB+b).

• For distance calls: a distance tax, d, is calculated: d=p*0.0341 (3.41%). This is truncated to exactly 2fractional digits (round-down), and then the total distance tax variable is incremented (sumD=sumD+d).

• The total price, t, is calculated (t=p+b, and, if a distance call, t=t+d).

• The total prices variable is incremented (sumT=sumT+t).


http://www.sympy.org/

http://speleotrove.com/decimal/telco.html

http://speleotrove.com/decimal/telcoSpec.html


• The total price, t, is converted to a string, s.

The Python benchmark is implemented with the decimal module.

See the Python decimal module (stdlib).

2.2.39 tornado_http

Benchmark HTTP server of the tornado module

See the Tornado project.

2.2.40 unpack_sequence

Microbenchmark for unpacking lists and tuples.

Pseudo-code:

a, b, c, d, e, f, g, h, i, j = to_unpack

where to_unpack is tuple(range(10)) or list(range(10)).

2.2.41 xml_etree

Benchmark the ElementTree API of the xml.etree module:

• xml_etree_generate: Create an XML document

• xml_etree_iterparse: Benchmark etree.iterparse()

• xml_etree_parse: Benchmark etree.parse()

• xml_etree_process: Process an XML document

See the Python xml.etree.ElementTree module (stdlib).


https://docs.python.org/dev/library/decimal.html

http://www.tornadoweb.org/

https://docs.python.org/dev/library/xml.etree.elementtree.html

CHAPTER 3

CPython results, 2017

This page lists benchmarks which became faster in CPython.

3.1 Optimizations

3.1.1 2016-12-14: speedup method calls

Optimization: Speedup method calls 1.2x, commit f2392133.

Benchmark 2016-12-01 (27580c1fb5e8) 2017-01-01 (67e1aa0b58be)call_method 14.1 ms 11.2 ms: 1.26x faster (-21%)call_method_slots 13.9 ms 11.1 ms: 1.25x faster (-20%)call_method_unknown 16.0 ms 14.3 ms: 1.12x faster (-11%)

3.1.2 2016-04-22: pymalloc allocator

Optimization: PyMem_Malloc() now uses the fast pymalloc allocator, commit f5c4b990.

Changes of at least 5%:

25

https://bugs.python.org/issue26110

https://github.com/python/cpython/commit/f2392133eba777f05947a8996c507690b95379c3


https://github.com/python/cpython/commit/f5c4b99034fae12ac2b9498dd12b5b3f352b90c8


Benchmark 2016-04-21 (5439fc4901db) 2016-04-22 (f5c4b99034fa)unpickle_list 10.4 us 7.64 us: 1.36x faster (-27%)json_dumps 28.0 ms 25.2 ms: 1.11x faster (-10%)unpickle_pure_python 741 us 678 us: 1.09x faster (-9%)unpickle 33.9 us 31.3 us: 1.08x faster (-8%)meteor_contest 197 ms 183 ms: 1.08x faster (-7%)mako 36.9 ms 34.3 ms: 1.07x faster (-7%)pathlib 41.0 ms 38.4 ms: 1.07x faster (-6%)call_method_slots 14.8 ms 13.9 ms: 1.07x faster (-6%)telco 19.5 ms 18.3 ms: 1.07x faster (-6%)scimark_lu 413 ms 388 ms: 1.07x faster (-6%)nqueens 221 ms 207 ms: 1.07x faster (-6%)fannkuch 937 ms 882 ms: 1.06x faster (-6%)regex_compile 319 ms 301 ms: 1.06x faster (-6%)raytrace 1.16 sec 1.09 sec: 1.06x faster (-5%)pickle_pure_python 1.11 ms 1.05 ms: 1.05x faster (-5%)genshi_text 70.1 ms 66.6 ms: 1.05x faster (-5%)

3.1.3 2015-12-07: Optimize ElementTree.iterparse(), xml_etree_iterparse

Optimization: Issue #25638: Optimized ElementTree.iterparse(); it is now 2x faster, commit 9ec5e25f2.

Benchmark 2015-12-01 (df144092a340) 2016-01-01 (71db90356390)xml_etree_iterparse 423 ms 206 ms: 2.05x faster (-51%)

3.1.4 2015-09-19: PGO uses test suite, pidigits

Optimization: Issue #24915: Add Clang support to PGO builds and use the test suite for profile data, commit 7188a3ef.


Benchmark 2015-09-18 (4b363e270108) 2015-09-18_22-13 (7188a3efe07b)pickle 33.7 us 26.4 us: 1.28x faster (-22%)pidigits 332 ms 286 ms: 1.16x faster (-14%)pickle_list 9.90 us 8.84 us: 1.12x faster (-11%)unpickle 37.2 us 33.3 us: 1.12x faster (-11%)unpickle_list 11.1 us 9.95 us: 1.11x faster (-10%)regex_dna 330 ms 297 ms: 1.11x faster (-10%)regex_effbot 6.43 ms 5.80 ms: 1.11x faster (-10%)pickle_dict 69.3 us 64.1 us: 1.08x faster (-8%)mako 39.1 ms 36.2 ms: 1.08x faster (-7%)call_simple 12.2 ms 11.6 ms: 1.05x faster (-5%)genshi_xml 175 ms 166 ms: 1.05x faster (-5%)

Changes of at least 5%, sadly two benchmarks also became slower:

26 Chapter 3. CPython results, 2017


https://github.com/python/cpython/commit/9ec5e25f26a490510bb5da5c26a276cd30a263a0


https://github.com/python/cpython/commit/7188a3efe07b9effdb760f3a96783f250214f0be


Benchmark 2015-09-18_14-32-master-4b363e270108

2015-09-18_22-13-master-7188a3efe07b

un-pickle_pure_python

776 us 821 us: 1.06x slower (+6%)

regex_v8 49.5 ms 52.6 ms: 1.06x slower (+6%)

3.1.5 2015-05-30: C implementation of collections.OrderedDict, html5lib

Optimization: Issue #16991: Add a C implementation of collections.OrderedDict, commit 96c6af9b.

Benchmark 2015-05-02 (3b4d30a27bd6) 2015-06-01 (41874c570cf3)html5lib 285 ms 233 ms: 1.23x faster (-19%)

3.1.6 2015-05-23: C implementation of functools.lru_cache(), sympy

Optimization: Issue #14373: Added C implementation of functools.lru_cache(), commit 1c858c35.


Benchmark 2015-05-23_19-15-master-c70908558d8e

2015-05-23_19-42-master-1c858c352b8c

sympy_expand 1.45 sec 1.14 sec: 1.27x faster (-21%)sympy_sum 308 ms 247 ms: 1.25x faster (-20%)sympy_str 621 ms 500 ms: 1.24x faster (-19%)sympy_integrate 54.2 ms 45.7 ms: 1.19x faster (-16%)scimark_lu 497 ms 471 ms: 1.06x faster (-5%)

pickle_dict is seen as 1.06x slower, but since pickle doesn’t use functools.lru_cache(), the change is ignored inthe table.

3.2 Slowdown

3.2.1 2016-09-11: regex_compile

Slowdown: convert re flags to (much friendlier) IntFlag constants (issue #28082), commit f93395bc.

Benchmark 2016-04-01(6b6abd4cf10e)

2016-07-01(355048970b2a)

2016-10-01(78a111c7d867)

regex_compile 339 ms 309 ms: 1.10x faster (-9%) 383 ms: 1.13x slower (+13%)

3.3 Timeline

3.3.1 April, 2016 -> May, 2016

2016-04-01 .. 2016-05-01:

3.2. Slowdown 27


https://github.com/python/cpython/commit/96c6af9b207c188c52ac53ce87bb7f2dea3f328b


https://github.com/python/cpython/commit/1c858c352b8c11419f79f586334c49378726dba8


https://github.com/python/cpython/commit/f93395bc5125c99539597bf134ca8bcf9707655b


Benchmark 2016-04-01 (dcfebb32e277) 2016-05-01 (f1e2671fdf88)nqueens 255 ms 207 ms: 1.23x faster (-19%)raytrace 1.31 sec 1.09 sec: 1.19x faster (-16%)float 290 ms 243 ms: 1.19x faster (-16%)chaos 273 ms 235 ms: 1.16x faster (-14%)hexiom 21.0 ms 18.6 ms: 1.13x faster (-11%)deltablue 16.4 ms 14.6 ms: 1.12x faster (-11%)go 557 ms 502 ms: 1.11x faster (-10%)nbody 254 ms 232 ms: 1.10x faster (-9%)

3.3.2 call_method

Timeline 2016-04-01 .. 2017-01-01:


2016-07-01(355048970b2a)

2016-10-01(78a111c7d867)

2017-01-01(67e1aa0b58be)

call_method 15.8 ms 14.9 ms: 1.06x faster(-6%)

14.1 ms: 1.13x faster(-11%)

11.2 ms: 1.42x faster(-29%)

call_method_slots 15.7 ms 15.2 ms: 1.03x faster(-3%)

14.0 ms: 1.13x faster(-11%)

11.1 ms: 1.42x faster(-29%)

call_method_unknown17.7 ms 15.9 ms: 1.11x faster(-10%)

15.6 ms: 1.13x faster(-11%)

14.3 ms: 1.23x faster(-19%)

3.3.3 crypto_pyaes

Benchmark 2016-04-01 (master) 2016-05-01 (master)crypto_pyaes 226 ms 205 ms: 1.10x faster (-9%)

2016-03-01 .. 2016-06-01:

Benchmark 2016-03-01 (13d09afff127) 2016-06-01 (d80ab7d94578)crypto_pyaes 231 ms 199 ms: 1.16x faster (-14%)

3.3.4 json_loads

Progress on 21 months, 2015-01-01 .. 2016-10-01:

Benchmark 2015-01-01 (52074ac866eb) 2016-10-01 (78a111c7d867)json_loads 64.0 us 56.6 us: 1.13x faster (-11%)

3.3.5 logging_silent

Benchmark 2016-01-01 (899b72cee21c) 2016-07-01 (355048970b2a)logging_silent 718 ns 606 ns: 1.18x faster (-16%)



3.3.6 pickle

pickle, 2016-08-02 .. 2016-09-08:

Benchmark 2016-08-02 (133138a284be) 2016-09-08 (10427f44852b)pickle 25.5 us 21.4 us: 1.19x faster (-16%)

pickle dict/list:

Benchmark 2016-04-01 (6b6abd4cf10e) 2016-10-01 (78a111c7d867)pickle_dict 64.5 us 57.7 us: 1.12x faster (-11%)pickle_list 9.06 us 7.79 us: 1.16x faster (-14%)

unpickle:

Benchmark 2015-07-01 (d7982beca93c) 2015-10-01 (30b7138fe12b)unpickle 36.9 us 32.8 us: 1.13x faster (-11%)

3.3.7 python_startup

2015-04-01 .. 2015-10-01:

Benchmark 2015-04-01 (4fd929b43121) 2015-10-01 (30b7138fe12b)python_startup 16.4 ms 17.2 ms: 1.05x slower (+5%)python_startup_no_site 8.65 ms 8.90 ms: 1.03x slower (+3%)

2016-04-01 .. 2017-01-01:

Benchmark 2016-04-01 (6b6abd4cf10e) 2017-01-01 (67e1aa0b58be)python_startup 17.3 ms 14.5 ms: 1.20x faster (-16%)python_startup_no_site 8.89 ms 8.39 ms: 1.06x faster (-6%)

3.3.8 regex_compile


2016-07-01(355048970b2a)

2016-10-01(78a111c7d867)

regex_compile 339 ms 309 ms: 1.10x faster (-9%) 383 ms: 1.13x slower (+13%)

3.3.9 telco

Bench-mark

2016-01-01(899b72cee21c)

2016-04-01(6b6abd4cf10e)

2016-07-01(355048970b2a)

2016-10-01(78a111c7d867)

2017-03-31 (cd-cac039fb44)

telco 19.6 ms 19.2 ms: 1.02xfaster (-2%)

18.3 ms: 1.08xfaster (-7%)

15.1 ms: 1.30xfaster (-23%)

13.9 ms: 1.41xfaster (-29%)

3.3. Timeline 29


3.3.10 scimark

2016-10-01 .. 2017-03-31:

Bench-mark

2016-10-01(78a111c7d867)

2017-01-01(67e1aa0b58be)

2017-03-31 (cd-cac039fb44)

scimark_lu 423 ms 378 ms: 1.12x faster (-11%) 318 ms: 1.33x faster (-25%)scimark_sor 426 ms 403 ms: 1.06x faster (-5%) 375 ms: 1.14x faster (-12%)

3.3.11 sqlalchemy_declarative

Benchmark 2014-10-01 (5a789f7eaf81) 2015-10-01 (30b7138fe12b)sqlalchemy_declarative 345 ms 301 ms: 1.15x faster (-13%)

3.3.12 sympy

2016-04-01 .. 2016-10-01:


2016-07-01(355048970b2a)

2016-10-01(78a111c7d867)

sympy_expand 1.10 sec 1.01 sec: 1.09x faster (-8%) 942 ms: 1.17x faster (-14%)sympy_integrate 46.6 ms 42.9 ms: 1.09x faster (-8%) 41.2 ms: 1.13x faster (-11%)sympy_sum 247 ms 233 ms: 1.06x faster (-6%) 199 ms: 1.24x faster (-19%)sympy_str 483 ms 454 ms: 1.07x faster (-6%) 427 ms: 1.13x faster (-12%)

3.3.13 xml_etree_generate

Bench-mark

2015-04-01(4fd929b43121)

2015-07-01(d7982beca93c)

2015-10-01(30b7138fe12b)

2016-01-01(899b72cee21c)

2016-07-01(355048970b2a)

xml_etree_generate282 ms 267 ms: 1.06xfaster (-5%)

256 ms: 1.10xfaster (-9%)

237 ms: 1.19xfaster (-16%)

212 ms: 1.33xfaster (-25%)


CHAPTER 4

Changelog

4.1 Version 1.0.2 (2021-05-11)

• Disable the henshi benchmark temporarily since is no longer compatible with Python 3.11.

• Reenable html5lib benchmark: html5lib 1.1 has been released.

• Update requirements.

• Replace Travis CI with GitHub Actions.

• The development branch master was renamed to main. See https://sfconservancy.org/news/2020/jun/23/gitbranchname/ for the rationale.

4.2 Version 1.0.1 (2020-03-26)

• Drop usage of the six module since Python 2 is no longer supported. Remove Python 2 specific code.

• Update dependencies:

– django: 3.0 => 3.0.4

– dulwich: 0.19.14 => 0.19.15

– mako: 1.1.0 = > 1.1.2

– mercurial: 5.1.1 => 5.3.1

– psutil: 5.6.7 => 5.7.0

– pyperf: 1.7.0 => 2.0.0

– sqlalchemy: 1.3.12 => 1.3.15

– sympy: 1.5 => 1.5.1

– tornado: 6.0.3 => 6.0.4

31

https://sfconservancy.org/news/2020/jun/23/gitbranchname/

https://sfconservancy.org/news/2020/jun/23/gitbranchname/


• Remove six, html5lib and mercurial requirements.

• pip-tools (pip-compile) is now used to update dependencies

4.3 Version 1.0.0 (2019-12-17)

• Enable pyflate benchmarks on Python 3.

• Remove spambayes benchmark: it is not compatible with Python 3.

• Remove 2n3:benchmark group.

• Drop Python 2.7 support: old Django and Tornado versions are not compatible with incoming Python 3.9.

• Disable html5lib benchmark temporarily, since it’s no longer compatible with Python 3.9.

• Update requirements:

– Django: 1.11.22 => 3.0

– Mako: 1.0.14 => 1.1.0

– SQLAlchemy: 1.3.6 => 1.3.12

– certifi: 2019.6.16 => 2019.11.28

– docutils: 0.15.1 => 0.15.2

– dulwich: 0.19.11 => 0.19.14

– mercurial: 5.0.2 => 5.1.1

– psutil: 5.6. => 5.6.7

– pyperf: 1.6.1 => 1.7.0

– six: 1.12. => 1.13.0

– sympy: 1.4 => 1.5

4.4 Version 0.9.1 (2019-07-29)

• Enable hg_startup on Python 3

• Fix compatibility with Python 3.8 beta 2


– certifi: 2019.3.9 => 2019.6.16

– Chameleon: 3.6.1 => 3.6.2

– Django: 1.11.20 => 1.11.22

– docutils: 0.14 => 0.15.1.post1

– Mako: 1.0.10 => 1.0.14

– mercurial: 5.0 => 5.0.2

– pathlib2: 2.3.3 => 2.3.4

– psutil: 5.6.2 => 5.6.3

– SQLAlchemy: 1.3.4 => 1.3.6

32 Chapter 4. Changelog


4.5 Version 0.9.0 (2019-05-29)

• Project renamed from “performance” to “pyperformance”

• Upgrade pyperf from version 1.6.0 to 1.6.1. The project has been renamed from “perf” to “pyperf”. Updateimports.

• Issue #54: Update Genshi to 0.7.3. It is now compatible with Python 3.8.


– Mako: 1.0.9= > 1.0.10

– SQLAlchemy: 1.3.3 => 1.3.4

4.6 Version 0.8.0 (2019-05-10)

• compile command: Add “pkg_only” option to benchmark.conf. Add support for native libraries that are installedbut not on path. Patch by Robert Grimm.

• Update Travis configuration: use trusty image, use pip cache. Patch by Inada Naoki.

• Upgrade tornado to 5.1.1. Patch by Inada Naoki.

• Fix compile command on Mac OS: no program extension. Patch by Anthony Shaw.


– Chameleon: 3.4 => 3.6.1

– Django: 1.11.16 => 1.11.20

– Genshi: 0.7.1 => 0.7.2

– Mako: 1.0.7 => 1.0.9

– MarkupSafe: 1.0 => 1.1.1

– SQLAlchemy: 1.2.12 => 1.3.3

– certifi: 2018.10.15 => 2019.3.9

– dulwich: 0.19.6 => 0.19.11

– mercurial: 4.7.2 => 5.0

– mpmath: 1.0.0 => 1.1.0

– pathlib2: 2.3.2 => 2.3.3

– perf: 1.5.1 => 1.6.0

– psutil: 5.4.7 => 5.6.2

– six: 1.11.0 => 1.12.0

– sympy: 1.3 => 1.4

– tornado: 4.5.3 => 5.1.1

4.5. Version 0.9.0 (2019-05-29) 33


4.7 Version 0.7.0 (2018-10-16)

• python_startup: Add --exit option.


– certifi: 2017.11.5 => 2018.10.15

– Chameleon: 3.2 => 3.4

– Django: 1.11.9 => 1.11.16

– dulwich: 0.18.6 => 0.19.6

– Genshi: 0.7 => 0.7.1

– mercurial: 4.4.2 => 4.7.2

– pathlib2: 2.3.0 => 2.3.2

– psutil: 5.4.3 => 5.4.7

– SQLAlchemy: 1.2.0 => 1.2.12

– sympy: 1.1.1 => 1.3

• Fix issue #40 for pip 10 and newer: Remove indirect dependencies. Indirect dependencies were used to installcffi, but Mercurial 4.0 doesn’t depend on cffi anymore.

4.8 Version 0.6.1 (2018-01-11)

• Fix inherit-environ: propagate to recursive invocations of performance in compile and compile_allcommands.

• Fix the --track-memory option thanks to the update to perf 1.5.

• Update requirements

– certifi: 2017.4.17 => 2017.11.5

– Chameleon: 3.1 => 3.2

– Django: 1.11.3 => 1.11.9

– docutils: 0.13.1 => 0.14

– dulwich: 0.17.3 => 0.18.6

– html5lib: 0.999999999 => 1.0.1

– Mako: 1.0.6 => 1.0.7

– mercurial: 4.2.2 => 4.4.2

– mpmath: 0.19 => 1.0.0

– perf: 1.4 => 1.5.1 (fix --track-memory option)

– psutil: 5.2.2 => 5.4.3

– pyaes: 1.6.0 => 1.6.1

– six: 1.10.0 => 1.11.0

– SQLAlchemy: 1.1.11 => 1.2.0



– sympy: 1.0 => 1.1.1

– tornado: 4.5.1 => 4.5.3

4.9 Version 0.6.0 (2017-07-06)

• Change warn to warning in bm_logging.py. In Python 3, Logger.warn() calls warnings.warn() to log a dep-recation warning, so is slower than Logger.warning().

• Add again the logging_silent microbenchmark suite.

• compile command: update the Git repository before getting the revision


– perf: 1.3 => 1.4 (fix parse_cpu_list(): strip also NUL characters)

– Django: 1.11.1 => 1.11.3

– mercurial: 4.2 => 4.2.2

– pathlib2: 2.2.1 => 2.3.0

– SQLAlchemy: 1.1.10 => 1.1.11

4.10 Version 0.5.5 (2017-05-29)

• On the 2.x branch on CPython, compile now pass --enable-unicode=ucs4 to the configure scripton all platforms, except on Windows which uses UTF-16 because of its 16-bit wchar_t.

• The float benchmark now uses __slots__ on the Point class.

• Remove the following microbenchmarks. They have been moved to the pymicrobench project because they aretoo short, not representative of real applications and are too unstable.

– pybench microbenchmark suite

– call_simple

– call_method

– call_method_unknown

– call_method_slots

– logging_silent: values are faster than 1 ns on PyPy with 2^27 loops! (and around 0.7 us on CPython)


– Django: 1.11 => 1.11.1

– SQLAlchemy: 1.1.9 => 1.1.10

– certifi: 2017.1.23 => 2017.4.17

– perf: 1.2 => 1.3

– mercurial: 4.1.2 => 4.2

– tornado: 4.4.3 => 4.5.1

4.9. Version 0.6.0 (2017-07-06) 35

https://github.com/vstinner/pymicrobench


4.11 Version 0.5.4 (2017-04-10)

• Create a new documentation at: http://pyperformance.readthedocs.io/

• Add “CPython results, 2017” to the doc: significant performance changes, significant optimizations, timeline,etc.

• The show command doesn’t need to create a virtual env anymore.

• Add new commands:

– pyperformance compile: compile, install and benchmark

– pyperformance compile_all: benchmark multiple branches and revisions of Python

– pyperformance upload: upload a JSON file to a Codespeed

• setup.py: add dependencies to perf and six modules.

• bm_xml_etree now uses “_pure_python” in benchmark names if the accelerator is explicitly disabled.

• Upgrade requirements:

– Django: 1.10.6 -> 1.11

– SQLAlchemy: 1.1.6 -> 1.1.9

– mercurial: 4.1.1 -> 4.1.2

– perf: 1.1 => 1.2

– psutil: 5.2.1 -> 5.2.2

– tornado: 4.4.2 -> 4.4.3

– webencodings: 0.5 -> 0.5.1

• perf 1.2 now calibrates the number of warmups on PyPy.

• On Python 3.5a0: force pip 7.1.2 and setuptools 18.5: https://sourceforge.net/p/pyparsing/bugs/100/

4.12 Version 0.5.3 (2017-03-27)

• Upgrade Dulwich to 0.17.3 to support PyPy older than 5.6: see https://github.com/jelmer/dulwich/issues/509

• Fix ResourceWarning warnings: close explicitly files and sockets.

• scripts: replace Mercurial commands with Git commands.


– dulwich: 0.17.1 => 0.17.3

– perf: 1.0 => 1.1

– psutil: 5.2.0 => 5.2.1

4.13 Version 0.5.2 (2017-03-17)


– certifi: 2016.9.26 => 2017.1.23


http://pyperformance.readthedocs.io/

https://sourceforge.net/p/pyparsing/bugs/100/

https://github.com/jelmer/dulwich/issues/509


– Chameleon: 3.0 => 3.1

– Django: 1.10.5 => 1.10.6

– MarkupSafe: 0.23 => 1.0

– dulwich: 0.16.3 => 0.17.1

– mercurial: 4.0.2 => 4.1.1

– pathlib2: 2.2.0 => 2.2.1

– perf: 0.9.3 => 1.0

– psutil: 5.0.1 => 5.2.0

– SQLAlchemy: 1.1.4 => 1.1.6

4.14 Version 0.5.1 (2017-01-16)

• Fix Windows support (upgrade perf from 0.9.0 to 0.9.3)


– Chameleon: 2.25 => 3.0

– Django: 1.10.3 => 1.10.5

– docutils: 0.12 => 0.13.1

– dulwich: 0.15.0 => 0.16.3

– mercurial: 4.0.0 => 4.0.2

– perf: 0.9.0 => 0.9.3

– psutil: 5.0.0 => 5.0.1

4.15 Version 0.5.0 (2016-11-16)

• Add mdp benchmark: battle with damages and topological sorting of nodes in a graph

• The default benchmark group now include all benchmarks but pybench

• If a benchmark fails, log an error, continue to execute following benchmarks, but exit with error code 1.

• Remove deprecated benchmarks: threading_threaded_count andthreading_iterative_count. It wasn’t possible to run them anyway.

• dulwich requirement is now optional since its installation fails on Windows.


– Mako: 1.0.5 => 1.0.6

– Mercurial: 3.9.2 => 4.0.0

– SQLAlchemy: 1.1.3 => 1.1.4

– backports-abc: 0.4 => 0.5

4.14. Version 0.5.1 (2017-01-16) 37


4.16 Version 0.4.0 (2016-11-07)

• Add sqlalchemy_imperative benchmark: it wasn’t registered properly

• The list command now only lists the benchmark that the run command will run. The list command getsa new -b/--benchmarks option.

• Rewrite the code creating the virtual environment to test correctly pip. Download and run get-pip.py if pipinstallation failed.


– perf: 0.8.2 => 0.9.0

– Django: 1.10.2 => 1.10.3

– Mako: 1.0.4 => 1.0.5

– psutil: 4.3.1 => 5.0.0

– SQLAlchemy: 1.1.2 => 1.1.3

• Remove virtualenv dependency

4.17 Version 0.3.2 (2016-10-19)

• Fix setup.py: include also performance/benchmarks/data/asyncio.git/

4.18 Version 0.3.1 (2016-10-19)

• Add regex_dna benchmark

• The run command now fails with an error if no benchmark was run.

• genshi, logging, scimark, sympy and xml_etree scripts now run all sub-benchmarks by default

• Rewrite pybench using perf: remove the old legacy code to calibrate and run benchmarks, reuse perf.RunnerAPI.

• Change heuristic to create the virtual environment, tried commands:

– python -m venv

– python -m virtualenv

– virtualenv -p python

• The creation of the virtual environment now ensures that pip works to detect “python3 -m venv” which doesn’tinstall pip.

• Upgrade perf dependency from 0.7.12 to 0.8.2: update all benchmarks to the new perf 0.8 API (which introducesincompatible changes)

• Update SQLAlchemy from 1.1.1 to 1.1.2



4.19 Version 0.3.0 (2016-10-11)

New benchmarks:

• Add crypto_pyaes: Benchmark a pure-Python implementation of the AES block-cipher in CTR mode usingthe pyaes module (version 1.6.0). Add pyaes dependency.

• Add sympy: Benchmark on SymPy. Add scipy dependency.

• Add scimark benchmark

• Add deltablue: DeltaBlue benchmark

• Add dulwich_log: Iterate on commits of the asyncio Git repository using the Dulwich module. Adddulwich (and mpmath) dependencies.

• Add pyflate: Pyflate benchmark, tar/bzip2 decompressor in pure Python

• Add sqlite_synth benchmark: Benchmark Python aggregate for SQLite

• Add genshi benchmark: Render template to XML or plain text using the Genshi module. Add Genshidependency.

• Add sqlalchemy_declarative and sqlalchemy_imperative benchmarks: SQLAlchemy Declar-ative and Imperative benchmarks using SQLite. Add SQLAlchemy dependency.

Enhancements:

• compare command now fails if the performance versions are different

• nbody: add --reference and --iterations command line options.

• chaos: add --width, --height, --thickness, --filename and --rng-seed command line op-tions

• django_template: add --table-size command line option

• json_dumps: add --cases command line option

• pidigits: add --digits command line option

• raytrace: add --width, --height and --filename command line options

• Port html5lib benchmark to Python 3

• Enable pickle_pure_python and unpickle_pure_python on Python 3 (code was already compatiblewith Python 3)

• Creating the virtual environment doesn’t inherit environment variables (especially PYTHONPATH) by defaultanymore: --inherit-environ command line option must now be used explicitly.

Bugfixes:

• chaos benchmark now also reset the random module at each sample to get more reproductible benchmarkresults

• Logging benchmarks now truncate the in-memory stream before each benchmark run

Rename benchmarks:

• Rename benchmarks to get a consistent name between the command line and benchmark name in the JSON file.

• Rename pickle benchmarks:

– slowpickle becomes pickle_pure_python

– slowunpickle becomes unpickle_pure_python

4.19. Version 0.3.0 (2016-10-11) 39


– fastpickle becomes pickle

– fastunpickle becomes unpickle

• Rename ElementTree benchmarks: replace etree_ prefix with xml_etree_.

• Rename hexiom2 to hexiom_level25 and explicitly pass --level=25 parameter

• Rename json_load to json_loads

• Rename json_dump_v2 to json_dumps (and remove the deprecated json_dump benchmark)

• Rename normal_startup to python_startup, and startup_nosite topython_startup_no_site

• Rename threaded_count to threading_threaded_count, rename iterative_count tothreading_iterative_count

• Rename logging benchmarks:

– silent_logging to logging_silent

– simple_logging to logging_simple

– formatted_logging to logging_format

Minor changes:

• Update dependencies

• Remove broken --args command line option.

4.20 Version 0.2.2 (2016-09-19)

• Add a new show command to display a benchmark file

• Issue #11: Display Python version in compare. Display also the performance version.

• CPython issue #26383; csv output: don’t truncate digits for timings shorter than 1 us

• compare: Use sample unit of benchmarks, format values in the table output using the unit

• compare: Fix the table output if benchmarks only contain a single sample

• Remove unused -C/–control_label and -E/–experiment_label options

• Update perf dependency to 0.7.11 to get Benchmark.get_unit() and BenchmarkSuite.get_metadata()

4.21 Version 0.2.1 (2016-09-10)

• Add --csv option to the compare command

• Fix compare -O table output format

• Freeze indirect dependencies in requirements.txt

• run: add --track-memory option to track the memory peak usage

• Update perf dependency to 0.7.8 to support memory tracking and the new --inherit-environ commandline option

• If virtualenv command fail, try another command to create the virtual environment: catch virtualenverror



• The first command to upgrade pip to version >= 6.0 now uses the pip binary rather than python -m pipto support pip 1.0 which doesn’t support python -m pip CLI.

• Update Django (1.10.1), Mercurial (3.9.1) and psutil (4.3.1)

• Rename --inherit_env command line option to --inherit-environ and fix it

4.22 Version 0.2 (2016-09-01)

• Update Django dependency to 1.10

• Update Chameleon dependency to 2.24

• Add the --venv command line option

• Convert Python startup, Mercurial startup and 2to3 benchmarks to perf scripts (bm_startup.py,bm_hg_startup.py and bm_2to3.py)

• Pass the --affinity option to perf scripts rather than using the taskset command

• Put more installer and optional requirements into performance/requirements.txt

• Cached .pyc files are no more removed before running a benchmark. Use venv recreate command toupdate a virtual environment if required.

• The broken --track_memory option has been removed. It will be added back when it will be fixed.

• Add performance version to metadata

• Upgrade perf dependency to 0.7.5 to get Benchmark.update_metadata()

4.23 Version 0.1.2 (2016-08-27)

• Windows is now supported

• Add a new venv command to show, create, recrete or remove the virtual environment.

• Fix pybench benchmark (update to perf 0.7.4 API)

• performance now tries to install the psutil module on CPython for better system metrics in metadata andCPU pinning on Python 2.

• The creation of the virtual environment now also tries virtualenv and venv Python modules, not only thevirtualenv command.

• The development version of performance now installs performance with “pip install -e <path_to_performance>”

• The GitHub project was renamed from python/benchmarks to python/performance.

4.24 Version 0.1.1 (2016-08-24)

• Fix the creation of the virtual environment

• Rename pybenchmarks script to pyperformance

• Add -p/–python command line option

• Add __main__ module to be able to run: python3 -m performance

4.22. Version 0.2 (2016-09-01) 41


4.25 Version 0.1 (2016-08-24)

• First release after the conversion to the perf module and move to GitHub

• Removed benchmarks

– django_v2, django_v3

– rietveld

– spitfire (and psyco): Spitfire is not available on PyPI

– pystone

– gcbench

– tuple_gc_hell

4.26 History

Projected moved to https://github.com/python/performance in August 2016. Files reorganized, benchmarks patched touse the perf module to run benchmark in multiple processes.

Project started in December 2008 by Collin Winter and Jeffrey Yasskin for the Unladen Swallow project. The projectwas hosted at https://hg.python.org/benchmarks until Feb 2016

Other Python Benchmarks:

• CPython: speed.python.org uses pyperf, pyperformance and Codespeed (Django web application)

• PyPy: speed.pypy.org uses PyPy benchmarks

• Pyston: pyston-perf and speed.pyston.org

• Numba benchmarks

• Cython: Cython Demos/benchmarks

• pythran: numpy-benchmarks

See also the Python speed mailing list and the Python pyperf module (used by pyperformance).

pyperformance is not tuned for PyPy yet: use the PyPy benchmarks project instead to measure PyPy performances.

Image generated by bm_raytrace (pure Python raytrace):


https://github.com/python/performance

https://hg.python.org/benchmarks

https://speed.python.org/

https://github.com/tobami/codespeed/

http://speed.pypy.org/

https://bitbucket.org/pypy/benchmarks

https://github.com/dropbox/pyston-perf

http://speed.pyston.org/

http://numba.pydata.org/numba-benchmark/

https://github.com/cython/cython/tree/master/Demos/benchmarks

https://github.com/serge-sans-paille/numpy-benchmarks

https://mail.python.org/mailman/listinfo/speed

http://pyperf.readthedocs.io/

https://foss.heptapod.net/pypy/benchmarks


4.26. History 43

Documents

Python Performance Benchmark Suite Documentation