Memory DebuggingIn Python, Cython, and C/C++ On Linux

Alain Leon

github.com/xerebz

xerebz@gmail.com

Pindrop

BackgroundMemory was found to be leaking somewhere in our Python scripts

There are a bunch of tools I had never used before to find and diagnose the problem

Good Ol’ Top (or htop)Run your code and watch resident memory increase

● top -> shift+F -> n -> enter

● top -> shift+M

● watch -n 1 ps -o rss -C PROCESSNAME

● watch -n 1 ps -o rss -p PID

Python Memory Tools● Objgraph - objgraph.show_growth()

● Guppy/Heapy - hpy.heap()

● Memory_profiler - @profile

● Dowser

● Many others

C/C++ Memory ToolsValgrind (Memcheck, Massif, DHAT) - suite of tools, replace calls to malloc

GDB-Heap - unplanned memory usage debugging, adds heap helpers to GDB

Heaptrack - LD_PRELOAD (DLL Injection)

Google Heap Profiler - uses thread-caching malloc

sudo ltrace -e malloc,calloc,realloc,free -p $(pidof extract_all)

ValgrindPronounced Val-Grinned

Memcheck - default tool, lists out possible memory problems

Massif - heap profiler, describes heap size over time

DHAT - Dynamic Heap Allocation Tool, heap allocation tool, process lifetime leaks

Python allocates itself its own memory block (PyMalloc)

Compile Python with --without-pymalloc

docker pull xerebz/pygrind

GDBgdb python -> r test.py

python2.7-dbg test.py & -> gdb -p $(pidof python2.7-dbg)

Add libpython debugging symbols and GDB helper functions like py-bt

http://grapsus.net/blog/post/Low-level-Python-debugging-with-GDB

http://goo.gl/MRr3Od

Linux Virtual Memory Manager

Linux Memory System Callsbrk - set the program break, adds memory pages to the top of the heap

sbrk - increments the program break

mmap - map independent page of memory

munmap - unmaps memory pages

strace -e brk,mmap,munmap -p $(pidof PROCNAME)

Compiler flag -O0 removes optimizations

Haiku Intermission

Exampleltrace strace

At Pindrop

Before (Massif output)

After (Massif output)

The End!

Alain Leon

github.com/xerebz

xerebz@gmail.com

Pindrop