SANER 2015 ERA track: Differential Flame Graphs

3/5/15

Challenge the future

DelftUniversity ofTechnology

Understanding Software Performance Regressions Using Differential Flame GraphsCor-Paul Bezemer, Johan Pouwelse, Brendan Gregg

2Understanding Software Performance Regressions Using DFGs

Regression Testing


Regression Testing Cycle

1)Write code 2)Deploy to continuous integration system3)Test4)Deploy to production code or return to 1)


Performance Regression Testing

• Functional testing is difficult, but:• Functionality is either OK or broken

• Performance testing is even more difficult:• Depends on requirements of application

• A library may be fast enough for one application, while it's too slow for another

• How much performance regression is acceptable?



What to Measure

• Built-in performance metrics (or counters)• Response time, throughput, queue lengths, utilization• Usually system-wide

• Profilers• Per-function info on execution time, bytes written, etc.

• Higher granularity == more expensive

• After measuring: tons of data! :) / :(



Detecting I/O Regression

• Previous work:• “Detecting and analyzing I/O performance regressions”,

C. Bezemer, E. Milon, A. Zaidman, J. Pouwelse (Journal of Software: Evolution and Process 26 (12), 1193-1212)

• Approach for finding out which function exhibits I/O write regression over two software versions

• Results promising, but sometimes difficult to interpret


Work in Progress: Visualization


Flame Graphs (1)

def a ( ) : if some condition : b() …

def b ( ) : if some condition : c () ...

def c( ): ...

def d ( ) : if some condition : c () ...

Stack trace, CPU timea(), 100a()→b(), 25a()→b()→c(), 10d(), 50d()→c(), 10


Flame Graphs (2)


Flame Graphs (3)


Differential Flame Graphs (1)


Demo


Differential Flame Graphs (2)

• You can do this with performance profiles..

• But also with other data?• Ideas:

• Website clickpaths• Parallel/distributed computing• ...

• Let us know! http://corpaul.github.io/flamegraphdiff/

• (they use this at Netflix, so it must be pretty cool!)

http://corpaul.github.io/flamegraphdiff/


Discussion

• Data collection is difficult• No out of the box solutions• Expensive

• Data variation between executions can be considerable• Especially for 'non-stable' metrics (CPU, memory)


[email protected]://corpaul.github.io/flamegraphdiff/

mailto:[email protected]