Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access

Heap Growth Detection in C++GrowthTracker

1

Heap Growth Detection in C++Motivation

• Scalable City needs to run continuously– Many months without intervention/access– Had slow growth of memory

• Caused crash after several weeks

– Available analysis tools reported no leaks!• Software frees all memory correctly!• Had different kind of undetected memory issue

2

What is a Memory Leak?

3


• Has become a broad term for memory mismanagement• Definition depends on the programming language

– Does it have Garbage Collection?

4


• C/C++ Programmers generally subscribe to traditional definition:

Def. 1: A memory leak occurs iff all references to a region of memory are lost before the memory is freed.

5


• Java/C# Programmers generally have a different definition:

Def. 2: A memory leak occurs when a program’s memory is unexpectedly growing.

– Using Def. 1, Java/C# don’t have memory leaks• Garbage Collector frees memory that is no longer referenced

6


7

Def .2(unbounded growth)

Def .1(lost refs)

C/C++ Java/C#

Def .2(unbounded growth)

Def .1(lost refs)


• We need new terminology!– Java/C# programmers

• Concerned with unbounded memory growth, not leaks– References exist to all memory growth

• Have clearer understanding of their problem

– C/C++ programmers• Multiple definitions causes confusion• False sense of security when tools report no memory

leaks– Unbounded memory growth may still exist!

• A term for unbounded memory growth not caused by memory leaks, would clearly parse the 2 problems

8

New “Leak” Terminology

• Properties of new term– Unbounded Heap Growth– Reference(s) retained– Manifests as dynamically growing data structure– Will eventually kill the process– Doesn’t fit leaky pipe metaphor

• Memory Tumor– Structure of cells that exhibits unbounded growth

9

Memory Tumor Example

void main(){

queue q;

while( key != ESC ) // exits when ESC key pressed

q.push(0);

}

• q will grow constantly while the program runs• there are no memory leaks

– if the user hits ESC, all memory is freed when q goes out of scope

10

Leak vs. Tumor

11

X

X

X

X

X

Leak Tumor

heap heap

Separation of Concerns

12

Memory Leak(Def 1: lost refs)

C/C++/Java/C#

Memory Tumor

Current C++ Tumor Detection Tools

• SWAT and Hound– Only ones we’re aware of– Both use Staleness detection

• Misses some tumors by design• Memory can be accessed, but not needed

– Investigate at allocation level• Don’t need to modify source code

– Not open source or commercially available

13

Memory Tumor Detection Theory

• Tumors – Data Structures that grow without bound

• Healthy data structures– Will grow– Maximum size stabilizes

14

Memory Tumor Detection Challenges

• Detect all growth that doesn’t stabilize – Don’t dismiss non-stale growth

• Tests must exhibit the growth that exists in a program’s implementation

• Support Multithreaded programs

15

Memory Tumor Detection Approach

• Growth Tracker Tool– Container Tracking

• Keep references to all data structures in memory

– Growth Tracking• Track data structure size changes over time• Identify those with unbounded growth

• Automated Test– Cyclically execute all code paths (user created)

16

Growth Tracker Tool

• Container tracking– CAT (Central Aggregate Tracker)

• Maintains references to all aggregates in the system– Create wrappers for each aggregate type in

system• Templated constructors, multiple inheritance• Add to CAT on construction, remove on destruction

– Namespace replacement to enable wrappers• Find and replace to apply new namespace• Wrappers disabled with compile time flag• Example: trak::std::vector<int>

17

Growth Tracker Tool

• Growth Tracking– Take periodic samples of the CAT– Exponentially increasing interval sizes

• Reduces false positives & negatives over time– Report growing aggregates at each sample

18

Growth Tracking Heuristic

• Take periodic samples of the CAT• Two Interval Analysis

– 1st interval establishes aggregate age, gives time to stabilize– 2nd interval proves stability, non-tumors shouldn’t grow– 2nd interval becomes the 1st for next more accurate test

• Exponentially increasing interval sizes– Reduces false positives & negatives over time

• Monitor size maximums– Reduces size fluctuation false positives

• At each interval report all aggregates that:– Increased their size maximum– Have existed for two full intervals

• Prioritize results by size & reporting frequency

19


• Two interval analysis

20

time

memory

1 2 3 4

Not reported(growth stabilized)

A Data Structure Memory Footprint

Diagnosing Unbounded Heap Growth in C++ Detection Approach

• Two interval analysis

21

time

memory

1 2 3 4

Reported as tumor(false positive)

Not reported(growth stabilized)








22


• Growth Tracking– Exponentially increasing interval size

23

time

memory

1 2 3 4

In this example: constant intervals would not report growth half the time

erik

new slide








24


• Growth Tracking– Max size variable

25

1 2 3

time

memory

ceiling

Growth would be reported without

max size

erik

new slide








26

Growth Tracker Targets

• Multi-threaded Applications– Initial CAT implementation works

• Requires locking for each aggregate constructor• Potential to diminish multi-threaded performance• Good starting point

– Need new CAT implementation• Eliminate Locks

– Multiple bucket approach – Map aggregate construction from different threads to buckets

• Design can accelerate sampling process as well

27

Growth Tracker Drawbacks

– Source code modification• Tracking requires compilation with our wrappers• Allows consideration of Objects not just allocations.

– Limited information about identified tumors• Full type string & allocation number• Code location possible

– requires stack tracing (slower)

– Reliance on the user• Must identify custom data structures• Must run feature complete and cyclic test

28

Growth Tracker Drawbacks

– Multi-threaded potential slow down– Persistent buckets

• Example: Linear hash table with std::vector buckets

• More useful to include child bucket sizes in parent’s output and stop reporting individual children

– Multiple instances of same tumor reported• Parent report including children would resolve

29

Growth Tracker Results

– Scalable City• Identified tumor• Eliminated memory growth

– Ogre3D Rendering Engine• Identified 2 tumors• Our fix integrated into their code base

– Bullet Physics Engine• Tests revealed no tumors in Core• 1 tumor found in demo framework

30

Growth Tracker Results

– Google Chrome / Chromium• Identified 21 tumors• Fixed the fastest growing tumor ourselves

– WebKit (Safari Browser, etc.)• Identified 2 tumors• Submitted fix to code base

31

Growth Tracker Paper

• Recently accepted for publication• IEEE International Conference on

Software Testing, Verification and Validation (ICST 2013)

32

Growth Tracker Proposed Work

– Resolve Multithreaded locking limitations• Solution designed, needs implementation

– Reduce tracking of temporaries• Detect stack-based data structures• Multi-layer CAT to separate entries by age

– Will reduce overhead of CAT insertion/removal

33


• Automation– Reduce reliance on the user– Detect custom data structures

• Automatically create wrappers when possible

– Improvements to code transformation process• After initial code conversion, detect when wrapper

is forgotten.

34


• Prioritize tracking parent data structures– Would address persistent bucket problem– Would reduce reports of multiple instances of

same tumor– Must identify relationships between data

structures.

35

Documents

Heap Growth Detection in C++ GrowthTracker 1. Heap Growth Detection in C++ Motivation Scalable City needs to run continuously –Many months without intervention/access