Automatic Detection of Performance Anomalies in Task ...€¦ · I Post-mortem (o ine) analysis...

Automatic Detection of Performance Anomalies inTask-Parallel Programs

Work in progress on the Aftermath trace analysis tool

Andi Drebes

Universite Pierre et Marie CurieLaboratoire d’Informatique de Paris VI

andi.drebes@lip6.fr

Joint work with:Antoniu Pop, Karine Heydemann

Albert Cohen, Nathalie Drach

RACING’14, May 30th, 2014

Open tream

Context

Hardware and software environment

I Multi-core / many-core systems

I How to exploit the hardware efficiently?

I Task-parallel languages based on fine-grained tasks

Performance debugging

I Requires analysis of complex interactions at execution time:Application / Run-time / Machine

I Possible solution: Record dynamic events to a trace file

I Post-mortem (offline) analysis

Aftermath

I Trace visualization and support for manual analysis

I Originally developed for OpenStream language & run-time

I Work in progress: Automate repetitive tasks

Andi Drebes – Automatic Detection of Performance Anomalies in Task-Parallel Programs 1 / 16

Context

Aftermath

Context

Aftermath

Outline

1. Aftermath

2. Insufficient parallelism and its causes

3. Performance anomalies during task execution

4. Work in progress: Status

5. Summary & Questions

Andi Drebes – Automatic Detection of Performance Anomalies in Task-Parallel Programs

Aftermath

TimelineTimeline

Detailed text viewDetailed text view

Menu bar: derived metricsMenu bar: derived metrics

Aftermath

Procesors

Activityduringexecution

Aftermath

Task execution (dark blue)

Aftermath

Task Task Task

Instance Instance Instance

Task execution (dark blue)

Aftermath

Task creation (white)

Aftermath

Searching for work (light blue)

Aftermath

Basic statistics for run-time states

Aftermath

Heatmap indicating task duration (white: fast, red: slow)

Aftermath

NUMA heatmap indicating locality of memory accesses(blue: local, pink: remote)

Navigating through a trace

I Huge amounts of high-dimensional data

I Lots of features → lots of possibilities for analysis

I Where to look? What to look for?

I Expertise & lots of time required

Guide the user through performance analysisI Refine analysis in several steps

I Start by analyzing parallelismI Analyze what happens inside tasks

I Automate repetitive tasks

Navigating through a trace

I Huge amounts of high-dimensional data

I Lots of features → lots of possibilities for analysis

I Where to look? What to look for?

I Expertise & lots of time required

Guide the user through performance analysisI Refine analysis in several steps

I Start by analyzing parallelismI Analyze what happens inside tasks

I Automate repetitive tasks

Detecting insufficient parallelism

Task execution

Other states

Ideal situationAll CPUs are in task executionstate without any interruption

Pn-1...

Task execution

Other states

Realistic scenarioTask creation, memoryallocation, idle time,over-synchronization, . . .

Detect insufficient parallelism / high overhead automatically

Task execution

Other states

Pn-1...

Task execution

Other states

Task execution

Other states

Pn-1...

Task execution

Other states

Threshold-based analysis of parallelism

Task execution

Other states

d : Duration of the intervalde,i : Time that processor i spends in task execution statete : Threshold for task execution, e.g. te = 0.95

Consider that there is sufficient parallelism if inequation holds:

n∑i=1

de,i > te · n · d

Task execution

Other states

d : Duration of the interval

de,i : Time that processor i spends in task execution statete : Threshold for task execution, e.g. te = 0.95

n∑i=1

de,i > te · n · d

Task execution

Other states

de,0+ + + +

d : Duration of the intervalde,i : Time that processor i spends in task execution state

te : Threshold for task execution, e.g. te = 0.95

n∑i=1

de,i > te · n · d

Task execution

Other states

n∑i=1

de,i > te · n · d

Task execution

Other states

de,2+ + + +

n∑i=1

de,i > te · n · d

Task execution

Other states

de,n-1+

n∑i=1

de,i > te · n · d

Task execution

Other states

n∑i=1

de,i > te · n · d

Task execution

Other states

n∑i=1

de,i > te · n · d

Detecting the cause of insufficient parallelism

Multiple stages during analysis

I If inequation does not hold, find out why

I Possible causes: task creation overhead, memory allocation,not enough tasks available for execution, . . .

I Use thresholds for associated states:tc (task creation), ti (idle time)

1 2 3 4 5 6 7 8 9 10

Interval selection

I Multiple intervals: initialization, termination, etc.

I Repeat analysis for different intervals

Detecting the cause of insufficient parallelism

Multiple stages during analysis

I If inequation does not hold, find out why

I Possible causes: task creation overhead, memory allocation,not enough tasks available for execution, . . .

I Use thresholds for associated states:tc (task creation), ti (idle time)

1 2 3 4 5 6 7 8 9 10

Interval selection

I Multiple intervals: initialization, termination, etc.

I Repeat analysis for different intervals

Per-interval analysis of parallelism & overhead

ChooseInterval

Analyze taskexecution time

AboveSufficientparallelism

No intervalleft

ChooseInterval

Analyze task

synchronizationAnalyze task

creation time...

No intervalleft

Analyzeidle time

ChooseInterval

Analyzeidle time

Analyze task

synchronizationAnalyze task

creation time...

Inefficientsynchroni-

zation

Notenough

parallelismexposed

High taskcreationoverhead

BelowAbove BelowAbove BelowAbove BelowAbove

No intervalleft

Detecting performance anomalies during task execution

During task executionI Performance anomaly possible even at 100% task execution

(ineffective use of caches, remote memory accesses, branchmisprediction)

Number of tasks

Duration

Number of tasks

Duration

Impact on the distribution of task durationI Slowdown of all tasksI Different groups / peaks

Detecting performance anomalies during task execution

During task executionI Performance anomaly possible even at 100% task execution

(ineffective use of caches, remote memory accesses, branchmisprediction)

Number of tasks

DurationNumber of tasks

Duration

Impact on the distribution of task durationI Slowdown of all tasksI Different groups / peaks

Using performance counters

Hardware performance counters

I Implemented in hardware, no slowdown of the application

I Low tracing overhead if sampled at beginning / end of a task

I Dozens of hardware events can be monitored

Automatic analysis of performance counters

I Which hardware events are relevant?

I Manual testing tedious & time consuming

Using performance counters

Hardware performance counters

I Implemented in hardware, no slowdown of the application

I Low tracing overhead if sampled at beginning / end of a task

I Dozens of hardware events can be monitored

Automatic analysis of performance counters

I Which hardware events are relevant?

I Manual testing tedious & time consuming

Analyzing performance counters

Counter value

v(c,i,t)Pi

Per-CPU performance counterI Absolute value v(c , i , t), monotonically increasing

I c : Counter (e.g. cache misses)I i : Processor identifierI t: Timestamp

I Sampled at the beginning and end of a task

Break down counter evolution to task instances

I Increase of c by task T : Nc,T = v(c, i , e)− v(c , i , s)

Counter value

v(c,i,t)Pi

Counter value

v(c,i,t)Pi

Counter value

v(c,i,t)Pi Nc,T

Linear regression

Duration

Nc,TjPerformance indicator value

Perform linear regression

I Assume linear model: dTj= α · Nc,Tj

+ β (α and βconstant)

I Compare coefficient of determination with threshold

Linear regression

Duration

Linear regression

Duration

Shortcuts & refinement

Variation of task duration

I Determine coefficient of variation for task duration

I Only perform analysis if significant

Task typesI Different task types in an application

I Auxiliary tasks: initialization, terminationI Work tasks: matrix multiplication, decomposition, etc.

I Performance anomaly not necessarily present in all types

Topology of the machine

I Anomaly only present on subset of processors

I Example: Memory accesses local for one NUMA node, remoteon another

Choose task type

Choose task typeChoose set

of processors

of processorsCheck varation

of task duration

Choose set of

performance

counters

of task duration

LowHigh

Choose set of

performance

counters

of processors

Break down values

to task instances

Check varation

of task duration

LowHigh

Choose set of

performance

counters

of processors

Correlate per-task

instance values

and duration

Break down values

to task instances

Check varation

of task duration

LowHigh

Choose set of

performance

counters

of processors

Correlate per-task

instance values

and duration

Break down values

to task instances

Check varation

of task duration

Event set

irrelevant

LowHigh

Choose set of

performance

counters

of processors

Correlate per-task

instance values

and duration

Break down values

to task instances

Check varation

of task duration

Event set

irrelevant

Event set

relevant

LowHigh

Choose set of

performance

counters

of processors

Correlate per-task

instance values

and duration

Break down values

to task instances

Check varation

of task duration

Event set

irrelevant

Event set

relevant

LowHigh

Choose set of

performance

counters

of processors

Correlate per-task

instance values

and duration

Break down values

to task instances

Check varation

of task duration

Event set

irrelevant

Event set

relevant

No set left

LowHigh

Choose set of

performance

counters

of processors

Correlate per-task

instance values

and duration

Break down values

to task instances

Check varation

of task duration

No set left

Event set

irrelevant

Event set

relevant

No set left

LowHigh

Example: K-means branch misprediction

Task duration [cycles]

0.0×100

2.0×106

4.0×106

6.0×106

8.0×106

1.0×107

1.2×107

1.4×107

1.6×107

1.8×107

2.0×107

Task d

tion [

Branch mispredictions

0.0×100

2.0×106

4.0×106

6.0×106

8.0×106

1.0×107

1.2×107

1.4×107

1.6×107

1.8×107

2.0×107

Task d

tion [

Branch mispredictions

Low mispre-diction rateLow mispre-diction rate

High mispre-diction rateHigh mispre-diction rate

Work in progress: Status

Analysis of parallelism

I Per-interval analysis of time spent on task execution

I Per-interval analysis of time spent in run-time states

I Support for thresholds

I Loop performing analysis on set of intervals

Correlation of performance indicators

I Support for performance counters

I Task duration histogram

I Analysis of the variation of task durations

I Breaking down performance counter values to task instances

I Linear regression

Summary

Aftermath

I Tool for trace-based analysis of task-parallel programs

I Currently provides only support for manual analysis

I Available at http://openstream.info/aftermath

Automatic analysis of parallelism based on thresholds

I Amount of time spent on task execution sufficiently high?

I If not, perform subsequent threshold-based analysis for statesassociated with overhead of the run-time system

Automatic correlation of performance indicators

I Indicate which events are relevant

I Break down counter evolution to task instances

I Correlate with task duration

Summary

Aftermath

Summary

Aftermath

Automatic Detection of Performance Anomalies in Task ...€¦ · I Post-mortem (o ine) analysis...

Documents

The Great War World War I & Its Aftermath 1914-1930

Chronic bronchitis coal miners: ante-mortem/post-mortem

CHAPTER 16 WORLD WAR I & ITS AFTERMATH (1914 - 1918)

World War I & the Aftermath

THE ANTE-MORTEM/ POST-MORTEM DATABASEPost-Mortem Cases (Human Remains) includes all information obtained during post-mortem examination, including detailed pathology, anthropology

The Aftermath of Civil Warreynal/The Aftermath of Civil War.pdfThe Aftermath of Civil War I. Introduction War has devastating consequences for a country, including death, displacement

World War I and Its Aftermath

Cardign Post-Mortem

Unit 8 World War I and Its Aftermath- Answers

Chapter 14 World War I and Its Aftermath

Chapter 19, “World War I and Its Aftermath”

Chapter 27 – World War I and Its Aftermath

World War I and Its Aftermath - saintjoehigh.com€¦ · 2014-01-08 · World War I and Its Aftermath SECTION 1 The United States Enters World War I SECTION 2 The Home Front SECTION

XCOM_Post Mortem

Pre Mortem Retrospectives

Post Mortem

World War I and the Aftermath

Unit 6:World War I and its Aftermath

Post-mortem Toxicology

World War I: End and Aftermath