Performance Debugging for Distributed Systems of Black Boxes Marcos K. Aguilera Jeffrey C. Mogul Janet L. Wiener HP Labs Patrick Reynolds, Duke Athicha

Performance Debugging for

Distributed Systems of Black Boxes

Marcos K. AguileraJeffrey C. MogulJanet L. Wiener

HP Labs Patrick Reynolds, Duke

Athicha Muthitacharoen, MIT

WISP 200411 November 2004

page 220 October 2003 Project5 - SOSP

Example multi-tier system

client

web server

client

web serverweb server

database server

database server

application server

application server

authentication server


Motivation

• Complex distributed systems are built from black box components

• These systems may have performance problems

• High or erratic latency

• Locating the causes of these problems is hard

• We can’t always examine or modify system components

• We need tools to infer where bottlenecks are

• Choose which black boxes to open


Contributions of our work

• Tools to highlight which black boxes have problems

• Require only passive information, such as packet traces

• Infer where most of time is spent from traces

• Person can then use more invasive tools to examine those boxes

• Reduce time and cost to debug complex systems

• Improve quality of delivered systems


Example causal path

client

web server

client

web serverweb server

database server

database server

application server

application server

authentication server

100ms


Goals of our tools

• Find high-impact causal paths through a distributed systemCausal path: series of nodes that sent/received

messages– Each message is caused by receipt of previous

message– Some causal paths occur many times

High impact:– Occurs frequently– Contributes significantly to overall latency

• Without modifications or semantic knowledge• Report per-node latencies on causal paths


Overview of our approach

• Obtain traces of messages between components– Ethernet packets, middleware messages, etc.– Collect traces as non-invasively as possible

• Analyze traces using algorithms

• Visualize results and highlight high-impact paths

• Requires very little information: [timestamp, source, destination]


Outline

• Problem statement & goals• Overview of our approach• Algorithm• Experimental results• Related work• Conclusions


The convolution algorithm: input

Time From To

0.01 A B

0.02 A B

0.04 B D

0.05 C F

...


The convolution algorithm: output

A

C DB

E FE FFE

G G G G G G

.15.10 0 0

.10.10

0 0 0 0 00


Basic idea

• Creates a “time signal” for messages from each node

• Given time signals S1(t)=(AB) and S2(t)=(BX)

Computes convolution of S2(t) and S1(–t) = S1 * S2

(can be computed quickly using fast fourier transforms)

S1(t)=(AB msgs)

1 2 3 4 5 6 7 time


S1(t)=(AB msgs)

S2(t)=(BX msgs)

S1 * S2=conv(S2(t), S1(-t))

• Spikes suggest causality between ASpikes suggest causality between AB and BB and BX msgsX msgs• Time shift of a spike indicates its characteristic delayTime shift of a spike indicates its characteristic delay


Details: first step

• Choose starting node A• Use trace to add edges from it

Time From To 0.01 A B 0.02 A B 0.04 A C 0.05 A B

A

B C


Continuing

Time From To … B D … B E … B F … B G

A

B C

??

(AB)*(BE) (AB)*(BD)

d


How

Time From To t1 A B t2 A B t3 A B t4 A B

Time From To

… t1+d B D … t2+d B D … t3+d B D t3+d B E … t4+d B D


Heuristic to find spikes

threshold 1: n1 stddev over meanthreshold 2: n2 stddev over mean

n1 = 2n2 = 1.5


Recursing to continue

• Observations: 1. (BD) are not all msgs from B to D (only those caused by A)

2. Stop recursion when too few messages left or no more spikes found

A

B

D

d

??


Outline

• Problem statement & goals• Overview of our approach• Algorithm• Experimental results• Conclusions


Results: email service delays

• Jeff logged all email headers for two months • Parsed 80K Received headers in 12K messagesReceived: from cceexg11.americas.cpqcorp.net ...by wera.hpl.hp.com ... ; Fri, 4 Apr 2003 15:35:54 -0800

– Yields (timestamp, sender, receiver) trace records• Used Convolution Algorithm to

– Reconstruct message paths– Find typical delays

• Note: this is NOT the most direct way to use email headers– We made the problem harder so as to test our algorithm


Email trace: output

60

39 6737

40 3840 383840

41 41 41 41 41 41

4890,15

7380,10

4600

7660 523

0,10

7680,10

4780

5940

4390

6260

5120

6350


Results: Petstore

• Sun’s demo application for J2EE

• Stanford’s PinPoint project provided us with traces– One trace has a node that is

artificially slowed down


Future work

•Automate trace gathering and conversion•Sliding-window versions of algorithms

– Find phased behavior– Reduce memory usage of nesting algorithm– Improve speed of convolution algorithm

•Validate usefulness on more complicated systems

•What are limits of our approach?


Conclusion

• Looking for bottlenecks in black box systems

• Use signal processing techniques to find causal pathsin the network and its delays

• For more information– http://www.hpl.hp.com/research/project5/

• Contact us if you have multi-hop message traces!

Documents

Performance Debugging for Distributed Systems of Black Boxes Marcos K. Aguilera Jeffrey C. Mogul Janet L. Wiener HP Labs Patrick Reynolds, Duke Athicha