Sizing Router Buffers

Sachin Katti, CS244 Slides courtesy: Nick McKeown

2

Routers need Packet Buffers

§  It’s well known that routers need packet buffers §  It’s less clear why and how much

§  Goal of this work is to answer the question:

How much buffering do routers need?

Given that queueing delay is the only variable part of packet delay in the Internet, you’d think we’d know the answer already!

3

How much Buffer does a Router need?

§  Universally applied rule-of-thumb: §  A router needs a buffer size:

§  2T is the two-way propagation delay (or just 250ms) §  C is capacity of bottleneck link

§  Context §  Mandated in backbone and edge routers. §  Appears in RFPs and IETF architectural guidelines.. §  Usually referenced to Villamizar and Song: “High Performance

TCP in ANSNET”, CCR, 1994. §  Already known by inventors of TCP [Van Jacobson, 1988] §  Has major consequences for router design

CTB ×= 2

C Router Source Destination

2T

4

TCP

C C’ > C

Only W=2 packets may be outstanding

§  TCP Congestion Window controls the sending rate §  Sender sends packets, receiver sends ACKs §  Sending rate is controlled by Window W, §  At any time, only W unacknowledged packets may be outstanding

§  The sending rate of TCP is

Router Source

RTTWR =

Dest

5

Single TCP Flow Router with large enough buffers for full link utilization

B

Dest C C’ > C Source

maxW

2maxW

t

Window size RTT

For every W ACKs received, send W+1 packets

RTTWR =

6

Required buffer is height of sawtooth

t

B

0

7

Buffer = rule of thumb

8

Over-buffered Link

9

Under-buffered Link

10

Origin of rule-of-thumb §  Before and after reducing window size, the sending rate of the

TCP sender is the same

§  Inserting the rate equation we get

§  The RTT is part transmission delay T and part queueing delay B/C . We know that after reducing the window, the queueing delay is zero.

newold RR =

new

new

old

old

RTTW

RTTW

=

TW

CBTW oldold

22/

/2=

+ ⇔ CTB ×= 2

)1( 1noldnew WW −= CTB n ×=

−2)( 1

1

11

Rule-of-thumb

§  Rule-of-thumb makes sense for one flow §  Typical backbone link has > 20,000 flows §  Does the rule-of-thumb still hold?

§  Answer: §  If flows are perfectly synchronized, then Yes. §  If flows are desynchronized then No.

12

Outline

§  The Rule of Thumb §  The buffer requirements for a congested router

§  Synchronized flows §  Desynchronized flows §  The 2T×C/sqrt(n) rule

§  Buffer requirements for short flows (slow-start) §  Experimental Verification §  Conclusion

13

If flows are synchronized

§  Aggregate window has same dynamics §  Therefore buffer occupancy has same dynamics §  Rule-of-thumb still holds.

2maxW

t

max

2W

∑

maxW∑

maxW

14

When are Flows Synchronized?

§  Small numbers of flows tend to synchronize

§  Large aggregates of flows are not synchronized §  For > 200 flows, synchronization disappears § Measurements in the core give no indication

of synchronization

15

If flows are not synchronized

Probability Distribution

B

0

Buffer Size

∑W

16

Quantitative Model §  Model congestion window of a flow as random variable

)(tWi model as )(][ xfxWP i ==iW where

§  For many de-synchronized flows §  We assume congestion windows are independent §  All congestion windows have the same probability distribution

2]var[][ WiWi WWE σµ ==

§  Now central limit theorem gives us the distribution of the sum of the window sizes

)1,0()( NnntW WWn

i σµ +→∑

17

Buffer vs. Number of Flows for a given Bandwidth

§  If for a single flow we have

§  Standard deviation of sum of windows decreases with n

)1,0(1)( 11 Nn

tW nnn

i == +→∑ σµ

§  Thus as n increases, buffer size should decrease

nBB n 1=→

211 ]var[][ == == nn WWE σµ

§  For a given C, the window W scales with 1/n and thus 2

11 ]var[][ ⎟⎠

⎞⎜⎝

⎛== ==

nW

nWE nn σµ

18

Required buffer size

2T Cn×

Simulation

19

Summary §  Flows in the core are desynchronized

§  For desynchronized flows, routers need only buffers of

nCTB ×

=2

20

Experimental Evaluation Overview

§  Simulation with ns2 §  Over 10,000 simulations that cover range of settings

§  Simulation time 30s to 5 minutes §  Bandwidth 10 Mb/s - 1 Gb/s §  Latency 20ms -250 ms,

§  Physical router §  Cisco GSR with OC3 line card §  In collaboration with University of Wisconsin

§  Experimental results presented here §  Long Flows - Utilization §  Mixes of flows - Flow Completion Time (FCT) §  Mixes of flows - Heavy Tailed Flow Distribution §  Short Flows – Queue Distribution

21

Long Flows - Utilization (I) Small Buffers are sufficient - OC3 Line, ~100ms RTT

2T Cn×

99.9%

98.0%

99.5% 2T Cn×

2×

22

23

24

25

26

27

28

29

30

Impact on Router Design §  10Gb/s linecard with 200,000 x 56kb/s flows

§  Rule-of-thumb: Buffer = 2.5Gbits §  Requires external, slow DRAM

§  Becomes: Buffer = 6Mbits §  Can use on-chip, fast SRAM §  Completion time halved for short-flows

§  40Gb/s linecard with 40,000 x 1Mb/s flows §  Rule-of-thumb: Buffer = 10Gbits §  Becomes: Buffer = 50Mbits

§  For more details… §  “Sizing Router Buffers – Guido Appenzeller, Isaac Keslassy and

Nick McKeown, to appear at SIGCOMM 2004