A Look at TCP Behavior & Bufferbloat - Wild...

Produced by and Copyright, 2016Philip Hippensteel

A Look at TCP Behavior & Bufferbloat:

Understanding, Detecting and Mitigating Performance Issues

NY and New EnglandTelephone Associations: 2/1/17

Dr. Phil Hippensteelpjh15@psu.edu(717) 448-0552

2Produced by and Copyright, 2016Philip Hippensteel

Why Look at this Topic?

Bufferbloat creates inefficiency in worker performance if they use the network or the Internet. It is not a security issue (although exploits

may be possible). IT generally is not unaware of the issue. Also, your ISP will likely be clueless.

How Did This problem Develop?

Due to poor understanding of network operations, particularly the TCP protocol, we: Developed the notion that dropping packets was

always bad. Didn’t really study and understand TCP, even though it

carried 90% of the traffic across our networks. Increased link speed dramatically on both the LAN and

the WAN. Increased speed differences at network connections.

What is Bufferbloat?

The effect that oversized buffers have on network latency. It causes Slow responses to web requests. Decreases in throughput of file transfers when the

return path is congested. Decreases in quality of adaptive bit rate video

(Netflix, Hulu, etc.) Can be easily confused with the affects of

malware.

The Background

Understanding bufferbloat means understanding TCP operation The role of DNS How buffers work and are managed

Understanding it’s impact means understanding DNS HTTP operation

Normal TCP Operation(Send Policy)

Station A Station Bsyn

syn/ackack

datadata

data ack

datadata

Cwnd=2

Cwnd=4

Cwnd= 8

VIP: The TCP sequence numbers increase by one per byte transferred. i.e. The seq # measures the number of TCP bytes

delivered.

data ack

TCP Exchange

* View generated with Viavi Observer Gigastor.

The Receive Policy

When packets arrive in order: Ack a packet if either: 200 ms. has transpired since last ack OR Ack second packet received but unacked.

When packet arrives out-of-order Ack previously received in-order packet

Error Free Ideal TCP Performance

TCP send action has three phases. * Assuming no jitter, uniform delay, no packet loss, etc. Steady-state is a goal that can only be reached under ideal

conditions. Slow-start assumes cwnd ≤ ssthresh (later).

Segments sent

slowstart

CongestionAvoidance

Steady State*

Wireshark Shows Slow Start

Note delta time

Ideal TCP Performance w/ Packet loss

Segments sent

slowstart

Congestion Avoidance Steady

Dropped packet

slowstart

More on Send/Receive Policies

Sender Policy: cwnd: the number of segments that may

currently be sent before receiving an ack. rwnd: the number of incoming receiver buffer

bytes available number of segments that can be sent:

Min(cwnd, rwnd)

Cwnd and Latency

Cwnd depends on RTT. Therefore increased latency slows the

grow of cwnd (the ability to send).

163264

1 RTT 2 RTT 3 RTT 4 RTT 5 RTT

Error Handling (more)

When a packet arrives out-of-order, the receiver:(1) immediately ack the previous packet (sender gets dup ack)

(2)Next packet out-of-order, receiver sends same ack. (sender has second dup ack)

(3) Next packet out-of-order, receiver sends same ack (sender receives third dup ack)

(4) Sender retransmits packet requested and proceeds with packet loss policy.

(5) Typically sender cuts cwnd to half and restarts slow-start.

slowstart

CongestionAvoidanc

slowstart

Wireshark Shows Dup-acks

TCP Stack Variations

Tahoe TCP Reno TCP New Reno Compound – current MS implementation Cubic – current Linux distros

Real Example of Uncongested TCP:Netflix

This is a connection to Netflix. DSL: 20M down, 2 M up.

Real Example of Uncongested TCP:Ubuntu Download

Connection to a Ubuntu repository server.

DSL: 20M down, 2 M up.

Real Example of TCP:Netflix(SSL) & Web Browsing(HTTP)

DSL: 20M down, 2 M up.

Real Example of TCP:Ubuntu Download & Dropbox Upload

• DSL: 20M down, 2 M up.

• So, at t=55, why doesn’t TCP use the available total bandwidth of 22Mb/sec?

** I believe this slide should be memorized!!!

PART 2: HTTP: HYPERTEXT TRANSFER PROTOCOL

HTTP Has Been Around

HTTP versions 0.9,1.0,1.1 Essentially a client request-server fulfill

protocolClient ServerStart TCP Session

Start TCP sessionGet Request(s)

Get Response (200 OK)Response Transmitted

TCP Session CloseTCP Session Cloe

HTTP 1.1 Typical Operation

Client Browser Web ServerGet index file

Index file transmittedGet page components

Other components sent

* * A recent web development text quoted a study in 2013 that showed that the average web request involved 90 retrievals of text, images, scripts, icons, etc. Many of these will involve a DNS query/response.

PART 3: BUFFERS AND BUFFERBLOAT

Where Are Buffers

In the stack between layers. In the NIC drivers. In the edge routers (especially wireless

routers). In boundary devices between service

providers.

Purpose of Buffers

Smooth flow of packets. Allow QOS policies to be implemented. Make packet drops more unlikely. Allow for security policy implementations.

Flows Though Different BW Links

Above: height is BW, width is time used. So, area is data carried. Ack rate is dependent on slowest link.

receiver

High BWLow BW

Buffer Action

1) Group of segments arrive

2) Segments buffered3) Segments begin to exit

buffer at slower speed4) Another group of

segments arrive5) Buffer full6) Last to arrive are

dropped (tail-drop)discards

Flow direction

Buffers and File Transfer

Ubuntu Download Starts

A Buffer Overfills

When is segment Retransmitted?

Note: 413 dup acks received before retranmission!

BUFFERS, DNS, HTTP AND SLOW START

A Practical View of DNS Use

request number dns queries average response time (ms)

get vanguard 15 24 get ebay 26 28 get google 18 25 get irs.gov 6 35 get highmark 2 25 get highmark mission values 3 36 get washington.edu 22 80

Buffers and DNS

Test with wired Ethernet/DSL

Buffers and DNS with WiFi/DSL

DNS and Upload on Wi-fi

No Upload

SimultaneousUpload

Summarizing Bufferbloat Affects

Slows slow start Decreases file transfer throughput by

delaying acks Increases DNS response times, increasing

response times for loading web pages

Part 3: Mitigation

Queue Management:Commonly Deployed Drop Policies

RED: random early drop WRED: weighted RED CoDel: drop based on queue occupancy fq-codel: codel with fairness to flows.

upstream

downstream

discard

Good Queues, Bad Queues

Bad Queue

Good Queue

packets

CoDel Operation

The critical metric is minimum queue sojourn time. A second parameter, target, is also used. Drop Policy: if a packet sojourns have a

minimum that exceed target for more than one interval time, then randomly drop a packet. In practice, interval = 100 ms and target =

5-10% of RTT.

CoDel Illustrated

Per packet sojourn time

target

interval

fq-codel Operation

1: Incoming Flow 2: Random assignment to up to 1024 sub queues. 3: CoDel applied per queue. 4. One MTU per queue scheduled out, round robin.

●●●

Summary:Big Buffer Problems

Slow ack on large file transfers. Slow DNS response resulting in decreased

web performance. Collaboration Voice/RTP video: without VLANs or QOS, it can

cause packet loss (due to jitter buffer overflow) Video conferencing: cause SVC based VC to

degrade Adaptive video: cause decrease in resolution

or playback pause.

TESTING FOR BUFFERBLOAT

ICSI Netalyzr

www.dslreports/speedtest/

Manual Test

Start a continuous ping (ping www.google.com –t) Run test such as www.speedtest.net Observe change in ping times

Another Manual Test(continuous ping with upload)

Upload starts

Upload stops

Some Final Thoughts

Does QOS mitigate bufferbloat? No. It can determine which flows are affected

and by how much they are affected. If fq-codel is a mitigation, where should it

be implemented? At the edge router or in front of it.

Can traffic shaping mitigate bufferbloat? Yes, but at the expense of some throughput.

AQM with Wireless

Source: Controlling Queue Delay, ACM Queue, Kathy Nichols and Van Jacobson, 2012.

Questions?

A Look at TCP Behavior & Bufferbloat - Wild...

Documents

NYSTA 2012 Annual Conference Telcom Insurance Group Presented by: Joyce Hermann, AU, CISR Sr. Account Executive Insure IT, Manage IT But Never Ignore IT…

I. Executive Summary - CMU · VOICES Executive Summary: Gilbride-Brown, J., Hippensteel, H. & Sutkus, J. (2019 Oct 1) 3 • Culturally Relevant: Participant responses among culturally

Scott˜Hippensteel Rocks and Rifles · 2019. 7. 10. · battles, where the rocks played a lesser role, are mentioned only in passing (i.e., Chancellorsville, Wilderness, Shiloh)

Bufferbloat is alll Wet!

Reduzindo os Efeitos do Bufferbloat sobre Multi-Caminhos em … · 2016-04-25 · Reduzindo os Efeitos do Bufferbloat sobre Multi-Caminhos em Redes Sem Fio Heterogeneasˆ Benevid

Congestion Management at the Network Edgemdh.diva-portal.org/smash/get/diva2:754020/FULLTEXT01.pdf · Bufferbloat. Bufferbloat is manifested at the bottleneck of the network path

THE FAMILY WAY - · PDF fileBy accepting delivery of this document, the recipient agrees that the recipient will ... BLAYZE HIPPENSTEEL “Young Crawford

Bufferbloat and Beyond: Removing Performance Barriers in ... · Bufferbloat and Beyond - Removing Performance Barriers in Real-World Networks ISBN 978-91-7063-973-9 (pdf) ISBN 978-91-7063-878-7

PIE: A Lightweight Scheme to Address the Bufferbloat Problem

Bufferbloat - Dark Buffers in the Internet · 2011-06-14 · Bufferbloat Dark Buffers in the Internet or: how I became paranoid, and you should be paranoid too Jim Gettys Bell Labs

Scott P. Hippensteel ABSTRACT · The most studied battleground from the American Civil War, from a geologi cal perspective, is the rolling terrain surrounding Gettysburg, Pennsylvania

Thruway Authority: Inspecting Highway Bridges and Repairing ......caused by vehicle impacts, or soil erosion around the bridge supports. NYSTA has six weeks to take appropriate action

Passive bufferbloat measurement exploiting transport layer ...wpage.unina.it/pescape/doc/utp_globecom_13.pdf · 1 Passive bufferbloat measurement exploiting transport layer information

What is Bufferbloat?

Understanding Bufferbloat in Cellular Networksweb.cs.wpi.edu/~rek/Adv_Nets/Fall2014/CellBufferbloat_F14.pdf · Understanding Bufferbloat in Cellular Networks Haiqing Jiang, Zeyu Liu,

Tackling Bufferbloat in 3G/4G Mobile Networks

The state of Bufferbloat in the Netherlands · Opleiding Informatica The state of Bufferbloat in the Netherlands Bernardus A. Jansen Supervisors: prof. dr. H.A.G. Wijshoff & dr. K.F.D

Analisis Bottleneck dan Bufferbloat pada AQM Droptail, RED

Bufferbloat - Dark Buffers in the Internet · Bufferbloat is now understood to be a serious problem in the technical community, but problems are all over the Internet, from end-to-end

Optimal Delay Control for Combating Bufferbloat in …kcleung/papers/conferences/adaptive_CoDel:... · Optimal Delay Control for Combating Bufferbloat in the Internet ... management