Upload
devante-turrill
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Tuning and Evaluating TCP End-to-End Performance in LFN Networks
P. Cimbál* [email protected] was supported by Sven Ubik** [email protected]
Motivation • today‘s network traffic consists about 90% of TCP• majority of data transfers is unidirectional and elastic• network paths allow high throughputs and wide bandwidth• end hosts can produce and consume high dataflows
How to utilise all available bandwidth?• extend TCP window must to >RTT*BWb
• implement SACK for more flexible retransmit• protect SEQ number wrapping
All today‘s commodity systems can be tuned to use all mentioned extensions. But, this tuning isn‘t simple, and TCP can still show very low performance. So, methods for low-level debugging,measuring and analysis are needed.
Long Fat Network
RN
R1
RN-1R2 ...
- minimal bandwidth along path BWb
- round-trip delay RTT
Measured ParametersThe problem of bandwidth utilisation in LFN networks is always caused by a TCP stack, unable to keep sufficient sending window. The resulting window is a product of sending application’s writes, sender‘s cogestion mechanism, network path bottleneck BWb, round-trip delay RTT, additional queueing delays, advanced queueing-policies on routers, receiver‘s ACK policy, and receiving application‘s reads.
Path Properties• bottleneck bandwidth BWb• size of wire-pipe WP• round-trip delay RTT• available space in router queues on path QS• additional queueing dealy (part of RTT)• packet-loss ratio p and queue management
TCP Stack Properties• RCV and SND windows, RFC1323 scaling• AIMD implementation, ACK delaying• TX queues on interfaces (Linux specific)• internal limits and known oddities
Methods of Measurement
offline on host
system monitor
inline
offline on path
patched application,debug-outs, debug-calls
kernel accessing, WEB100, socket states
packet dumper on local interface, overhead
packet sniffing, anotherhost, promiscuous mode
Configuration for MeasuringBecause of many oddities in buffer-settings, observed in commodity system, packet analysis is always necessary. To have knowledge about injected bandwidth, RTT, and succes of segment-sending, packet dump must be provided at least on sender side. To avoid wide gaps in captured stream (sending application has machine-time priority over sniffer), mirroring on sender side was chosen. The state of the sender‘s cogestion mechanism was monitored using WEB100 interface, to be used with offline packet analysis later.
SW
Long Fat Network2x P III / 850
netserver, iperfftp, scp server...
tcpdump logs, latertcptrace, own scripts...
netperf, iperfftp, scp client...
lab2.cesnet.cz
lab1.cesnet.cz
tcp4-ge.uninett.no
OC-192, OC-48,Gigabit Ethernet NIC12 hops path, ~ 50 ms, >600 mbps, ~ 6MB pipelocal mirroring
Typical TCP BehaviorAs shown in the graph (typical example, throughput decrease for windows bigger than pipe was common for all measurements), available bandwidth 600 mbps wasn‘t utilised, regardless to sufficient window size.
Závislost dosa itelné jednosmìrné propustnosti TCP spojení na velikosti okénka�
0,00
50,00
100,00
150,00
200,00
250,00
300,00
1000 10000 100000 1000000 10000000
Velikost okénka [B]
Pro
pu
stn
ost
[Mb
ps]
Throughput Decrease Details
The effect strength depends on transmit queue of local output NIC. But, the decrease can be observed for correct size of the queue too, and also for highly overestimated queues even. It‘s caused by the AIMD mechanism on sender side, which tends to fix very low sstresh. For smaller receive windows, AIMD is clamped by standart flow-control. Thus, all queued cogestions are limited to the difference between advertised window and wire-pipe size.
Packet Analysis DetailsMentioned effect was studied later, using packet dumps. Connection was reconstructed by specialised scripts and tools, proving the former explanation of throughput decrease.
Cogestion Avoidance
This should be the expected behavior of all stable TCP connections. Rate ~ 0.75 * BWb .
Aggressive Cogestion
This is the cause of decrease. Initial aggressive burst starts huge cogestion, fixing low sstresh by packet drops.
Fixed Small Window Details
Detailed case of previous effect, showing the unused advertised window, due to slow CA and low sstresh. It‘s an oscillation of the AIMD cogestion mechanism.
Theory in Background
Stable state of the TCP connection can be described as a serie of CA sawtooth cycles.
For a packet-loss rate p, maximum throughput will be clamped by AIMD to
BW <= MSS * sqrt(1.5 / p) / RTT
For cogestion-limited throughput, utilised bandwidth is due to CA cycles
BW <= 0.75 * BWb
But, for flow-control limited connection, we can achieve
BW <= BWb
Notice that the formulas don‘t consider multiple AIMD decrease flaws.
Current Research and Proposed SolutionsThe TCP‘s AIMD behavior in LFN environment is one of the most important aims in today‘s network research. There are many solutions, how to increase CA‘s cycle efficiency, such as HSTCP modification, or Tom Kelly‘s stack. Such solutions perform more agressive slowstart, and/or faster CA sawtooth slope. Result is the bigger average window for the same loss-rate of the physical link.
But, our problem is different, because packet-drops are caused by cogestion, not by physical errors on the link. The most important cause of low data throughput effect is a huge cogestion, leading to bursty packet-drops later. Without this cogestion, we can achieve 0.75 * BWb rates with standart TCP stack. Thus, mentioned TCP stacks with modified AIMD, designed for high-speed networks, won‘t solve this problem.
There are some tries, such as ETCP, solving AIMD‘s impact on cogestion. Unfortunately, the solution relies on the Explicit Cogestion Notification mechanism which must be implemented on all routers along the path.
Conclusion and Further ResearchTCP stacks, implemented in all today‘s commodity systems, are ready for LFN networks. There are many oddities (described in TCP Performance Tuning Guide, related to this project), which can influence the accuracy of tuning, but can be avoided to have minimal performance impact.
Unfortunately, AIMD mechanism can‘t cope with LFN networks well. In the case, when the physical loss-rate limits the transfer, there are new TCP stacks, such as HSTCP or Tom Kelly‘s stack. But, in the case of packet-loss bursts, restarting the AIMD more than once, we must avoid huge cogestion. There are only few solutions, relying on not very popular ECN mechanism.
Aims of our Further Research
The detailed background of packet-loss bursts must be explored. After that, we will try to develop cogestion-predictive AIMD clamping, solving low performance in LFN, without ECN.
*) CTU, Faculty of Electrical Engineering, Dept. of Computer Science and Engineering,Karlovo náměstí 13, 121 35 Praha 2
**) CESNET z. s. p. o. o. Zikova 4, 160 00 Praha 6