Make Protocol Ready for Gigabit

Scopes

• In this presentation, we will present various protocol design and implementation techniques that can allow a protocol to function correctly on Gbps or deliver Gbps performance to the user application or the system output.

• (In the previous presentation, what we presented were operating system design and implementation techniques for supporting Gbps network)

Protect Against Wrapped Sequence Number

Sequence Number Wrapping Around• TCP uses sequence numbers to help it detect packet loss and

duplicate packets.• TCP’s sequence number is in bytes, rather than in packets. The

length of the sequence number field is 32 bits.• On a gigabit network, it would take only 32 seconds to wrap around

a sequence number! The TTL field in the IP header just limits the maximum number of hops a packet can traverse. It does not limit the maximum amount of time a packet can stay in the network.

• Wrapping around a sequence number can result in wrong comparisons of the freshness of two sequence numbers. This can have a very bad effect.

Problem 1: Sequence Number Drops to Zero

• Suppose that the length of the sequence number field is n bits, then when the sequence number grows from 0, 1, … to 2^n, the sequence number will wrap and drop to zero.

• As a result, when we compare two sequence numbers a and b where b > a, the comparison result may be wrong.

• The effect of the wrong comparison is that a more recent packet carrying b will be rejected and discarded because it is considered older than the the packet (carrying a) that is already received.

Sequence Number Wheel to Avoid Problem 1

• To avoid the comparison problem, we can use a sequence number wheel scheme.• The sequence number space (N = 2 ^n ) is divided into two parts each of which is (N/2) large.• The division line is not fixed. It is floating with the sequence number (e.g., a) to be compared

with.• One part represents all the sequence numbers that are considered as larger than a.

– a < b if |a – b| < N/2 and a < b, or |a – b| > N/2 and a > b

• The other part represents all the sequence numbers that are considered as smaller than b.– Otherwise.

Sequence Number Wheel

Problem 2: Sequence Number Wraps and Grows Up

• In Gbps network, in just 32 seconds, a sequence number can wrap and grow to the same number (e.g., a -> 0 -> a -1).

• This means that an outdated packet (carrying a) that stays in the network for a long time (e.g., 32 seconds) may look like that it is exactly the next packet that the receiver expects to receive. (because the last packet received carries a – 1).

• This problem may result in a corrupted received file.• This problem cannot be solved by the sequence number w

heel scheme.

PAWS Used to Detect Problem 2

• PAWS (Protect against wrapped sequence numbers: RFC 1323) is a scheme used in tcp_input() to detect problem 2.

• PAWS is based on the premise that the 32-bit timestamp values wrap around at a much lower frequency than the 32-bit sequence number, on a high-speed network.– The TCP timestamp option is assumed to be used in the TCP

header.– Right now in FreeeBSD 4.x, one tick used in timestamp repr

esents 1 ms. Therefore, it needs about 24 days (1193 hours) to change the sign bit.

PAWS Used to Detect Problem 2

• Therefore, when tcp_input() receives a packet, it will first check whether the new packet’s timestamp is older (smaller) than the timestamp of the lastly received packet.

• Of course, if the TCP connection has been idle for more than 24 days, its timestamp may have wrapped around, which can make the comparison wrong.– In this case, the packet will not be dropped.

• Otherwise, the packet will be dropped.

TCP Checksum Offloading

Turn off Checksum Computation• Computing checksum is very expensive.

– Every byte of a packet need to be read from memory to the CPU, be added, and then written fro memory back to memory.

– CPU cycles are wasted.– The bandwidth of the CPU-memory bus or the memory syste

m (depending on which one is smaller) is wasted.– Therefore, on Gbps networks, how to avoid or reduce the che

cksum cost becomes an important topic.

• Solution 1: Do not calculate checksum at all– E.g., right now on FreeBSD 4.x, you can turn off UDP packe

ts’ checksum computation.

Checksum Offloading• Solution 2: Let the network interface card do the checksum com

putation. (IP header checksum and TCP data payload checksum)• Nowadays almost every Gigabit Ethernet NIC supports computi

ng checksum on the NIC. For example, the 3COM if_ti.c driver.• To take advantage of this hardware function, the NIC device driv

er needs to communicate with TCP code so that tcp_output() and tcp_input() know whether they should compute checksums.

• Checksum offload risks that the errors occurring between the TCP layer and the device driver cannot be detected by the receiver!

Free Checksum Computation on some RISC Processors

• Sometimes, on a computer system with a RISC processor, the checksum computation can be performed without any cost.

• Some researchers observed that most RISC processors can perform two instructions per clock cycle, of which only one operation can be loading data from memory or storing data to memory.

• Thus, there is space for two instructions in the following copy loop instructions– Load [%r0] %r2– Store %r2 [%r1]

Free Checksum Computation on some RISC Processors

• As a result, if we add two instructions that calculate checksum in between these two load and store instructions, the checksum can be calculated for free. (No other work can be done in between the load and store instructions anyway)– Load [%r0] %r2– Add %r5 %r2 %r5 ! Add to running sum in r5– Addc %r5 #0 %r5 ! Add carry into r5– Store %r2 [%r1]

• This example also shows that programmed I/O is not always worse than DMA.

TCP Header Prediction for Gigabit Networks

TCP Implementation is Complicated• In FreeBSD 4.2, tcp_input.c has 2797 lines of C code and tcp_o

utput.c has 939 lines. There are 339 lines of “if” statements in tcp_input.c and 126 lines in tcp_output.c.

• These numbers show:– TCP processing is complicated.– TCP input processing is more complicated than TCP output processing.

• Previously, we presented that locality is important for good cache performance

• And, because conditional branches can hurt the performance of a pipelined CPU a lot, their bad effect should be minimized.

TCP Header Prediction • Header prediction looks for packets that fit the profile

of the packets that the receiver expects to receive next.• If a packet meets the header prediction condition, it

will be handled in just a few instructions. Otherwise, it will be handled by a general-processing code.

• Actually, this is an old design principle – optimize for the common case!

• TCP header prediction scheme can improve TCP transfer throughput because it improves instruction locality, which can improve cache performance.

Two Common Cases• If TCP is sending data, the next expected segment for t

his connection is an ACK for outstanding data.• If TCP is receiving data, the next expected segment for

this connection is the next in-sequence data segment.• On LAN, where packet losses are rare, the header pred

iction works between 97 and 100%. On WAN, where packet losses are more possible, the percentage drops to between 83% and 99%.

• The code for processing these two common cases is placed at the beginning of tcp_input(). This results in a better cache performance.– There is no information about how well this scheme can imp

rove TCP transfer throughput though.

Measuring RTTs on Gigabit Networks

Measuring RTTs Is Important• To more efficiently retransmit lost packets, the RTT of a

connection should be correctly and precisely measured.– When a packet is lost, we do not want to wait unnecessarily lo

ng before resending it.

• To increase the accuracy of measurements, the first step is to use a high-resolution clock.– Before FreeBSD 3.0, the clock resolution for TCP RTT meas

urement is 500 ms.– Now it becomes 1 ms.

• The second step is to use more RTT measurement samples to calculate the average RTT.

Timer Management• Timers are extensively used to measure RTTs.

– When a packet is sent, a timer is started. When the corresponding ACK returns, the timer is stopped. The elapsed time of the timer represents one RTT sample.

• The above approach can be used on a low speed network such as 10 Mbps. On a gigabit network, where a 1500-byte packet is sent every 12 microseconds, this approach is infeasible.

• To reduce the high frequency of timer setup/cancel operations, the original solution is to get a RTT sample every RTT, rather than getting a RTT sample for every sent packet.– Simple to implement. Just do not set up another timer until the previou

s timer is cancelled.– However, the accuracy suffers.

TCP Timestamp Option• To get a RTT sample for every sent packet while

avoiding the need to setup/cancel timers at a high frequency, TCP uses a timestamp option (RFC 1323).

• The sender places a timestamp in every sent segment. The receiver sends the timestamp back in the ACK. This allows the sender to calculate the difference and use it as the RTT sample.– This option must be supported by both the TCP sender and

receiver.– However, the original design does not need support from the

receiver.

TCP Window Scale Option for Gigabit Networks

TCP Maximum Throughput• A TCP connection’s maximum achievable throughput

is limited by the minimum of the TCP sender’s socket send buffer and the TCP receiver’s socket receive buffer.– Min(socket send buffer on the sender, socket receive buffer

on the receiver) / RTT.

• Although we can use setsockopt() to enlarge the socket send and receive buffer to a big value, the advertised window field in the TCP header is only 16 bits, which means a maximum window size of 64 KB only.– On gigabit networks, clearly this is not enough.

TCP Window Scale Option• In this scheme, the definition of the TCP window is enlarged

from 16 to 32 bits.• The window field in the header still uses 16 bits, but a option

is defined that applies a scaling operation to the 16-bit values.– During the 3-way handshaking phase, this option is carried in the

SYN and SYN+ACK packets to indicate whether the option is supported.

• In TCP implementation, the real window size is internally maintained as a 32-bit value.

• The shift field in the option is 1-byte long.– 0 means no scaling is performed.– 14 is the maximum, allowing 64KB * 2^14.

TCP Window Scale Option• This option can only appear in a SYN packet. Therefore, the scale factor

is fixed in each direction when the connection is established.• The shift count is automatically calculated by TCP, based on the size of

the socket receive buffer.– Keep dividing the buffer size by 2 until the resulting number is less

than 64 KB. • Each host thus maintains two shift counts – S for sending and R for

receiving.– Every 16-bit advertised window that is received from the other end is

left-shifted by R bits to obtain the real advertised window.– When we need to send a window advertisement to the other end, the

real window size is right-shifted by S bits, and the resulting 16-bit value is placed in the TCP header.

Congestion Control on Gigabit Networks

Congestion Control • Congestion control is more difficult on Gigabit networks.

– Although the absolute control delay (a connection’s RTT) may remain about the same regardless of the link bandwidth (either 10, 100, or 1000 Mbps), the cost of the control delay becomes higher on higher-bandwidth network.

– Why? During the same RTT, a larger amount of data has been injected into the network, before the control packet arrives at the traffic source to reduce its sending rate.

– For example, in Gigabit Ethernet 802.3x PAUSE flow control scheme, one RTT is required for the pause packet to take effect. Therefore, as the bandwidth increases, more data will be sent before the congestion control starts to take effect.

– The result is that congestion control becomes less and less effective on high-bandwidth networks. (No solution!)