The TIME-WAIT state in TCP and its Effect on Busy Servers Theodore Faber University of Southern California

Xindian Long

Outline TIME-WAIT state and its function Performance problems on busy web

server Solution: move the TIME-WAIT to

clients End points negotiation TCP, HTTP modification

Experiments Conclusion

Delayed Packet Problem

TIME-WAIT State Blocking connections between the

same address/port pair Holding a Time-Wait State at one

endpoint 2MSL The end doing active close holds

the TIME-WAIT State

TIME-WAIT State

TIME-WAIT state in the server The server actively closes the connection,

and maintains the TIME-WAIT state Server sending data TCP connection closure as end-of-transaction

marker Simple protocol, fast response

Otherwise:Knowing the content lengthExtra end-of-transaction marker requires masking & restoring the marker

Performance Problems on Busy Server Too many connections in TIME-

WAIT State TCB Consumes memory Slows active connections A shorter MSL weaken the protection Solutions like persistent connection

are not enough

Solution: TIME-WAIT in Clients Blocking at one end is enough Move the TIME-WAIT to the Client

End points negotiating TCP Modification HTTP Modification

Scales better than persistent connection

TIME-WAIT Negotiation

SYN (TW-Negotiate)

Client Server

SYN, Ack (TW-Negotiate set to its choice)

Ack (TW-Negotiate set to the same value)

If Acceptable

If not Acceptable RST

Non-simultaneous connection establishment

TIME-WAIT Negotiation

SYN (TW-Negotiate)

endpoint1

endpoint2

Simultaneous connection establishment SYN (TW-Negotiate)

SYN-RCVD SYN-RCVD

Value Known Value To Use

Either Contains No Option

No Option

Same IP Address That IP Address

Different IP Address Larger IP Address

TIME-WAIT Negotiation Advantage

Makes post-connection memory requirement explict

Transparent to applications No hidden performance penalty

Disadvantage Significant change to TCP state machine Information from connection establishment

affects the closure Negotiating at closure reduces endpoints’

control over their resource

TCP Level Solution Modify the clients’ TCP implementation

Sends a RST packet to the server Puts itself into a TIME-WAIT state

Client

Server

Final Ack

RSTTW

TW

closed

Modified TCP State Machine

TCP Level Solution Only works for system that accepts

<RST> in TIME-WAIT state Performance limited by the way

<RST> is processed Changes the meaning of <RST>

packet

Application Level Solution Decouple the end-of-connection with

end-of-transaction indication HTTP 1.1 Modification

Content Length as the end of transaction New Extension Request: CLIENT_CLOSE

1. The client opens a connection2. Sending series of requests3. Collecting the response4. Sending a Client-Close, closing connection5. The server closes its end without responding

CLIENT_CLOSE Extention

Client

Server

data

CLIENT_CLOSE

Detect the end of data

closeFIN

.

.

.

Client

Server

data

close

.

.

.

[Conection: close] line

Last request:Connection: close

FIN

Application Level Solution CLIENT-WAIT notify the server about

the client close Requires change only on the client side Conforms to HTTP 1.1 Requires no change to other protocols Creates no new security vulnerability ONLY reduces the load by HTTP

Experiments Environment

Implemented under SUNOS 4.1.3 Evaluation using custom benchmark

program and WebSTONE benchmark 640 Mb/s Myrinet LAN

Three Tests Demonstration of Worst-case Server Loading HTTP Experiment Time wait avoidance and Persistent Connections

Demonstration of Worst-CaseServer Loading Test Configuration

A Server Two clients doing simultaneous bulk transfer Server loaded with TIME-WAIT state by a

fourth machine Constructing a worst case scenario

Client connections are put at the end of the list of TCBs

Two clients are used to neutralize the simple caching behavior

Two distinct clients allow bursts from different clients to interleave

Demonstration of Worst-CaseServer Loading

HTTP Load Experiments

System Throughput Mb/s

Conn/s TCP Mem (Kb)

Unmodified

20.97 49.09 722.7

TCP Mods 26.40 62.02 23.1

HTTP Mods

31.73 74.70 23.4

Two workstations each running 20 clientsFile Size: 9KB-5MB

System Throughput Mb/s

Conn/s TCP Mem (Kb)

Unmodified

Fails Fails Fails

TCP Mods 1.14 223.8 16.1

HTTP Mods

1.14 222.4 16.1

8 clients on 4 workstations, small files

TIME-WAIT Avoidance andPersistent Connection

2 Client machine, 1 Server5 HTTP request bursts10 request / Connection

Throughput vs. Clients

TIME-WAIT Avoidance andPersistent Connection

Connection Rate vs. Clients

TIME-WAIT Avoidance andPersistent Connection

Memory vs. Clients

Conclusion TCP With TCP With CLIENT_CLOSE TIME-WAIT Client HTTP Negotiation <RST> ExtensionReduces TIME-WAIT Loading Yes Yes YesCompatible With Current Protocols Yes Yes YesChanges Are Effective If Only No Yes YesThe Client Is ModifiedAllows System To Prevent Yes No YesTIME-WAIT AssassinationNo Changes To Transport Protocol No No YesNo Changes To Application Protocols Yes Yes NoAdds No Packet Exchanges To Yes No NoModified ProtocolTIME-WAIT Allocation Is A Requirement Yes No Noof Connection Establishment