Proper Use of Networks
Congestion
– A system is loaded beyond its capacity
In a lightly loaded network, network throughput and
delay increase linearly as offered load increases
Beyond the breakup
point, throughput
decreases even though
offered load increases
– Delay increases
exponentially
OfferedLoad
Delay
Throughput
SNU SCONE lab. 3
Congestion Control & Resource
Allocation The only solution to the congestion problem is to
throttle the packets entered into the network
– Little’s theorem
Two ways to solve the congestion problem
– Congestion control
– Resource allocation
Congestion control
– Send packets, if congestion occurs reduce the sending rate
– Mostly performed at hosts
Resource allocation
– Request resources (e.g. bandwidth) before sending packets
– Limits the sending rate to the agreed amount
Congestion Avoidance
SNU SCONE lab.
Evaluation of CC Algorithms Efficiency & Fairness
– Easy to achieve one while scarifying the other
Efficiency
– How efficiently(properly) uses network resources?
– In most cases, as throughput increases, delay increases also
– Power = Throughput / Delay
Consider M/M/1/∞ system
Throughput = λDelay = 1/(μ- λ)
Power = λ∙(μ- λ)
Too pessimistic
Fairness
No unanimously agreed definition
– Are resources fairly allocated to users?
One popular fairness index: Jain’s fairness index
SNU INC lab. 5
f x x x x n xn i i( , ,..., ) ( )1 22 2
f1
f2
How to allocate bandwidth to f1 and f2?
Xi: throughput of flow(user) i
6
Congestion Control TCP provides a reliable transport service
– Layer 4 error control mechanism based on Go-Back-N ARQ
– ACK, window size, TimeOut, …
Differences of Layer 2/4
Characteristics of layer 4
– Some parts of networks are
congested while other parts are not
– Indirect sensing
Layer 2 Layer 4
Network structure Single link Internet consists of many
physical networks
Sender One (P2P) Many
Resource sharing None, MAC Congestion control
Taxonomy
Closed-loop Control mechanism
– Monitor the network condition
Feedback
– Adapt the sending rate to the
observed network condition
Congestion detection (Monitor)
– Explicit vs Implicit
Inflow control
– Window vs. Rate
SNU SCONE lab. 7
Response
(ACK)
SR
SNU SCONE lab. 8
Congestion Detection
How to obtain network information?
Explicit
– Network (routers) informs the network status explicitly
– Good performance
– Overhead of information acquisition
Implicit
– Guess the network condition by looking at various symptoms
– Less accurate, but simpler & smaller overhead
9
Input Control - 1
How to regulate the packet inflow?
Window based– Like the ARQ window
• Limit the number of packets that can be sent without ACKs
– Vary window size according to the network conditions
Plusses for window– No need for fine-grained timer
– Self-limiting
Window size & Rate ?
S D
System
N=λ∙T
Input Control - 2
Rate-based– Adjust the packet sending rate
• Needs a timer
Plusses for rate– Better control (finer grain)
– De-coupling of congestion control and error control
Safeness of control mechanism– Robust against error, malfunction, …
SNU SCONE lab. 10
SNU SCONE lab. 12
TCP CC Background
The Internet was fully operational since 1980
At the beginning, no congestion control, just flow
control
A sender transmit as much bytes as possible up to AW
(Advertised Window) Congestion
Van Jacobson introduced the congestion control
mechanism in late 1980
cwnd: Congestion window
AW: Advertised Window (Or rwnd)
S D
rcvbuf: Socket interface receiver buffer
AW = rcvbuf – occupied bytes
TCP CC & Error Control
TCP provides end-to-end reliable delivery service
– Go-back-N ARQ & optional Selective repeat
• Learn SACK (Selective ACK) option
For each segment, a receiver returns ACK to a sender• Learn delayed ACK option
Sender measures RTT Use it for setting TO value
SNU SCONE lab. 13
S R
DATA segment
ACK segment
Window size N ≡ Number of in-flight packets & ACKs
TCP Network Model
Empire state building elevators
– Two tandem elevators, one from ground
to the 80th floor and another from the 80th
to the observation deck
Only the first has a long line
If a path is congested, only one link
has a (long) queue
Single bottleneck link
14
S R
TCP Network Model - 2
Constant RTT
– RTTs are about the same whether the path is congested or not
– Routers have a (relatively) very small buffer (e.g. buffer size is
< 1000 packets)
– Link speed is ~1 Tbps
Queueing delay can be ignored for long distance paths
SNU SCONE lab. 15
Constant RTT assumption is not valid anymore
Buffer bloat problem
N=λ∙T
16
TCP Congestion Control - Feedback Implicit feedback
What are the symptoms of network congestion?– Increased RTT
– Packet loss (drop)
– Out-of-order delivery (≅ dropped segment)
A congested node (router) receives packets more than its capacity– Buffer will increase and packets will be dropped eventually
A sender waits for an ACK after sending a segment– If an ACK arrives before TO, judge the network is not congested
– In the case of TO, guess the segment was dropped due to network congestion
Is the feedback reliable?
SNU SCONE lab.
TCP Congestion Control - AIMD
Additive Increase (AI)– All ACKs arrive before TO Not congested
– Increase window size (cwnd, N) by one MSS (Max. Segment Size) at each RTT
– Increment rule
• inc = MSS * (MSS / cwnd)
• cwnd += inc
Multiplicative Decrease (MD)– TO Congested
– Decrease cwnd in the case of TO
– cwnd = cwnd / 2
– TCP may drop cwnd to 1 MSS
Note: cwnd unit is byte
SNU SCONE lab. 18
Behavior of AIMD
Oscillating on the fairness line
Fairness Line
Efficiency Line
f1
f2S1
S2
R1
R2
f1
f2
RTT1 = RTT2
Redo for RTT1 = 2*RTT2
19
Slow Start
AI is too conservative
– Takes one RTT to increase cwnd one MSS
Slow start
– Increase cwnd multiplicatively
Two situations to apply the slow
start
– Initialization
• No network information
– Recovery from TO
• Use previous network information
• Multiplicative increase up to threshold =
(half of the cwnd prior to the TO)
Let RTT =0.1 sec. & BW = 10 Gbps, How long will
it take until fully use the BW? (MSS=10,000 bits)
SNU INC lab. 20
Example Trace
Problem: Coarse timer granularity
=> Waste bandwidth waiting for timeouts
Multiple packet drops
Suppose the available BW is 20 KB
If cwnd is 16, then all packets will
be delivered
Slow start increases cwnd to 32
12 KB would be lost
No transmission waiting for Ack
TO when cwnd = 22
Slow start up to 11 and then
AI
Single packet loss during AI
SNU SCONE lab. 21
Fast Retransmission
Solve the problem of long TO
Receiving duplicate ACKs
– Out-of-order delivery
Packet loss
Differences? btw
SNU SCONE lab. 22
Fast Retransmission - 2
Fast retransmit
– If duplicate ACKs arrive, retransmit unacked segments
immediately not waiting for TO
How many duplicate ACKs trigger FR?
Window size?
SNU SCONE lab. 24
Fast Recovery & TCP Reno
Mechanism
– Remove the slow start phase in the case of Fast Retransmit
– Go directly to half the last CW
– Increase cwnd additively from the threshold
TCP Reno
– In addition to TCP Tahoe, add fast recovery and (header
prediction + delayed ACK) mechanisms
What is Header Prediction?
Learn CUBIC, BIC by Injong Rhee
SNU SCONE lab. 26
Congestion Avoidance
Prevent congestion
Detect the symptoms when congestion may occur soon
and adjust the sending rate
Many methods rely on (explicit) feedbacks from
routers
Resource
Allocation
Congestion
Control
Congestion
Avoidance
SNU SCONE lab. 27
DECbit Flow Control
Mechanism
– Every packet has a bit in header
– Intermediate routers set “congestion bit” if average queue
length >= ?
– The destination node copies the congestion bit to ACK
– The sender monitors ACKs and adjust cwnd according to
AIMD
DECbit
How to judge Congestion?
Condition to trigger congestion avoidance?
SNU SCONE lab. 28
Average Queue Length Computed over queue regeneration cycles
Balance between sensitivity and stability
If AvgLen > 1, mark the bit with 50% probability
SNU SCONE lab. 29
Source Actions
Observe bits over past + present window size
– Should not take control actions too fast!
– Wait for past change to take effect
If more than 50% set, then decrease window, else
increase
Additive increase, multiplicative decrease
– cwnd = cwnd + 1
– cwnd = 0.875 * cwnd
SNU SCONE lab. 30
RED (Random Early Detection) TCP
– Packet drops congestion feedbacks
Early random drop
– Drop packets before the buffer is full
– Early feedback to avoid congestion
• Implicit (Indirect) feedback
No need to change the TCP protocol (hosts)
Drop packets in advance
31
Packet Drop - 1
Adjust drop probability according to the severity
of congestion (Queue length)
Queueing mechanism
Case A
enqueue the packet
Case B
drop packet with probability P ∝ (AvgLen,..)
Case C
drop packet
C B A
Average queue length: EWMAAvgLen=(1-W)*AvgLen + W*SampleLen
SNU SCONE lab. 32
TCP Vegas
Congestion
Queued packets increases
– How to estimate the number of packets queued inside
the network?
Idea: symptoms that congestion will happen soon
– RTT is growing (Note: Constant RTT assumption)
– Sending rate flattens
SNU SCONE lab. 34
TCP Vegas
Number of packets (bytes) in the network (= cwnd)
= In-transit packets (Ns) + Queued packets (NQ)
Time in the system (= RTT)
= In-transit time + Queueing time
In-transit time = the RTT when there is no queueing– BaseRTT
– Approximate it w/ the Minimum RTT that’s been observed
S R
Estimate the # of packets
in the queue
We know them
What other values can we know(estimate)?
TCP Vegas - 2
Current input rate (sending rate)= cwnd / RTT ( = λ)
Number of in-transit packets (Ns) Ns = λ * BaseRTT
NQ = N-Ns= cwnd - λ * BaseRTT
= cwnd – (cwnd/RTT) * BaseRTT
= cwnd * BaseRTT * (1/BaseRTT – 1/RTT)
= BaseRTT * (cwnd/BaseRTT – cwnd/RTT)
= BaseRTT * (Expected Rate – ActualRate)
SNU INC lab. 35
S Rλ λNs
Ts
NQ
TQ
N = λ∙T
cwnd = λ∙RTT
N = λ∙T
Ns = λ∙BaseRTT
SNU SCONE lab. 36
TCP Vegas – Adjustment
Diff = ExpectedRate - ActualRate
if Diff <
-->increase CW linearly
else if Diff >
-->decrease CW linearly
else
-->leave CW unchanged
SNU SCONE lab. 37
Expected Rate
Actual Rate
Expected Rate-αExpected Rate-β
ExpectedRate - ActualRate < α
> β
SNU SCONE lab. 38
Packet Pair
Assumption– Routers use round-robin scheduling
How to measure (my) capacity at a bottleneck link?
Send two packets back to back
Then, spacing between packets at receiver (= ackspacing) = 1/(rate of slowest server)
1n 21 …
SNU SCONE lab. 40
Algorithm
N(cwnd) = Nq + Ns
Nq / Ns = # packets in bottleneck buffer/in service(transit)
b = bottleneck rateNs = RTT*b
Nq = cwnd - RTT*b (assuming no losses)
Let Tq is the target queue length
cwnd = r*RTT = Nq + RTT*b
To have the target queue length, adjust input rate to r’– r’*RTT = Tq + RTT*b
r(k+1) = r(k) + (Tq - Nq)/RTT
S R
1/b
1/b
Bottleneck link rate
1/r
Sending rate
r: sending rate
SNU SCONE lab. 42
QoS (Quality of Service)
Real-time service
– Service that requires strict delay and loss performance
– Ex: A voice call requires the delay of less than 200-300 ms
Performance aspects
– Delay
– Loss
– Jitter
– Bandwidth
SNU SCONE lab. 43
Application Taxonomy
Applications
Elastic Real Time
Intolerant Tolerant
Nonadaptive Adaptive
Rate Adaptive Delay Adaptive
Tolerate occasional losses?
Adapt to network condition?
Adjust playback delayAdjust sending rate
Learn DASH (Dynamic Adaptive Streaming over HTTP)
SNU SCONE lab. 44
Queueing
When a packet arrives at a router
– The router determines an outgoing link & enqueues the packet to
the interface’s buffer
– Waiting for its service according to the queueing principle
Queueing
Queueing is a complex mechanism
– Structure of buffers(queues)
– Methods of inserting arriving packets to queues
– Discard of packets
– Scheduling
• Service order: Who will be served first?
Queue in data
structure
Queue in
network
SNU SCONE lab. 45
FIFO & Drop Tail
Simplest queueing mechanism– A single queue that is shared by
all flows (sources)
– Serve the oldest packet first
Drop tail– When the buffer is full, discard
new packets
FIFO problems– No isolation (separation)
• A user that generates many packets get more services
– No priority
• All packets are equally treated
46
Multiple Queues Isolation
Multiple queues each dedicated to– Priority
– Flow
A packet is inserted to the queue of its priority/flow
Insertion & dropping to/from each queue may be same as a single queue
Scheduling– Priority scheduling
• Serve all high priority packets before serving low priority packets
– Fair scheduling
• Serve packets fairly (equally)
What is a flow?
?
47
RR (Round Robin) Fair service
– Allocate resources equally to all customers
Round Robin (RR)– Serve each queue once per round
What is the service (scheduling) unit?
Packet by packet RR– Send one packet from each flow per round
– Unfair if packet sizes are different
Bit by bit RR– Send one bit from each flow per round
– Fair but Impossible
How to emulate bit by bit RR?
SNU SCONE lab. 48
FQ (Fair Queueing)
Emulation of bit by bit RR
Virtual clock
– An imaginary clock that moves one tick when one bit from
all active queues are transmitted
Compute the finish time of each packets and
transmit the packet with the earliest finish time
– No preemption
• Finish a transmission once it started
Finish time: Fi
– Fi = Si + Pi
Start time: Si
– Si = max (Fi-1, Ai)
49
FQ – 2
WFQ(Weighted Fair Queueing)- Allocate different weights to flows
Implementation- Computation of virtual clock
- Selection of the earliest finish time packet
: Heap sorting, O(log N)
There are thousands of simultaneous flows!!
SNU SCONE lab. 50
QoS Support Architectures Mechanisms to support QoS
– Separation (Classification)
– Different treatment (Scheduling)
– Resource reservation
Blocking (admission control) & Regulation (Policing)
Two QoS support approaches
IntServ (Integrated Service) architecture
– Fine-grained, Reserve resources for each flow
– Strict QoS support
DiffServ (Differentiated Service) architecture
– Coarse-grained, Group flows into a priority class
– Scalability
SNU SCONE lab. 51
IntServ Classify traffic into three categories
– GS (Guaranteed Service)
• Strict support of QoS requirements
• e.g. VoIP
– CS (Controlled-load Service)
• Performance of lightly loaded network
• e.g. Streaming
– BS (Best-effort Service)
Mechanisms
– Admission control & Reservation
– Classification
– Scheduling
– Policing (Regulation)
SNU SCONE lab. 52
Call Admission Control (CAC) Procedure
– Before sending packets, request if the network can support
the flow
– Admit the flow if there are unused resources to support
requested QoS
• At each link on the end-to-end path
Flowspec
– A flow should specify how much traffic it will generate &
the level of QoS
– Performance requirements (Rspec)
• For simplicity, use deterministic requirements
• No packet loss, maximum allowable delay
– Traffic characteristics (Tspec)
• Characteristics of traffic that the flow generate
SNU SCONE lab. 53
Traffic Specification CBR(Constant Bit Rate) and VBR(Variable)
– Traffic generated by realtime applications are highly
variable
Leaky bucket representation
– Burst size = bucket size
– Average flow rate = token rate
σ bytes
Tokens enter
at rate ρ
Stop if there are
not enough tokens
Size s packet
consumes s tokens
Tokens stored more
than σ are spilled over
A(t) ≤ σ + ρ∙t
SNU SCONE lab. 54
Parekh-Gallager Theorem
How to satisfy the strictest QoS requirement?
Worst-case end-to-end delay (no packet loss)– Assume that bw are allocated to a flow at each WFQ scheduler along
its path, so that the least bw it is allocated is g
– Let it be leaky-bucket regulated such that # bits sent in time [t1, t2] <=
ρ(t2 - t1) +
– Let the connection pass through K routers(schedulers), where the k-th
scheduler has a rate r(k)
– Let the largest packet size in the network be P
1
1 1
)(///___K
k
K
k
krPgPgdelayendtoend
SNU SCONE lab. 55
RSVP (Resource reSerVation Protocol)
A signaling protocol that reserves bandwidth (buffer)
for QoS guarantee on the best-effort Internet
Examine if the links on the end-to-end path have
enough resources
– Parekh-Gallager Theorem
Features
– Soft state
– Multicast
– Receiver-oriented
– Filter
SNU SCONE lab. 56
RSVP Procedure Sender initiates a session by sending PATH message to receiver
– Includes Tspec
Receiver determines the supportable traffic parameter and desirable QoS level
Receiver determines the
resource requirements (Rspec)
and returns RESV message
along the pinned route
Intermediate nodes determine
if they can support the
flowspec (Tspec + Rspec)
– Maintain states for packet
classification and resource
reservation
SNU SCONE lab. 57
DiffServ
Motivations– Scalable QoS support network
• IntServ is too complicated
– Emphasize good network planning
• Well-engineered networks usually provide good performance
• Minimize traffic control
Architecture– Distribute functions to core devices and edge devices
– Edge devices
• Flow by flow processing
• Classification
• Packet marking/shaping
– Core devices
• Simple priority queueing based on packet classes
SNU SCONE lab. 58
DiffServ
Service classes
– EF (Expedited Forwarding)
• Transmit the packet before any other packets
– AF (Assured Forwarding)
• Usually receive good performance
– BF
Mechanisms
– Admission control
• For EF and AF services
– Regulator
• EF
– Marker
• AF
SNU SCONE lab. 59
DiffServ – Edge Router
Edge router
– Identify flows – Packet classification
– Regulation
• Discard violating packets
– Marking
• Mark violating packets as OUT (Out of profile) packets
Classifier
Flow 1
Flow N
Shaper
Marker
SNU SCONE lab. 60
DiffServ – Core Router
Process packets based on classes
PHB(Per Hop Behavior)
– EF
• Priority Queueing
– AF
• RIO (RED with In and Out)
EF
AF, BF