A Hybrid Systems Modeling Framework for Communication

Network

Junsoo Lee (USC)

Prof. Stephan BohacekProf. Joao HespanhaProf. Katia Obraczka

Sep 8, 2004

2

Motivation• Problems in network research:

– Design, test and analyze protocols (throughput, fairness, security).– Validate designs (scalability, performance).– Tune network/protocol parameters (Queue size, bandwidth, timers, etc.).

• Study of networks and network protocols have used:

– Analytical• Fast• Significant accuracy loss• Only applicable to limited application

– Simulation• Accurate • Long simulation time• Large memory overhead

– Combination

3

Publications

• A Hybrid Systems Modeling Framework for Fast and Accurate Simulation of Data Communication Networks. In Proc. SIGMETRICS, June 2003

• Analysis of a TCP hybrid model. In Proc. of the 39th Annual Allerton Conf. on Comm., Contr., and Computing, Oct. 2001

• Hybrid Modeling of TCP Congestion Control. In Maria

Domenica Di Benedetto, Alberto Sangiovanni-Vincentelli, Hybrid Systems: Computation and Control, number 2034 in Lect. Notes in Comput. Science, pages 291-304, Mar. 2001

4

Outline

• Related work

• Simple case studies (e.g., UDP, TCP variations).

• Generalized Hybrid Model Framework

• Validation

• Scalability of Hybrid Systems

• Software Tools

• Conclusion

5

Related Work: Packet Model

• Track individual data packets.

• Computationally intensive.

• Complexity depends on the number of events.

• Does not scale to high bandwidths and complex topologies.

• NS-2 (NS00)

• Pdns (Riley99)

• QualNet

• Opnet (Desbrandes93)

• SSFNET

• Dummynet, Nistnet

6

Related Work: Fluid Model

• Track time/ensemble-average packet rates.

• Computationally efficient.

• Complexity depends on the rate changes.

• Only suitable to model many flows.

• Does not explicitly model individual events.

• ATM (Kesidis96)

• Time driven (Yan99)• Stochastic Differential Equation

(Misra99,20)

• Time-Stepped Hybrid Simulation (Guo00)

• Fluid-Simulation using SSF (Nicol98)

• More efficient and larger scale (Liu03)

7

Our Approach: Hybrid System• Track packet rates for each flow averaged over small time

scales.

• Capture dynamics of each flow

• Explicitly models some discrete events (drops, queues becoming empty, etc.).

• Time accuracy of a few milliseconds (round-trip time).

8

Related Work: Other Models

• Hybrid models:– Discrete event + analytical technique (Schetman78).– Packet (foreground) + fluid model (back-ground) (Melamed01).– Packet (edge) + fluid model (backbone) (Riley02, Yu04)) .– Hybrid buffer model (Cameron03).

• Simulation, Sampling & Analysis:– SHRiNK (Psounis03)

• Abstraction:– Abstract Technique (Huang99)– Packet Train (Ahn96)

9

Talk Outline

• Related work

• Simple case studies (e.g., UDP, TCP variations).

• Generalized Hybrid Model Framework

• Validation

• Scalability of Hybrid Systems

• Software Tools

• Conclusion

10

Simple Hybrid Model Example

State 1

State 2?bx

?ax

0:x

transition enabling condition

state reset

[Shaft00]

11

Cwnd of TCP

Slow Start

Fast Recovery

Congestion Avoidance

12

Cwnd of TCP

Slow StartFast

Recovery

Congestion Avoidance

13

Queue Size

Queue Empty

Queue Full

Queue Not Full

14

Queue Size

Queue Empty

Queue Not Full

Queue Full

15

Dumbbell topology

When i ri exceeds B the queue fills and data is lost (drops)

drop (discrete event)

r1 bps

r2 bps

r3 bpsq( t ) = queue size

queue (temporary storage for data)

f1

f2

f3

f1

f2

f3

rate = B bps

16

Window-based rate adjustmentwf (window size) = number of packets that can remain

unacknowledged for by the destination

1st packet sent

e.g., wf = 3

t

2nd packet sent3rd packet sent 1st packet received & ack. sent

2nd packet received & ack. sent3rd packet received & ack. sent1st ack.

received )4th packet can be sent

t

source f destination f

wf effectively determines the sending rate rf :

t0

t1

t2

t3

0

1

2

propagation delay

time in queueuntil transmission

round-trip time

17

TCP Sack Congestion Control1. wf (window size)= number of packets that

can remain unacknowledged for by the destination

2. While there are no drops, increase wf by 1 on each RTT

3. When a drop occurs, divide wi by 2Queuing model TCP controllers

drop

RTT

rf

Consider only CA for now for the simplicity

18

On-Off UDP model

On:

Off:

transition enabling condition

state reset

?0timt

)exp(: pttim ?0timt

)exp(: qttim

19

Talk Outline

• Related work

• Simple case studies (e.g., UDP, TCP variations).

• Generalized Hybrid Model Framework

• Validation

• Scalability of Hybrid Systems

• Software Tools

• Conclusion

20

General Topology f1

f2

f1

f2

F := { f1, f2, … } : set of end2end flows

N := { n1, n2, … } : set of nodes

L := { 1, 2, … } : set of links

n1

n2

n3

n4

n5

n6

1

2

3

4

5

B = bandwidth of link T = prop. delay of link

21

Congestion Controlrouting

queue dynamics (Droptail)

sendingrates

drops

out-queuerates

in-queue rates

TCP

RTTs

UDPHSTCP

REDWireless-Droptail

22

Queue Dynamics

total queue size queue size due to flow f

the packets of each flow are assumed uniformly distributed in the queue

in-queue rates out-queue rates

…drop rates

Queue dynamics:

23

Queue Dynamics

queue not empty/full

queue full

queue empty

same in and out-queue rates

out-queue rates proportional to fraction of packets in the queue

no drops

drops proportional to fraction in-queue rates

in-queue rates out-queue rates

…drop rates

24

Hybrid Queue Modeltransition enabling condition

-queue-not-full

-queue-full

exporteddiscrete event

-queue-empty

Discrete State

25

TCP: AIMD

congestion-avoidance

set of links transversed by flow f

propagation delays

1. While there are no drops, increase wf by 1 on each RTT (additive increase)

2. When a drop occurs, divide wf by 2 (multiplicative decrease)

(congestion controller constantly probe the network for more bandwidth)

importeddiscrete event

26

TCP: Slow Start

3. Until a drop occurs (or a threshold ssthf is reached), double wf on each RTT4. When a drop occurs, divide wf and the threshold ssthf by 2

cong.-avoid.slow-start

27

TCP: Timeout, Fast Recovery

dropf

f

dropf

ntw

tw

Floorntwn

2

)(1

2

)(1

log:)),((0

0

20

}42,2max{ dropdropf nnw

2/)( 0 twn fdrop

12/)( 0 twn fdrop dropdropf nFloorntwn 20 log1:)),((

6. wf is initialized to 1 and ssthresh becomes wf /2

7. During fast recovery, data is sent at a rate consistent with a window size of wf /2

8. Duration of fast recovery (RTT) for Tcp-sack

5. Timeout occurs when

28

Full TCP: Sack

Timeout, Delay between drop occurrence and detection are considered

29

Other Models

TCP-FAST Wireless-Droptail

TCP-NewReno, TCP-Reno, TCP-Tahoe, HSTCP, STCP, RED, Drop-rotation, UDP,etc

Reacts to congestion by using queuing delay in addition to packet loss

Packet loss occurs in queue-empty or queue-not-full because of wireless medium

30

Talk Outline

• Related work

• Simple case studies (e.g., UDP, TCP variations).

• Generalized Hybrid Model Framework

• Validation

• Scalability of Hybrid Systems

• Software Tools

• Conclusion

31

Validation MethodologyCompared simulation results from• ns-2 packet-level simulator.• Hybrid models implemented in Modelica and Shift.

Plots in the following slides refer to test topologies below:

• 10ms propagation delay• drop-tail queuing• 5-500Mbps bottleneck throughput

• 45,90,135,180ms propagation delays• drop-tail queuing• 5-500Mbps bottleneck throughput• 0-10% UDP on/off background traffic

Y-shapedumbbell

32

Validation MethodologyParking-lot

45,90,135,180ms propagation delaysdrop-tail queuing5-500Mbps bottleneck throughput0-10% UDP on/off background traffic from R7 to R80-10% UDP on/off background traffic from R9 to R10

33

Slow Start : Dumbbell• Single TCP flow.• 5Mbps bottleneck throughput.• No background traffic.

34

4 Flows : Dumbbell• Four competing TCP flows.• 5Mbps bottleneck throughput.• No background traffic.

hybrid model ns-2

Hybrid model accurately captures flow synchronization

35

4 Flows with BG:Y-shape

hybrid model

• Four competing TCP flows.• 5 Mbps bottleneck throughput.• 10% UDP background traffic

(exponentially distributedon-off times).

ns-2

36

Average Throughput and RTT

• Four competing TCP flows.• 5Mbps bottleneck throughput.• 20 trials with 10 minutes

simulation time.• 10% UDP background traffic

(exponentially distributedon-off times).

Hybrid model accurately captures TCP unfairness in 10% relative error for different propagation delays

• 45,90,135,180ms propagation delays• drop-tail queuing

Thru. 1 Thru. 2 Thru. 3 Thru. 4 RTT1 RTT2 RTT3 RTT4

ns-2 1.913 1.134 0.817 0.668 0.091 0.136 0.182 0.225

hybrid model 1.816 1.162 0.876 0.680 0.093 0.138 0.183 0.228

relative error 5.0% 2.4% 6.7% 1.0% 2.1% 1.5% 0.5% 1.3%

37

Average throughput and RTT

Thru. 1 Thru. 2 Thru. 3 Thru. 4 RTT1 RTT2 RTT3 RTT4

relative error (5M) 5.0% 2.4% 6.7% 1.0% 2.1% 1.5% 0.5% 1.3%

relate error (500M) 1.2% 4.1% 9.4% 4.1% 0.5% 1.6% .1% .1%

• Four competing TCP flows• 500Mbps bottleneck throughput.• 5 trials with 133 minutes

simulation time.• 10% UDP background traffic

(exponentially distributedon-off times).

Hybrid model accurately captures TCP unfairness in 10% relative error with 500 Mbps for different propagation delays

• 45,90,135,180ms propagation delays• drop-tail queuing

38

Empirical Distribution (5M Y-shape)

hybrid model ns-2

Hybrid model captures distribution of congestion windows and queue size

L1-distance/2 cwnd1 cwnd2 cwnd3 cwnd4 bottleneck queue

Y-shape (5M) .05 .035 .11 .02 .165

Y-shape (500M) .11 .095 .095 .09 .06

39

0 2000 4000 6000 8000 100000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1x 10

-3

P

Cwnd & Queue size

• Four competing TCP flows.• 500Mbps bottleneck throughput.• 10% UDP background traffic

(exponentially distributedon-off times).

hybrid model

0 2000 4000 6000 8000 100000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1x 10

-3

P

Cwnd & Queue size

ns-2

Hybrid model can reproduce similar probability densities compared to ns-2 for y-shape.

Empirical Distribution (500M Y-shape)

40

• Four competing flows.• 500Mbps bottleneck.• 10% UDP background.

0 2000 4000 6000 8000 100000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1x 10

-3

P

Cwnd & Queue size0 2000 4000 6000 8000 10000

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1x 10

-3

P

Cwnd & Queue size

hybrid model ns-2

Hybrid model can reproduce similar probability densities compared to ns-2 for parking-lot.

Empirical Distribution (500M Parking-lot)

41

Talk Outline

• Related work

• Simple case studies (e.g., UDP, TCP variations).

• Generalized Hybrid Model Framework

• Validation

• Scalability of Hybrid Systems

• Software Tools

• Conclusion

42

Execution Time Speedup: Dumbbell

• ns-2 complexity approximately scales with

• hybrid simulator complexity approximately scales with

number of flows

per-flow throughput

(# packets)

hybrid models are particularly suitable for large, high-bandwidth simulations (satellite, fiber optics, backbone)

0.1

1

10

100

1000

10000

100000

10 100 1000

Bottleneck Bandwidth (Mbps)

Exe

cutio

n T

ime

Spe

edup

1 flow

10 flows

20 flows

30 flows

Exe

cuti

on t

ime

of n

s-2

/ hyb

rid m

odel

43

Execution Time Speed-upY-shape and Parking-Lot

Execution speed-ups of parking lot is less than that of y-shape because of frequent discrete transition. 200 times faster with parking-lot and 2.5 Gbps.

0.1

1

10

100

1000

10 100 500 2500

Bottleneck Bandwidth (Mbps)

Exe

cutio

n T

ime

Spe

edup

Y-shape

parking-lot

Exe

cuti

on t

ime

of n

s-2

/ hyb

rid m

odel

44

Memory Usage: Y-shape

Hybrid model needs memory for each variable and state whereas ns-2 needs memory for each packet.

1

10

100

1000

10 100 500 1000 2500

Bottleneck Bandwidth (Mbps)

Mem

ory

Usa

ge (

MB

)

NS-2

Hybrid Model

45

Talk Outline

• Related work

• Simple case studies (e.g., UDP, TCP variations).

• Generalized Hybrid Model Framework

• Validation

• Scalability of Hybrid Systems

• Software Tools

• Conclusion

46

Network Description Script (NDS)

• Script similar to OTCL in ns-2.• Easy to reconfigure network. • Example NDS primitives:

– Define• Define qsize 2000

– Node• Node node1 node2

– Link with various drop policies• Link node0 node1 bandwidth q-type q-size q-opt

– Tcp-Sack• TcpSack count start-time finish-time time-opt opt-para src path

– UDP• UDP count start-time finish-time time-opt dist rate on-time off-time

src path

47

Example of NDS: Internet2.nds

# NDS script fot internet-2define qsize 250000define qtype Droptaildefine band 10000M

node STTL DNVR KSCY IPLS CHIN NYCM WASH ATLA HSTN LOSA SNVA

link STTL DNVR band 0.0257 qtype qsizelink SNVA DNVR band 0.0250 qtype qsizelink DNVR KSCY band 0.0107 qtype qsize

link HSTN KSCY band 0.0155 qtype qsizelink ATLA IPLS band 0.0110 qtype qsize

TcpSack 10 0 40000 0 0 ATLA IPLS CHINTcpSack 10 0 40000 0 0 HSTN KSCY IPLS CHINTcpSack 10 0 40000 0 0 SNVA DNVR KSCY IPLS CHIN NYCM

48

NDS Translator

• Parses and translates Network Description Script into Modelica.

• Written in Perl.

• Reads NDS input file (*.nds) and generates Modelica output file (*.mo).

• Extracts only necessary modules from netlib.mo.

• Creates new module which connects extracted modules.

• Netlib.mo is written in Modelica (TCP variations, RED, Droptail, Drop-rotation, Wireless, Fast-Tcp, HSTCP, Round-robin-combine, functions, connectors, etc).

49

Case Study: Abilene Backbone

Src Dest delay

Seattle Denver 25.684

Sunnyvale Denver 25.01

Denver Kansas City 10.674

Kansas City Indianapolis 9.34

Indianapolis Chicago 3.99

Chicago New York 20.464

Sunnyvale Los Angeles 7.772

Los Angeles Houston 31.624

Houston Atlanta 19.756

Atlanta Washington 15.938

Washington New York 4.412

Sunnyvale Seattle 16.852

Houston Kansas City 15.504

Atlanta Indianapolis 10.95

Two-way propagation delay

set Num flows Min RTT Path

One 10 15 ms ATLA to CHIN

Two 10 28.8 ms HSTN to CHIN

Three 10 69.5 ms SNVA to NYCM

TCP traffic on Internet-2

12

3

50

Example of NDS: Internet2.nds

# NDS script fot internet-2define qsize 250000define qtype Droptaildefine band 10000M

node STTL DNVR KSCY IPLS CHIN NYCM WASH ATLA HSTN LOSA SNVA

link STTL DNVR band 0.0257 qtype qsizelink SNVA DNVR band 0.0250 qtype qsizelink DNVR KSCY band 0.0107 qtype qsize

link HSTN KSCY band 0.0155 qtype qsizelink ATLA IPLS band 0.0110 qtype qsize

TcpSack 10 0 40000 0 0 ATLA IPLS CHINTcpSack 10 0 40000 0 0 HSTN KSCY IPLS CHINTcpSack 10 0 40000 0 0 SNVA DNVR KSCY IPLS CHIN NYCM

51

0

0.5

1

1.5

2

2.5

3

3.5

25000 50000 75000 100000 125000 150000

Buffer Size (packets)

Fairn

ess

Rat

io

FR1,2FR2,3FR1,3

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

25000 50000 75000 100000 125000 150000

Buffer Size (packets)

Rou

nd T

rip T

ime

(ms)

Set 1Set 2Set 3

Results: Abilene Backbone

12

3

• Buffer size between 25,000 and 150,000 packets.

• Throughputs computed by averaging 10 flows in each set.

• Fairness Ratio is large when queue size is small.

• Average round trip time shows queuing delay.

• Further analysis on the fairness ratio can be found in thesis (Section 5.4).

52

TCP for High-Speed Network

0 10 20 30 40 50 60 70 800

2

4

6

8

10

12x 10

4

Time (seconds)

Cw

nd (

pack

ets)

TCP-SackSTCP HSTCP FASTTCP

• FAST-TCP, HSTCP, STCP.

• LA to New York with 71.7 ms RTT.

• 10 Gbps bandwidth with buffer size of 5,000 packets.

• TCP-SACK increase window too slow in congestion avoidance.

• HSTCP,STCP use aggressive increase in the congestion avoidance.

• Fast TCP can be in steady state without packet loss.

53

6

6.5

7

7.5

8

8.5

9

9.5

10

10.5

2500 5000 10000 30000 60000 90000 120000

Buffer Size (packets)

Thr

oug

hput

(G

bp

s)

TCP- SackFastTCPSTCPHSTCP

Average Throughput

• Average throughput for the first 3,600 seconds

• TCP-Sack shows 76% of throughput when buffer size is 2,500

• TCP-Sack fully utilize when queue size is 90,000

• FAST-TCP’s utilization is 100% when queue is 2,500

54

Talk Outline

• Related work

• Simple case studies (e.g., UDP, TCP variations).

• Generalized Hybrid Model Framework

• Validation

• Scalability of Hybrid Systems

• Software Tools

• Conclusion

55

Conclusion

• Hybrid Systems provide a promising approach to model network traffic– Retain the low-dimensionality of continuous

approximations to traffic flow– Represent event based control mechanisms with

high accuracy, even at small time-scales– Complexity scales inversely with throughput

and RTT– Amenable to formal analysis

56

Future Work

• Verify and Improve FAST-TCP protocols for high-speed network using hybrid systems

• Study on the impact of bursty background traffic to the TCP fairness in the high-speed network

• Develop new models– Predicting voice-over-IP performance

– Network planning to accommodate voice-over-IP

– Modeling data communication over mobile phone networks

– Effect of congestion to the routing

– Modeling the impact of routing anomalies such as route flap

57

Thank You!

http://www-rcf.usc.edu/~junsool/hybrid/

58

Comparison of Hybrid Model Simulation Environments

Simulator SHIFT DYMOLA

Language SHIFT MODELICA

Institution UC Berkeley Dynasim

Solver Fixed time-step Fixed/Variable time-step

Analysis Tool No Yes

Object Oriented Yes Yes

Speed Slow Fast

Platform Linux/win32 Redhat/win32

Public Yes No

Dymola has variety of solvers and efficient methods for determining when discrete events occur

59

Drop probability vs. fraction of arrival rate

Blue flow gets most of the drops, in spite of using a smaller fraction of bandwidth when synchronization occurs, with sufficient randomness drop probability

60

TCP Sack congestion control (Slow-Start)

)()()(1

log

tmwtweRTTtW ff

dtRTT

m

ff

fRTTt

t f

f

ff RTT

wr

1. While there are no drops, increase wi exponentially, being multipliedby 2 on each RTT

ff

f wRTT

mw

log

2. For an appropriately define constant m. If was constantwe get

fRTT

3. Since wf packets are sent each round-trip time, sending rate is

61

Hybrid Queue Model (RED)

Random Early Dropactive queuing

stochastic counter-queue-not-full

-queue-full

discrete modes

62

Publications• Hybrid Systems Modeling

– A Hybrid Systems Modeling Framework for Fast and Accurate Simulation of Data Communication Networks. In Proc. SIGMETRICS, June 2003

– Analysis of a TCP hybrid model. In Proc. of the 39th Annual Allerton Conf. on Comm., Contr., and Computing, Oct. 2001

– Hybrid Modeling of TCP Congestion Control. In Maria Domenica Di Benedetto, Alberto Sangiovanni-Vincentelli, Hybrid Systems: Computation and Control, number 2034 in Lect. Notes in Comput. Science, pages 291-304, Mar. 2001

63

Publications• Stochastic Multipath Routing

– Enhancing security via stochastic routing. In Proc. of the 11th IEEE Int. Conf. on Comput. Communications and Networks

• TCP-PR: – A New TCP for Persistent Packet Reordering-

TCP-PR, Accepted for publication, Transaction on Networking

– TCP-PR: TCP for Persistent Packet Reordering. In Proc. of the IEEE 23rd Int. Conf. on Distributed Computing Systems, pages 222-231, May 200

64

High Speed TCP (2)

5.0

loglog

loglog5.0

rate) droppacket :)(( 2

2)(2

LowHigh

Low

WW

WwseHighDecreawb

wpwb

wbwpwwa

The whole point is that a(w) increases and b(w) decreases as cwnd becomes larger.Example: behavior when cwnd = 80,000 packets:

TCP HSTCP

a(w) 1 72

b(w) 0.5 0.1