Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 Building Blocks for Engineering QoS Expectations over Best-Effort Networks David Harrison, Yong

Shivkumar KalyanaramanRensselaer Polytechnic Institute

1

Building Blocks for Engineering QoS Expectations over Best-Effort Networks

David Harrison, Yong Xia, Omesh Tickoo, Hema T. Kaur, Shiv Kalyanaraman

I E

I

EI

E

Logical FIFO

B

A

ED

CB

F2

21

3

1

1

2

53

5

2

: “shiv rpi”Sponsors: DARPA-NMS, NSF, Intel, AT&T Labs Research


2

Context: Rest-in-Peace QoS ? (SIGCOMM RIPQoS 2003)

QOSQOS


3

Our Goals…

HISTORY: TCP offers reliability over unreliable networks

Our focus: Building blocks for …

… performance expectations …

over (I.e. as an overlay)

…multi-domain inter-networks …

...that do not guarantee performance… (I.e. best-effort networks)


4

What is Quality-of-Service (QoS)? “Better performance” as described by a set of parameters or

measured by a set of metrics.

Generic parameters: Bandwidth Delay, Delay-jitter Packet loss rate (or loss probability)

QoS as perceived by the application Transport/Application-specific parameters:

TimeoutsPercentage of “important” packets lost


5

What is QoS (contd) ? These parameters can be measured at several

granularities: “micro” flow, aggregate flow, population.

QoS considered “better” if a) more parameters can be specified a priori b) QoS can be specified at a finer granularity.

QoS vs CoS: CoS maps micro-flows to classes and may perform optional resource reservation per-class

QoS spectrum:Best Effort Leased Line


6

QoS: Fundamental Problems

In a FIFO service discipline, the performance assigned to one flow is convoluted with the arrivals of packets from all other flows! Cannot get QoS with a “free-for-all” Need to use new scheduling disciplines which provide

“differentiation” or “isolation” of performance from arrival rates of background traffic

B

Scheduling DisciplineFIFO

B


7

QoS: Fundamental Problems Conservation Law

(Kleinrock): (i)Wq(i) = K

Irrespective of scheduling discipline chosen: Average backlog (delay) is

constant Average bandwidth is

constant

Zero-sum game => need to “set-aside” resources for premium services


8

Talk Overview

Part 3: E2E Expected QoSPipe Abstraction:

H2H Multimedia Apps

Part 1:I E

I

EI

E

Logical FIFO

B

Expected QoSUsing Closed-loopTechniques

Part 2:A

ED

CB

F2

21

3

1

1

2

53

5

2

BANANAS Multi-path & TE Framework: incrementallydeployable

[Eg: MSN/Yahoo + Akamai]


9

Part 1: Overlay Closed-loop QoS

FIFO

B

Loops: differentiate service on an RTT-by-RTT basis using edge-based policy configuration.

Differentiation/Isolation meaningful in steady state only…

B

Priority/WFQ

Scheduler: differentiates service on a packet-by-packet basis


10

Architectural Advantages of Closed Loops Traffic management consolidated at edges

(placement of functions in line with E2E principle)

End system

Bottleneckqueue

Edge system

Architectural Potential: Edge-based (distributed) QoS services, Edge plays in application-level QoS


11

Expected Min Rate (EMR) Service: Sample Steady State Behavior

Flow 1 with 4 Mbps assured + 3 Mbps best effort

Flow 2 with 3 Mbps best effort

Issues: Transients, steady state oscillations/tracking errors


12

Overlay QoS w/ Closed Loops

and therefore,

QoS can be posed in Kelly’s optimization framework

Challenges:

1. What about the non-concavity of QoS utility functions?

2. Can we avoid admission control? Graceful degradation instead.

Ans:

1. Define & Solve an Auxiliary Optimization Problem

2. Alternative Convex Constraints in Lagrange Domain can avoid need for admission control, allowing graceful service degradation

QoS can be viewed as a congestion control problem: extension of the idea of fairness


13

Kelly’s Optimization Formulation

)log( ss xwMaximize

Network N optimizes

Each user s optimizes

subject to capacity constraints

ss wxU )(Maximize

Resulting System optimizes

)( sxUMaximize

weightUser s’s send rate

User s’s utility function

subject to capacity constraints

ws can be interpreted

as price per unit time or as congestion in the network.


14

Two User Utility Functions

optimum

)log()log()()( yxyUxU Maximize


15

Issue: QoS => Non-concave User Utility Functions A single user with a minimum rate QoS expectation (gracefully degrading into a

weighted service) can be modeled with a non-concave utility function. But this kind of U-function cannot be plugged directly into Kelly’s non-linear

optimization formulation!

log(xs)

expected minimum =0.410log(xs)

log(xs-0.4)


16

Luckily, the Sum of Non-Concave U-fns is not what we want to Optimize!

U(z) = log(z-0.6) if z > 0.6 (expected minimum rate)

10logz if z <= 0.6 (graceful degradation to weighted svc)

Two maxima

Desired Allocation(oversubscribed)not anyof the 2 maxima!

• Can use strictly concave functions and define multiple optimization problems for the same QoS problem &

• Dynamically choose a different optimization problem when oversubscribed


17

No Over-Subscription Case: Auxiliary Problem

Let i.e. flow i: two virtual sub-flows on same path

Think of a modified network with modified link capacities

Provide proportional fairness on residual

network capacity iell xCC

~

Capacity, C1

xie

Capacity,

ieiip xxx

lC~

N~

N~

N


18

Handling both under- and over-subscription…

• For ai, xi: (primary problem)

Exp. Min Rate:Effective when:

ai < Ai

lQq ll ,

lcx lIi

ie

l

,

Weighted fairnessEffective when: ai = Ai

lQq ll ,

• For aip, xip: (auxiliary problem)

If under-subscribed, solve the aux-problem; and the primary problem is automatically solved (note: aip = constant)


19

Accumulation: Per-flow Backlog in a Series of Routers

J

j

J

jkkiji dtqta

1

1

)()(

11,)()( 1, Jjitdt jijij Assuming: Define accumulation:

which is a time-shifted, distributed sum of buffered bits of flow i at all routers 1 through J

Note: accumulation is an abstract dynamical concept. Vegas and Monaco attempt to provide estimators for accumulation.

1 j j+1 J

μij Λi,j+1

dj

fi

Λiμi

ingress egress


20

Accumulation: Definition & Physical Meaning

1 j j+1 J

μi

j

Λi,j+1

djfi

Λ

i

μi

… …

20time

)(1f

ii dtq )(

1

J

jkkij dtq

)(tqiJ

1 j j+1 J

jd 1Jd

),( tdtI fii

)(tai

)( ttai

),( ttOi

fid

t

J

j

J

jkkiji dtqta

1

1

)()(


21

Accumulation-based Control Policy1 j j+1 J

μij Λi,j+1

dj

fi

Λiμi

0)( * i

atai control objective : keep

if goal , no way to probe increase of available b/w;0)( tai

ttttdtttarecall

thenataif

thenataif

if

iii

ii

ii

i

i

)],(),([),(:

)(

)(*

*

control algorithm :

0,0)0(,

))(()( *

kfonlyfwhere

atafktw iii

Example control algorithm :


22

Monaco: Robust Accumulation Estimation

1 j j+1 J

μij Λi,j+1

dj

fi

Λiμi

… …

time

)(1f

ii dtq )(

1

J

jkkij dtq

)(tqiJ

1 j j+1 J

jd 1Jd

)(taq im out-of-band

in-band ctrl pkt

),,(1

J

jkkq dtjit


23

Monaco Accumulation Estimator

1 jf Jf

μij Λi,j+1

djf

fi

Λiμi

Jb jb+1 jb 1djb ctrl

data

jf+1

ctrl

out-of-bd ctrl

in-band ctrl,data pkt

classifier fifo

Interior routers provide two priority fifo queues :

1) high priority queue for out-of-band control packet

2) low priority queue for in-band control packet and data packet

Can be done w/ IP precedence

on existing routers in Internet!!


24

ACC: Fairness & Linux Implementation Results


25

“Accumulation”: Steering Parameter Accumulation-based congestion control (ACC) is a nonlinear optimization, where

user i maximizes: Ui(xi) = ai lnxi

Accumulation (ai) is the weight (wi) of the weighted prop. fair allocation Accumulation is hence a “steering” parameter:

Equilibrium accumulation allocation => Equilibrium rate allocation! Dynamics of CC scheme decoupled from equilibrium spec (unlike AIMD)

Accumulation has a physical meaning: sum of buffered bits of the flow in the path Accumulation is related to the lagrange multiplier, I.e., ai = pl

For two flows i,k sharing the same path, ai / ak = xi / xk FIFO queues => arrival order decides departure order => buffer occupancy decides rate allocation

q1

q2 x1x2


26

Recall: Handling both under- and over-subscription…

• For ai, xi: (primary problem)

Exp. Min Rate:Effective when:

ai < Ai

lQq ll ,

lcx lIi

ie

l

,

Weighted fairnessEffective when: ai = Ai

lQq ll ,

• For aip, xip: (auxiliary problem)

If under-subscribed, solve the aux-problem; and the primary problem is automatically solved (note: aip = constant)


27

Accumulation

Control Law

Expected Min Rate (EMR) Service Building Block

iieiip dxxa )(

iii dxa

iieie dxa ix

ipx

iexdi

) - A, a - a(a - w iiipipi*max

Accumulation limit

Target

Estimated accumulationin network


28

Range of Expected Minimum Rates


29

Mean Queue Length for EMR

A=30KB A=3000KBRIO

No AQM

AVQ+VD


30

Service Multiplexing Topology

100M100M

0,20,2

0,0

0,3

0,1

0,0

0,1

0,3

…

100M

…

3,0 ~ 3,3

…

…

2,0 ~ 2,3

…

500 webA users

500 webB users

…

1,0 ~ 1,3

1,0 ~ 1,3

…

…web B

3,0 ~ 3,3

…

…web A

2,0 ~ 2,3

1-100ms 1-100ms

- Bandwidth for all unlabelled links are 1Gbps; Delay 1ms; - AQM+VD at router R1, no AQM at other routers

1ms

34ms

67ms

100ms

R0 R1 R2 R3

30M60K

35M75K

50M60K

10M15K

2,0 has weight 3

1-100ms


31

No oversubscription<m00=30,A00=60KB><m03=10,A03=15KB>

Service Multiplexing: Expected Min Rate

Moderate oversubscription<m01=35,A01=75KB>

Gross oversubscription<m02=50,A02=60KB>

Loop 02,03stop sending

Web


32

Queue Lengths

Non-AQM: Q-length Q-length w/ virtual accumulation support

More details: Y.Xia et al, “Accumulation-based Congestion Control,” to appear in IEEE/ACM Transactions of Networking, early 2005.


33

Talk Overview


H2H Multimedia Apps

Part 1:I E

I

EI

E

Logical FIFO

B

Expected QoSUsing Closed-loop

Techniques [Eg: MSN/Yahoo + Akamai]

Part 2:A

ED

CB

F2

21

3

1

1

2

53

5

2


[Eg: Akamai]


34

Part 2/3: Best-Effort Path Multiplicity

EthernetWiFi (802.11b)802.11a

USB/802.11a/b

Firewire/802.11a/b

Phone modem

InternetAS1

ISP-1

ISP-n

.

.

.


35

Part 2: Multiplicity: Flows, Paths, Exits/Interfaces

Multi-paths withina routing domain

Multi-AS-paths across domainsW/o specifying intra-domain paths Multiple exits from an AS

w/o specifying intra-domain paths

Multiple E2E Flows

Multi-homed interfaces & ISP/AS peers

BANANAS: A Single Abstract Framework To Exploit These Forms of Multiplicity In the Internet and Future Networks


36

Isn’t multi-path routing an old subject?

Lots of old work: Multi-path algorithms/protocols [5, 6, 7, 8]*, Internet signaling architectures [9, 10, 11, 12, 13]* Novel overlay routing methods [14, 15]* Transport-level approaches for multi-homed hosts [16,

17]*

But newer goals: Traffic engineering (reliability, availability, survivability, re-route):

Separation of traffic trunking from route selection Packet level or aggregate (micro-flow or macro-flow level)

Best-effort e2e service composition… Why hasn’t it happened after 2

decades of work?

*[5-17] In SIGCOMM Future Directions of Network Architectures (FDNA) 2003 paper


37

Missing Architectural Concepts An evolutionary partial deployment strategy

Allows partial deployment, incremental upgrades Incentives: increasing value with increasing deployment Fits the connectionless nature of dominant routing protocols,

Must not require signaling (unlike ATM, MPLS etc)

Abstract enough to be applicable to other scenarios: Overlay networks, last-mile multi-hop (mesh or community)

wireless networks, ad-hoc/sensor networks…

Flexible enough to have other realizations/semantics: Different placement of functions (edge vs core) Tunneling/Label Stacking Geographic/Trajectory routing


38

Connectionless TE: Questions Can we do multi-path & explicit routing ?

without signaling (I.e. in a connectionless context) without variable (and large) per-packet overhead being backward compatible with OSPF & BGP allowing incremental network upgrades

Non-Goals: Monitoring, Traffic trunking/mapping

Shortest Path MPLS…

BANANAS-TE

Signaled TE

Traffic Engineering (TE) Spectrum


39

What can we learn from ATM and MPLS ? MPLS label = Path identifier at each hop

Labels is a local identifier… Signaling maps global identifiers (addresses, path

specifications) to local identifiers

Miami

Seattle

SanFrancisco(Ingress)

New York(Egress)

1321

5

120

IP 1321

IP 120

IP 0

IPLabe

l


40

Global Path Identifiers? Instead of using local path identifiers (labels in MPLS),

consider the use of “global” path identifiers Constructed out of global variables a node already knows! Eg: Link/Router IP addrs, Link weights, ASNs, Area Ids, GPS location

Avoid the need for signaling to establish a mapping!

10

Miami

Seattle

9

27

SanFrancisco(Ingress)

New York(Egress)

18

1

5

4

3

5

IP

IP PathId

4

IP 36

IP 27

IP 0


41

Global Path Identifier (continued)

Path = {i, w1, 1, w2, 2, …, wk, k, wk+1, … , wm, j} Sequence of globally known node IDs & Link weights Global PathID: hash of this sequence: computable w/o signaling!

Canonical method: MD5 hashing of the subsequence of nodeIDs followed by a CRC-32 to get a 32-bit hash value (MD5+CRC)

Low collision (i.e. non-uniqueness) probability

Note: Different PathID encodings have different architectural implications

i

k

j

m-11

2w1

w2

wm

Path su

ffix


42

Abstract Forwarding Paradigm Forwarding table (Eg; at Node k):

[Destination Prefix, ] [Next-Hop, ] [j, ] [k+1, ]

i

k

j

m-1

1

2w1

w2

wm

Path su

ffix

Packet Header:

[j, H{k, k+1, … , m-1} ]

PathID SuffixPathID

H{k, k+1, … , m-1} H{k+1, … , m-1}

[j, H{k+1, … , m-1} ]


43

BANANAS TE: Partial Deployment Only red nodes are upgraded

“Virtual hop” between upgraded nodes Black nodes compute single-shortest-path

10

Miami

Seattle

9SanFrancisco(Ingress)

New York(Egress)

28

1

5

4

30

1

IP

4

IP 27

IP 0

27

1

3

2

IP 27

IP 27

X


44

Outline

Motivation: Best-effort path multiplicity

BANANAS: Data-plane

BANANAS: Control-plane

BANANAS: Mapping to BGP

Not covered: details in SIGCOMM FDNA paper


45

Putting It Together: Integrated OSPF/BGP Simulation


46

Multiplicity: Take-Home Message…

Multi-paths withina routing domain

Multi-AS-paths across domainsW/o specifying intra-domain paths Multiple exits from an AS

w/o specifying intra-domain paths

Multiple E2E Flows

Multi-homed interfaces & ISP/AS peers

BANANAS: A Single Abstract Framework To Exploit These Forms of Multiplicity In the Internet and Future Networks


47

Talk Overview


H2H Multimedia Apps

Part 1:I E

I

EI

E

Logical FIFO

B

Expected QoSUsing Closed-loop

Techniques [Eg: MSN/Yahoo + Akamai]

Part 2:A

ED

CB

F2

21

3

1

1

2

53

5

2


[Eg: MSN/Yahoo/Google/AOL]


48

Part 3: Multimedia Apps: Leverage E2E Performance Diversity


49

Part 3: Introduction Motivation: Video over best-effort networks (wired/wireless)

Broadband => more access bandwidth End-to-end (E2E) => constraints due to path congestion Virtual extension of broadband access pipe E2E using

multi-paths => HOME-to-HOME (H2H) multimedia

Path Diversity: dimensions Aggregate Capacity Delay diversity Loss diversity Correlations in path performance characteristics

Key: Match inherent content diversity to path diversity


50

Motivation: Internet Path Congestion limits E2E bandwidth

Internet

Server Access Link

Client Access Link

Per

form

ance

Access Link Speed

Performance Saturation (even w/ many flows/path)


51

Multi-paths using (lousy) Peers

Overlays or peers can provide path diversity even if multi-paths not available natively

Issue: diversity of performance (b/w, delay, loss), possible correlations…


52

Path (Flow) Aggregator/ Multiplexer

Path (Flow) Aggregator/ De-multiplexer

Internet

E2E Broadband Virtual Pipe Abstraction!!

Server Access Link

Client Access Link

Per

form

ance

Access Link Speed

Smart Multi-path Capacity Aggregation (SMCA): Motivation

Performance Scaling


53

Time

Lossy

Low Capacity

High Delay/Jitter

Network paths usually have:• low e2e capacity, • high latencies and • high/variable loss rates.

Single path issues: capacity, delay, loss…


54

SMCA: Leverage Diversity!

Low Perceived Delay/Jitter

Low Perceived Loss

High Perceived Capacity


55

Delay Diversity Unit

Loss Diversity Unit

Network

Receive Buffer

Content

SMCA: Framework


56

Paths Ranked by LatencyApplication Data

Low DelayRANK

High DelayRANK

SMCA: Delay Diversity Unit


57

Paths Ranked by LatencyApplication Data

Low DelayRANK

High DelayRANK

Early deadline packets mapped to low-delay paths



58

Paths Ranked by LatencyTransmit Queue

Low DelayRANK

High DelayRANK

Early deadline packets (in order of rank) mapped to low-delay paths (in order of rank)



59


Low DelayRANK

High DelayRANK

Late deadline packets mapped to high-delay paths…

Note: these packets leave the sender roughly at the same time as the early-deadline packets



60


Low Delay

High Delay

Consider a delay-based group of paths and the associatedpackets…

SMCA: Delay Diversity Loss Diversity


61


Low Delay

High Delay

Consider a delay-based group of paths and the associatedpackets…

SMCA: Delay Diversity Loss Diversity


62

Paths Ranked by Loss Raten GOPs

Low LossRANK

High LossRANK

Re-rank Paths within this group based upon packet loss rates

SMCA: Loss Diversity Unit


63

I


Low LossRANK

High LossRANK

Enlarged View of Packets (with content labels) and Paths

P

BBPBB



64

I


Low LossRANK

High LossRANK

P

BBPBB

Map high priority packets (eg: I-frame packets) to low loss rate rank paths



65

I


Low LossRANK

High LossRANK

P

BBPBB

Continue map packets to low loss rank paths based upon priority(Eg: P-frames get the next set of loss-ranked paths)



66

I


Low LossRANK

High LossRANK

P

BBPBB

Lowest priority packets get high loss rate paths(within the delay-based group of paths)



67

I


Low LossRANK

High LossRANK

P

BBPBB

FEC (unequal FEC) for a GOP mapped within the same delay-group, but mapped to the higher loss paths

SMCA: Loss Diversity Unit + FEC

I-FEC

P-FEC


68

SMCA: Performance with increasing number of Paths

Num. Of

Paths

1

2

3

4

5

PSNR (dB)

20.98

22.48

25.42

26.02

28.04

Table 1. Average PSNR Variation with Number of Paths

Background traffic

generatorBackground traffic sink

Content Source Content Sink

Performance with Path Diversity + Matching


69

SMCA gains with delay diversity

Avg. Delay(ms)

SMCAPSNR(dB)

PTPSNR(dB)

OPMSPSNR(dB)

300 21.78 18.73 11.03

100 25.12 24.21 19.19

50 28.32 29.46 24.33

30 30.12 31.63 27.96

Table 3. Gains with Delay Variation

Better comparative perf. when average delay and jitter is high


70

SMCA gains with loss diversity

Avg. Loss Prob.

SMCAPSNR(dB)

PTPSNR(dB)

OPMSPSNR(dB)

0.4 22.78 20.31 11.64

0.35 26.32 26.86 18.21

0.1 29.03 29.02 24.43

0.05 29.32 31.82 26.06

Table 2. Gains with Loss Variation

Better comparative perf. when avg loss and loss variation is high


71

Video Results: Original Video


72

Video Results: SMCA vs Simple Mapping

SMCA Multiplexing Simplistic Mapping


73

Part 3: Summary Multi-path performance diversity can be leveraged E2E

Key: must be mapped to content diversity (Similar to lessons learnt from content-driven unequal FEC protection

vs uniform FEC protection)

Ideas: Map late deadline packets to high latency paths Map higher priority packets to lower loss rate paths (within a delay-

based group of paths) FEC packets sent on paths different from that of associated content

(FEC: lower priority)

Our scheme can scale to handle lots of paths Possible with p2p networks (eg: 10-100 kbps from single path, but 10s

of paths) Does not require MD coding, or high complexity optimization


74

Perspective: Traditional QoS: Control/Data Planes

Internetw ork or W ANW orkstationRouter

Router

RouterW orkstation

Control Plane: Signaling + Admission Control orSLA (Contracting) + Provisioning/Traffic Engineering

Data Plane: Traffic conditioning (shaping, policing, markingetc) + Traffic Classification + Scheduling, Buffer management

Our Overlay QoS BBlocks: primarily in the data-plane.

Note: Skype, Kazaa/Napster/DHTs are control-plane ideas that can be combined w/ such overlay QoS


75

Perspective: Overlay QoS: > best-effort

Shortest Path MPLS…

BANANAS-TE

Signaled TE

Traffic Engineering

Best Effort Leased Line…

OverQoS, Overlay QoS w/ Closed Loop Control

DPS/CSFQ, Diff-Serv

Int-Serv

QoS spectrum

Overlay QoS: * Performance Expectations (not guarantees), * Edge/end-systems take charge of QoS just like reliability/availability* Perceived QoS by diversity matching


76

Thanks !

: “shiv rpi”

Papers, PPTs, Audio talks:

Ps: VIDEOS of all my networking courses available freely at the above web site


77

Global Path Identifier: Key Ideas

ik j

m-11

2w1

w2

wm

IP PathId(i,j)

IP PathId(1,j)

Key ideas (take-home!): 1. Global pathids (computed from global variables) instead of local labels!2. Avoid inefficient path encoding (IP) AND 3. Avoid signaling (MPLS)4. Incrementally deployable: w/ control-plane modifications

Documents

Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 Building Blocks for Engineering QoS Expectations over Best-Effort Networks David Harrison, Yong