Ben Teitelbaum January 25th, 2001 TEQUILA Workshop on Internet Design for SLS Delivery Service Specification and TE for the QBone

Ben Teitelbaum <[email protected]>January 25th, 2001TEQUILA Workshop on Internet Design for SLS Delivery

Service Specification and TE for the QBone

Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 2

architecture

deployment

How We Got Here (short version)

• Began chanting: “enable advanced applications,…”

• Assessed requirements• Recommended DiffServ• Selected “Premium” service to meet demands of loss/jitter sensitive apps

• Charted QBone initiative • Specified QBone architecture• Now proceeding to implement it and tweak the architecture


IntServ/RSVP

IntServ/RSVP vs DiffServ

BB BB

•Per-flow service state at every hop

•Abstract/manage each cloud’s resources (BBs) •Packets colored to indicate forwarding “behavior”•Focus on aggregates not individual flows•Policing at edge to get services

•Scalability problems

DiffServ

•Focus on multipoint multicast


DiffServ Overview

Applications contract for specific QoS profiles • Policing at network periphery• “Color” packets with a few simple, differentiated per-hop forwarding behaviors (PHBs)

–Indicated in packet header–Applied to PHB traffic aggregates

• PHBs + policing rules = range of services

DS domains contract with each other for aggregate QoS traffic profiles

• Policing at cloud-cloud boundary• Supports simple, bilateral business agreements

Exploits edge/core distinction for scalability


Example Service #1: Premium

Assurance: like a leased line

PHB: Expedited Forwarding (RFC 2598)• EF in separate queue configured with minimum departure rate

• Example mechanisms: strict priority, MDRR, WFQ

Policing: police to a specified peak rate and drop out-of-profile packets; effectively a leaky bucket with depth 1 MTU


Example Service #2: Controlled Load

Assurance: network looks “lightly-loaded” for conforming traffic

PHB: Assured Forwarding (RFC 2597)• 4 independent AF classes• 3 drop preference levels within each class• Example mechanisms: WRED, WFQ

Policing: police to specified rate and burst profile, remarking out-of-profile packets to have higher drop probability


Example Service #3: CoS

Assurance: “better than Joe”

PHB: “drop the lower classes first ” (AF or class selector PHBs)

Policing: could be based on anything (e.g. higher priority for the CEO)

A.K.A.“Olympic” classes of BE service (e.g. Gold, Silver, Bronze)


QBone Architecture

A Service: QBone Premium Service• Built on Expedited Forwarding (EF) (RFC 2598)• Assurance: near-zero loss & low, bounded jitter for marked traffic conforming to a specified peak rate

–a.k.a. “virtual leased line”, “virtual wire”

Reservation Setup Protocol• Now: long-lived, manual setup• Proposed: SIBBS protocol between QBone domains; RSVP end-to-end between hosts

QBone Measurement Architecture• Uniform collection of QoS metrics• Uniform dissemination interface


QBone Measurement Architecture1/2

Collection metrics, EF and BE...• Active metrics (paths)

– One-way delay-variation

– One-way loss

– Traceroutes

– e.g. IPPM Surveyors

• Passive metrics (interfaces)– Load

– EF reservation load

– Discards (suggested)

– Link bandwidths (suggested)

– e.g. OCxMon, RTFM, MIBs

Boundary Router

Intra-Domain Premium Path

Inter-Domain Premium Path

ActiveMeasurements

MIB-basedstatistics

PassiveMeasurements

PassiveMeasurements

PM node PM node

AM node

QBone Domain2

QBone Domain1

QBone Domain3


QBone Measurement Architecture2/2

Dissemination • Standard URL query syntax:

• whois server to learn canonical names for QBone domains, routers, sniffers, etc

label ::= <alphanum> { <alphanum> }router ::= label"-ROUTER”probe ::= label"-PROBE”sniffer ::= label "-SNIFFER”host ::= router | probe | snifferpath ::= host "/" hostPHB ::= "BE" | "EF”metric ::= "LOSS" | "ONEWAY" | "PING" | "IPDV" | "LOAD" | "TRACEROUTE" | "COMMITMENT" | "RESERVATION"year ::= digit digit digit digitmonth ::= "01" | "02" | ... | "12”day ::= "01" | "02" | ... | "31”YYYYMMDD ::= year month dayaggregation ::= <unsigned_integer>prefix ::= “http” | “ftp” | <other>query ::= prefix ”://" path "/" PHB "/" metric "/” YYYYMMDD "/" metric "/" aggregation


QBone E2E Picture

GigaPoPA

CampusA

CampusB

GigaPoPB

CampusC

CampusD

Backbone

X Kbps of QPS from hither to………..yon

Key


SC2000 Interdomain QoS DemoNovember 6-9, 2000

Premium service over two wide-area paths• LBNL-ESnet-Abilene-SCinet-Internet2 booth• Stanford-CalREN2-Abilene-SCinet-Internet2 booth

Congestion induced at multiple points

CD-quality interactive audio application shown with/without QoS

ESnet and Abilene QoS capabilities will form nucleus of QBone

SC2000 Network Challenge Winner: Award for "Most Captivating and Best Tuned" Demo



Project Overview

Internet2 & NetworkResearch

Internet2 QoS

Abilene QoS

Future Directions

Questions

Abilene QoS...

0

15

30

45


Abilene TopologyJanuary, 2001

•47 connectors •183 participants

•34 connections to 20 peer networks


Miami

NYC (60 Hudson)

LA

AbileneSingaren

SEATTLE

NYC (Telehouse

25 Broadway)

STARTAP

ArgentinaBrazilChileColumbiaBrazil

Developing International Peering

Courtesy: Linda Winkler, STAR TAP

CERNSINET

AbileneESnetvBNS

JAnetSURFnetNORDUnet

AbileneESnetCA*net3AARnetCERNET

CUDIHARNETSINET

AbilenevBNSESnetDRENNRENNISN

CA*net3SURFnetMIRnetNORDUnetREUNA

IUCCAPANCERNSINETSingarenTanet

GEMNETRenater

DFNINFNDANTECA*net3


Abilene Load Snapshot


APS Participation

Goal:• Make APS a reference implementation of the QBone architecture

Current Participants• MAGPI (U. Penn)• iCAIR• PSC (Penn State)• OARNet (Ohio State)• NASA EOS

Others in the wings• TF-NGN (through DANTE)• MIRnet• Various other international

• ANL• UIUC• DOE Science GRID (peering transit network)


Initial Engineering Plan (obsolete)

Measurement

Edge Policing

Manual Setup

EF Core Forwarding

Automated Setup

EF Edge Forwarding

Shaping

Sweetwater Midland Odessa Pecos

(GTS)

(Surveyor + SNMP + HTTP)

(“Firehose” CAR)

(Whiteboard + CLI)

(MDRR)

(MDRR)

(BB)


APS Measurements

Status: • Collecting EF/BE loads and CAR conform/exceed stats• Not currently monitoring IPDV, but Abilene Surveyor nodes now OC-3 connected and operational

• Ohio-ITEC hosting APS measurements and QBone-wide whois server

Near Future: • IPDV along edge-to-edge QBone paths• Abilene Surveyor timing improvements

–Better NTP–New CDMA timing sources (can't get GPS in Qwest POPs)

• Collection of AS-level traffic matrices


Committed Access Rate (CAR)

Classifies traffic based on certain matching criteria (interface, DSCP, or ACL) and meters it to a leaky bucket traffic profile

Depending on metering result, different actions applied (drop, transmit, set DSCP,…)

Syntax:rate-limit {input | output} [access-group [rate-limit] acl-index] bps burst-normal burst-max conform-action action exceed-action action


CAR Experience

For the most part, CAR is exactly what the DiffServ doctor ordered

However, there are some limitations…• PPS performance cost• Quirky constraints on token bucket depths• Not easy to do "virtual trunk" style classification


CAR Limitation 1: Performance

Problem:• On E0 GSR edge cards, lack of ASIC support for CAR results in non-trivial pps hit

Solution:• Really out of our hands; must wait for E3 edge cards, which will have CAR in hardware

• Load on access interfaces is still light, so performance hit not a big issue for current participants


CAR Limitation 2: Policing Granularity

Problem:• Cisco CAR Doc:

–“burst-normal Normal burst size in bytes. The minimum value is bps divided by 2000”

• i.e. burst-normal max (mtu, bps/2000)• But, QPS demands token bucket depth of 1 MTU!• Implication: bps 3Mbps (for Ethernet MTU)

Solution:• Again, out of our hands• Have raised concern with Cisco and are hoping E3 cards will address this limitation too


Solution: with additional rope,CAR can also classify by qos-group (Cisco proprietary)

• Packets assigned to QoS groups through QoS Policy Propagation via BGP (QPPB)

CAR Limitation 3: Classification

Problem: we want...

…is what we have

("virtual trunk")

("firehose")But this...


Abilene Architecture Limitation: “Porous” Edge Problem

DoQoS problem with current architecture

MDRR (EF forwarding) on interior interfaces easily subverted by unpoliced connectors


How to “Crisp” the Edge?

Problem: EF requires that all connectors be policed

Solution:• Short term: Stochastically detect illegal EF traffic with NetFlow and/or OCxMon passive monitoring; gigaPoP would be asked to police

• Longer term: Wait for E3 edge cards, deploy them aggressively, and police everywhere

• Road not taken:Re-write all non-participant traffic with DSCP 000000 (using PIRC hack); need to pass some DSCP values (reason why coming up…)


Current Engineering Plan

Measurement

Edge Policing

Manual Setup

EF Core Forwarding

Automated Setup

EF Edge Forwarding

Shaping

Sweetwater Midland Odessa Pecos

(GTS)

(Surveyor + SNMP + HTTP + whois + traffic matrices)

(CAR + QPPB)

(Whiteboard + CLI)

(MDRR)

(MDRR)

(SIBBS? + DSTE?)


Project Overview


Internet2 QoS

Abilene QoS

Future Directions

Questions

Future Directions...

0

15

30

45


QBone Premium Service Outlook

Good News:• DiffServ functionality in most modern routers• Many hosts support QoS signaling • Lots of isolated testbed trials• Some partial backbone implementations

Bad News:• Deploying QPS "requires upgrading the world"• Low demand for QPS (app/net chicken/egg dynamic)• Vendor implementations don’t always live up to hype• Requires a lot of elbow grease, which is in short supply in most campus IT organizations

Bottom line:• Progress can be made towards end-to-end real-time services (QPS), but it is going to be slow


Looking forward...

QBone Premium• Revise/complete architecture (joint work with TF-NGN)• Make measured progress on deployment• Focus on ad hoc QoS solutions where it needed, but deploy in a way consistent with QPS architecture

Big measurement push• QBone measurement architecture• Broader Internet2 measurement push• E2E Performance Initiative

Reap lower-hanging fruits of DiffServ• Internet2 Scavenger Service (I2SS)


Internet2 Scavenger Service

Basic idea• A lower priority class of best-effort• Voluntary marking hints to network that degraded service is OK (think of a "nice" for the network)

Intended uses• Non-time-critical traffic (e.g. server-to-server NNTP, anonymous FTP, network backups)

• Bulk data transfers using TCP• Non-mission applications (e.g. Napster, games, etc)• New kinds of distributed applications that attempt to use idle network capacity


I2SS Service Specification

Rigorously defining the E2E I2SS is difficult!• Hoping to define relative to best effort

I2SS traffic indicated by DSCP 001000• Modification of class selector PHB• Note that the I2SS codepoint has global significance

I2SS domain requirements• Traffic leaving must be marked I2SS, if it entered so

Router requirements (all SHOULDs)• Forward I2SS independently giving it a lower probability of timely forwarding OR forward in same manner as BE

• Offer I2SS a very small minimum departure rate• Offer I2SS all used bandwidth


Project Overview


Internet2 QoS

Abilene QoS

Future Directions

Questions

Any Questions?

0

15

30

45


For more information...

Internet2 Home:• http://www.internet2.edu/

Internet2 QoS Working Group Home:• http://www.internet2.edu/wg/qos/

QBone Home:• http://qbone.internet2.edu/

Abilene Premium Service Home:• http://www.internet2.edu/abilene/qos/



What is ''QBone"?

Internet2 effort to build an interdomain diff-serv testbed

Service objective: Premium/VLLOther objectives: thorough instrumentation

and open dissemination of measurements Implementation status: embryonic

• Abilene: PSC (Penn State), UIUC, iCAIR, OARNet (Ohio State), ANL, MAGPI (U. Penn), NASA (Ames and Goddard)

• ESnet: ANL, LBNL

We hope to be a place where experience with QoS SLSs is gained and shared!


SLS Basics

An SLS is the technical component of an SLA

SLSs are bilateral and private

An SLS is not a reservation

SLSs can be arbitrarily weird; for example...• Per {sourceContinent, destContinent} RTT bounds• All VLL requests under 1 Mbps will be rejected• 100 Mbps VLL to MIT OK with 7 days notice

SLA

SLS Contractual Goop


Our Solution

Define services and reservations; punt on SLSs

Services • Globally well-known and defined• Include many parameters from draft-tequila-sls-00.txt (scope, traffic envelope, performance guarantees, etc)

Reservations• Identify a service ID• Specify service parameters

Treat SLS as black box: can this reservation be extended across the peering; yes or no?


Simple Interdomain Bandwidth Broker Signalling (SIBBS)

BB BB BB

H H

DS Domain1 DS Domain3DS Domain2

S LS1-2 S LS2-3

R RRR

Control

Data


SIBBS - Simple Interdomain Bandwidth Broker Signaling

Basic Idea• Simple protocol for one QBone domain to request of another QBone domain an increase or decrease in an aggregate reservation of a globally well-known service

Design Goals• Simple & extensible• New protocol for now; looking at mapping to COPS

Status• Protocol draft nearing completion (see: http://qbone.ctit.utwente.nl/BBroker/bboutline2.html)


BB Conceptual Model

Major Blocks• Intra-domain protocol• Inter-domain protocol• User interface• Network Management

Interface• Routing interface• Policy functions• Local resource and

configuration data


SIBBS: Basic Reservation Setup

BB

BB

BB

RTR ES RTR RTR RTR ES

RAR RAR

RAA RAA

Web interface RSVPDIAMETERCOPSRYO

CLISNMPCOPS


DRAFT SIBBS RAR

SIBBS version RAR ID

Sender ID Sender Signature

Source Prefix Destination Prefix

Ingress Router ID Start Time

Stop Time Flags

GWS ID Service Paramter TLV

Other (Optional) TLVs


Core Tunnels &Virtual Peerings

BB BB BB SIBBS SIBBS

BB BB BB

Aggregate Resevation Src RTR RTR Dest

Virtual Peering


Conclusions

Limited diff-serv implementation experiences

Even less experience with diff-serv peering

Need to specify a few internet-wide services

TEQUILA draft could be recast as a framework for specifying services

Formal specification language for SLSs is probably premature

Automated negotiation of SLSs is premature


For More Info...

http://qbone.internet2.edu/architecture.shtml

Documents

Ben Teitelbaum January 25th, 2001 TEQUILA Workshop on Internet Design for SLS Delivery Service Specification and TE for the QBone