1

Resource Optimization in Hybrid Core Networks with 100G Links

Malathi VeeraraghavanUniversity of VirginiaDate: Feb. 4, 2010

(Collaborator: Admela Jukan)• Outline

– What is a hybrid network?– Scheduled Dynamic Circuit Service (SDCS)– Single or separate hybrid networks?– SDCS usage– ASCR project software testing on DOE ANI network testbed– Summary

Sponsored by DOE ASCR grant DE-SC0002350

What is a hybrid network?

• A network with – packet switches and circuit switches?– Connection-oriented (CO) and

connectionless (CL) modes of operation?

– All-optical switches and electronic switches?

– “Optical and IP”? [dissimilar items]

2

A classification of switches/gateways

• A circuit switch is necessarily connection-oriented as positions have to be allocated to a communication session before data transfer can start

3

Line cards

(multiplexing)

Controller

(admission

control or not)

Circuit mode(position-based mux:time, frequency, port)

Packet mode

(header-based mux)

Connectionless (CL)(no admission control)

e.g., IP routers, Ethernet switches

Connection-oriented (CO)(admission control)

e.g., SONET, WDM switches

Virtual-circuit switches (VLAN, MPLS)

ESnet: IP-routed network(2006 network)

4

• Is this a hybrid network? – Leased lines between

routers pass through SONET circuit switches

– Thus, packet and circuit switches

– Leased-lines: CO mode

– CO and CL modes– Some switches could

be all-optical (metro rings)

– Thus all-optical and electronic

•Not quite!

From ESnet document dated Oct. 08, 2006

Our understanding of“hybrid network”

• Supports two types of services:– IP-routed service– Dynamic Circuit Service

• To do so, what is required of the network?– Support for connection-oriented (CO) and

connectionless modes– CO mode should support dynamic bandwidth sharing

with control-plane software such as OSCARS schedulers, not just leased-line service

– Circuit or virtual-circuit switches to support the dynamic circuit service

– IP routers– As link speeds and switching capacities increase, all-

optical switches may be required 5

Scheduled Dynamic Circuit Service (SDCS)

• Instead of advance-reservation dynamic circuit service, we propose the name SDCS because– circuit requests can be for immediate usage– so cannot limit it to “advance-reservation” or book-ahead– scheduler (e.g., OSCARS) returns a start time for the

circuit• hence “Scheduled Dynamic Circuit Service”

• Two types of SDCS requests:– User-specified start times (USST): co-scheduling apps

• with instrument or HPC or for a teleconference

– Earliest start time (EST): immediate usage requested, but scheduler returns an assigned start time based on resource availability: suitable for large file transfers

6

How does SDCS fit in with other connectivity services?

7RDP: Rate Duration Product

RDP = 1500B

RDP = 1.4MB (DS0*3 min)

RDP = 5.9TB (T1*1 year)

Gap to fill

How does SDCS differ from leased line service?

8

Leased lineCircuit/VC switch

Customer devicee.g., IP router

Customer devicee.g., cluster computer

SDCS access link SDCS access link

Dynamic circuit

Customer devicee.g., LCD display

Differences

Service Leased line service

Scheduled dynamic circuit service

Components of contract

One contract:• duration• rate• specific endpoints

Two contracts:•SDCS access link of certain rate (like IP access link)•as needed, requests for circuits of certain duration, rate, between any two SDCS endpoints

Duration limit?

No Maximum limit required for acceptable blocking rate/start-time delay with moderate utilization

Call blocking?

No Yes; to achieve moderate utilization, some call blocking should be tolerated

9

• Difficult to articulate differences between a “leased line” and a “dynamic circuit”

• BUT difference between leased-line service and dynamic-circuit service is clear

10

Outline check

• Outline– What is a hybrid network?– Scheduled Dynamic Circuit Service (SDCS) Single or separate hybrid networks?– SDCS usage– ASCR project software testing on DOE ANI network testbed– Summary

Single or separate networks?

• The term “hybrid networking” is more general than “hybrid network”– Hybrid networking: Deploy two

separate networks to support the two services • IP-routed service• Scheduled dynamic circuit service (SDCS)

– Hybrid network: seems to imply a single network

11

ESnet4 (from 2006 talk by Johnston & Burrescia)Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),

500-600 Gbps by 2011-2012 (100 Gb/s circuits)

Cle

vela

nd

Europe(GEANT)

Asia-Pacific

New York

Chicago

Washington DC

Atl

anta

CERN (30+ Gbps)

Seattle

Albuquerque

Au

stra

lia

San Diego

LA

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN (30+ Gbps)Canada(CANARIE)

Europe(GEANT)

Asia

-Pac

ific

Asia Pacific

GLORIAD (Russia and

China)

Boise

Houston Jacksonville

Tulsa

Boston

Science Data Network Core

IP Core

Kansa

s City

Au

stra

lia

Core network fiber path is~ 14,000 miles / 24,000 km

1625

mil

es /

2545

km

2700 miles / 4300 km

Sunnyvale

Production IP core (10Gbps)

SDN core (20-30-40-50 Gbps)

MANs (20-60 Gbps) or backbone loops for site access

International connections

IP core hubs

Primary DOE LabsSDN hubs

High speed cross-connectswith Ineternet2/AbilenePossible hubs

USLHCNet

“Single” hybrid network• What does a “single” network mean:

– One node at each PoP– One set of inter-PoP links– One set of customer (e.g., site) access links

• Today’s deployment:– One node at each PoP for ESnet (IP-routed service) and one node

for SDN (dynamic circuit service)– Separate inter-PoP links– Separate site access links– Reasons: Costs of MX vs. M-series interfaces, funding, etc.

• Expenditures saved:– Inter-PoP and site access link costs– Maintenance costs for nodes

• What are the negatives?

13

What is required for a“single” hybrid network

deployment? • Node equipment should support

– IP routing capability– Circuit/VC capability (e.g. VLAN, MPLS,

GMPLS)

• To support IP-routed service and SDCS on a single interface– Circuit/VC based rate enforcement– e.g., VLAN rate policing/limiting

• OSCARS IDC for circuit scheduling

14

Single node supports both IP-forwarding and virtual-circuit capabilities

15

IP (Layer 3)

Ethernet/VLAN (L2)

Access links to sites

and peers

Inter-PoP ESnet links

Leased-linefor IP-routed

service

Dynamic circuit

• Existing equipment already available– Juniper MX series

switches – Cisco 6500 series– Carrier-grade Ethernet

switches

• Separate SDN was proof-of-concept for dynamic circuit service?– Since same type of

node (MX) used in both IP and SDN networks, both services can be offered on one substrate

Part of the ESnet topology (potential single network offering both services)

16

ALBU

DENVPNWG

SUNN

ELPA100 GbE

PNNLIP

Ethernet/VLAN

IP

Ethernet/VLAN

IP

Ethernet/VLAN

IP

Ethernet/VLAN

PE router

PE router

Leased-linefor IP-routed

service

Dynamic circuit

OSCARSIDC

SpectrumNMS

LANL

IP

Ethernet/VLAN

17

Outline check

• Outline– What is a hybrid network?– Scheduled Dynamic Circuit Service (SDCS)– Single or separate hybrid networks? SDCS usage– ASCR project software testing on DOE ANI network testbed– Summary

SDCS deployment

• Successful deployment in the core– important: inter-domain demonstrations– [Scheduled Circuit Routing Protocol: like BGP for

SDCS]

• Status on deployment in:– site networks?

• With site deployments, development of new applications (remote instrument control, WAFS) can be encouraged

• Without data source-to-data sink (end-to-end) reach of SDCS:– usage scenarios for core SDCS service

18

Core network SDCS usage scenarios

• Usage I: Lambdastation– Applications on user hosts (dCache/SRM) communicate

with Lamba Station Server, which runs at the “edge” of sites

– Lambda Station Server communicates with core network IDC to reserve/provision dynamic circuit, and sets PBR in CE router to forward packets corresponding to that particular application flow to the circuit

• Usage II: Automatic long flow detection at PE routers and rerouting to dynamic circuits

• Usage III: Use SDCS instead of leased-line service for creating IP-routed topology, and resize/reroute these router-to-route circuits as aggregate IP loads on these circuits change– Spectrum monitors SNMP data and initiates changes by requesting

these of the IDC 19

Two issues with Usage III• Current ESnet weathermap shows light loading (5%-

20%) on links between IP routers– If SNMP loads indicate a change from 5% to 25%, should

the Spectrum NMS react and increase the rate of the circuit between the two routers?

• Commercial providers note:– Operations divisions typically have strong resistance to

change the network topology because of the potential for:• route flaps, and • drastic changes in the end-end packet latency (e.g., >

10ms)– For these reasons, while theoretically management-plane

traffic and network engineering is a potential user of core SDCS, we expect its use to be limited mainly to failure recovery.

20

Our ASCR project

• Focus on Usage II• Long-flow identification methods:

– 5-tuple rule based flow classification– payload based classification– machine learning algorithms

21

Software implementation and testing on

DOE ANI network testbed

• MFDB: Monitored Flow Data Base• Flows in MFDB mirrored by router to HYNES server• When one of these flows is detected, circuit setup is

initiated and PBR set in router to route flow to circuit 22

Summary

• Proposed name “Scheduled Dynamic Circuit Service (SDCS)” to cover both – advance-reservation – immediate-request

• Hybrid networks support IP-routed service and SDCS

• SDCS: comparison with leased-line service• Single hybrid network deployment• Three usage scenarios for SDCS• HYNES software testing on DOE ANI network

testbed23