A User Driven Dynamic Circuit Network Implementation

Evangelos ChaniotakisNetwork Engineering Group

DANMS 2008November 30 2008

Energy Sciences NetworkLawrence Berkeley National Laboratory

Networking for the Future of Science

Contents

• Introduction• ESnet Network Architecture• Virtual Circuit Implementation• User-Driven VCs• Layer 2 and 3 support• Path Computation• Authentication and Authorization• Oversubscription and soft reservations• Collaboration• Network use• Future work• Acknowledgments• Questions

Introduction

• ESnet's mission: provide the network infrastructure for DOE researchers

• Rapid growth in scientific computing

• Highly distributed collaboration reaching the global scale – LHC, eVLBI

• Distribution of large data sets becoming more and more common (40Tb / day projected for LHC)

• ESnet must reliably and economically accommodate large flows and regular Internet traffic

• But: Large flows don't work too well al TCP/IP

• Our solution: Isolate large flows into VCs

• Provides predictable bandwidth, allows impolite protocols without disruption to other traffic

A Multi-Domain Environment• End points will be at independent institutions – campuses or

research institutes - that are served by ESnet, Abilene, GÉANT, and their regional networks– Complex inter-domain issues – typical circuit will involve five or more

domains

– For example, a connection between FNAL and DESY involves five domains, traverses four countries, and crosses seven time zones

FNAL (AS3152)[US]

ESnet (AS293)[US]

GEANT (AS20965)[Europe]

DFN (AS680)[Germany]

DESY (AS1754)[Germany]

ESnet Network Architecture

• A core 10G best-effort IP network

• A logically distinct Science Data Network

• Virtual circuits are generally engineered and provisioned only on SDN links

• Engineered OSPF metrics ensure that best effort traffic uses IP core and avoids SDN

• In case of IP network bifurcation, the SDN network will be used by best-effort traffic.

• QoS is used to engineer this backup mechanism

ESnet 4 Core Network – December 2008

Las Vegas

Seattle

Su

nn

yv

ale

LA

San D

iego

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash. DCSLC

Port.

BatonRougeHouston

Pitts.

NY

C

Boston

Atlanta

Nashville

ESnet IP coreESnet Science Data Network core (N X 10G)ESnet SDN core, NLR links (backup paths)Lab supplied linkLHC related linkMAN linkInternational IP Connections

Layer 1 optical nodes - eventual ESnet Points of Presence

ESnet IP switch/router hubs

ESnet SDN switch hubs

Layer 1 optical nodes not currently in ESnet plans

Lab site

SDSC

StarLight

20G

20G

20G

20G

20G

MA

N L

AN

(Aof

A)

Lab site – independent dual connect.

USLHC

USLHC

GA

LLNL

LANL

ORNL

FNAL

BNL

PNNL

PhilDenver

?

LHC/CERN

ESnet aggregation switch

Chicago

Virtual Circuit Implementation

Source

Sink

MPLS labels are attached onto packets from Source andplaced in separate queue to ensure guaranteed bandwidth.

Regular production traffic queue.Interface queues

SDN SDN SDN

IP IP IPIP Link

IP L

ink

SDN LinkRSVP, MPLS, LDPenabled on

internal interfaces

standard,best-effort

queue

high-priority queue

LSP between ESnet border routers is determined using topology information from OSPF-TE. Path of LSP is explicitly directed to take SDN network where possible.

On the SDN Ethernet switches all traffic is MPLS switched (layer 2.5).

Layer 3 VC Service: Packets matching reservation profile IP flow-spec are filtered out (i.e. policy based routing), “policed” to reserved bandwidth, and injected into an LSP.Layer 2 VC Service:Packets matching reservation profile VLAN ID are filtered out (i.e. L2VPN), “policed” to reserved bandwidth, and injected into an LSP. Label Switched Path

SDN Link

QoS parameterization

• Classes of service in ESnet:– network control,

– expedited-forwarding,

– best-effort,

– scavenger

Table IQoS Queue Percentages

ScavengerIP Core 5% 20% 74% 1%SDN 5% 74% 20% 1%

Network Control

Expedited Forwarding

Best Effort

User-driven Virtual Circuits

• On-demand Secure Circuit Advance Reservation System

– Virtual circuits are requested by end-users

– Parameters: endpoints, bandwidth, duration

– OSCARS decides on the VC path, implements the VCs inside ESnet, and forwards requests to other domains

– Web interface for general users

– SOAP interface for automated provisioning tools

– Advance reservations allow orchestration

Authentication and Authorization

• SOAP API

– Signed messages using X.509 certs

– User id determined by the cert subject

• Web Interface

– Username and password

• Authorization:

– Complex underlying resource and privilege system.

– Simplified with roles: user, engineer, site admin, operator

– Support for one-time authorization tokens

Layer 2 and Layer 3 VCs

• Ethernet Layer 2 VCs

– VLAN id can be requested by the user or assigned by the system

– Multi-domain negotiation is done

– Coordination with end-sites needed

• IP layer 3 VCs

– User provides flow specs

– Source & destination IP, port, protocol, DSCP

– CE router injects matching packets in LSP

Path Computation

• OSCARS periodically harvests full topology information for ESnet

• When a path needs to be computed for a new VC request, a topology graph is populated from that data as well as all concurrent VCs.

• Then, all links that cannot satisfy the new VC are pruned.

• Finally, a Djikstra shortest-path algorithm is run on the pruned graph

• The base graph currently stands at ~1000 nodes and 1500 edges.

Automated Device Configuration

• After a VC has been reserved the network devices must be configured

• Cisco and Juniper platforms are supported

• Users can use the SOAP API to signal VC setup and teardown

• OSCARS has a scheduler component that periodically checks for pending configuration tasks

• A platform-specific configuration template is filled out and pushed to the routers.

• Currently 10-100 seconds are needed to instantiate a circuit in this manner.

Over-subscription and Soft Reservations

• Original concept did not allow for any kind of over-subscription or over-booking.

• Emerging user requirements:

– User-managed load-balancing

– Redundant VCs

• We decided to allow users to oversubscribe their VCs.

• Packets below reserved bandwidth are marked expedited-forwarding (normal VC traffic)

• Any packets exceeding that are marked as scavenger.

Collaboration

• DICE: Dante, Internet2, Caltech/USLHCNet, ESnet

• Close partnership with Internet2

• Interoperability with AutoBAHN, Phosphorus

• Automated provisioning with TeraPaths, LambdaStation and Phoebus

• Standardization efforts:

– OGF: • NSI WG, • NML WG, • NM WG

– GLIF: • GNI API WG

Network Use

• Currently in pre-production.

• 16 long-term VCs, total ~40 Gbps reserved

– Almost all related to LHC T0-T1 and T1-T2

– Almost all are “soft” reservations

• Primary users: Fermilab, Brookhaven

• Our users consistently demand production-quality availability for LHC T0-T1 and T1-T2 VCs.

• Cross-domain VCs with Internet2 using LambdaStation and Terapaths

• Demos at SC07, SC08, multiple Joint Techs and I2 Member Meetings

• VCs minimally disrupted during full replacement of network gear in two of our PoPs.

OSCARS Managed Production VCs

Future work

• Outage management

– Automated VC rerouting based on network management system data, and scheduled or unscheduled outages

• Multi-layer VCs

– Integrated solution for services provisioned across multiple layers – ie an L3 service over a L2 circuit over a L1 lightpath.

• Optimizations

– Support for short-lived just-in-time VCs (<15 min)

– Provisioning and instantiation speed-up

Acknowledgments

• Tom Lehman, ISI East

• John Vollbrecht, Internet 2

• Andrew Lake, Internet 2

• Afrodite Sevasti, AutoBAHN project

• Guy Roberts, DANTE

• Radek Krzywania, PSNC

Thank you!

• Questions?

Authors

• Chin P. Guok, ESnetchin@es.net

• David W. Robertson, LBNLdwrobertson@lbl.gov

• Evangelos Chaniotakis, ESnethaniotak@es.net

• Mary R. Thompson, LBNLmrthompson@lbl.gov

• William E. Johnston, ESnet wej@es.net

• Brian Tierney, ESnettierney@es.net