OSCARS Roadmap OGF 28 Munich, Germany Mar 15, 2010

Supporting Advanced Scientific Computing Research • Basic Energy Sciences • Biological and Environmental Research • Fusion Energy

Sciences • High Energy Physics • Nuclear Physics

OSCARS Roadmap

OGF 28Munich, Germany

Mar 15, 2010

Chin Guok ([email protected])

Energy Sciences Network (ESnet)

Lawrence Berkeley National Laboratory

OSCARS Overview

2

Path Computation• Topology

• Reachability• Contraints

Scheduling• AAA

• Availability

Provisioning• Signaling• Security

• Resiliency/Redundancy

OSCARSGuaranteedBandwidth

Virtual Circuit Services

OSCARS Design Goals

3

• Configurable– The circuits must be dynamic and driven by user requirements (e.g. termination end-points, required bandwidth, etc)

• Schedulable– Premium service such as guaranteed bandwidth will be a scarce resource that is not always freely available and

therefore should be obtained through a resource allocation process that is schedulable

• Predictable– The service should provide circuits with predictable properties (e.g. bandwidth, duration, etc) that the user can leverage.

• Usable– The service must be easy to use by the target community

• Reliable– Resiliency strategies (e.g. reroutes) should be largely transparent to the user

• Informative– The service should provide useful information about reserved resources and circuit status to enable the user to make

intelligent decisions

• Scalable– The underlying network should be able to manage its resources to provide the appearance of scalability to the user– The service should be transport technology agnostic (e.g. 100GE, DWDM, etc)

• Geographically comprehensive– The R&E network community must act in a coordinated fashion to provide this environment end-to-end

• Secure– The user must have confidence that both ends of the circuit is connected to the intended termination points, and that the

circuit cannot be “hijacked” by a third party while in use

• Provide traffic isolation– Users want to be able to use non-standard/aggressive protocols when transferring large amounts of data over long

distances in order to achieve high performance and maximum utilization of the available bandwidth

Network Mechanisms Underlying OSCARS

4

Best-effort IP traffic can use SDN, but under

normal circumstances it does not because the

OSPF cost of SDN is very high

Sink

MPLS labels are attached onto packets from Source and

placed in separate queue to ensure guaranteed bandwidth.

Regular production (best-effort)traffic queue.Interface queues

SDN SDN SDN

IP IP IPIP Link

IP L

ink

SDN LinkRSVP, MPLS, LDP

enabled oninternal interfaces

standard,best-effort

queue

high-priority queue

LSP between ESnet border (PE) routers is determined using topology information from OSPF-TE. Path of LSP is explicitly directed to take SDN network where possible.

On the SDN all OSCARS traffic is MPLS switched (layer 2.5).

explicitLabel Switched Path

SDN Link

Layer 3 VC Service: Packets matching reservation profile IP flow-spec are filtered out (i.e. policy based routing), “policed” to reserved bandwidth, and injected into an LSP.

Layer 2 VC Service:Packets matching reservation profile VLAN ID are filtered out (i.e. L2VPN), “policed” to reserved bandwidth, and injected into an LSP.

ESnet WANbandwidth

policer

AAAS

Ntfy APIs

Resv API

WBUI

OSCARS Core

PSSNS

OSCARSIDC PCE

Source

Production OSCARS

• OSCARS is currently being used to support production traffic movement• Operational Virtual Circuit (VC) support

– As of 3/2010, there are 30 long-term production VCs instantiated• 24 VCs supporting High Energy Physics

– LHC T0-T1 (Primary and Backup), T1-T2– Soudan Underground Laboratory

• 3 VCs supporting Climate– GFDL– ESG

• 2 VCs supporting Computational Astrophysics– OptiPortal

• 1 VC supporting Biological and Environmental Research– Genomics

• Short-Term Dynamic VCs• Between 1/2008 and 10/2009, there were roughly 4600 successful VC reservations

– 3000 reservations initiated by BNL using TeraPaths– 900 reservations initiated by FNAL using LambdaStation– 700 reservations initiated using Phoebus

• The adoption of OSCARS as an integral part of the ESnet4 network was a core contributor to ESnet winning the Excellence.gov “Excellence in Leveraging Technology” award given by the Industry Advisory Council’s (IAC) Collaboration and Transformation Shared Interest Group (Apr 2009)

5

OSCARS Interoperability Efforts

• As part of the OSCARS effort, ESnet worked closely with the DICE (DANTE, Internet2, CalTech, ESnet) Control Plane working group to develop the InterDomain Control Protocol (IDCP) which specifies inter-domain messaging for end-to-end VCs

• The following organizations have implemented/deployed systems which are compatible with the DICE IDCP:

– Internet2 ION (OSCARS/DCN)– ESnet SDN (OSCARS/DCN)– GÉANT AutoBHAN System– Nortel DRAC– Surfnet (via use of Nortel DRAC)– LHCNet (OSCARS/DCN)– Nysernet (New York RON) (OSCARS/DCN)– LEARN (Texas RON) (OSCARS/DCN)– LONI (OSCARS/DCN)– Northrop Grumman (OSCARS/DCN)– University of Amsterdam (OSCARS/DCN)– MAX (OSCARS/DCN)

• The following “higher level service applications” have adapted their existing systems to communicate using the DICE IDCP:

– LambdaStation (FNAL)– TeraPaths (BNL)– Phoebus (University of Delaware)

6

OSCARS Collaborative Research Efforts

• LBNL LDRD “On-demand overlays for scientific applications”– To create proof-of-concept on-demand overlays for scientific applications that

make efficient and effective use of the available network resources

• GLIF GNI-API “Fenius”– To translate between the GLIF common API to

• DICE IDCP: OSCARS IDC (ESnet, I2)• GNS-WSI3: G-lambda (KDDI, AIST, NICT, NTT)• Phosphorus: Harmony (PSNC, ADVA, CESNET, NXW, FHG, I2CAT, FZJ, HEL IBBT,

CTI, AIT, SARA, SURFnet, UNIBONN, UVA, UESSEX, ULEEDS, Nortel, MCNC, CRC)

• DOE Project “Virtualized Network Control”– To develop multi-dimensional PCE (multi-layer, multi-level, multi-technology,

multi-layer, multi-domain, multi-provider, multi-vendor, multi-policy)

• DOE Project “Integrating Storage Management with Dynamic Network Provisioning for Automated Data Transfers”– To develop algorithms for co-scheduling compute and network resources

• DOE Project “Hybrid Multi-Layer Network Control”– To develop end-to-end provisioning architectures and solutions for multi-layer

networks 7

OSCARS 0.6 Design / Implementation Goals

• Support production deployment of service, and facilitate research collaborations• Distinct functions in stand-alone modules

• Supports distributed model• Facilitates module redundancy

• Formalize (internal) interface between modules• Facilitates module plug-ins from collaborative work (e.g. PCE)• Customization of modules based on deployment needs (e.g.

AuthN, AuthZ, PSS)

• Standardize external API messages and control access• Facilitates inter-operability with other dynamic VC services (e.g.

Nortel DRAC, GÉANT AuthBAHN)• Supports backward compatibility of IDC protocol

8

OSCARS 0.6 Architecture (Target 3/10)

Notification Broker• Manage Subscriptions• Forward Notifications

AuthN• Authentication

Path Setup• Network Element

Interface

Coordinator• Workflow Coordinator

PCE• Constrained Path

Computations

Topology Bridge• Topology Information

Management

WS API• Manages External WS

Communications

Resource Manager• Manage Reservations

• Auditing

Lookup• Lookup service

AuthZ*• Authorization

• Costing

*Distinct Data and Control Plane Functions

Web Browser User Interface

50%

80%

50%

95%

50%

95%

20%

50%70%

90%

60%

9

OSCARS 0.6 PCE Features

• Creates a framework for multi-dimensional constrained path finding

• Plug-in architecture allowing external entities to implement PCE algorithms: PCE modules.

• Dynamic, Runtime: computation is done when creating/modifying a path.

• PCE modules organized as a graph (PCE, Aggregators)

• PCE modules uses OSCARS 0.6 new PCE framework providing API (SOAP) and language independent bindings.

10

OSCARS 0.6 Standard PCE’s

• OSCARS implements a set of default PCE modules (supporting existing OSCARS deployments)

• Default PCE modules are implemented using the PCE framework.

• Custom deployments may use, remove or replace default PCE modules.

• Custom deployments may customize the graph of PCE modules.

11

OSCARS 0.6 PCE Framework Workflow

12

Graph of PCE Modules And Aggregation

Aggregate Tags 3,4

Aggregate Tags 3,4

Aggregate Tags 1,2

Aggregate Tags 1,2

PCERuntime

PCE 1PCE 1

Tag 1Tag 1

PCE 3PCE 3

Tag 1Tag 1

PCE 2PCE 2

Tag 1Tag 1

PCE 4PCE 4

Tag 2Tag 2

PCE 5PCE 5

Tag 3Tag 3

PCE 6PCE 6

Tag 4Tag 4

PCE 7PCE 7

Tag 4Tag 4

User + PCE1 + PCE2 + PCE3 Constrains

(Tag=1)


(Tag=1)

User + PCE1 + PCE2 Constrains (Tag=1) User + PCE1 + PCE2 Constrains (Tag=1)

User + PCE1 Constrains (Tag=1)


User ConstrainsUser Constrains







(Tag=4)


(Tag=4)


User ConstrainsUser Constrains

*Constraints = Network Element Topology Data

Intersection of [Constrains (Tag=3)] and [Constraints

(Tag=4)] returned as Constraints (Tag =2)

Intersection of [Constrains (Tag=3)] and [Constraints

(Tag=4)] returned as Constraints (Tag =2)

13

Composable Network Services Framework

14

• Motivation– Typical users want better than best-effort service but are unable

to express their needs in network engineering terms– Advanced users want to customize their service based on

specific requirements– As new network services are deployed, they should be

integrated in to the existing service offerings in a cohesive and logical manner

• Goals– Abstract technology specific complexities from the user– Define atomic network services which are composable– Create customized service compositions for typical use cases

Atomic and Composite Network Services Architecture

15

Atomic Service (AS1)




Composite Service (S2 = AS1 + AS2)

Composite Service (S3 = AS3 + AS4)

Composite Service (S1 = S2 + S3)

Ser

vice

Abs

trac

tion

Incr

eas

es

Ser

vice

Usa

ge

Sim

plif

ies

Network Service Plane

Service templates pre-composed for specific applications or customized by advanced users

Atomic services used as building blocks for composite services

Network Services Interface

Multi-Layer Network Data Plane

Examples of Atomic Network Services

16

Scheduling resources to facilitate workflow pipelines

Security (e.g. encryption) to ensure data integrity

Measurement to enable collection of usage data and performance stats

Monitoring to ensure proper support using SOPs for production service

1+1

Store and Forward to enable caching capability in the network

Topology to determine resources and orientation

Path Finding to determine possible path(s) based on multi-dimensional constraints

Connection to specify data plane connectivity

Protection to enable resiliency through redundancy

Restoration to facilitate recovery

Examples of Composite Network Services

17

1+1

LHC: Resilient High Bandwidth Guaranteed Connection

Protocol Testing: Constrained Path Connection

Reduced RTT Transfers: Store and Forward Connection

Atomic Network Services Currently Offered by OSCARS

18

ESnet OSCARS

Network Services Interface

Multi-Layer Multi-Layer Network Data Plane

Scheduling of guaranteed bandwidth connections in granularity of minutes

Connection creates virtual circuits (VCs) within a domain as well as multi-domain end-to-end VCs

Path Finding determines a viable path based on time and bandwidth constrains

Monitoring provides critical VCs with production level support

Conclusion

Questions?

Comments?

19

Documents

OSCARS Roadmap OGF 28 Munich, Germany Mar 15, 2010