Dan NaeDan Nae

California Institute of Technology California Institute of Technology

US LHCNet UpdateUS LHCNet Update

DEN

ELP

ALB

ATL

Metropolitan Area Rings

SEA

Major DOE Office of Science Sites

SNV

MetropolitanAreaRings

10Gb/s

20-30-40-50 Gb/s

20-30-40 Gb/s

NYC

CHI

DC

SDNetwork core, 20-30-40-50 Gbps circuit based transportMajor international

Production IP ESnet core, 10 Gbps enterprise IP traffic

LHCNet Data Network

NSF/IRNC circuit; GVA-AMS connection via Surfnet or Geant2

HOU

TUL

KAN

SDG

LA

ESNet4Science Data Network Core

(20-50 Gbps)

ESnet4IP Core (≥10

Gbps) US LHCNet Data Network

(3x10Gbps to the US)

NYC

BNL

FNAL

CERN

AMS

US LHCNet + ESnet (2007)US LHCNet + ESnet (2007)

2007

Connections to ESnet Hubs in New-York and Chicago Redundant “light-paths” to BNL and FNAL Redundant 10 Gbps peering with Abilene Access to USNet/HOPI for R&D

2006

Four providers:Colt QwestGlobal Crossing GEANT

Atlantic Ocean

Multiple Fiber Paths: Multiple Fiber Paths: Reliability Through DiversityReliability Through Diversity

NYC 111 8th

Pottington (UK)

VSNL South

NY60 Hudson

Highbridge

VSNL North

AMS-SARAAC-2

Bude

GVA-CERN

FrankfurtVSNL

Wal, NJ

London

Global Crossing

Qwest

Colt

GEANT

NYC-MANLAN

CHI-Starlight

Paris

Bellport Whitesands

LCG Availability LCG Availability requirement: 99.95%requirement: 99.95%

Brookhaven

LONAC-1

EASTWEST

Additional SlidesAdditional Slides

Equipment DiversityEquipment Diversity

Single point of failure for the CERN-FNAL

traffic

US LHCNet, ESnet and the two US Tier1s (FNAL and BNL) are working to achieve complete equipment diversity for the primary and backup paths

Equipment Diversity (cont.)Equipment Diversity (cont.)

The new setup allows for independent paths and can survive the failure of any single piece of equipment

Great advantage in case of hardware of software maintenance Similar setup for the CERN-BNL connection

Ciena Transitions (Today)Ciena Transitions (Today)

Two parallel networks, one Force10 and one Ciena Today the main links (CERN-BNL, CERN-FNAL) go over the Force10s (proven

reliability, stable configuration) Circuit oriented services development for the Cienas

GEANT-ESnet Peering, FNAL-GridKa, BNL-

GridKa

Planned Configuration (2008)Planned Configuration (2008)

Robust fallback at layer 1 + next-generation hybrid optical network: Robust fallback at layer 1 + next-generation hybrid optical network: Dynamic Dynamic circuit-oriented network services with BW guarantees circuit-oriented network services with BW guarantees

Emerging Emerging StandardsStandards

VCAT, LCASVCAT, LCAS

Provisioned circuits over a failed SONET link can be re-routed according to priorities and preempt lower priority circuits.

Fallback is automatic and very fast

(<50ms once failure is detected)

Ciena “Mesh Restoration” of a CircuitCiena “Mesh Restoration” of a Circuit

Network ForecastNetwork Forecast

Ports at Each PoP 10 GbE Ports OC-192/STM-64 Ports

AMS 2 7

GVA 10 9

NYC 4 16

CHI 4 12

Geneva

Chicago

4 x 10GBE

4 x 10GBE

10 x 10GBE

4 x 10G SDH

ESnet SDN

3 x 10G SDH

New York

4 x 10G SDH

Amsterdam

2 x 10GBE

3 x 10G SDH

Circuit-enabled regional network

Circuit-enabled regional network

Geneva

Chicago

New York

4 x 10GBE

4 x 10GBE

8 x 10GBE

3 x 10G SDH

3 x 10G SDH

ESnet SDN

2 x 10G SDH 2 x 10G SDH

Amsterdam

2 x 10GBE

Circuit-enabled regional network Circuit-enabled

regional network

Ports at Each PoP 10 GbE Ports OC-192/STM-64 Ports

AMS 2 5

GVA 8 7

NYC 4 12

CHI 4 10

2009 2010

Double density cards and matrix

40 Gbps wavelengths?

Two port 10 GbE card

LHCNet connections to ESnet: FY09/FY10LHCNet connections to ESnet: FY09/FY10

NLR wavegear sites

NLR regeneration / OADM sites

ESnet via NLR (10 Gbps waves)LHCNet (10 Gbps waves)

Denver

Seattle

Su

nn

yva

le

LA

San Diego

Chicago

PittsWash DC

Raleigh

Jacksonville

Atlanta

KC

Baton Rouge

El Paso - Las Cruces

Phoenix

Pensacola

Dallas

San Ant.Houston

Albuq. Tulsa

New YorkClev

Boise

CERN (Geneva)

LHCNet: To ~80 Gbps by 2009-10Routing + Dynamic managed

circuit provisioning

Network MonitoringNetwork Monitoring

MonALISA (TL1 Module) Spectrum (CERN first line) Various open source tools

(cricket, nagios, rancid, syslog-ng, etc)

perfSONAR (GEANT E2ECU)

True end-to-end (host-to-host) monitoring using MonALISA

“Network intelligence” or the ability to reconfigure the circuits based on performance, changing network conditions or high-priority scheduled transfers

US LHCNet Working Methods US LHCNet Working Methods

Production NetworkDevelop and build next generation networks

Networks for ResearchD0, CDF, BaBar, CMS, Atlas

GRID applications PPDG/iVDGL, OSG, WLCG,

DISUN

LHCOPN

Interconnection of US and EU Grid domains

VRVS/EVO

High performanceHigh bandwidthReliable network

HEP & DoE Roadmaps

Testbed for Grid Development

Pre-ProductionN x 10 Gbps transatlantic testbed

New Data transport protocols

Interface and kernel setting

HOPI / UltraScience Net / Ultralight / CHEPREO /

LambdaSation

Lightpath technologies

Vendor Partnerships

Four Continent Testbed and FacilityFour Continent Testbed and Facility

Building a global, network-aware end-to-end managed real-time Grid

Caltech, Florida, FIU, UMich,

SLAC,FNAL, CERN, Internet2, NLR, UERJ(Rio),

USP, CENIC, Starlight, Cisco

http://ultralight.caltech.eduhttp://ultralight.caltech.eduhttp://ultralight.caltech.eduhttp://ultralight.caltech.edu

Network Services for Managed Network Services for Managed End-to-End Data TransfersEnd-to-End Data Transfers

Robust Network Services based on

Bandwidth guarantees Virtual Circuits

Scheduled Transfers Transfer Classes Priorities

Monitoring of all components end-to-end Network Elements End-Hosts

Interface to other circuit-oriented systems Be part of

heterogeneous end-to-end infrastructure

Problem Finding and Resolution Problem Finding and Resolution Problems encountered today are

hard to track due to missing the global view of the system

Example situation: the system recognizes an end-host problem during the transfer and takes mitigating actions, re-scheduling transfers and notifying operators

Developed in the field-proven MonALISA Framework

End-to-end MonitoredManaged Transfers

Track problem to the source Network / End-host

Take appropriate action Change transfer path Adjust end-host parameters Re-schedule transfer

Provide experts with relevant (real-time) information Keep the user/application up-to-date on transfer progress Progessive automation: Target

optimal resource utilization

Tom Lehman, GLIF 2007 Winter Workshop

http://www.glif.is/meetings/2007/winter/controlplane/lehman-dynamic-services.pdf

GLIFConcep

t

US LHCNet MilestonesUS LHCNet Milestones

““Pre-production”:Pre-production”: The new infrastructure initially deployed The new infrastructure initially deployed 2007 will offer circuit-based services intended to provide 2007 will offer circuit-based services intended to provide redundant paths and on-demand, high bandwidth end-to-end redundant paths and on-demand, high bandwidth end-to-end dedicated circuits. Circuit-switched services will be used to dedicated circuits. Circuit-switched services will be used to directly interconnect the DOE laboratories to CERN and will be directly interconnect the DOE laboratories to CERN and will be available on demand to policy-driven, data-intensive available on demand to policy-driven, data-intensive applications, managed by MonALISA servicesapplications, managed by MonALISA services

End of 2007:End of 2007: initial deployment of our circuit oriented network initial deployment of our circuit oriented network services on US LHCNet; simple scheduler with fixed bandwidth services on US LHCNet; simple scheduler with fixed bandwidth circuits for site to site on-demand data set transfers.circuits for site to site on-demand data set transfers.

Spring 2008:Spring 2008: interaction with the data transfer application of the interaction with the data transfer application of the experiments, as well as with other intra-domain and inter-experiments, as well as with other intra-domain and inter-domain (LambdaStation, TeraPaths, DRAGON, Oscars) control domain (LambdaStation, TeraPaths, DRAGON, Oscars) control plane services in order to provide an end-to-end path plane services in order to provide an end-to-end path reservation.reservation.

LHC Startup:LHC Startup: July 2008: July 2008: We will begin to exercise the network We will begin to exercise the network and services with real data, in close cooperation with the LHC and services with real data, in close cooperation with the LHC experiments. experiments.