Shawn McKee University of Michigan University of Michigan UltraLight: A Managed Network Infrastructure for HEP CHEP06, Mumbai, India February 14, 2006

Shawn McKeeShawn McKee University of MichiganUniversity of Michigan

UltraLight: A Managed Network UltraLight: A Managed Network Infrastructure for HEPInfrastructure for HEP

CHEP06, Mumbai, IndiaCHEP06, Mumbai, IndiaFebruary 14, 2006February 14, 2006

S. McKee, UM - CHEP06 2

IntroductionIntroduction

The UltraLight Collaboration was formed to The UltraLight Collaboration was formed to develop the network as a managed develop the network as a managed component of our infrastructure for High-component of our infrastructure for High-Energy PhysicsEnergy Physics

(See Rick Cavanaugh’s overview from yesterday)(See Rick Cavanaugh’s overview from yesterday)

Given a funding level at 50% of our request we Given a funding level at 50% of our request we had to reduce our scope while maintaining had to reduce our scope while maintaining the core network infrastucture. the core network infrastucture.

This talk will focus on the UltraLight network and This talk will focus on the UltraLight network and the activities of the UL network group to the activities of the UL network group to enable effective use of 10 Gbps networks.enable effective use of 10 Gbps networks.


OverviewOverview

Overview of UltraLight Network EffortOverview of UltraLight Network Effort

Network Engineering Plans and StatusNetwork Engineering Plans and Status

MonitoringMonitoring

Kernel/End-host Development Kernel/End-host Development

Status and SummaryStatus and Summary


S. McKeeS. McKee ( (UM,UM, Team LeaderTeam Leader))

S. Ravot (S. Ravot (LHCNetLHCNet))

D. Nae (D. Nae (LHCNetLHCNet))

R. Summerhill (R. Summerhill (Abilene/HOPIAbilene/HOPI))

D. Pokorney (D. Pokorney (FLRFLR))

S. Gerstenberger (S. Gerstenberger (MiLRMiLR))

C. Griffin (C. Griffin (UFUF))

S. Bradley (S. Bradley (BNLBNL))

J. Bigrow (J. Bigrow (BNLBNL) )

The UltraLight Network The UltraLight Network Engineering TeamEngineering Team

J. Ibarra (J. Ibarra (WHREN, AWWHREN, AW))

C. Guok (C. Guok (ESnetESnet))

L. Cottrell (L. Cottrell (SLACSLAC))

C. Heerman (C. Heerman (I2/HOPII2/HOPI))

D. Petravick (D. Petravick (FNALFNAL) )

M. Crawford (M. Crawford (FNALFNAL))

R. Hockett (R. Hockett (UMUM))

E. Rubi (E. Rubi (FIUFIU))

Many individuals have worked deploying and maintaining the UltraLight Network


UltraLight BackboneUltraLight Backbone

UltraLight has a UltraLight has a non-standard core networknon-standard core network with dynamic with dynamic links and varying bandwidth inter-connecting our nodes. links and varying bandwidth inter-connecting our nodes. Optical Hybrid Global NetworkOptical Hybrid Global Network

The core of UltraLight is dynamically evolving as function of The core of UltraLight is dynamically evolving as function of available resources on other backbones such as NLR, available resources on other backbones such as NLR, HOPI, Abilene and ESnet. HOPI, Abilene and ESnet.

The main resources for UltraLight:The main resources for UltraLight: LHCnet (IP, L2VPN, CCC)LHCnet (IP, L2VPN, CCC) Abilene (IP, L2VPN)Abilene (IP, L2VPN) ESnet (IP, L2VPN)ESnet (IP, L2VPN) UltraScienceNet (L2)UltraScienceNet (L2) Cisco NLR wave (Ethernet) Cisco NLR wave (Ethernet) Cisco Research Service (L3)Cisco Research Service (L3) HOPI NLR waves (Ethernet; provisioned on demand)HOPI NLR waves (Ethernet; provisioned on demand) UltraLight nodes: Caltech, SLAC, FNAL, UF, UM, UltraLight nodes: Caltech, SLAC, FNAL, UF, UM,

StarLight, CENIC PoP at LA, CERN, StarLight, CENIC PoP at LA, CERN, SeattleSeattle


UltraLight Network UltraLight Network Infrastructure ElementsInfrastructure Elements

Trans-US 10G Trans-US 10G s Riding on NLR, Plus CENIC, FLR, MiLRs Riding on NLR, Plus CENIC, FLR, MiLR LA – CHI (4 Waves): HOPI (2 Waves), LA – CHI (4 Waves): HOPI (2 Waves), USNUSN, and Cisco , and Cisco

ResearchResearch CHI – JAX (Florida Lambda Rail/NLR)CHI – JAX (Florida Lambda Rail/NLR)

Dark Fiber Caltech – L.A.: 2 X 10G Waves (One to WAN In Lab);Dark Fiber Caltech – L.A.: 2 X 10G Waves (One to WAN In Lab); 10G Wave L.A. to Sunnyvale for UltraScience Net Connection 10G Wave L.A. to Sunnyvale for UltraScience Net Connection Dark Fiber with 10G Waves (Dark Fiber with 10G Waves (22 Waves): StarLight – Fermilab Waves): StarLight – Fermilab Dedicated Wave StarLight (1 + Dedicated Wave StarLight (1 + 2 Waves2 Waves) – Michigan Light Rail) – Michigan Light Rail SLAC: ESnet MAN to Provide Links (from July): SLAC: ESnet MAN to Provide Links (from July): One for Production, and One for Research One for Production, and One for Research Partner with Advanced Research & Production NetworksPartner with Advanced Research & Production Networks

LHCNet (Starlight- CERN), Abilene/HOPI, ESnet, NetherLight, LHCNet (Starlight- CERN), Abilene/HOPI, ESnet, NetherLight, GLIF, UKLight, CA*net4GLIF, UKLight, CA*net4

Intercont’l extensions: Brazil (CHEPREO/WHREN), GLORIAD, Intercont’l extensions: Brazil (CHEPREO/WHREN), GLORIAD, Tokyo, AARNet, Taiwan, ChinaTokyo, AARNet, Taiwan, China


UltraLight Network UltraLight Network Infrastructure ElementsInfrastructure Elements

Trans-US 10G Trans-US 10G s Riding on NLR, Plus CENIC, FLR, MiLRs Riding on NLR, Plus CENIC, FLR, MiLR LA – CHI (4 Waves): HOPI (2 Waves), LA – CHI (4 Waves): HOPI (2 Waves), USNUSN, and Cisco , and Cisco

ResearchResearch CHI – JAX (Florida Lambda Rail/NLR)CHI – JAX (Florida Lambda Rail/NLR)

Dark Fiber Caltech – L.A.: 2 X 10G Waves (One to WAN In Lab);Dark Fiber Caltech – L.A.: 2 X 10G Waves (One to WAN In Lab); 10G Wave L.A. to Sunnyvale for UltraScience Net Connection 10G Wave L.A. to Sunnyvale for UltraScience Net Connection Dark Fiber with 10G Waves (Dark Fiber with 10G Waves (22 Waves): StarLight – Fermilab Waves): StarLight – Fermilab Dedicated Wave StarLight (1 + Dedicated Wave StarLight (1 + 2 Waves2 Waves) – Michigan Light Rail) – Michigan Light Rail SLAC: ESnet MAN to Provide Links (from July): SLAC: ESnet MAN to Provide Links (from July): One for Production, and One for Research One for Production, and One for Research Partner with Advanced Research & Production NetworksPartner with Advanced Research & Production Networks

LHCNet (Starlight- CERN), Abilene/HOPI, ESnet, NetherLight, LHCNet (Starlight- CERN), Abilene/HOPI, ESnet, NetherLight, GLIF, UKLight, CA*net4GLIF, UKLight, CA*net4

Intercont’l extensions: Brazil (CHEPREO/WHREN), GLORIAD, Intercont’l extensions: Brazil (CHEPREO/WHREN), GLORIAD, Tokyo, AARNet, Taiwan, ChinaTokyo, AARNet, Taiwan, China


UltraLight Points-of-PresenceUltraLight Points-of-Presence

StarLight (Chicago)StarLight (Chicago)

HOPI (2 x 10GE), USNet (2 x 10GE), NLR (4 x 10GE)HOPI (2 x 10GE), USNet (2 x 10GE), NLR (4 x 10GE)

UM (3 x 10GE), TeraGrid, ESnet, AbileneUM (3 x 10GE), TeraGrid, ESnet, Abilene

FNAL, US-LHCNet (2 x 10GE), TIFR (FNAL, US-LHCNet (2 x 10GE), TIFR (MumbaiMumbai) )

MANLAN (New York)MANLAN (New York)

HOPI (2 x 10GE), US-LHCNet (2 x 10GE), BNL,HOPI (2 x 10GE), US-LHCNet (2 x 10GE), BNL,

Buffalo (2 x 10GE), Cornell, NevisBuffalo (2 x 10GE), Cornell, Nevis

SeattleSeattle

GLORIAD, JGN2, Pwave, NLR (2 x 10GE)GLORIAD, JGN2, Pwave, NLR (2 x 10GE)

CENIC (Los-Angeles)CENIC (Los-Angeles)

HOPI (2 x 10GE), NLR (4 x 10GE)HOPI (2 x 10GE), NLR (4 x 10GE)

Caltech (2 x 10GE), PwaveCaltech (2 x 10GE), Pwave

Level3 (Sunnyvale)Level3 (Sunnyvale)

USNet (2 x 10GE), NLR, SLACUSNet (2 x 10GE), NLR, SLAC


International PartnersInternational Partners

One of the UltraLight program’s strengths is the large One of the UltraLight program’s strengths is the large number of important international partners we have:number of important international partners we have:

UltraLight is well positioned to develop and coordinate UltraLight is well positioned to develop and coordinate globalglobal advances to networks for LHC Physics advances to networks for LHC Physics


• Global ServicesGlobal Services support management / co-scheduling of support management / co-scheduling of multiple resource types with a strategic end-to-end view.multiple resource types with a strategic end-to-end view.

Provide strategic recovery mechanisms from system failuresProvide strategic recovery mechanisms from system failuresSchedule decisions based on CPU, I/O, Network capability and Schedule decisions based on CPU, I/O, Network capability and

End-to-end task performance estimatesEnd-to-end task performance estimates, incl. loading effects, incl. loading effectsConstrained by Constrained by local and global policieslocal and global policies

• Global Services Consist of: Global Services Consist of: Network and System Resource MonitoringNetwork and System Resource Monitoring Network Path Discovery and Construction ServicesNetwork Path Discovery and Construction Services Policy Based Job Planning ServicesPolicy Based Job Planning Services Task Execution ServicesTask Execution Services

• These types of services are These types of services are requiredrequired to deliver a managed to deliver a managed network. network.

UltraLight Global ServicesUltraLight Global Services

See “VINCI” talk on Thursday


GOAL:GOAL: Determine an effective mix of bandwidth- Determine an effective mix of bandwidth-management techniques for this application-space, management techniques for this application-space, particularly:particularly:

Best-effort and “scavenger” using Best-effort and “scavenger” using “effective” protocols“effective” protocolsMPLSMPLS with with QOS-enabled QOS-enabled packet switchingpacket switchingDedicated pathsDedicated paths arranged with TL1 commands, GMPLS arranged with TL1 commands, GMPLS

PLAN: PLAN: Develop, Test the most cost-effective integrated Develop, Test the most cost-effective integrated combination of network technologies on our unique combination of network technologies on our unique testbed:testbed:

1.1.Exercise UltraLight Exercise UltraLight applications applications on NLR, Abilene and campus on NLR, Abilene and campus networks, as well as LHCNet, and our international partnersnetworks, as well as LHCNet, and our international partners

2.2.Deploy and systematically study Deploy and systematically study ultrascale protocolultrascale protocol stacks stacks (such as FAST) addressing issues of performance & fairness (such as FAST) addressing issues of performance & fairness

3.3.Use MPLS/QoS and other forms of Use MPLS/QoS and other forms of BW managementBW management, to optimize , to optimize end-to-end performance among a set of virtualized disk servers end-to-end performance among a set of virtualized disk servers

4.4.Address Address “end-to-end” issues“end-to-end” issues, including monitoring and end-hosts, including monitoring and end-hosts

UltraLight Network EngineeringUltraLight Network Engineering


UltraLight: Effective ProtocolsUltraLight: Effective Protocols

The The protocolsprotocols used to reliably move data are a critical used to reliably move data are a critical component of Physics “end-to-end” use of the networkcomponent of Physics “end-to-end” use of the networkTCP is the most widely used protocol for reliable data TCP is the most widely used protocol for reliable data transport, but is becoming ever more ineffective for transport, but is becoming ever more ineffective for higher and higher bandwidth-delay networks.higher and higher bandwidth-delay networks.UltraLight is exploring extensions to TCP (UltraLight is exploring extensions to TCP (HSTCP, HSTCP, Westwood+, HTCPWestwood+, HTCP, , FAST, MaxNetFAST, MaxNet) designed to maintain ) designed to maintain fair-sharing of networks and, at the same time, to allow fair-sharing of networks and, at the same time, to allow efficient, effective use of these networks.efficient, effective use of these networks.We identified a need to provide an “UltraLight” kernel to We identified a need to provide an “UltraLight” kernel to make protocol testing easy among the UltraLight sites.make protocol testing easy among the UltraLight sites.

UltraLight plans to identify the most effective fair UltraLight plans to identify the most effective fair protocol and implement it in support of our “Best Effort” protocol and implement it in support of our “Best Effort” network components.network components.


FAST

others

Gigabit WANGigabit WAN

5x higher utilization5x higher utilization

Small delaySmall delay

FAST: 95%

Reno: 19%

Random packet lossRandom packet loss

10x higher thruput10x higher thruput

Resilient to random lossResilient to random loss

FAST Protocol ComparisonsFAST Protocol Comparisons


MPLS/QoS for UltraLightMPLS/QoS for UltraLight

UltraLight plans to explore the full range of UltraLight plans to explore the full range of end-to-end end-to-end connectionsconnections across the network, from across the network, from best-effortbest-effort, , packet-packet-switchedswitched through through dedicated end-to-end light-pathsdedicated end-to-end light-paths. . MPLS paths with QoSMPLS paths with QoS attributesattributes fill a middle ground in fill a middle ground in this network space and allow fine-grained allocation of this network space and allow fine-grained allocation of virtual pipes, sized to the needs of the application or virtual pipes, sized to the needs of the application or user.user.

UltraLight, in conjunction UltraLight, in conjunction with the DoE/MICS with the DoE/MICS funded TeraPaths and funded TeraPaths and OSCARS effort, is OSCARS effort, is working toward working toward extensible solutions for extensible solutions for implementing such implementing such capabilities in next capabilities in next generation networksgeneration networks

TeraPaths SC|05 QoS Data Movement BNL→UM

See “Terapaths” talk in previous session today


Optical Path DevelopmentsOptical Path Developments

Emerging “light path” technologies are arriving:Emerging “light path” technologies are arriving: They can extend and augment existing grid computing They can extend and augment existing grid computing

infrastructures, currently focused on CPU/storage, to include the infrastructures, currently focused on CPU/storage, to include the network as an integral Grid component. network as an integral Grid component.

Those technologies seem to be the most effective way to offer Those technologies seem to be the most effective way to offer network resource provisioning on-demand between end-network resource provisioning on-demand between end-systems.systems.

We have developed a multi-agent system for secure light path We have developed a multi-agent system for secure light path provisioning based on dynamic discovery of the topology in distributed provisioning based on dynamic discovery of the topology in distributed networks. [networks. [See VINCI talk on Thursday, Feb. 16See VINCI talk on Thursday, Feb. 16]]

We are working to further develop this distributed agent system and to We are working to further develop this distributed agent system and to provide integrated network services capable to efficiently use and provide integrated network services capable to efficiently use and coordinate shared, hybrid networks and to improve the performance and coordinate shared, hybrid networks and to improve the performance and throughput for data intensive grid applications. throughput for data intensive grid applications.

This includes services able to dynamically configure routers and to This includes services able to dynamically configure routers and to aggregate local traffic on dynamically created optical connections.aggregate local traffic on dynamically created optical connections.


Realtime end-to-endRealtime end-to-end Network monitoring is Network monitoring is essentialessential for UltraLight. for UltraLight. You can’t manage what you can’t see!You can’t manage what you can’t see!

We need to understand our network infrastructure and We need to understand our network infrastructure and track its performance both historically and in real-time track its performance both historically and in real-time to enable the network as a managed robust component to enable the network as a managed robust component of our infrastructure. of our infrastructure.

MonALISA MonALISA http://monalisa.cern.chhttp://monalisa.cern.ch

IEPM IEPM http://www-http://www-iepm.slac.stanford.edu/bwiepm.slac.stanford.edu/bw//

We have a new effort to push monitoring to the “ends” We have a new effort to push monitoring to the “ends” of the network: the hosts involved in providing of the network: the hosts involved in providing services or user workstations.services or user workstations.

Monitoring for UltraLightMonitoring for UltraLight

http://monalisa.cern.ch/

http://www-iepm.slac.stanford.edu/bw/




MonALISA UltraLight RepositoryMonALISA UltraLight Repository

The UL repository: The UL repository: http://monalisa-ul.caltech.edu:8080/http://monalisa-ul.caltech.edu:8080/

http://monalisa-ul.caltech.edu:8080/


Host Monitoring with UltraLightHost Monitoring with UltraLight

Many “network” problems are actually endhost problems: Many “network” problems are actually endhost problems: misconfigured or underpowered end-systemsmisconfigured or underpowered end-systems

The LISA application (see Iosif’s talk later) was designed to The LISA application (see Iosif’s talk later) was designed to monitor the endhost and its view of the network.monitor the endhost and its view of the network.

For SC|05 we developed a Perl script to gather the relevant For SC|05 we developed a Perl script to gather the relevant host details related to network performance and host details related to network performance and integrated the script with ApMon (an API for MonALISA) integrated the script with ApMon (an API for MonALISA) to allow us to “publish” this data to a MonALISA to allow us to “publish” this data to a MonALISA repository.repository.

Information on the system information, TCP configuration Information on the system information, TCP configuration and network device setup was gathered and accessible and network device setup was gathered and accessible from one site.from one site.

Future plans are to coordinate this with LISA and deploy this Future plans are to coordinate this with LISA and deploy this as part of OSG. The Tier-2 centers are a primary target.as part of OSG. The Tier-2 centers are a primary target.

Network Device InformationTCP SettingsHost/System Information


UltraLight KernelUltraLight Kernel

Kernels and the associated device drivers are critical to the achievable performance of hardware and software.

The FAST protocol implementation for Linux requires a modified kernel to work. .

We learned to deal with many pitfalls in the configuration and varying versions of linux kernels, particularly how they impact the performance of the system on the network. Currently we are working on a new version based upon 2.6.15.3. Will include FAST, NFSv4, newest drivers for 10GE NICs and RAID cards.

Because of the need to have FAST easily available, and the desire to achieve the best performance possible, we plan for all UltraLight “hosts” to install and utilize this Kernel.


End-Systems performance End-Systems performance

Latest disk to disk over 10Gbps WAN: Latest disk to disk over 10Gbps WAN: 4.3 Gbits/sec (536 MB/sec)4.3 Gbits/sec (536 MB/sec) - 8 TCP - 8 TCP streams from CERN to Caltech; windows, 1TB file, 24 JBOD disksstreams from CERN to Caltech; windows, 1TB file, 24 JBOD disks

Quad Opteron AMD848 2.2GHz processors with 3 AMD-8131 chipsets: 4 64-Quad Opteron AMD848 2.2GHz processors with 3 AMD-8131 chipsets: 4 64-bit/133MHz PCI-X slots.bit/133MHz PCI-X slots.

3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA disks3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA disks Local Disk IO – Local Disk IO – 9.6 Gbits/sec9.6 Gbits/sec (1.2 GBytes/sec read/write, with <20% (1.2 GBytes/sec read/write, with <20%

CPU utilization)CPU utilization)10GE NIC10GE NIC

10 GE NIC – 9.310 GE NIC – 9.3 Gbits/sec Gbits/sec (memory-to-memory, with 52% CPU (memory-to-memory, with 52% CPU utilization, PCI-X 2.0 Caltech-Starlight)utilization, PCI-X 2.0 Caltech-Starlight)

2*10 GE NIC (802.3ad link aggregation) – 2*10 GE NIC (802.3ad link aggregation) – 11.1 Gbits/sec11.1 Gbits/sec (mem-2-mem) (mem-2-mem) Need PCI-Express?, TCP offload enginesNeed PCI-Express?, TCP offload engines Use 64 bit OS? Which architectures and hardware?Use 64 bit OS? Which architectures and hardware?

GOAL: Single server to server at 1 GByte/secGOAL: Single server to server at 1 GByte/secDiscussions are underway with 3Ware, Myricom to try to prototype viable Discussions are underway with 3Ware, Myricom to try to prototype viable

servers capable of driving 10 GE networks in the WAN.servers capable of driving 10 GE networks in the WAN.Targeting Spring 2006Targeting Spring 2006


UltraLight Network SummaryUltraLight Network Summary

The network technical group has been hard at work on The network technical group has been hard at work on implementing UltraLight. Network and basic services are implementing UltraLight. Network and basic services are deployed and operating.deployed and operating.

Significant progress has been made in Significant progress has been made in monitoringmonitoring, , kernelskernels, , prototype servicesprototype services and and disk-to-disk transfersdisk-to-disk transfers in the WAN. in the WAN.

Our global collaborators are working with us on achieving Our global collaborators are working with us on achieving the UltraLight vision.the UltraLight vision.

2006 will be busy, 2006 will be busy, criticalcritical (for LHC) and productive year (for LHC) and productive year working on UltraLight!working on UltraLight!

Documents

Shawn McKee University of Michigan University of Michigan UltraLight: A Managed Network Infrastructure for HEP CHEP06, Mumbai, India February 14, 2006