Embed Size (px)
Citation preview
Shawn McKee / University of MichiganShawn McKee / University of MichiganUSATLAS Tier1 & Tier2 Network Planning MeetingUSATLAS Tier1 & Tier2 Network Planning Meeting
December 14, 2005 - BNLDecember 14, 2005 - BNL
UltraLight OverviewUltraLight Overview
The UltraLight ProjectThe UltraLight Project
UltraLight isUltraLight is A four year $2M NSF ITR funded by MPS.A four year $2M NSF ITR funded by MPS. Application driven Network R&D.Application driven Network R&D. A collaboration of A collaboration of BNL, Caltech, CERN, Florida, FIU, BNL, Caltech, CERN, Florida, FIU,
FNAL, Internet2, Michigan, MIT, SLAC.FNAL, Internet2, Michigan, MIT, SLAC. Significant international participation: Brazil, Japan, Significant international participation: Brazil, Japan,
Korea amongst many others.Korea amongst many others.
Goal:Goal: Enable the network as a managed resource. Enable the network as a managed resource.
Meta-Goal:Meta-Goal: Enable physics analysis and discoveries Enable physics analysis and discoveries which could not otherwise be achieved.which could not otherwise be achieved.
UltraLight BackboneUltraLight Backbone
UltraLight has a non-standard core network with dynamic UltraLight has a non-standard core network with dynamic links and varying bandwidth inter-connecting our nodes. links and varying bandwidth inter-connecting our nodes. Optical Hybrid Global NetworkOptical Hybrid Global Network
The core of UltraLight is dynamically evolving as function of The core of UltraLight is dynamically evolving as function of available resources on other backbones such as NLR, available resources on other backbones such as NLR, HOPI, Abilene or ESnet. HOPI, Abilene or ESnet.
The main resources for UltraLight:The main resources for UltraLight: LHCnet (IP, L2VPN, CCC)LHCnet (IP, L2VPN, CCC) Abilene (IP, L2VPN)Abilene (IP, L2VPN) ESnet (IP, L2VPN)ESnet (IP, L2VPN) Cisco NLR wave (Ethernet)Cisco NLR wave (Ethernet) Cisco Layer 3 10GE NetworkCisco Layer 3 10GE Network HOPI NLR waves (Ethernet; provisioned on demand)HOPI NLR waves (Ethernet; provisioned on demand) UltraLight nodes: Caltech, SLAC, FNAL, UF, UM, UltraLight nodes: Caltech, SLAC, FNAL, UF, UM,
StarLight, CENIC PoP at LA, CERN StarLight, CENIC PoP at LA, CERN
UltraLight Layer3 ConnectivityUltraLight Layer3 Connectivity
Shown (courtesy of Dan Nae) Shown (courtesy of Dan Nae) is the current UltraLight is the current UltraLight Layer 3 connectivity as of Layer 3 connectivity as of Mid-October 2005Mid-October 2005
UltraLight Network UsageUltraLight Network Usage
UltraLight SitesUltraLight Sites
UltraLight currently has 10 UltraLight currently has 10 participating core sites (shown participating core sites (shown alphabetically)alphabetically)
Details and diagrams for each Details and diagrams for each site will be reported Tuesday site will be reported Tuesday during “Network” dayduring “Network” day
SiteSite MonitorMonitor TypeType StorageStorage Out of Out of BandBand
BNLBNL MonalisaMonalisa OC48, OC48,
10 GE 0610 GE 06
TBDTBD YY
CaltechCaltech MonalisaMonalisa 10 GE10 GE 1 TB 1 TB YY
CERNCERN MonalisaMonalisa OC192OC192 9 TB9 TB YY
FIUFIU MonalisaMonalisa 10 GE10 GE TBDTBD YY
FNALFNAL ?? 10 GE10 GE TBDTBD YY
I2I2 ?? MPLSMPLS
L2VPNL2VPN
TBDTBD YY
MITMIT MonalisaMonalisa OC48 OC48 TBDTBD YY
SLACSLAC MonalisaMonalisa
IEPMIEPM
10 GE10 GE TBDTBD YY
UFUF ?? 10 GE10 GE 1 TB 1 TB YY
UMUM MonalisaMonalisa 10 GE10 GE 9 TB9 TB YY
Implementation via “sharing” Implementation via “sharing” with HOPI/NLRwith HOPI/NLR
Also LA-CHI Cisco/NLR Also LA-CHI Cisco/NLR Research WaveResearch Wave
DOE UltraScienceNet Wave DOE UltraScienceNet Wave SNV-CHI SNV-CHI (LambdaStation)(LambdaStation)
Connectivity to FLR to Connectivity to FLR to be determined be determined
MIT involvement welcome, but MIT involvement welcome, but unfundedunfunded
AMPATHUERJ, USP
UltraLight Network: UltraLight Network: PHASE IPHASE I
Plans for Phase I from Oct. 2004
Move toward multiple Move toward multiple “lambdas”“lambdas”
Bring in FLR, as well Bring in FLR, as well as BNL (and MIT)as BNL (and MIT)
General comment: We are General comment: We are almost here!almost here!
AMPATHUERJ, USP
UltraLight Network: UltraLight Network: PHASE IIPHASE II
Move into productionMove into production
Optical switching fully Optical switching fully enabled amongst enabled amongst primary sitesprimary sites
Integrated international Integrated international infrastructure infrastructure
Certainly reasonable Certainly reasonable sometime in the next sometime in the next few years…few years…
AMPATHUERJ, USP
UltraLight Network: UltraLight Network: PHASE IIIPHASE III
UltraLight envisions a 4 year program to deliver a new, UltraLight envisions a 4 year program to deliver a new, high-performance, network-integrated infrastructure:high-performance, network-integrated infrastructure:
Phase I will last Phase I will last 12 months12 months and focus on and focus on deploying the initial deploying the initial network infrastructurenetwork infrastructure and and bringing up first services bringing up first services
Phase II will last Phase II will last 18 months18 months and concentrate on and concentrate on implementing all the needed servicesimplementing all the needed services and and extending the extending the infrastructure to additional sites infrastructure to additional sites
Phase III will complete UltraLight and last Phase III will complete UltraLight and last 18 months18 months. The . The focus will be on a focus will be on a transition to production in support of transition to production in support of LHC Physics (+ eVLBI Astronomy, +<insert your e-LHC Physics (+ eVLBI Astronomy, +<insert your e-Science here>?)Science here>?)
Workplan/Phased DeploymentWorkplan/Phased Deployment
We are HERE!
GOAL:GOAL: Determine an effective mix of bandwidth-management Determine an effective mix of bandwidth-management techniques for this application-space, particularly:techniques for this application-space, particularly:
Best-effort/“scavenger” using effective ultrascale protocolsBest-effort/“scavenger” using effective ultrascale protocolsMPLS with QOS-enabled packet switchingMPLS with QOS-enabled packet switchingDedicated paths arranged with TL1 commands, GMPLSDedicated paths arranged with TL1 commands, GMPLS
PLAN: PLAN: Develop, Test the most cost-effective integrated combination Develop, Test the most cost-effective integrated combination of network technologies on our unique testbed:of network technologies on our unique testbed:
1.1.Exercise Exercise UltraLight applicationsUltraLight applications on NLR, Abilene and campus on NLR, Abilene and campus networks, as well as LHCNet, and our international partnersnetworks, as well as LHCNet, and our international partners
Progressively enhance Abilene with QOS support to Progressively enhance Abilene with QOS support to protect production traffic protect production traffic
Incorporate emerging NLR and RON-based lightpath Incorporate emerging NLR and RON-based lightpath and and lambda facilitieslambda facilities
2.2.Deploy and systematically study ultrascale protocol stacksDeploy and systematically study ultrascale protocol stacks (such as FAST) addressing issues of performance & fairness (such as FAST) addressing issues of performance & fairness
3.3.Use MPLS/QoS and other forms of BW management, and Use MPLS/QoS and other forms of BW management, and adjustments of optical paths, to optimize end-to-end adjustments of optical paths, to optimize end-to-end performance performance among a set of virtualized disk serversamong a set of virtualized disk servers
UltraLight Network EngineeringUltraLight Network Engineering
UltraLight: Effective ProtocolsUltraLight: Effective Protocols
The The protocolsprotocols used to reliably move data are a critical used to reliably move data are a critical component of Physics “end-to-end” use of the networkcomponent of Physics “end-to-end” use of the network
TCP is the most widely used protocol for reliable data TCP is the most widely used protocol for reliable data transport, but is becoming ever more ineffective for transport, but is becoming ever more ineffective for higher and higher bandwidth-delay networks.higher and higher bandwidth-delay networks.
UltraLight is exploring extensions to TCP (UltraLight is exploring extensions to TCP (HSTCP, HSTCP, Westwood+, HTCPWestwood+, HTCP, , FASTFAST) designed to maintain fair-) designed to maintain fair-sharing of networks and, at the same time, to allow sharing of networks and, at the same time, to allow efficient, effective use of these networks.efficient, effective use of these networks.
Currently FAST is in our “UltraLight Kernel” (a Currently FAST is in our “UltraLight Kernel” (a customized 2.6.12-3 kernel). This was used in SC2005. customized 2.6.12-3 kernel). This was used in SC2005. We are planning to broadly deploy a related kernel with We are planning to broadly deploy a related kernel with FAST. Longer term we can then continue with access to FAST. Longer term we can then continue with access to FAST, HS-TCP, Scalable TCP, BIC and others.FAST, HS-TCP, Scalable TCP, BIC and others.
UltraLight Kernel DevelopmentUltraLight Kernel Development
Having a standard tuned kernel is very important for a Having a standard tuned kernel is very important for a number of UltraLight activities:number of UltraLight activities:
1.1. Breaking the 1 GB/sec disk-to-disk barrierBreaking the 1 GB/sec disk-to-disk barrier2.2. Exploring TCP congestion control protocolsExploring TCP congestion control protocols3.3. Optimizing our capability for demos and performanceOptimizing our capability for demos and performance
The current kernel incorporates the latest FAST and The current kernel incorporates the latest FAST and Web100 patches over a 2.6.12-3 kernel and includes the Web100 patches over a 2.6.12-3 kernel and includes the latest RAID and 10GE NIC drivers.latest RAID and 10GE NIC drivers.
The UltraLight web page (The UltraLight web page (http://www.ultralight.orghttp://www.ultralight.org ) has a ) has a Kernel page which provides the details off the Kernel page which provides the details off the Workgroup->Network pageWorkgroup->Network page
MPLS/QoS for UltraLightMPLS/QoS for UltraLight
UltraLight plans to explore the full range of UltraLight plans to explore the full range of end-to-end end-to-end connectionsconnections across the network, from across the network, from best-effortbest-effort, , packet-packet-switchedswitched through through dedicated end-to-end light-pathsdedicated end-to-end light-paths. . MPLS paths with QoSMPLS paths with QoS attributesattributes fill a middle ground in fill a middle ground in this network space and allow fine-grained allocation of this network space and allow fine-grained allocation of virtual pipes, sized to the needs of the application or virtual pipes, sized to the needs of the application or user.user.
Network QoS with Three Classes: Best Effort, Class 4 and EF
0
200
400
600
800
1000
1200
0 100 200 300 400 500 600 700 800 900 1000
Time (Seconds)Best Effort Class 4 Express Forwarding TOTAL Wire Speed
UltraLight, in conjunction UltraLight, in conjunction with the DoE/MICS with the DoE/MICS funded TeraPaths effort, funded TeraPaths effort, is working toward is working toward extensible solutions for extensible solutions for implementing such implementing such capabilities in next capabilities in next generation networksgeneration networks
TeraPaths Initial QoS test at BNL
Terapaths URL: http://www.atlasgrid.bnl.gov/terapaths/
Optical Path PlansOptical Path Plans
Emerging “light path” technologies are becoming Emerging “light path” technologies are becoming popular in the Grid community:popular in the Grid community:
They can extend and augment existing grid computing They can extend and augment existing grid computing infrastructures, currently focused on CPU/storage, to infrastructures, currently focused on CPU/storage, to include the network as an integral Grid component. include the network as an integral Grid component.
Those technologies seem to be the most effective way to Those technologies seem to be the most effective way to offer network resource provisioning on-demand between offer network resource provisioning on-demand between end-systems.end-systems.
A major capability we are developing in Ultralight is the A major capability we are developing in Ultralight is the ability to dynamically switch optical paths across the ability to dynamically switch optical paths across the node, bypassing electronic equipment via a fiber cross node, bypassing electronic equipment via a fiber cross connect.connect.The ability to switch dynamically provides additional The ability to switch dynamically provides additional functionality and also models the more abstract case functionality and also models the more abstract case where switching is done between colors (ITU grid where switching is done between colors (ITU grid lambdas).lambdas).
MonaLisa to Manage LightPathsMonaLisa to Manage LightPaths
Dedicated modules to Dedicated modules to monitor and control monitor and control optical switchesoptical switches
Used to control Used to control CALIENT switch @ CALIENT switch @
CITCIT GLIMMERGLASS GLIMMERGLASS
switch @ CERNswitch @ CERN
ML agent systemML agent system Used to create Used to create
global pathglobal path Algorithm can be Algorithm can be
extended to include extended to include prioritisation and prioritisation and pre-allocationpre-allocation
Network monitoring is Network monitoring is essentialessential for UltraLight. for UltraLight. We need to understand our network infrastructure and We need to understand our network infrastructure and track its performance both historically and in real-time to track its performance both historically and in real-time to enable the network as a managed robust component of enable the network as a managed robust component of our overall infrastructure. our overall infrastructure. There are two ongoing efforts we are leveraging to help There are two ongoing efforts we are leveraging to help provide us with the monitoring capability required: provide us with the monitoring capability required:
IEPM IEPM http://www-http://www-iepm.slac.stanford.edu/bwiepm.slac.stanford.edu/bw// MonALISA MonALISA http://monalisa.cern.chhttp://monalisa.cern.ch
We are also looking at new tools likeWe are also looking at new tools like PerfSonar PerfSonar which which may help provide a monitoring infrastructure for may help provide a monitoring infrastructure for UltraLight.UltraLight.
Monitoring for UltraLightMonitoring for UltraLight
MonALISA UltraLight RepositoryMonALISA UltraLight Repository
The UL repository: The UL repository: http://monalisa-ul.caltech.edu:8080/http://monalisa-ul.caltech.edu:8080/
End-Systems performance End-Systems performance
Latest disk to disk over 10Gbps WAN: Latest disk to disk over 10Gbps WAN: 4.3 Gbits/sec (536 MB/sec)4.3 Gbits/sec (536 MB/sec) - 8 TCP - 8 TCP streams from CERN to Caltech; windows, 1TB file, 24 JBOD disksstreams from CERN to Caltech; windows, 1TB file, 24 JBOD disks
Quad Opteron AMD848 2.2GHz processors with 3 AMD-8131 chipsets: 4 64-Quad Opteron AMD848 2.2GHz processors with 3 AMD-8131 chipsets: 4 64-bit/133MHz PCI-X slots.bit/133MHz PCI-X slots.
3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA 3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA disksdisks Local Disk IO – Local Disk IO – 9.6 Gbits/sec9.6 Gbits/sec (1.2 GBytes/sec read/write, with <20% (1.2 GBytes/sec read/write, with <20%
CPU utilization)CPU utilization)10GE NIC10GE NIC
10 GE NIC – 10 GE NIC – 7.5 Gbits/sec7.5 Gbits/sec (memory-to-memory, with 52% CPU (memory-to-memory, with 52% CPU utilization)utilization)
2*10 GE NIC (802.3ad link aggregation) – 2*10 GE NIC (802.3ad link aggregation) – 11.1 Gbits/sec11.1 Gbits/sec (memory-to- (memory-to-memory)memory)
Need PCI-Express, TCP offload enginesNeed PCI-Express, TCP offload engines Need 64 bit OS? Which architectures and hardware?Need 64 bit OS? Which architectures and hardware?
Discussions are underway with 3Ware, Myricom and Supermicro to try to Discussions are underway with 3Ware, Myricom and Supermicro to try to prototype viable servers capable of driving 10 GE networks in the WAN.prototype viable servers capable of driving 10 GE networks in the WAN.
• Global ServicesGlobal Services support management and co-scheduling of multiple support management and co-scheduling of multiple resource types, and resource types, and provide strategic recovery mechanisms from provide strategic recovery mechanisms from system failuressystem failures
• Schedule decisions based on CPU, I/O, Network capability and Schedule decisions based on CPU, I/O, Network capability and End-End-to-end task performance estimatesto-end task performance estimates, incl. loading effects, incl. loading effects
• Decisions are constrained by Decisions are constrained by local and global policieslocal and global policies• Implementation: Implementation: Autodiscovering, multithreaded services, service-Autodiscovering, multithreaded services, service-
engines to schedule threadsengines to schedule threads, making the system scalable and robust, making the system scalable and robust• Global Services Consist of: Global Services Consist of:
Network and System Resource MonitoringNetwork and System Resource Monitoring,, to provide pervasive to provide pervasive end-to-end resource monitoring info. to HLSend-to-end resource monitoring info. to HLS
Network Path Discovery and Construction Services, Network Path Discovery and Construction Services, to provide network to provide network connections appropriate (sized/tuned) to the expected useconnections appropriate (sized/tuned) to the expected use
Policy Based Job Planning Services,Policy Based Job Planning Services, balancing policy, efficient balancing policy, efficient resource use and acceptable turnaround timeresource use and acceptable turnaround time
Task Execution Services,Task Execution Services, with job tracking user interfaces, with job tracking user interfaces, incremental re-planning in case of partial incompletionincremental re-planning in case of partial incompletion
• These types of services are required to deliver a managed network. These types of services are required to deliver a managed network. Work along these lines is planned for OSG and future proposals to Work along these lines is planned for OSG and future proposals to NSF and DOE.NSF and DOE.
UltraLight Global ServicesUltraLight Global Services
UltraLight Application in 2008UltraLight Application in 2008
Node1> fts –vvv –in mercury.ultralight.org:/data01/big/zmumu05687.root –out Node1> fts –vvv –in mercury.ultralight.org:/data01/big/zmumu05687.root –out venus.ultralight.org:/mstore/events/data –prio 3 –deadline +2:50 –xsumvenus.ultralight.org:/mstore/events/data –prio 3 –deadline +2:50 –xsum
FTS: Initiating file transfer setup…FTS: Initiating file transfer setup…FTS: Remote host responds readyFTS: Remote host responds readyFTS: Contacting path discovery serviceFTS: Contacting path discovery servicePDS: Path discovery in progress…PDS: Path discovery in progress…PDS:PDS: Path RTT 128.4 ms, best effort path bottleneck is 10 GEPath RTT 128.4 ms, best effort path bottleneck is 10 GEPDS:PDS: Path options found:Path options found:PDS:PDS: Lightpath option exists end-to-endLightpath option exists end-to-endPDS:PDS: Virtual pipe option exists (partial)Virtual pipe option exists (partial)PDS:PDS: High-performance protocol capable end-systems existHigh-performance protocol capable end-systems existFTS: Requested transfer 1.2 TB file transfer within 2 hours 50 minutes, FTS: Requested transfer 1.2 TB file transfer within 2 hours 50 minutes,
priority 3priority 3FTS: Remote host confirms available space for FTS: Remote host confirms available space for [email protected]@ultralight.orgFTS: End-host agent contacted…parameters transferredFTS: End-host agent contacted…parameters transferredEHA: Priority 3 request allowed for EHA: Priority 3 request allowed for [email protected]@ultralight.orgEHA: request scheduling detailsEHA: request scheduling detailsEHA: Lightpath prior scheduling (higher/same priority) precludes useEHA: Lightpath prior scheduling (higher/same priority) precludes useEHA: Virtual pipe sizeable to 3 Gbps available for 1 hour starting in 52.4 EHA: Virtual pipe sizeable to 3 Gbps available for 1 hour starting in 52.4
minutesminutesEHA: request monitoring prediction along pathEHA: request monitoring prediction along pathEHA: FAST-UL transfer expected to deliver 1.2 Gbps (+0.8/-0.4) averaged EHA: FAST-UL transfer expected to deliver 1.2 Gbps (+0.8/-0.4) averaged
over next 2 hours 50 minutesover next 2 hours 50 minutes
EHA: Virtual pipe (partial) expected to deliver 3 Gbps(+0/-0.3) during reservation; EHA: Virtual pipe (partial) expected to deliver 3 Gbps(+0/-0.3) during reservation; variance from unprotected section < 0.3 Gbps 95%CLvariance from unprotected section < 0.3 Gbps 95%CL
EHA: Recommendation: begin transfer using FAST-UL using network identifier #5A-3C1. EHA: Recommendation: begin transfer using FAST-UL using network identifier #5A-3C1. Connection will migrate to MPLS/QoS tunnel in 52.3 minutes. Estimated Connection will migrate to MPLS/QoS tunnel in 52.3 minutes. Estimated completion in 1 hour 22.78 minutes. completion in 1 hour 22.78 minutes.
FTS: Initiating transfer between mercury.ultralight.org and venus.ultralight.org FTS: Initiating transfer between mercury.ultralight.org and venus.ultralight.org using #5A-3C1using #5A-3C1
EHA: Transfer initiated…tracking at URL: fts://localhost/FTS/AE13FF132-FAFE39A-44-EHA: Transfer initiated…tracking at URL: fts://localhost/FTS/AE13FF132-FAFE39A-44-5A-3C15A-3C1
EHA: Reservation placed for MPLS/QoS connection along partial path: 3Gbps beginning EHA: Reservation placed for MPLS/QoS connection along partial path: 3Gbps beginning in 52.2 minutes: duration 60 minutesin 52.2 minutes: duration 60 minutes
EHA: Reservation confirmed, rescode #9FA-39AF2E, note: unprotected network section EHA: Reservation confirmed, rescode #9FA-39AF2E, note: unprotected network section included.included.
<…lots of status messages…><…lots of status messages…>FTS: Transfer proceeding, average 1.1 Gbps, 431.3 GB transferredFTS: Transfer proceeding, average 1.1 Gbps, 431.3 GB transferredEHA: Connecting to reservation: tunnel complete, traffic marking initiatedEHA: Connecting to reservation: tunnel complete, traffic marking initiatedEHA: Virtual pipe active: current rate 2.98 Gbps, estimated completion in 34.35 EHA: Virtual pipe active: current rate 2.98 Gbps, estimated completion in 34.35
minutesminutesFTS: Transfer complete, signaling EHA on #5A-3C1FTS: Transfer complete, signaling EHA on #5A-3C1EHA: Transfer complete received…hold for xsum confirmationEHA: Transfer complete received…hold for xsum confirmationFTS: Remote checksum processing initiated…FTS: Remote checksum processing initiated…FTS: Checksum verified—closing connectionFTS: Checksum verified—closing connectionEHA: Connection #5A-3C1 completed…closing virtual pipe with 12.3 minutes remaining EHA: Connection #5A-3C1 completed…closing virtual pipe with 12.3 minutes remaining
on reservationon reservationEHA: Resources freed. Transfer details uploading to monitoring nodeEHA: Resources freed. Transfer details uploading to monitoring nodeEHA: EHA: Request successfully completedRequest successfully completed, transferred 1.2 TB in 1 hour 41.3 minutes , transferred 1.2 TB in 1 hour 41.3 minutes
(transfer 1 hour 34.4 minutes)(transfer 1 hour 34.4 minutes)
Supercomputing 2005Supercomputing 2005
The Supercomputing conference (SC05) in Seattle, The Supercomputing conference (SC05) in Seattle, Washington held another “Bandwidth Challenge” Washington held another “Bandwidth Challenge” during the week of Nov 14-18during the week of Nov 14-18thth
A collaboration of high-energy physicists from Caltech, A collaboration of high-energy physicists from Caltech, Michigan, Fermilab and SLAC (with help from BNL: Michigan, Fermilab and SLAC (with help from BNL: thanks Frank and John!) won achieving thanks Frank and John!) won achieving 131 Gbps 131 Gbps peakpeak network usage. network usage.
This SC2005 BWC entry from HEP was designed to This SC2005 BWC entry from HEP was designed to preview the scale and complexity of data operations preview the scale and complexity of data operations among many sites interconnected with many 10 among many sites interconnected with many 10 Gbps linksGbps links
Total Transfer in 24 hoursTotal Transfer in 24 hours
BWC Take Away SummaryBWC Take Away Summary
Our collaboration previewed the IT Challenges of the next generation Our collaboration previewed the IT Challenges of the next generation science at the High Energy Physics Frontier (for the science at the High Energy Physics Frontier (for the LHCLHC and other and other major programs): major programs): Petabyte-scale datasets Petabyte-scale datasets Tens of national and transoceanic links at 10 Gbps (and up) Tens of national and transoceanic links at 10 Gbps (and up) 100+ Gbps aggregate data transport sustained for hours; We reached 100+ Gbps aggregate data transport sustained for hours; We reached
a Petabyte/day transport rate for real physics data a Petabyte/day transport rate for real physics data
The team set the scale and learned to gauge the difficulty of the global The team set the scale and learned to gauge the difficulty of the global networks and transport systems required for the LHC mission networks and transport systems required for the LHC mission Set up, shook down and successfully ran the system in < 1 week Set up, shook down and successfully ran the system in < 1 week
Substantive take-aways from this marathon exercise: Substantive take-aways from this marathon exercise: An optimized Linux (2.6.12 + FAST + NFSv4) kernel for data transport; An optimized Linux (2.6.12 + FAST + NFSv4) kernel for data transport;
after 7 full kernel-build cycles in 4 days after 7 full kernel-build cycles in 4 days A newly optimized application-level copy program, bbcp, that A newly optimized application-level copy program, bbcp, that
matches the performance of iperf under some conditions matches the performance of iperf under some conditions Extensions of Xrootd, an optimized low-latency file access Extensions of Xrootd, an optimized low-latency file access
application for clusters, across the wide area application for clusters, across the wide area Understanding of the limits of 10 Gbps-capable systems under stress Understanding of the limits of 10 Gbps-capable systems under stress
UltraLight and ATLASUltraLight and ATLAS
UltraLight has deployed and instrumented an UltraLight network and made good UltraLight has deployed and instrumented an UltraLight network and made good progress toward defining and constructing a needed ‘progress toward defining and constructing a needed ‘managed networkmanaged network’ ’ infrastructure.infrastructure.
The developments in UltraLight are targeted at providing needed capabilities and The developments in UltraLight are targeted at providing needed capabilities and infrastructure for LHC.infrastructure for LHC.
We have some important activities which are ready for additional effort:We have some important activities which are ready for additional effort: Achieving 10GE disk-to-disk transfers using single serversAchieving 10GE disk-to-disk transfers using single servers Evaluating TCP congestion control protocols over UL linksEvaluating TCP congestion control protocols over UL links Deploying embryonic network services to further the UL visionDeploying embryonic network services to further the UL vision Implementing some forms of MPLS/QoS and Optical Path control as part of standard Implementing some forms of MPLS/QoS and Optical Path control as part of standard
UltraLight operationUltraLight operation Enabling automated end-host tuning and negotiationEnabling automated end-host tuning and negotiation
We want to extend the footprint of UltraLight to include as many interested sites as We want to extend the footprint of UltraLight to include as many interested sites as possible to help insure its developments meet the LHC needs.possible to help insure its developments meet the LHC needs.
Questions?Questions?
