Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Building a Data Infrastructure(A WAN Provider’s Perspective)
1
Chin GuokESnet Planning & Architecture Group LeadLawrence Berkeley National Laboratory
Data on the NetworkComputing and Software RoundtableVirtual MeetingApr 6, 2021
What Do You Need for a Data (Network) Infrastructure?
• Capacity - Planning and deploying of the right amount of capacity in the right place, and at the right time
• Capability - Building services that will be integral to a Data Infrastructure that is beyond just networking
• Coordination - Enabling the orchestration of resources to build end-to-end solutions whose whole is greater than the sum of its parts
2
ESnet Data Infrastructure Activities
• Capacity– ESnet6 CY2022 Planned Capacity
• Capability– In-Network Caching Pilot– DTN-as-a-Service efforts– Edge Computing
• Coordination– SENSE: SDN for End-to-end Networking Science at the Exascale– ASCR Integrated Research Infrastructure Task Force
3
ESnet Data Infrastructure Activities
• Capacity– ESnet6 CY2022 Planned Capacity
• Capability– In-Network Caching Pilot– DTN-as-a-Service efforts– Edge Computing
• Coordination– SENSE: SDN for End-to-end Networking Science at the Exascale– ASCR Integrated Research Infrastructure Task Force
6
In-Network Caching Pilot Experiment
• In-network temporary data cache for data sharing
• Collaboration with UCSD, US CMS, Open Science Grid (OSG)– ESnet cache node as a part of SoCal Petabyte scale cache for US CMS
– Petabyte scale cache is deployed/operated by UCSD and Caltech
– ESnet: Provide a storage host to US CMS
– UCSD: Operation support for the ESnet node
• Goals
– Study how network cache storage helps network traffic performance and the overall application performance
– Accumulate experience on how the US DOE scientific experiments and simulations share data among their users
7 Sim et al.
In-Network Caching Pilot Data Movements
8 Sim et al.
Data Transfers from Source to Xcache - Cache Misses
2020-05 2020-112020-06 2020-07 2020-08 2020-09 2020-100TB
2TB
4TB
2020-05 2020-112020-06 2020-07 2020-08 2020-09 2020-100TB
2TB
4TB
6TB
8TB
10TB
12TB
Shared Data Access from Xcache to Applications - Cache Hits
Total Data Transfer = 168.083TB
Total Shared Data Access= 322.748TB
Network demand is reduced by a factor of ~3
In-Network Caching Pilot File Reuse Rates
9 Sim et al.
• Data file reuse rate = (sum of reuses) / (total number of unique files)– Sum of reuses = all shared data access counts of the day (cache hits)– Total number of unique files = number of unique files for the day
In-Network Caching Pilot Summary
• 1,286,748 accesses from May 2020 to Oct 2020– Total 490.831 TB of client data access (first time reads and repeated reads)
– Transferred/cached 168.08 TB (from remote sites to cache)
– Saved 322.748 TB of network traffic volume (repeated reads only)
– Network demand reduced by a factor of ~3
• Future work
– Deployment of 2 additional caches in summer 2021
– Cache miss rate study
– Cache utilization study
10 Sim et al.
DTN-as-a-Service
• Exploring an ESnet-managed data movement service
– Support a pool of transfer software images that “just work” across the ESnet footprint
– Reduce reliance on varying levels of (site) network/system expertise for DTN deployments
• Investigate opportunities to expand data movement software availability and to support more flexible service deployments
• A potential candidate for supporting the ESnet6 smart services edge
11 Kissel et al.
1. Data mover software in containers
2. Network and storage performance optimization
3. Configuration and tuning flexibility
4. Lightweight service orchestration
12
DTN-as-a-Service Prototype Service
Kissel et al.
DTN-as-a-Service Use Cases and Ongoing Engagements
• Transitioning the ESnet 10G/40G testing DTNs to DTNaaS
• Deployed in the ESnet 100G Testbed to manage experimenter images
• Evaluation of data mover software stacks supported by DTNaaS– For example - Zettar/zx1
• Investigating a deployment model to support In-Network Data Caching effort
• Seeking partnerships for service trial– Using DTNaaS to accelerate an existing data workflow challenge– Outreach to JGI, FRIB, EMSL
13 Kissel et al.1 E. Kissel, C. Fang, Zettar zx Evaluation for ESnet DTNshttps://www.es.net/assets/Uploads/zettar-zx-dtn-report.pdf
Edge Computing for Real Time Workflows - Today
14 Yatish et al.
Instrument
Custom DAQ
A/D+
Trigger+
Timing
DAQ data reformatting
Pre-filtering
Stream assembly for
event/particle extraction
Compute cluster
GPUs
Fast Storage
Batch mode event/particle
extraction
N x100G
Up to 1 Tbps Today,100Tbps Next Generation
Servers + DDR3 + 100G NIC cards
The Molecular Foundry (TMF)Advanced Light Source (ALS)
Linac Coherent Light Source (LCLS)
Continuous Electron Bean Accelerator Facility (CEBAF)
Edge Computing for Real Time Workflows - Future
15 Yatish et al.
Instrument
Custom DAQ
A/D+
Trigger+
Timing
DAQ data reformatting
Pre-filtering
Stream assembly for
event/particle extraction
Compute cluster
GPUs
Fast Storage
Batch mode event/particle
extraction
Out of order reassembly
Local compute load balancing
N x100G
Up to 1 Tbps,100Tbps NxGen
100% link load balancing(Out of order UDP protocols)Real time data compression
FPGA HW laid balancing intocluster. Group DAQ data by
timestamp and frame forparallel data processing
Low cost scalable FPGApacket processors toreplace expensiveCPU+DRR solutions
The Molecular Foundry (TMF)Advanced Light Source (ALS)
Linac Coherent Light Source (LCLS)
Continuous Electron Bean Accelerator Facility (CEBAF)
ESnet Data Infrastructure Activities
• Capacity– ESnet6 CY2022 Planned Capacity
• Capability– In-Network Caching Pilot– DTN-as-a-Service efforts– Edge Computing
• Coordination– SENSE: SDN for End-to-end Networking Science at the Exascale– ASCR Integrated Research Infrastructure Task Force
16
SENSE: SDN for End-to-End Networked Science at the Exascale- Coordination of End-to-End Network Cyberinfrastructure
WAN
Application Workflow Agents
SDX
End Site
SDN Layer
ComputeDTNs Storage Instruments
End Site
Compute DTNsStorageInstruments
Regional
SDMZ
Regional
SDMZ
WAN
SENSE
Types of Interactions● What is possible?● What is recommended?● Requests with negotiation● Scheduling
17
● Intent Based APIs● Resource Discovery● Service Life Cycle
Monitoring/Troubleshooting● Deterministic Networking
Lehman et al.
SENSEArchitecture
18 Lehman et al.
SENSE-O SENSE-O
Application Workflow Agent (e.g., CMS)
Application Workflow Agent (e.g., ATLAS)
Application Workflow Agent(s)
Collaboration / Project Centric
Resource Administrative Domain Centric
SENSE-NRM
(NSI) Domain Controller (e.g.,
OpenNSA)
SENSE-DTN-RM
DTN
NSIDomain
SENSE-NRM
(NSI) Domain Controller (e.g.,
OpenNSA)
(NSI) Domain Controller (e.g.,
OpenNSA)
(NSI) Aggregator (e.g., Safnari)
SENSE-NRM
(NSI) Domain Controller (e.g.,
OESS)
Domain Controller (e.g.,
OSCARS)
SENSE-NRM
Domain with network using NSI under SENSE control (e.g., Internet2)
Domain with network under SENSE (native) control (e.g., ESnet)
Domain with DTN under SENSE control (e.g., Caltech, UCSD)
Federated domains using NSI under SENSE control (e.g., AutoGOLE)
Domain with network and DTN under SENSE control (e.g., CENIC, WIX, MANLAN)
SENSE-DTN-RM
DTN
Application Agent:SENSE-O = N:1
SENSE-O:SENSE-RM = N:N
O: OrchestratorRM: Resource ManagerDTN-RM: Data Transfer Node RMNRM: Network RM
SENSE ExaFEL Use Case - Superfacility Automation Prototype
19 Lehman et al.
DTNs
FFB Layer (nVRAM)
ESNET
DAQ (1 per instrument, 10 total)
Online System (1 instance per experimental hall, 2 total)
Compressor Nodes
PGP
PGP
PGP
PGP
Eth
IB
Beam Line Data
LCLSLCLS
ESNET
Exascale HPC
1 TB/s
DTNs
NERSCRouter
Burst Buffer nVRAM: Streaming Analysis
Software ETH Routers
NGF / Lustre: Offline from HDF/XTC files.
IB
Exascale HSN
HPC System with SDN & Burst Buffer Supporting
(1) file-based transfer path (2) stream-based data transfer path
Aggregate detector data, EPICS data, beamline data - selection and compression
10 GB/s - 1Tb/sEPICSData
Timing
Online Monitoring & FFB Nodes
Event builder nodes
ExaFEL Data Flow
LCLS Data Transfer Workflow SENSE-O
SENSE-NRM SENSE-DTN-RMSuperfacility
(SF) API*
*NB: Superfacility (SF) API under development by NERSC
SENSE Future Domain Science Integrations
20
• RUCIO/FTS expected to be used in multiple domain science workflows
• SENSE Service and Orchestration model designed for adaptation and customization for different infrastructures, service requirements, and workflows
Lehman et al.
ASCR Integrated Research Infrastructure Task Force
21
ALCFNERSCOLCFESnet
AL
DA
APSC
WF
AR
PB
AC
Experimental Facility
2
ALCFNERSCOLCFESnet
AL
DA
APSC
WF
AR
PB
AC
Experimental Facility
Experimental Facility
Experimental Facility
3
ALCF
AL
DA
APSC
WF
AR
PB
AC
ESnet
AL
DA
APSC
WF
AR
PB
AC
OLCF
AL
DA
APSC
WF
AR
PB
AC
NERSC
AL
DA
APSC
WF
AR
PB
AC
Experimental Facility
1
AL Allocations
AccountsAC
DA
AP
SC
WF
AR
PB
Data
Workflows
Applications
Scheduling
Archiving
Publication
Toward a Seamless Integration of Computing, Experimental, and
Observational Science Facilities: A Blueprint to Accelerate Discovery*
1. Today, an experimental facility must arrange separate bespoke interactions with individual HPC/HPN facilities.
2. A future paradigm with common interfaces could simplify integration of an experimental facility with multiple HPC/HPN facilities.
3. In turn, these common interfaces could support expansion and integration across multiple experimental facilities and HPC/HPN facilities
Brown et al.*NB: Whitepaper to be released soon.