Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
©2018 DataDirect Networks, Inc. DDN Confidential
Death to the Shared POSIX File System!?
Long Live the Shared POSIX File System!!
HPC Storage and the HPC Data Center
this is a vendor presentation!
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Classical HPC Data Centers
• Everything is bare metal
•Most data is stored in shared POSIX file systems that largely remain on HDDs
• SSD-based systems are slowly coming, but capacities remain small
• People have played with Hadoop and objects stores, but very few systems are actually in production
• Special performance and capacity tiers are used mainly by the large centers, but not the broader market
Cloud Data Centers
• Everything is virtualized, containerized, dynamic, automated, etc.
•Most date lives on local file systems in virtual machines (or containers), either on SSDs or HDDs
•Monolithic web applications use large, distributed object stores
• Enterprise storage still being used for various specific requirements, both on premise and hosted in the cloud
• Tiers exists for redundancy, data back-up, data retention, etc.
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Example: 2015 DDN User Analysis
Work Data Mixed Use Archive
Cloud
HPC Work
WeatherClimate
CAEChemical
General Academic
Genomics
Big DataScience
Security
Finance
Energy
Tier 2HPCCloud
Cloud
Work31%
Data33%
Mixed22%
Cloud2%
Archive12%
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Example: 2018 Lustre End User Survey
0% 5% 10% 15% 20% 25% 30% 35% 40%
Weather/Climate
Other
Media
Manufacturing
Life Sciences
Government
Financial Services
Energy
Education
Defense
AI/Machine Learning
Primary Usage of Lustre
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
HPC Data Centers: The Past Decade
►“Research Big Data” is a very big market for storage (and a much larger than the classical tightly-coupled simulations) and it is quite different from classical HPC
►Many data center dedicated to analytics and deep learning look like large cloud environments, but their IO requirements are much closer to Research Big Data
►Some “Research Big Data” end users have moved to on-premise cloud environments using Open Stack etc.
►More HPC and Analytics customers are running applications in the cloud or in cloud-like environments, with shared file systems running in the cloud
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
This architecture was imperiled by SSD economics
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
This architecture becomes imperiled by tape economics
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
This architecture becomes imperiled by Lang’s Law!
”DOE doesn’t want tiers. Tiers are an unfortunate accident of economics. DOE wants infinite memory and a system without
unplanned interrupts.
Just remember this:The fewer tiers, the fewer tears.”
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Predictions From LANL in 2016: Serving Data to the Lunatic Fringe
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Predictions from NERSC: Storage 2020-2025
Four Tiers Three Tiers Two Tiers
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
The Foreseeable Future Remains Tiered
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
DDN ConfidentialDDN Storage | ©2019 DataDirect Networks
Tiering Tiering – Data Schmiering
All due respect to Lang’s Law (‘fewer tiers, fewer tears’),tiering is a (mostly) solved problem.
Buffer-caching is a (mostly) solved problem!
Russel Kirsch developed it for the SEAC in 1952.
DDN ConfidentialDDN Storage ©2019 DataDirect Networks, Inc.
Flash Acceleration Layer Usage and Tiering Workflows in Traditional HPCLee Ward, Use Cases or BB Roles, Informal Burst Buffer Presentation via Sandia National Laboratories, 2015.
Challenges and Considerations for Utilizing Burst Buffers in High-Performance Computing, Melissa Romanus, Robert Ross, Manish Parashar, 2018.
Development of a Burst Buffer System for Data-Intensive Applications,Teng Wang, Sarp Oral, Michael Pritchard, Kevin Vasko, Weikuan Yu, 2015.
An Operational Perspective on a Hybrid and Heterogeneous Cray XC50 System.Sadaf Alam, Nicola Bianchi, Nicholas Cardo, Matteo Chesi, Miguel Gila, Stefano Gorini, Mark Klein, Colin McMurtrie, Marco Passerini, Carmelo Ponti, Fabio Verzelloni, 2017.
1. Checkpoint-Restart
2. In-situ/transit viz/analysis
3. Out-of-core
4. Accelerated reads (pre-stage)
5. Random-read centric applications (lots of them!)
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Data Tiering: A Subtle Shift in Perception
Applications
Performance Storage
Capacity Storage
Unnecessarily Strict Tiering Relaxed Tiering
Applications
Capacity Storage
Performance Storage
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
“What?”
“Do you have any water?”
“Build an object store!”
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Typical Object Requirements (“Object Schmobject; long live POSIX”)
► Immutable, transactional get/put, trillions of objects
► Named objects
► Group objects into logical collections
► Nest logical collections within each other
► Have the same object appear within multiple collections
► Multi-threaded writes
► Tag objects
►Object is a subset of file• There is no application which uniquely requires
object semantics
•O_TMPFILE and rename are useful primitives
►Object requirements grow as humans use them• Eventually they become file requirements
►We do not live on a deserted desert island•We have two decades experience building parallel
file systems
►RELEVANT LESSON FROM OBJECT STORES?• POSIX relaxation is useful
Bent’s Law for HPC Storage:
The future is bright and mostly as we predicted it.
Don’t be scared of POSIX; but embrace relaxations.
Don’t be scared of tiering; but embrace relaxations.
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
ML/DL Storage Scenarios: Small-to-Large
Local NVMe
Node-Local “Shared” NVMe
Global/Capacity
Global NVMe
OnDemandNamespaces Remote NVMe
(NVMeoF)
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Dynamically-Provisioned File Systems
► Job-Specific File Systems• LLNL Proposal (2014): file system per JOB with data staging
• Isolation from other jobs
• Increased metadata performance
► Loop-back devices for increased metadata Performance
• K-Computer/Fujitsu: Very large-scale Implementation
• NERSC Library (e.g. for use of SPARC on Lustre)
• Lustre on Amazon/Azure/Google (since 2013)
• Amazon Lustre Service (November 2018)
►Future• Client Container Image (CCI) Feature in Lustre (integrates look-back
devices into Lustre)
• Lustre-on-Demand Feature (first deployments in early 2019!)
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Amazon FSx for Lustre
►Dynamically-provisioned Lustre file systems
►Application-specific
►Focused on analytics and DL applications
►Connector to Amazon S3
►But: the usual limitations still apply;-)
DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.
Next Decade of HPC Storage: Back to the Future?
►Still PFS…
•… but with increasing portions that are dynamically allocated and integrated into the compute platform
• e.g. a file server turns into a containerized process run anywhere
►Still POSIX…
•… but relaxed where needed
►Still Tiers…
•… but relaxed to reflect actual application workflows
Relaxed Tiering with Dynamically-Managed,
System-integrated Storage
Applications
Capacity Storage
Performance ENS ENS
CapS