View
220
Download
0
Category
Tags:
Preview:
Citation preview
The ADAMANT Project:Linking Scientific Workflows and Networks
“Adaptive Data-Aware Multi-Domain Application Network Topologies”
Ilia Baldine, Charles Schmitt, University of North Carolina at Chapel Hill/RENCIJeff Chase, Duke University
Ewa Deelman, University of Southern California
Funded by NSF under the Campus Cyberinfrastructure – Network Infrastructure and Engineering (CC-NIE) Program
The Problem
• Scientific data is being collected at an ever increasing rate• The “old days” -- big, focused experiments– LHC, LIGO, etc..
-- big data archives– SDSS, 2MASS, etc..• Today “cheap” DNA sequencers – and an increasing number of
them in individual laboratories
• The complexity of the computational problems is ever increasing
• Local compute resources are often not enough (too small, limited availability)
• The computing infrastructure keeps changing• Hardware, software, but also computational models
Computational workflow--managing application complexity
• Helps express multi-step computations in a declarative way
• Can support automation, minimize human involvement– Makes analyses easier to run
• Can be high-level and portable across execution platforms
• Keeps track of provenance to support reproducibility
• Fosters collaboration—code and data sharing
• Gives the opportunity to manage resources underneath
Large-Scale, Data-Intensive Workflows
• Montage Galactic Plane Workflow– 18 million input images (~2.5 TB)– 900 output images (2.5 GB each, 2.4 TB total)– 10.5 million tasks (34,000 CPU hours)
•An analysis is composed of a number of related workflows– an ensemble• Smart data/network provisioning are important
4
John Good (Caltech)
× 17
CyberShake PSHA Workflow
239 Workflows• Each site in the input map
corresponds to one workflow• Each workflow has: 820,000 tasks
Description Builders ask seismologists: “What will the peak
ground motion be at my new building in the next 50 years?”
Seismologists answer this question using Probabilistic Seismic Hazard Analysis (PSHA)
Southern California Earthquake Center
MPI codes ~ 12,000 CPU hours, Post Processing 2,000 CPU hoursData footprint ~ 800GB
Coordination between resources is needed
EnvironmentHow to manage complex workloads?
Data Storage
Campus Cluster
XSEDE
Open Science Grid
Amazon Cloud
Work definition
Local Resource
Use Given Resources
Data Storage
Campus Cluster
FutureGrid
XSEDE
Open Science Grid
Amazon Cloud
Work definitionAs a WORKFLOW
Workflow Management System
Local Resource
work
data
Workflow Management
• You may want to use different resources within a workflow or over time• Need a high-level workflow specification• Need a planning capability to map from high-level to
executable workflow• Need to manage the task dependencies• Need to manage the execution of tasks on the
remote resources
• Need to provide scalability, performance, reliability
Pegasus Workflow Management System (est. 2001)
Pegasus makes use of available resources, but cannot control them
• A collaboration between USC and the Condor Team at UW Madison (includes DAGMan)
• Maps a resource-independent “abstract” workflow onto resources and executes the “concrete” workflow
• Used by a number of applications in a variety of domains• Provides reliability—can retry computations from the point of
failure• Provides scalability—can handle large data and many
computations (kbytes-TB of data, 1-106 tasks)• Infers data transfers, restructures workflows for performance• Automatically captures provenance information• Can run on resources distributed among institutions, laptop,
campus cluster, Grid, Cloud
A way to make it work better
Data Storage
Work definition
Pegasus WMS
Local Resource
work
data
ResourceProvisioner
Virtual Resource Pool
Resources requests
Resources: compute, data, networks
Grids and Clouds
• ORCA is a “wrapper” for off-the-shelf cloud and circuit nets etc., enabling federated orchestration:+ Resource brokering+ VM image distribution+ Topology embedding+ Stitching+ Authorization
o Deploys a dynamic collection of controllerso Controller receive user requests and provisions
resources
Open Resource Control Architecture
Jeff Chase, Duke University
What is missing
• Tools and systems that can integrate the operation of workflow-driven science applications on top of dynamic infrastructures that link campus, institutional and national resources
• Tools to manage workflow ensembles• Need to
– orchestrate the infrastructure in response to the application
– monitor various workflow steps and ensemble elements– expand and shrink resource pools in response to
application performance demands– integrate data movement/storage decisions with
workflows/resource provisioning to optimize performance
Summary: ADAMANT will• Focus on data-intensive applications: astronomy,
bioinformatics, earth science• Interleave workload management with resource
provisioning– Emphasis on storage and network provisioning
• Monitor the execution and adapt resource provisioning and workload scheduling
• Experiment on exoGeni
– http://networkedclouds.org– http://geni-orca.renci.org– http://pegasus.isi.edu
Recommended