Leveraging Distributed Research Cloud Infrastructures for Domain Science Research and Experimentation

Anirban Mandal, Cong Wang, Paul Ruth, Komal Thareja (RENCI, UNC – Chapel Hill)

Ewa Deelman, George Papadimitriou (USC/ISI)

Michael Zink, Eric Lyons (UMass Amherst)

Ivan Rodero, J. J. Villalobos (Rutgers University)

Funded by the National Science Foundation

Grant #1826997

Drew Angerer, Getty ImagesOpen Cloud Workshop, Boston University, March 2020

Distributed Research Clouds for Domain Sciences

• Develop novel algorithms and mechanisms to offer optimized data-flows across different kinds of national CI.

• Dynamic multi-cloud resource provisioning - ExoGENI, Chameleon, XSEDE JetStream, MOC.

• Network-centric platform to bridge the abstraction gap.

• Data-aware scheduling in Workflow Management Systems (Pegasus).

• Deploy solutions for use in observational science communities - CASA and OOI.

Major challenge is integration of data CI to science workflows: data movement across diverse infrastructure, complex workflows, distributed repositories, and elastic application compute/storage/network requirements.

Multi-cloud Provisioning and Network Orchestration

• Resource requirements from applications are generated using a Gantt chart abstraction

• Mobius network-centric platform• Multi-cloud provisioning of compute and

storage resources• Layer 2 network provisioning• Resource monitoring and control• https://github.com/RENCI-NRIG/Mobius• Exposes a REST API for Applications and

Workflow Management Systems• Management, Optimization & Prioritization

of Data Flows with virtualized SDX

CASA OOI

CASA: Collaborative Adaptive Sensing of the Atmosphere

• Traditional Next Generation Weather Radars (NEXRAD)

• High power, long range• Limited ability to observe the lower

part of the atmosphere because of the earth's curvature

• CASA• Network of short range Doppler

radars deployed in DFW area• Adjustable sensing modes in response

to quick weather changes• Suitable for near-ground weather

events: tornado, hail, high winds

CASA Workflows

> 7M people, >100K businesses, >1500 Corporate HQs

SC’19 SCinet Tech Challenge

Distributed Research Cloud Deployment with Mobiusfor CASA Applications

CASA Operational Workflows on Chameleon and ExoGENIwith Layer 2 Provisioned Data Flows

• Distributed testbeds• Heterogeneous: ExoGENI and

Chameleon testbeds

• ExoGENI• 11 workers (VMs)• 4 cores and 12 GB RAM• NFS storage

• Chameleon• 4 workers (bare metal)• 24 cores and 192 GB RAM

• Radar data repository• UNT via L2 stitching port

• Connected via 10 Gbpsnetwork.

CASA Hail Workflow

Multi-workflow display of hail (orange) and wind (red) contours, with GIS boundaries and infrastructure overlaid during a severe weather event.

Pegasus workflow for Hail

CASA radar data

Current Research Thrusts

• Optimization & prioritization of science data flows with virtualized SDX.

• Active monitoring and control for maintaining QoS.

• Supporting wider federation of clouds, including EC2, CloudLab, and OCT.

• Data-aware workflow scheduling. • Deploying novel CASA workflows (e.g.

drone path planning).• Support for streaming data and on-

demand workflows from Ocean Observatory Initiative (OOI) NSF Large Facility.

Thank you !

Funded by the National Science Foundation

Grant #1826997

Funded by the National Science FoundationGrant #1826997