21
Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S. Civil Servant

Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Earthdata Cloud Analytics Project

Chris Lynnes* and Rahul Ramachandran*NASA

*U.S. Civil Servant

Page 2: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

2

Earth Observing System Data and Information System (EOSDIS)

Applications

capture and clean

data downlink

Education

process

archive

subset

distributeResearch

Users

EOSDIS

Page 3: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

3

Over time, EOSDIS archive volumes increase exponentially

projected

Page 4: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

4

Distribution increases similarly to cumulative volume

Page 5: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

5

How do we support user analysis of very large data volumes?

Page 6: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

6

Solution: Data-proximal Analysis

Data Archive

Data Analysis

Page 7: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Goals

1. Enable big compute next to big data

2. Encourage user adoption of cloud for analytics

3. Maximum analytics capability at minimum costa. Use capabilities within NASA more effectively and efficiently

b. Leverage analytics capabilities of external partners

Page 8: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Key Features1. Satisfy a diverse user community

2. Support analysis in the cloud without egressing data

3. Facilitate multi-dataset comparison and fusion

4. Support batch, interactive and streaming modes

5. Support distributed filesystems and databases

6. Support cost constraints and cost-sharing

Page 9: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Earthdata Cloud Analytics Guiding Principles

1. Infusion- and innovation-friendly framework and building blocks

2. No monolithic systems

3. Open code and services

4. Interoperability and reuse

5. No unnecessary duplication (“undifferentiated heavy lifting”)

Page 10: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Architectural Concept

Earth Science Data Analytics the Cloud-Native Way:

Everything is a Service

This approach produces key important benefits for

the user community and EOSDIS

Page 11: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Abstract Analytics Workflow

ExtractTransform

Loaddata Analyze Visualize

Page 12: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

AODS1 as-a-service

Preprocessing as-a-serviceCumulus

Data Archive

Analysis as-a-service

Visualization as-a-service

Earthdata Cloud Analytics Reference Architecture

Extract Transform Load

1 Analytics Optimized Data Store

Page 13: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Preprocessing as-a-serviceCumulus

AODS1 as-a-service

Analysis as-a-service

Visualization as-a-service

Interactive Mode: Analytics-Optimized Storage

Cumulus Data

Archive

1 Analytics Optimized Data Store

Page 14: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Preprocessing as-a-serviceCumulus

Analysis as-a-service

Visualization as-a-service

Batch Mode

AODS1 as-a-service

Cumulus Data

Archive

1 Analytics Optimized Data Store

Page 15: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Preprocessing as-a-serviceCumulus

Event Analytics as-a-service

Visualization as-a-service

Streaming Mode

Cumulus Data

Archive

Page 16: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Open Pipeline Provides Outputs at Different Stages Appropriate for a Diverse User Base

End-User-Specific Analysis

Preprocessing as-a-serviceCumulus

AODS1 as-a-service

Analysis as-a-service

Visualization as-a-serviceCumulus

Data Archive

1 Analytics Optimized Data Store

Page 17: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

AODS1 as-a-service

Open Pipeline Provides Outputs at Different Stages Appropriate for a Diverse User Base

End-User-Specific Analysis

Preprocessing as-a-serviceCumulus

Analysis as-a-service

Visualization as-a-service

End-User Cloud-Native

Analysis

Cumulus Data

Archive

1 Analytics Optimized Data Store

Page 18: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Open Pipeline Provides Outputs at Different Stages Appropriate for a Diverse User Base

End-User-Specific Analysis

Preprocessing as-a-serviceCumulus

Analysis as-a-service

Visualization as-a-service

End-User Cloud-Native

Analysis

End-User Interpretation

Cumulus Data

Archive

AODS1 as-a-service

1 Analytics Optimized Data Store

Page 19: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Open Pipeline Provides Outputs at Different Stages Appropriate for a Diverse User Base

End-User-Specific Analysis

Preprocessing as-a-serviceCumulus

Analysis as-a-service

Visualization as-a-service

End-User Cloud-Native

Analysis

Data Exploration

End-User Interpretation

Cumulus Data

Archive

End-User Interpretation

AODS1 as-a-service

1 Analytics Optimized Data Store

Page 20: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Open Pipeline Enables Integration with Other Data, Scripts, and Workflows

End-User-Specific Analysis

Preprocessing as-a-serviceCumulus

Analysis as-a-service

Visualization as-a-service

End-User Cloud-Native

Analysis

Data Exploration

End-User Interpretation

Cumulus Data

Archive

End-User Interpretation

Data Exploration

AODS1 as-a-service

1 Analytics Optimized Data Store

Page 21: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11  · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S

Open Pipeline Enables Integration with Exploitation Platforms

Preprocessing as-a-serviceCumulus

AODS1 as-a-service

Analysis as-a-service

Visualization as-a-service

Exploitation Platforms

Cumulus Data

Archive

1 Analytics Optimized Data Store