Embed Size (px)
Citation preview
NASA's Earth Observing Data and Information System (EOSDIS)
Kevin Murphy EOSDIS System Architect NASA Goddard Space Flight Center
Copernicus Big Data Workshop
13 March 2014 Brussels, Belgium
Topics to be Covered
� Overview of NASA Earth Observation System Data and Information System (EOSDIS) • Architecture and Capabilities
� NASA International Collaboration
� Sentinel Mirror Site to Support U.S. Users
0 pt 1 pt 0 pt1 00 ptptpt
MISSION OPERATIONS SCIENCE OPERATIONS
EEARTH SCIENCE DATA OPERATIONS
DATA ACQUISITION FLIGHT OPERATIONS, DATA CAPTURE,
INITIAL PROCESSING, BACKUP ARCHIVE
DATA TRANSPORT TO DATA CENTERS/SIPSs
SCIENCE DATA PROCESSING, DATA MANAGEMENT, INTEROPERABLE DATA, ARCHIVE, AND DISTRIBUTION
DISTRIBUTION AND DATA ACCESS
Research
Education
Value-Added Providers
Earth System Models
International Partners
Decision Support Systems
Infrastructure (Search, Order, Distribution)
Tracking and Data Relay Satellite (TDRS)
Instrument Teams and Science Investigator-led Processing Systems (SIPSs)
White Sands Complex (WSC)
EOS Polar Ground Stations
Direct Broadcast (DB)
Direct Broadcast/ Direct Readout
Stations
EOSDIS Data Centers
EOS Data Operations System (EDOS) Data Processing
EOS Operations Center (EOC)
Mission Control
EOS Spacecraft
DATA ACQUISITION
TracRReReReRRR lay
WEEOS SSppaacceeccrraafftt
y
Interagency Data Centers
EOS Polar Ground Stations
/
Mission Con
National Aeronautics and Space Administration
NASA INTEGRATED SERVICES NETWORK
(NISN) MISSION SERVICES
� Earth Science Data are held at Distributed Active Archive Centers (DAACs) to provide knowledgeable curation and science-discipline-based support
� NASA provides high bandwidth
network connectivity to support production data flows and community access to data
� NASA develop tools for users to obtain needed data/information while minimizing burden associated with unwanted data
� NASA engages with multiple US
agency efforts to facilitate use of data by broadest possible community with minimal effort and maximal consistency with other data sources
Key EOSDIS Metrics
Unique Data Products (Collections) 6,861
Distinct Data and Service Users Per year 1.7 M
Average Daily Archive Growth 8.5 TB/day
Total Archive Volume 9.8 PB
End User Products Distributed per year 839 M
End User Average Daily Distribution Volume 22 TB/day
Key EOSDIS Metrics
NASA’s Earth Observation System Data and Information System
NASA’s DAACs
EOSDIS as a Seamless, Efficient User Driven System
� Present NASA’s EOSDIS as an interoperable system of systems where users can select, view, interact and download the data they need transparently from all subsystems in support of interdisciplinary Earth Science research.
� Supplement current data system capabilities with new interoperable technologies to create a foundation for future evolution. • Support technology infusion of tools developed by internal programs and by
industry
� EOSDIS capabilities and feature development is prioritized through DAAC User Working Groups (science experts) and input from community data system programs
Data Metadata as a Service
Imagery as a Service
Data as a Service
Partnerships commercial
clouds
Today Tomorrow Working Towards
Central Reusable Capabilities
� Earthdata: the EOSDIS website (http://earthdata.nasa.gov)
� Metadata Services
• ECHO: Searchable catalog of granule metadata for NASA datasets (OpenSearch, CSW, OGC interfaces)
• Global Change Master Directory (GCMD): Searchable catalog of over 26,000 NASA and International dataset collections
� User Tools (e.g.)
• Reverb – search and order tool
• Global Imagery Browse Services (GIBS) – full resolution imagery derived from NASA products in a standardized manner to any web-connected client (Open Sourced)
• Worldview - highly responsive interface to explore GIBS imagery and download the underlying data
• Giovanni – Quick-Start Exploratory Data Analysis
� Metrics System (EMS): collects and reports on data ingest, production, archive, and distribution across all EOSDIS data centers
� User Registration System: provides a centralized and mechanism for user registration and account management for all EOSDIS system.
DAAC
LANCE
SIPS
GCMD
ECHO
EMS
Earthdata Web Infrast.
GIBS
Web Applications & Services
Examples of Relevant DAAC Capabilities
� Sentinel 1 (ASF DAAC) • ASF DAAC offers a variety of SAR-derived higher level
products via easy-to-use Web interfaces • MapReady Toolkit
� Sentinel 3 (ODPS, LAADS, LPDAAC, …) • SeaDAS - science processing from Level-1B through
Level-3 with a host of NASA standard and alternative ocean product algorithms (including source code). Also data product analysis and visualization based on ESA's BEAM tool.
• Match-up to field data - Level-1/Level-2 browser will identify all data granules for which coincident field data exists in the NASA SeaBASS in situ bio-optical archive, and provide the data as a unified order.
• In situ SeaBASS archive for product validation and the NOMAD database for bio-optical algorithm development
� Sentinel 5p • Giovanni • Subsetting and reformatting 8
o s
,
ntt tt
8
EOSDIS Networks
� Provide end-to-end network connectivity between users and geographically distributed EOSDIS data centers. • Globally connected to serve the
diverse needs of NASA's worldwide science and research community.
� Network Operations • Monitor and operate all aspects of
network to meet required level of service
• Develop tools to identify and resolve bandwidth and connectivity issues
• Plan for future bandwidth needs
f
Examples of International Collaborations
� European Space Agency (ESA) - ESA/NASA Bilateral • MERIS – MODIS/SEAWiFS Data Exchange • NASA provided the entire Aqua MODIS and SeaWiFS
Level 1A data archives to ESA in exchange for MERIS Reduced Resolution and Full Resolution data (for redistribution to NASA user community)
• First time non-commercial 3rd party data distributor for ESA
� Japanese Aerospace Exploration Agency (JAXA) • Under agreement between NASA and JAXA , in 2010
the NASA ASF DAAC began acquiring PALSAR data from ALOS
• MapReady tool modified by ASF DAAC to support ALOS PALSAR data
• ASF DAAC integrated JAXA processor into DAAC processing stream
• Currently the ASF DAAC archives over 1 petabyte of ALOS PALSAR data, and has distributed 1.6 million scenes.
� Canadian Space Agency (CSA) • NASA ASF DAAC provides mission planning, downlink,
and data distribution for RADARSAT-1
Long-term record of Chla averaged over global deep-water region. Shown is the monthly anomaly after subtraction of the monthly climatological mean for consistently-processed SeaWiFS (black), MERIS (brown), MODIS-Aqua (blue), and NASA NPP VIIRS evaluation product (red). Gray region is estimated uncertainty.
0 pt 0 pt
COPERNICUS SERVICES NASA SCIENCE OPERATIONS
DDHuS Plugs into EOSDIS DATA TRANSPORT
SCIENCE DATA PROCESSING, DATA MANAGEMENT, INTEROPERABLE DATA, ARCHIVE, AND DISTRIBUTION
DISTRIBUTION AND DATA ACCESS
Research
Education
Value-Added Providers
Earth System Models
International Partners
Decision Support Systems
Infrastructure (Search, Order, Distribution)
EOSDIS Data Centers
Interagency Data Centers
NASA INTEGRATED SERVICES NETWORK
(NISN) MISSION SERVICES
Sentinel Mirror System
NASA Sentinel Mirror Leveraging Existing EOSDIS Capabilities
� For Sentinel 1, 3, 5P, NASA will leverage proven mirroring and redistribution capabilities, currently used for S-NPP • Single network interface relieves
bandwidth load on European networks
• Long term archival and end user distribution by DAACs (e.g. Sentinal-1 by ASF DAAC).
• Provides metric reports back to the EC/ESA on product distribution and usage
• Leverages entire suite of NASA’s EOSDIS capabilities including capturing and reporting metrics on distribution and usage of Sentinel products by U.S. scientists
edd
e d
Sentinel Mirror: Leveraging experience with SNPP
� Current Capabilities of NASA’s S-NPP Science Data System: • Server System dedicated to
• Acquiring data from multiple locations • Storing data temporarily (~ 30 days) • Making data available to six data processing centers
• Ingests 6 TB daily • Capability of distributing 2.5 times the ingest volume; routinely
distributes 15TB daily • Products available to data processing centers within 30 minutes of
receipt
Lessons Learned
� Data Acquisition and Archives • Pick the best protocol for high bandwidth circuits over long distances – NASA’s recent test
results indicate the most effective protocols are: • Bbftp (http://doc.in2p3.fr/bbftp/) • GridFtp (http://toolkit.globus.org/toolkit/data/gridftp/)
• Monitor system health, provide system failover, database failover and replication to protect against data loss resulting from system faults
• Detect data gaps and automatically reacquire missing products • Provide users with data gap reports and data archive status
• Provide data consumers with subscription capabilities and a manifest file to streamline data access
� User Experience
• Data on spinning disk is essential for providing interactive services
• Quality, consistency and flexibility of metadata services enables service orientated architectures
• Open source software and standards are vital for interoperability • Machine accessible APIs
• Engage users early and often
Conclusion
����
����
����
����
����
�����
�����
���� ���� ���� ���� ���� ��� ��� ���� ���� ��� ���� ���� ���� ����
����
���
����
�
��������� �!�����" #$�%���������������! �&'
In partnership with Copernicus program NASA’s EOSDIS is prepared to invest in capabilities to maximize Sentinel data utilization
Backup Slides
DC
2 x 10 g 10 g
College Park, MD
Greenbelt, MD
NASA GSFC
NASA Sentinel Reflector
MAX 10 g
10 g
10 g
NASA Sentinel Connectivity
10 g �ESA
������������� ���
����
GÉANT WIX
GSFC Users
EOS LAN 10 gbps
NASA Backbone
(NISN) 10 gbps
100 g
Universities
EROS
2.4 g
PNW
10 g
Internet2 100 gbps
1 g
Commodity Internet
Other NASA Users, e.g., LaRC,
JPL DAACs
Other Users
Seattle, WA
FedNets
Users at Other Federal
Agencies
10 g
Fairbanks, AK
Starlight
ASF
622Mbps
Sioux Falls, SD
Chicago, IL 1 g
Protocol Performance Test
� ESDIS Networks has recently completed an extensive test of OpenSource protocols • FTP, SFTP, FTPS, SCP, HTTP,
HTTPS, BBFTP, GridFTP
� Test variables: • Latency (low, medium, long) • Packet Loss (low, medium, high)
� Testing was completed in-lab • Using a delay and packet loss
simulator
� Lab test results validated by testing between GSFC and Alaska NASA facility.
• From our test results there are protocols which use high-bandwidth circuits more effectively over long distance.
• bbftp • http://doc.in2p3.fr/bbftp/
• GridFtp • http://en.wikipedia.org/wiki/GridFTP • http://toolkit.globus.org/toolkit/data/gridftp/ • http://toolkit.globus.org/toolkit/data/gridftp/
downloads.html
Exploratory Analysis of Remote Sensing Data with Giovanni*
Write Code
*Geospatial Interfactive Online Visualization and Analysis Interface: http://giovanni.gsfc.nasa.gov
Giovanni provides Quick-Start
Exploratory Data Analysis:
no coding necessary
Select Data
Find Download
Learn format
Read
Summarize / Analyze
Analyze
Quality Filter Subset
Publish Derive Conclusions
Visualize
Read
Subset
Filter Quality
Reformat
Explore
Regrid
Visualize
Extract Variables
Giovanni
Exploratory Data
Analysis
Main Analysis
Phase
linked interactive scatterplot + map
