Preservation of NASA’s Earth Observation Data
EOSDIS Science Operations, ESDIS Project Code 423Goddard Space Flight Center, Greenbelt, MD 20771
ESIP Federation Meeting, Santa Barbara, CA.July 9, 2009
EOSDIS and NASA’s Earth SciencePreservation Activities
•NASA’s Earth Science Data and Information System Project will maintain distributed active archives to continue servicing the science and applications communities.
–responsibilities for data management include, yet go beyond, the life of EOS missions and continue with missions recommended by Decadal Survey. –keep all data important to active science research and applications in place at the EOSDIS Data Centers.
04/20/23 2
04/20/23 3
Preservation of Earth Observation Data
• NASA has observational data that will need long term preservation
Selected data sets will be organized for the National Archives and Records Retention (NARA) based on mission end of life and other critical data needs.
Our data life cycle ends with a consolidated and organized permanent record which specifically means instrument data co-located with the product generation source code, allowing for regenerated products when necessary, documentation, submitted and accepted to NARA.
Organization
• A number of events need to occur with each instrument for a successful archive strategy, but in general:
– ESDIS will coordinate joint long term archive activities with each EOS Instrument team through appropriate working groups.
– The working group will agree on source of information, responsibility for inventory and content, standards to follow and any corresponding preparation tasks.
– ESDIS will align the long term archive plan with current NASA’s records retention schedule rules.
– The entire archive process will be a transparent activity to all involved, including the science community.
– The working group will stay active through the entire archive process to ensure goals are met.
04/20/23 4
04/20/23 5
Characterizing EOS Electronic Records
EOS Data Product categories:
Level 0 - Unprocessed instrument/payload data at full resolution.Level 1 - Instrument data with radiometric and geometric calibration and geo-referencesLevel 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data.Level 3 - Variables mapped on uniform space-time grid scales
Science Mission Drivers
• EP-TOMS and ICESat 1 where chosen as the first data sets to enter the long term archive process for the reasons below:
• EP-TOMS– Data flow discontinued in Dec 2005. Currently near the end of the project,
should be forming or have identified final records.– Team members should still be available to support a joint working group (OMI is
the continuity mission – different team with some overlap in membership).– A critical national historical dataset of ozone records for NASA in support of
Clean Air Act Amendments of 1990.
• ICESat 1– Instrument may fail soon and we may suddenly need to finalize records.– The instrument is a reasonable representation of the kind of information that
needs to be organized for the archive, and the learning curve faced by most other flight and science projects concerning use of preservation standards.
04/20/23 6
The value of the OAIS Ref. Model
• A common language that enables coordinated planning and implementation by partnering agencies and institutions for permanent records retention– NOAA, USGS, NARA
• A structure for linking representation information with the data– Example – MODIS LTA Handbook– Provides a logical structure, avoiding having to rethink
what goes into an AIP
04/20/23 7
Experience using OAIS RM
• Several of the data centers have used the OAIS Reference Model when evolving, updating or re-designing their archive– NASA Langley (LARC) Atmospheric Sciences Data
Center data system evolution– PO.DAAC (JPL) using OAIS Reference Model, ISO
19115 as design constraint– SEDAC and Columbia University using OAIS RM in
implementation of COTS packages Fedora and VITAL
04/20/23 8
Atmospheric Sciences Data CenterArchive, Next Generation (or ANGē)
• ANGē’s design is based on the Open Archival Information System (OAIS) Reference Model. • ANGē development is organized by each of the primary OAIS model component.• One of primary goals of ANGē’s design is to minimize the changes required for the interfaces to any current data providers.
An additional translation layer will be developed to allow current interface protocols to interact with ANGē’s OAIS compliant services. However, current data providers may elect to modify existing interfaces to take advance of better services (e.g. automated notification of data receipt). The diagram depicts this approach using nomenclature from the OAIS reference model.
04/20/23 9
CONCEPT DIAGRAM: Layered Approach to Insulate Core Functions from Varying Provider & User Needs
PRE-INGEST
CURRENT PRODUCER CONSUMERS
Requests; Responses; SIP: Data, Content Info., Preservation Descript. Info.
Data Request DIP
Queries; Result Sets
Orders; Data Packages Requests;
Data Input
Responses
DATA SEARCH
Search Parameters
Content Info.
Archive Admin.
Queries Statistics, Diagnostics
DATA ORDER
AIP AIP
AIP
Ingest Data Management Access
Archival Storage Administration Preservation Planning AIP
DIP
DIP
SIP
SIP media
database
CORE ARCHIVE
SIP
FUTURE PRODUCER
Model for Web-Based Data Submission and Workflow
DataProducer
ProvideFilesAndDescriptions
DataDeposit
DescribeData Set
DataDescription
GrantIntellectualPropertyRights
SubmissionAgreement
AppraiseAndApproveSubmissionInformationPackage
Review
TransformValues toXMLMetadata
Transformation
ArchivalInformationPackageInDigitalRepository
Ingest
Login with ID & PasswordAuthentication
One or More Submission Sessions
Notifications and Requests
Communication
Source: Downs and Chen. Creating a Trustworthy Digital Repository for a Long-Term Archive of Interdisciplinary Data: A Case Study. 21st International CODATA Conference, 5-8 October, 2008 Kyiv, Ukraine.
Socioeconomic Data CenterSEDAC
backup
04/20/23 11
04/20/23 12
EOS Data and Information System
• Earth Observing System (EOS) Data and Information System (EOSDIS) was conceived and built in the 1990’s as the spacecraft control, data acquisition, processing system for the EOS flight missions and the archive and distribution system for all NASA Earth science data
• EOSDIS consists of several Distributed Active Archive Centers (DAACs), and Science Investigator-led Processing Systems (SIPSs), and a Core System– DAACs provide processing, archive and distribution services for
EOS data.• Each DAAC holds different kinds of scientific data that includes
atmospheric, oceanic, land use/cover, space geodesy, cryospheric, and biospheric.
• The DAACs make EOS data publically available to all users, through online query, metadata, and retrieval systems directly to the user community.
– SIPSs produce standard products from EOS instrument data and send them to the DAACs for archiving
– Core System consists of components used to support major data flows at several DAACs and SIPSs
04/20/23 13
EOS electronic records life cycle
• Mission based Earth observing research programs– Many instruments and levels (maturity) of data– Many different activity timelines, retention is event driven
• Typical period for operating instrument (~3-15 yrs)– Nominally the design life plus optimal performance period– Ongoing data acquisition, product generation, archive & distribution– Ongoing science research and applications for societal benefit– Algorithm adjustments and reprocessing from beginning of mission can
occur– Provides inputs to next research & operational satellite programs
• Mission End-of-Life period, Decommissioning (~3-5 yrs)– Final reprocessing for products from the entire mission– Final research results, inputs to next missions– Organization of mission data for permanent long term retention
• Permanent electronic records are created during both periods
04/20/23 14
Earth Science Missions
2011 2012 2013 20142007 2008 2009 20102003 2004 2005 20061999 2000 2001 20021995 1996 1997 19981991 1992 1993 1994
2011 2012 2013 20142007 2008 2009 20102003 2004 2005 20061999 2000 2001 20021995 1996 1997 19981991 1992 1993 1994
10/1984 - 2 years planned mission life
10/1978 - 10 yrs m life
ERBS (SAGE II)
ERS-1
NIMBUS-7 (TOMS I)
Meteor 3 TOMS (TOMS II)
UARS
TOPEX/Poseidon
JERS-1
OrbView-2 (SeaStar)
ERS-2
Earth Probe TOMS (TOMS III)
RadarSat 1
ADEOS I early failure
Jason-1
ACRIMSAT
Aqua
ICESat
Aura
SORCE
Landsat 7
QuikSCAT
Terra
TRMM Heritage Missions
EOS Era Missions
No Planned EOL
KEY
Ongoing Data Access
GRACE
Transition to USGS
Extended Mission Life
Planned Mission Life
ADEOS II early failurePost Mission Support
April 2007 NASA Earth Science Senior Review
Science Data Available
CERES data only
Meteor 3M (SAGE III)
EOSDIS Manages DataFor all EOS Measurements
04/20/23 15
ACRIMSAT(12/99)
Solar Output
ACRIM
SORCE (1/03)Solar Irradiance
TIM, SIM, XPSSOLSTICE
Terra (12/99)EnergyBudget
CERES
Albedo,Aerosols,
Vegetation
MISR
SurfaceImaging
MODISASTER
LowerAtmospheric
Chemistry
MOPITT
Aqua (5/02)EnergyBudget
CERES
SurfaceImaging
MODIS
AtmosphericSounders
AMSR-EAIRS/AMSU/HSB
Aura (7/04)TraceGases
TES
AtmosphericDynamics and
Chemistry
HIRDLS,MLS, OMI
Mesosphere
Stratosphere
Troposphere
TRMM(11/97)
Energy Budget
CERES, LIS
EvaporationH2O
Sea Surface Winds
OceanBiology
LandBiology
Volcanology
Ecosystem Dynamics
O3
precursor gases,
aerosols
Mission & Science 04102007.ppt
O3
ClOBrOOH
trace gasesaerosols
Land Ice and Snow Cover
Rain
QuikScat(6/99)
Sea Surface Winds
SeaWinds
Temperature Moisture
Sea Ice
Fire Occurrence
METEOR 3M(12/01)
Aerosol Trace Gases
SAGE III
CloudSat(4/06)
Cloud Properties
CPR CALIPSO(4/06)
Cloud, AerosolProperties
CALIOP
Jason (12/01)Ocean Altimetry
Poseidon/JMR/DORIS
Rain
ICESat(1/03)
Ice Topographyand Altimetry
GLAS
04/20/23 16
Tracking & Data Relay Satellite
(TDRS)
Instrument Teamsand Science
Investigator-ledProcessing System
(SIPS)
EOSSpacecraft
Direct Broadcast/Direct Readout
Stations
DirectBroadcast
(DB)
EOS Data and Operations System
(EDOS) Data Processing
White SandsComplex
(WSC)
EOS PolarGround Stations
EOSDIS and Related Data
Centers
Internet
(Search, order,
distribution)
Research
Education
Value-AddedProviders
InteragencyData Centers
InternationalPartners
EarthSystem Models
Decision Support Systems
Distribution and DataAccess
Data Acquisition Flight Operations,Data Capture,
Initial Processing,Backup Archive
Data Transport
to Data Centers/
SIPSs
Science Data Processing,Data Mgmt, Interoperable
Data Archive, & Distribution
NASA Integrated Services Network (NISN)
Mission Services
Mission Operations Science Operations
EOS Operations Center (EOC)
Mission Control