Upload
emery-jenkins
View
214
Download
0
Embed Size (px)
Citation preview
Page 1
Ocean PEATE
Fred Patt
Page 2
Agenda
• Introduction• Design Overview
• Design Description
• Systems Engineering• Closing Remarks
•SDS Data Depository and Distribution Element (SD3E)•Ocean Product Evaluation and Analysis Tool Element (PEATE)•Land Product Evaluation and Analysis Tool Element (PEATE)•Project Science Office Element (PSOE)•NPP Instrument Calibration Support Element (NICSE)•Atmosphere Product Evaluation and Analysis Tool Element (PEATE)•Ozone Product Evaluation and Analysis Tool Element (PEATE)•Sounder Product Evaluation and Analysis Tool Element (PEATE)•Integration and Test System Element (I&TSE)
Page 3
Ocean PEATE Agenda
• Overview
• Interfaces and Design
• Gap Analysis
• ScheduleLandLand
SD3ESD3E
PSOEPSOE
OceanOcean
AtmosphereAtmosphere
OzoneOzone
SounderSounder
I&TSE
NICSE
Page 4
Element Overview
• Acquire VIIRS RDRs, SDRs, and Ocean EDRs from the SD3E and ADS/CLASS
• Assess the quality of the NPP Ocean EDRs for accomplishing NASA’s climate research requirements
• Provide suggested algorithm improvements to the IDPS via the Project Science Working Group (PSWG)
• Process selected data subsets in support of Evaluation and Validation activities
Page 5
Ocean PEATE Requirements (1 of 2)
• Acquire and ingest xDRs from the SD3E (Req 3.4.1.1 to 3.4.1.5)– VIIRS RDRs– VIIRS SDRs– OCC EDRs– SST EDRs– VIIRS IPs
• Acquire and ingest xDRs from ADS/CLASS (Req 3.4.1.6 to 3.4.1.12)
• Acquire and ingest NPP ancillary data sets (3.4.1.13)• Acquire and ingest prelaunch data sets (Req 3.4.1.14)• Catalog and manage NPP data sets (Req 3.4.2.1)• Support distribution of xDRs to VIIRS Ocean Science Team
(VOST) (Req 3.4.5.1 to 3.4.5.5)
Page 6
Ocean PEATE Requirements (2 of 2)
• Support evaluation processing of VIIRS data using IDPS operational code: SDRs, OCC EDRs, SST EDRs (Req 3.4.4.1, 3.4.4.5, 3.4.4.8, 3.4.4.9, 3.4.4.14 to 3.4.4.18)
• Support VIIRS calibration analysis (Req 3.4.4.4, 3.4.4.9, 3.4.4.23 and 3.4.4.24)
• Support evaluation of VIIRS Ocean EDRs by VOST (Req 3.4.4.10 to 3.4.4.13, 3.4.8.1 & 3.4.8.2)– Perform matchups of VIIRS data with SeaBASS data– Support cross-comparison of VIIRS ocean data with concurrent
sensor data sets– Support cross-comparison of VIIRS ocean data with
climatological data sets– Support cross-comparion of VIIRS ocean data with alternative
algorithms and products– Support internal consistency evaluation of VIIRS ocean data
Page 7
Matchup Analysis
• Ocean data granules in ODPS catalog are automatically matched with in situ data
• SeaWiFS Bio-optical Archive and Storage System (SeaBASS) stores and manages in situ holdings from field programs and supported investigators.
• Ocean staff acquire, QC and analyze new data samples
• Over 300,000 in situ samples stored
Page 8
Sensor Cross-Comparisons
• Level-3 parameters (e.g., nLw) compared for common spectral bands
• Common bins extracted and compared over the period of overlap between the sensors
• Comparisons are performed globally (deep water, clear water, coastal), zonally and for specified regions.
Page 9
Internal Consistency Analysis
• Global averages from successive years are overplotted to determine interannual repeatability.
Page 10
Assumptions – Role of I&TSE
• The Ocean PEATE will rely upon the SDS-provided I&TSE (mini-IDPS) to serve as the testbed for algorithm evaluation and potential improvements and will not attempt to duplicate efforts by running the operational code within the PEATE environment.
• The staff of the I&TSE will have the ability to modify any operational code as per Ocean PEATE specification and run the code within the operational environment against a set of PEATE-specified test data sets.
• The I&TSE will receive and install the most current version of the code and executable programs for all operational IDPS software required to produce NASA-evaluated EDRs, starting from RDRs.
• The I&TSE will maintain all processing code under configuration control. The PEATEs will have access to the configuration controlled code.
Page 11
Ocean PEATE Software Support
• The Ocean PEATE will maintain a working knowledge of the VIIRS SDR code and the EDR code for the Ocean Color and SST EDRs.
• The Ocean PEATE will maintain an understanding of the relevant IP characteristics, but will (initially) assume that the IPs are properly generated.
• The Ocean PEATE will design changes to the code in the I&TSE for the purpose of algorithm improvement or problem resolution.
• The Ocean PEATE will develop appropriate test cases and request runs to verify and evaluate the changes.
• The Ocean PEATE will provide recommended changes to the PSG, including a description of the proposed change, effect on the EDR performance, and the evaluation of the test runs.
Page 12
Ocean PEATE Product Improvement
• The Ocean PEATE will evaluate the means (algorithms, ancillary data sources, etc.) to produce the best-quality Ocean Color and SST products from VIIRS data.
• If approved by the PSG and NASA HQ, the Ocean PEATE will produce and evaluate these products alongside the official EDRs.
Page 13
IDPS VIIRS Ocean EDR Data Flow
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
Page 14
IDPS VIIRS Ocean EDR Data Flow
VIIRS_SDR_01RDR Decompression
VIIRS_GEO_01Geolocation
VIIRS_SDR.IM375m SDR
VIIRS_SDR.MOD750m SDR
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
VIIRSSDRVIIRS
SDR
VIIRSGeolocation
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
Page 15
IDPS VIIRS Ocean EDR Data Flow
VIIRS_GD_28Surface Pressure
Adjustment
VIIRS_GD_08750m Granulation
VIIRS_GD_25NAAPS Granulation
VIIRS_GD_27L/W Mask Granulation
VIIRS_GD_09GFS Granulation
VIIRS_GD_12BathymetryGranulation
VIIRS_GD_13TemperatureGranulation
ALL_GD_01Time Interpolation
VIIRS_SDR_01RDR Decompression
VIIRS_GEO_01Geolocation
VIIRS_SDR.IM375m SDR
VIIRS_SDR.MOD750m SDR
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
VIIRS_GD_11Ancillary Profile
VIIRSSDRVIIRS
SDR
VIIRSGeolocation
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
Page 16
IDPS VIIRS Ocean EDR Data Flow
VIIRS_LN_06Active Fires
VIIRS_CM_01Cloud Mask
VIIRS_GD_11Ancillary Profile
VIIRS_GD_28Surface Pressure
Adjustment
VIIRS_GD_08750m Granulation
VIIRS_GD_25NAAPS Granulation
VIIRS_GD_27L/W Mask Granulation
VIIRS_GD_09GFS Granulation
VIIRS_GD_12BathymetryGranulation
VIIRS_GD_13TemperatureGranulation
ALL_GD_01Time Interpolation
VIIRS_SDR_01RDR Decompression
VIIRS_GEO_01Geolocation
VIIRS_SDR.IM375m SDR
VIIRS_SDR.MOD750m SDR
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
VIIRSSDRVIIRS
SDR
VIIRSGeolocation
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
Page 17
IDPS VIIRS Ocean EDR Data Flow
VIIRS_OC_01Ocean Color /
Chlorophyll
VIIRS_ST_02 Surface Temp
VIIRS_SN_03Ice Concentration
VIIRS_ST_01Sea SurfaceTemperature
VIIRS_AR_01Aerosol Type
VIIRS_SN_02Ice Quality
VIIRS_LN_06Active Fires
VIIRS_CL_01Cloud Optical
Properties
VIIRS_CM_01Cloud Mask
VIIRS_GD_11Ancillary Profile
VIIRS_GD_28Surface Pressure
Adjustment
VIIRS_GD_08750m Granulation
VIIRS_GD_25NAAPS Granulation
VIIRS_GD_27L/W Mask Granulation
VIIRS_GD_09GFS Granulation
VIIRS_GD_12BathymetryGranulation
VIIRS_GD_13TemperatureGranulation
ALL_GD_01Time Interpolation
VIIRS_SDR_01RDR Decompression
VIIRS_GEO_01Geolocation
VIIRS_SDR.IM375m SDR
VIIRS_SDR.MOD750m SDR
SSTEDR
OCCEDR
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
VIIRSSDRVIIRS
SDR
VIIRSGeolocation
Page 18
ODPS MODIS Ocean Product Data Flow
ProcessingModule
MODISProduct
DynamicAncillary
Data
StaticAncillary
Data
MOD_PR01Level-0 to 1A
MOD_PR03Geolocation
MSl12Level-1B to 2
MOD_PR02Level-1A to 1B
MODISLevel-0
Land/WaterMask
PlatformATTEPH
Data
OzoneAncillary
Files
METAncillary
Data
MODIS1 km
Level-1B
MODISGeolocation
MODISLevel-1A
MODISSST
MODISOceanColor
Page 19
Ocean PEATE Interface Diagram
PSOESD3E
I&TSE
NICSE
VOST
CLASS(ADS)
AncillaryData
Providers
Ocean Science
Community
Casa-NOSA
xDRs, IPs, Ancillary Data
Ancillary Data
Management Direction
Calibration Updates and Evaluations
Interaction
xDR Eval. Results, Algorithm Updates
Pre-flight
Algorithms,
Data, Info
Software, Data
In Situ Data
OceanPEATE
Algorithm Updates, Test Requests &
Results
xDRs, IPs, Ancillary Data(if unavailable from
SD3E)
Analysis Results, Proposed
Algorithm Updates
CARS
Alternative Algorithm
Results
Page 20
Ocean PEATE External Interfaces (1 of 2)
• SDS Science Data Distribution and Depository Element (SD3E)– Provides NRT access to raw data– Primary source of RDRs– Provides selected SDRs and EDRs
• SDS Integration and Test System Element (I&TSE)– Build and test updates to operational code in mini-IDPS– Run tests on selected data per request of PEATE
• Archive Distribution Segment (ADS) – Primary source for archived data
• xDRs, IPs, Ancillary Data, Operational Algorithm/Source Code and Calibration Products
• Ancillary Data Providers (ADP)– Provides alternate ancillary data sets (e.g., ozone, meteorological data sets)
• CasaNOSA– Serves as the NPP pre-flight repository of Government held data for distribution to
Government user teams– Place to acquire pre-launch NPP algorithms and supported data files
Page 21
Ocean PEATE External Interfaces
• NASA VIIRS Ocean Science Team (VOST)– Coordinate activities with PEATEs and PSOE on xDR and recommended
algorithm improvements. Supports Independent Calibration Validation Activities• NPP Instrument Calibration Support Element (NICSE)
– Provides alternative calibration LUTs and recommended improvements to calibration algorithms
– PEATE provides results of LUT and algorithm tests• Project Science Office Element (PSOE)
– Provides management direction– Accepts algorithm update recommendations
• NASA Ocean CARS– Provides in situ data and results of alternative Ocean algorithms using VIIRS data
• Ocean Science Community– Relies on Ocean PEATE to provide evaluation products and results
Page 22
Ocean PEATE Design
● The NPP Ocean PEATE will be implemented within the framework and facilities of the current NASA Ocean Data Processing System (ODPS)
● This system has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions.
Page 23
Ocean PEATE Design Overview
• Fully automated, distributed data system for acquiring, processing, archiving, and distributing scientific data
• Highly scalable
• Easily adaptable to support multiple concurrent missions
• Graphical user interfaces for controlling and monitoring system functions and activity
• Non-platform specific
Page 24
Ocean PEATE Design Philosophy
• Building-Block approach• Programs are usually small and do one thing well• Programs are less complex and subsequently easy to maintain• Promotes reuse• Programs loosely coupled so testing and production can be done in the
same environment
• Adopt basic standards• ANSI, POSIX, C9x• Use existing technology when possible• Exit statuses indicate successful or failure conditions
Page 25
Components and Subsystems
Page 26
Components and Subsystems (1 of 2)
• RDBMS is the primary element that manages all system activity
• Generic core databases support system infrastructure and non-mission-specific functions
• Mission databases catalogue products and house mission-specific data and procedures
• High level of reuse possible for similar missions; e.g., MODIS Aqua/Terra, SeaWiFS, and OCTS are all ocean-color missions and have similar product suites and requirements
Page 27
Components and Subsystems (2 of 2)
• Relational Database Management System (RDBMS) supports all of the system components (subsystems)
• Scheduler is the primary controlling module within the system, supporting both time- and event-based tasks
• Other subsystems are independent modules, yet rely on the Scheduler for some their functions• Scheduler• Visual Database Cookbook (VDC)• Archive Device Manager (ADM)• Data acquisition and ingest• Level-3 Scheduler• File migration and management• Data distribution
Page 28
Architecture: Hardware
• Processing Servers• Intel-based dual Xeon / AMD-based dual Opteron• 8 GB RAM• Five 72 GB SCSI drives
• Storage Servers• Intel-based P4 / AMD-based single Opteron• 1 GB / 2 GB RAM• 1.5 TB IDE RAID 5 (3ware) / 5.5 TB SATA RAID 6
(Areca)• 2 hot spare drives
• Database Server• Sun V880• 8-16 GB RAM• 6-12 70 GB SCSI HDD
Page 29
Ocean PEATE Data Processing System Current Components
Processing Cluster 34 processing nodes 1.5 TB
Ingest Servers2 SeaSpace ground stations
5 storage nodes6 TB
Distribution Servers (ftp)6 storage nodes
7.2 TBDistribution Servers (web)1 large
server – 3 TB 68 storage nodes
120 TB
Testing Cluster13 test nodes
2 TB
Network Support Systems
Database Server1 large server
876 GB
Backup Servers1 large server
876 GB+2 TB tape5 storage nodes
6 TB
Extreme NetworksBlack Diamond 6816
Gigabit Ethernet switch
Development Servers1 processing node
2 storage nodes2.4 TB
User Desktops
Cal/Val & QC Systems
Mission Operations Systems
Page 30
ODPS COTS and Freeware
• Linux OS (CentOS 4.x)• Solaris OS• Sybase RDBMS• Subversion (source code management)• Pro-active DBA• Interactive Data Language (IDL)• Generic Mapping Tool (GMT)• Netpbm (graphic image toolkit)• HDF5 Library• Languages: C, PERL, SQL
Page 31
Ocean PEATE Gap Analysis
• Acquire, ingest and catalog NPP VIIRS data products: RDRs, SDRs and Ocean EDRs (Data Acquisition & Ingest and Data Cataloging).
• Process Ocean EDRs (SST and OCC) to Level-3 to support data product and algorithm evaluations (Level-3 Scheduler, VDC and Level-3 binner).
• Perform VIIRS EDR matchups with Ocean CARS in situ data (extract code)
• Process SDRs using best current algorithms to produce Ocean Color and SST products (MSl12).
• Produce VIIRS proxy data using VOST-developed software (Schedule and VDC).
• Support browse and distribution of data products for team members (Data Distribution).
Page 32
Basis of Estimate (1 of 2)
• Acquire, ingest and catalog NPP VIIRS data products:– 200 LOC (combined UNIX shell, SQL, C) per new
product, for each of the four VIIRS products
• Process Ocean EDRs (SST and OCC) to Level-3:– 300 C LOC to add VIIRS Ocean EDR input to existing
binning software (L2bin)– 300 SQL LOC per temporal range (10 ranges total)– 200 shell LOC for all ranges
• Perform VIIRS EDR matchups:– 100 C LOC + 10 PERL LOC to add VIIRS Ocean EDR
input to existing extraction software
Page 33
Basis of Estimate (2 of 2)
• Process SDRs using best current algorithms:– 1000 C LOC to add VIIRS SDR input to existing
Level-2 processing software (MSl12)
• Produce VIIRS proxy data:– 100 shell LOC for each processing stage
• Support browse and distribution of data products :– 3000 C LOC to implement browse image generation
– 100 PERL LOC to add VIIRS products to existing browse and order web site
Page 34
Data Storage Estimate
Data Type Daily 1 Year 5 Years
RDR 150 GB 53.5 TB 267.5 TB
SDR (M-band) 194 GB 7 TB (*) 35 TB (*)
OCC EDR 78 GB 2.8 TB (*) 14 TB (*)
SST EDR 17 GB 0.6 TB (*) 3 TB (*)
Inter. Products 70 GB N/A N/A
Ancillary Data 0.1 GB .04 TB .2 TB
Total 509 GB 63.9 TB 319.7 TB
Assumption: (*) Long-term storage is sized for 100% of RDRs
and 10% of SDRs and EDRs; packaged without geolocation.
Page 35
Ocean PEATERequirement Allocation (1 of 3)
SDS RqmtNumber
SDS Rqmt Title Subsystem
3.4 PEATE N/A
3.4.1 PEATE Ingest Data N/A
3.4.1.13.4.1.23.4.1.33.4.1.43.4.1.5
PEATE xDR Ingest from SD3E
PEATE Intermediate Product Ingest from the SD3E
PEATE Official Ancillary Data Ingest from the SD3E
PEATE Calibration Product Ingest from the SD3E
PEATE Request Listings
Data Acquisition and Ingest
3.4.1.63.4.1.73.4.1.83.4.1.93.4.1.103.4.1.113.4.1.12
PEATE Product subscription to ADS
PEATE Ad-hoc requests from ADS
PEATE xDR Ingest from ADS
PEATE Official Ancillary Data from ADS
PEATE Calibration Product Ingest from ADS
PEATE IP Product Ingest from ADS
PEATE ADS query and Request Status Ingest
Data Acquisition and Ingest
3.4.1.133.4.1.143.4.1.15
PEATE Alternate Ancillary Data Ingest
PEATE Pre-launch Data Ingest
PEATE Engineering Service Ingest
Data Acquisition and Ingest
3.4.2.1 Store/Catalog Data File Manage and MigrationArchive Device Manager
3.4.3.1 Manage Software Configuration Subversion
Page 36
Ocean PEATERequirement Allocation (2 of 3)
SDS RqmtNumber
SDS Rqmt Title Subsystem
3.4.4 Process Science Data N/A
3.4.4.1 Operational Algorithms Ingest from SD3E I&TSE
3.4.4.23.4.4.3
PEATEs Management Direction Ingest
PEATEs Instrument Service Report Ingest
PEATE Personnel
3.4.4.4 PEATE NICSE Cal LUT Ingest Subversion
3.4.4.5 PEATE I&TSE Algorithm Status PEATE Personnel
3.4.4.6 Pre-launch Assessment of xDRs and IPs Data Acquisition & IngestData Distribution
3.4.4.7 xDR Implementation Assessment Subversion
3.4.4.8 EDR Algorithms I&TSE
3.4.4.9 SDR and EDR Testing Environment I&TSE
3.4.4.10 Comparison Tools Development IDL
3.4.4.113.4.4.12
Comparing Against Other Satellite and Ground Truth Data
Comparing Against Other Products
Level-3 SchedulerData Distribution
3.4.4.13 Analyzing Algorithm Improvement/Enhancement Data Acquisition & IngestData Distribution
3.4.4.143.4.4.15
Generating SDRs with alternate LUTs
Generating EDRs with alternate LUTs
I&TSE
Page 37
Ocean PEATERequirement Allocation (3 of 3)
SDS RqmtNumber
SDS Rqmt Title Subsystem
3.4.4.16 Generate Test Data VDCData Distribution
3.4.4.173.4.4.183.4.4.19
Developing SDR (L1B) Algorithm Improvement/Enhancement
Developing EDR (L2 and L3) Algorithm Improvement/Enhancement
Testing Algorithm Improvement/Enhancement
I&TSEData Acquisition & Ingest
3.4.4.20 Documentation PEATE Personnel
3.4.4.21 PEATEs Status Report PEATE Personnel
3.4.4.22 PEATE Instrument Service Request PEATE Personnel
3.4.4.23 PEATE Cal Quality Control Response I&TSEData Acquisition & Ingest
3.4.4.24 PEATE Result of Cal Evaluation PEATE Personnel
3.4.5 PEATE Export Data N/A
3.4.5.13.4.5.2
PEATE Science Data Export
PEATE Data Compression
Data Distribution
3.4.8 Ocean PEATE Specific N/A
3.4.8.13.4.8.2
Ocean Product Assessment
Ocean EDR Evaluation
Level-3 SchedulerData Distribution
Page 38
Schedule
Initial Capability (L-18 months)
• All interfaces fully implemented and tested
• Verify initial versions of operational code ported and running in I&TSE
• L-3 product code developed and tested
• Data storage capacity for TBD months
• Initial test products generated for review by VIIRS Ocean Science Team
Full Mission Capability (L-12 months)
• Routine exercise of interfaces to acquire proxy, surrogate (Aqua?) and/or simulated data
• Verify pre-launch version of operational code running in I&TSE
• Browse and distribution capability developed and tested
• Test products routinely acquired based on simulated data and posted for access by VIIRS Ocean Science Team
• Data storage for TBD years
Page 39
Ocean PEATERequirement Mapping
• 100% of requirements (57) allocated to design
Requirements Met
0
5757 5757 57
0
10
20
30
40
50
60
PDR CDR Launch
# o
f R
eq
s
PDRCDRLaunch
Page 40
Ocean PEATERequirement Implementation
• 98% of requirements (56) implemented by full mission capability; add additional storage capacity before launch
Requirements Met
0
5353 5656 57
0
10
20
30
40
50
60
Initial Full Launch
# o
f R
eq
s
InitialFullLaunch
Page 41
Conclusion
• Ocean PEATE will meet all requirements
Next Up: Land PEATE – Ed Masuoka
NPP_SDS_Land.ppt
Page 42
Backup Slides
Page 43
RDBMS
Vendor Client LibraryVendor Library Module
Database Services Layer
C Interface FunctionsPerl DBI Module
PerlScripts
CPrograms
Goal: Isolate RDBMS from system software
To use a differentRDBMS vendor, swap
in a new DatabaseServices Layer
Components and Subsystems: RDBMS
RDBMS
Page 44
Scheduler
• C program with supporting database procedures
• Runs in a daemon-like state
• Primary system element responsible for coordinating most of the system activity
• Monitors task records in a to-do list database table
• Runs tasks according to defined task attributes
• Standard job-shell interface allows new programs to be quickly adapted for Scheduler control
Subsystems: Scheduler
Page 45
VDC
• Highly scalable, distributed infrastructure for concurrent processing of serial streams (e.g., L0 L1A L1B L2)
• Suite of C programs with supporting database procedures
• Uses recipes to encapsulate data-specific processing schemes, parameters, and pre-processing rules
• Virtual Processing Units (VPUs) serve as distinct distributed processing resources
• VPUs dynamically allocated based on available time and current OS load
• Comprehensive, user-defined processing priorities
Subsystems: VDC
Page 46
VDC: Ancillary Data Stager
• Runs in a daemon-like state
• Monitors entries in the processing queue and runs the ancillary-select procedure for each entry’s recipe
• Updates queue-entry status when ancillary data are available
• Governed by currently configured processing priorities
VDC: Ancillary Data Stager
Page 47
VDC: MakeVDC
• Selects processing-queue entries that have met pre-processing requirements
• Generate VDC job files according to configured priorities
• Runs as a Scheduler task, so it can easily be configured to run as often as needed to keep the VDC queue full
VDC: MakeVDC
Page 48
VDC: Engine
• Runs in a daemon-like state on each processing server
• Each instance of the VDC Engine actively competes for jobs that it is allowed to run, based on priority and length of time in the queue
• Monitors and manages processing resources
• Initializes processing streams
• Invokes recipe steps and monitors step-execution time
• Handles operator-requested stream actions
VDC: Engine
Page 49
Archive Device Manager
• User defines logical pools of storage devices
• Processes request a device in a specific pool
• ADM returns information for a storage device in the requested pool
• If auto-cycling is enabled, the ADM time-stamps the record for the selected device, so a different device within the pool will be selected for the next request
• Disk-monitor process polls all devices periodically to record usage statistics and invoke threshold handlers
Subsystems: Archive Device Manager (ADM)
Page 50
Data Acquisition and Ingest (1 of 2)
• Data types and sources are described in the database
• Active, passive, and periodic notification methods
• Active method scans remote systems for new files and populates the ingest queue
• Passive method waits for arrival of e-mail messages describing type and location of new file and populates the ingest queue
• Periodic method schedules transfers of files at user-specified intervals
Subsystems: Data Acquisition and Ingest
Page 51
Data Acquisition and Ingest (2 of 2)
• File transfers handled by ingest daemons and Scheduler tasks
• FTP, RCP, and SCP transfer protocols supported
• Generic script handles the file transfer and then hands off to data-specific post-ingest scripts
Subsystems: Data Acquisition and Ingest
Page 52
Level-3 Scheduler
• Responsible for scheduling processing for level-3 composite products
• Runs as a Scheduler task
• Configuration is database driven
• Mission-specific stored procedures are invoked to identify input files for a composite product
Subsystems: Level-3 Scheduler
Page 53
File Migration and Management
• Responsible for compressing files and migrating them to their various destinations
• Event- or time-based actions
• Queries associated with each action are run periodically by a Scheduler task to select files that are eligible for some type of migratory action and populate a migration queue
• Command-line queuing for file removal and delayed copies
• Migration daemons query the migration queue, perform registered actions on the files, and update catalog tables
Subsystems: File Migration and Management
Page 54
Data Distribution
• Interactive, web-based Data Ordering System, currently supporting SeaWiFS and MODIS Aqua
• Data Subscription System, currently supporting MODIS Aqua, allows users to define region and products of interest
• Order and Subscription Manager Daemons monitor the order and subscription queues and stage files on FTP servers (stage rate ~12 GB/hr)
• Near-real-time data extraction and image support
• Web-CGI applications that allow users to view and update their orders and subscriptions
Subsystems: Data Distribution
Page 55
New Data/Source Acquisition and Ingest
• Insert DB records for new data-source servers and data types
• Compose data-specific post-ingest scripts
• Configure ingest daemons if that method is going to be used for any of the new data types
• Define archive-device pools for product storage
Page 56
New Data Product Cataloging
• Insert record into the core tables (catalog DB) that describe the mission and products
• Create mission specific database and objects, reusing objects from existing mission databases where applicable
• Compose a program to provide geographical L1 meta-data information including granule start and stop times, day-night flag, and geographic coordinates, e.g., MSl1info
• Compose functions for DB-metaload program, so meta-data files can be read and product tables can be populated
• Configure file migration and management actions for new mission data
Page 57
New Data Processing Streams
• Define a recipe for each distinct serial processing stream• Insert records for each recipe and each recipe step• Compose a job template for each recipe• Compose a ancillary-selection procedure for each recipe
that requires ancillary data• Compose scripts for each science program associated
with the new mission's data• Insert record in recipe-constraints table for each
processing host allowed to run a recipe• Compose AP-load procedure for each base data type that
can be processed with a recipe• Update the reproc program to support each base data type
that has an AP-load procedure• Configure L3-Scheduler for desired composite processing
Page 58
VIIRS Ocean Level-3 Processing
• Develop data input routines to support processing of VIIRS Ocean EDRs using existing multi-mission Level-3 binning software
Page 59
New Data Product Distribution
• Update Subscription CGI to support new mission data
• Compose match-subscription procedure
• If data extraction and mapping is to be supported:• Compose extraction and mapping programs• Compose match-XM-requests procedure• Modify XM CGI to support new mission products
• Compose browser capabilities for new mission products
• Provide FTP access to new mission products