CERES FM-5 NPP Science Processing Delta Design Review GSFC, MD August 26, 2008 Erika Geier

CERES FM-5 Delta Design Review 1

CERES FM-5NPP Science Processing

Delta Design ReviewGSFC, MD

August 26, 2008

Erika GeierJim Closs

Denise CooperSunny Sun-Mack

Lee BoddenMichael Little

NASA Langley Research Center


Agenda

CERES Introduction

Operations Concept

Design Assumptions

Science Data Processing

Requirements

Instrument Subsystem Design

Clouds Subsystem Design

Testing Strategy

ASDC Implementation

Schedule

Risk Analysis

Issues/Concerns


CERES Mission Ground Systems

• NASA Langley Research Center (LaRC) provides critical cloud and Earth radiation budget climate data records to support global climate change research

• Past ERBE and current CERES instruments on TRMM (1), Aqua (2), and Terra (2), have provided 24 years of sustained radiation budget measurements

• LaRC has collected over 30 Instrument years of CERES data from TRMM, Terra, and Aqua and has processed over 90% of that data to date


CERES Current Capabilities

• Extensive LaRC CERES experience accumulated through the support of the five CERES instruments currently in orbit

• Existing scientific expertise for the development, calibration, management and application of CERES data products

• Data Center Operations already in place providing ingest, archival, production, management and distribution of CERES data products

• Existing Science Computing Facility (SCF) for scientific analysis, investigations and development of CERES data products

• Highly-experienced staff already in place supporting:– Ongoing algorithm investigation, development and enhancements– Implementation of algorithms into CERES Data Management System– Enhancements and development to current and new CERES data production systems– Production and distribution of CERES data products in a configured operations

• The CERES DM task was successfully appraised at CMMI Capability Level 3 /Equivalent Maturity Level 2


Science

CERES Climate Analysis Research System (CARS) Organization

Data Management

Team(DMT)

Atmospheric Sciences

Data Center(ASDC)

• Implements algorithms• Maintains software• Verifies data• Assists in validation• Provides CM and documentation support

• Ingests data• Places operational software in production• Produces data sets• Distributes data sets• Archives data• Provides User Services

• Derives & refines algorithms• Validates algorithms• Validates CERES data sets• Writes Quality Summary

Algorithm Development Algorithm Implementation Data Production


Required CERES CARS Functionality

• Provide Land PEATE sub-sampling software for aggregated

VIIRS SDRs (Science and DMT)

• Acknowledge data from SDS Land PEATE (ASDC)

• Ingest data from SDS Land PEATE & other sources (ASDC)

• Produce, archive, distribute data products (ASDC)

• Report status of collection (ASDC)

• Support user access to collection (ASDC)


CERES Operations Concept

• CERES instrument raw data transmitted to the C3S and IDPS

• IDPS creates CERES Raw Data Records (RDRs) and provides RDRs to the SDS SD3E and to ADS/CLASS

• SD3E provides CERES RDRs and VIIRS xDRs to the Land PEATE

• Land PEATE passes CERES RDRs along to ASDC

• Land PEATE runs VIIRS Sub-sampler; provides output to ASDC

• ASDC ingests and archives inputs coming from Land PEATE

• ASDC generates and archives higher-level CERES products

• ASDC distributes products to science researchers and public


CERES Operational Data Flow

SD3E L-PEATE

ADS

IDPS

SDS

All RDRs

All xDRs

CERES RDRs, VIIRS Sub-samples

& aerosols

C3S

Commands, Loads and requests

RDRsSDRsEDRs

IPs

CERES InstrumentOps Team

Svalbard

Cmd &

Tlm

Mission Data

Existing data, agreements

Data Providers

Mission Notices and Data

RDRs to PST

CERESCARS

Data and

Science Operations

Mission Notices and Data requests

Data Distribution

Data Users

• Reuse existing systems and interfaces• System enhancements for NPP CERES

NPOESS Systems

NPP Systems

NOAA Systems

CERES Systems


CERES Data Processing Design Assumptions

• CERES CARS data processing development and operations leverages the existing personnel, procedures, production systems, and infrastructure already in place supporting CERES instruments on EOS Terra and EOS Aqua Missions

• CERES CARS will use existing interfaces at the LaRC ASDC

• CERES CARS software development is extensively reusing existing Terra/Aqua processing codes

• Primary required NPP related enhancements are:– Use CERES raw data in new Raw Data Record (RDR) format:

Develop a preprocessor to distribute/sort RDR contents into Aqua-like files that can be used as inputs to existing CERES processing stream

– Use VIIRS data products: Use VIIRS sub-sampled calibrated radiance and aerosol products in place of

instantaneous MODIS data.


NOAA IDPS Assumptions

• Acquire CCSDS Packets

• HDF Wrap CCSDS Packets

• Generate & Stage RDRs for all APIDs for SDS & CLASS

• Maintain CDFCB

• Provide samples of each CERES RDR

• Provide 28 hours of CERES RDRs for testing CERES codes at LaRC


SDS/Land PEATE Assumptions

• Subscribe to SD3E for all CERES RDRs (Science, Diagnostic, and Housekeeping/Telemetry)

• Acquire CERES RDRs and forward to ASDC via existing Land PEATE interface

• Support CERES reordering of missing data• Produce aggregated VIIRS SDRs (MODIS HDF-EOS

format) in 5 minute granules (VIAE, VMAE, VDAE). Sub-sample and ship 5 minute granules (VIMD) to ASDC

• Produce VIIRS aerosol in 5 minute granules (VAOT) and ship to ASDC

• Post-launch: When VIIRS calibration and/or algorithm changes, reprocess VIIRS from covers-open forward and ship to ASDC to support CDRs


New Requirement Received, Logged,

andAnalyzed

Requirement Accepted

and Assigned

Subsystem Updates and Unit

Testing

Updates Integrated Into

Baseline

Subsystem Science Testing

Pre-CM Testing

Delivery to CM

CM Testing

Release to SIT

Operational Testing at

ASDC

ValRx Testing

RequirementsManagement

Subsystem Product Integration and

Verification

CM Validation and Product Integration ASDC Validation

Software Development Process

Software Development Process is in the following document currently under CERES Management review:

Clouds and the Earth’s Radiant Energy System (CERES)National Polar-orbiting Operational Environmental Satellite

System (NPOESS) Preparatory Project (NPP)Data Management System (DMS)

Software Management Plan Version 0


Science Data Processing Approach for CERES FM-5

Processing is divided into 2 Streams: • Instrument and ERBE-like

– Autonomous stream, dependent only on CERES data– In production within 48 hours of power-on– Aids in verifying Instrument health– Critical for establishing CERES calibration/validation– Produces ERBE-like data sets

Data sets exist for CERES instruments on TRMM, Terra, and Aqua Similar data sets exist for ERBE instruments on ERBS, NOAA-9 and NOAA-10

• Fused data sets beginning with Cloud/Convolution/Inversion processing and the SSF

– Use inputs from imager and other data sources Higher resolution imager data is Point Spread Function (PSF) weighted

– Produces higher quality climate data Data sets exist for CERES instruments on TRMM, Terra, and Aqua

– Will not be in production immediately after covers open Requirement is to ingest VIIRS radiance, geolocation, and aerosol data when it becomes

available from Land PEATE


Simplified CERES Processing Flow

SS4Compute Clouds,

Convolve withCERES, compare

TOA/surfacefluxes

CERES RDRs

SSI Instrument

SS2 Instantaneous

ERBE-like

SS3 ERBE-like

TISA

ES-4

ES-9

BDS ES-8

IES

Autonomous Subsystems affected by changing NPP inputs

Autonomous CERES Subsystems

VIIRS Radiances

GeolocationAerosols

SS9TISA

Gridding

SS10TISA

AveragingSRBAVGSSF SFC

SS5Compute fluxes

at levels

SS6TISA

Gridding

SS7 & 8TISA

Averaging

AVG

SYN

CRS FSW ZAVG

Subsystems affected by NPP input


Current CERES Climate Data Record Production Architecture

Grid TOA andSurface Fluxes

9

ERBE-likeAveraging to

Monthly TOAFluxes

3

Grid GEONarrowband

Radiances11

GEO:Geostationary

Narrowband Radiances

TimeInterpolate, Compute

Fluxes7

Grid Radiative

Fluxes andClouds

6

MOA:Meteorological, Ozone,

and Aerosol Data

ES-8: ERBE-likeInstantaneous

TOA Estimates

ERBE-likeInversion to

InstantaneousTOA Fluxes

2

RegridHumidity and

Temperature Fields12

BDS:BiDirectional

Scans

SRBAVG:Monthly

TOA/Surface Averages

SYNI:Intermediate

Synoptic RadiativeFluxes and Clouds

ComputeMonthly & Regional

TOA and SurfaceAverages

10

DetermineCloud Properties, TOA

and Surface Fluxes4

Geolocateand Calibrate

EarthRadiances

1

SSF: SingleScanner Footprint

TOA/Surface Fluxes and Clouds

CRS: Clouds and Radiative

Swath

MODIS CID: VIIRS CID:Cloud Imager

Data

SURFMAP:Surface Map

INSTR:Instrument

Production Data Set

EID6:ERBE-like

Regional Data

AVG:Monthly Regional Radiative

Fluxes & CloudsZAVG:

Monthly Zonal & Global Radiative Fluxes & Clouds

ComputeRegional, Zonal and

Global Averages8

GGEO:Gridded GEO

Narrowband RadiancesFSW: Monthly

Gridded Radiative Fluxes and Clouds

IES: Instrument Earth Scans

CRH:Clear Reflectance

History

GAP:Gridded Analysis

Product

OPD:Ozone

Profile Data

MWH:Microwave

Humidity

APD:Aerosol Data

SFC: MonthlyGridded

TOA/SurfaceFluxes and Clouds

ES-9:ERBE-like

Monthly Regional Averages

ES-4:ERBE-like Monthly

Geographical Averages

ComputeSurface &

Atmospheric Radiative

Fluxes5

SYNSynoptic RadiativeFluxes & Clouds


CERES Input Data

Type of Data Parameter

Description

Freq Source Comments

CERES RDRs Instrument level 0 data, attitude, and ephemeris

~131/day Land PEATE Expecting 3 types of RDRs: Science, Telemetry, and Diagnostic

VIIRS L1B Calibrated Radiance Data

Imager radiances & Geolocation

Aerosols

288/day

~144/day

Land PEATE Have identified radiance subsets and will provide associated code

Aerosol data Aerosol (Coln) Optical thickness, type/size

1/day MODAPS For Terra/Aqua using MODIS MOD08 and MATCH. Plan to do same for NPP

Ozone data Ozone Profile 1/day SMOBA

1/day OMI

NCEP SMOBA

OMI Meteorological data 3-D Met Data

2-D atmospheric data

2-D constants

4/day

24/day

1

GMAO

Precipitable Water 2-D constants 2/day Global Hydrology Resource Center (GHRC)

Geostationary data MCIDAS data from 5 geostationary satellites per month

120/day University of Wisconsin Space Science and Engineering Center (SSEC)

Only every 3rd hour is used for production

SURFMAP(Snow/Ice) Snow/Ice Map 4/day NCEP/NESDISSURFMAP(Snow/Ice) Snow/Ice Map 1/day NSIDC

Existing Data Source


CERES Publicly Available Output Products

Data Product Data Product Name File Freq

BDS Bidirectional Scan 1/day

ES-8 ERBE-Like Inst TOA Filtered Radiances 1/day

ES-9 ERBE-Like Monthly Regional Averages 1/mo

ES-4 ERBE-like Monthly Geographical Averages 1/mo

SSF Single Scanner Footprint TOA/Surface Fluxes & Clouds 1/hr

SFC Monthly Gridded Radiative Fluxes & Clouds 36/mo

SRBAVG Monthly TOA/Surface Averages 5/mo

CRS Clouds and Radiative Swath 1/hr

FSW Monthly Gridded Radiative Fluxes & Clouds 60/mo

SYN Synoptic Radiative Fluxes and Clouds 1/day

AVG Monthly Regional Radiative Fluxes and Clouds 1/mo

ZAVG Monthly Zonal and Global Radiative Fluxes & Clouds 1/mo


Data Production Scenarios

• CERES strives to receive all possible FM-5 level 0 data and VIIRS data• Instrument

– Ingest manager at ASDC to monitor missing RDR files and request those files from Land PEATE– Land PEATE has agreed to provide CERES 99% of RDR files produced by IDPS within 1 month– Data is actually processed three times

Baseline1-QC run ~ 6 hours after end of the day. Used for quick instrument assessment. Files not needed after Edition1-CV processes.

Edition1-CV run ~5-20 days after the end of the month. Assume all available data has been ingested. Calibration/Validation data set used to compute gains and spectral response functions.

Edition2 run 6-12 months later using Edition1-CV as input. Applies best available instrument gains and spectral response functions.

• Cloud/Convolution/Inversion processing of the SSF– Ingest manager at ASDC monitors missing MODIS files and requests missing files from MODAPS.

VIIRS files to be handled in similar fashion. – Once in routine production, Land PEATE has agreed to provide CERES 99% of available VIIRS files

within 2 months of data date.– Requires CERES and VIIRS radiance/geolocation data to avoid data gaps in SSF files

If either one missing, a data gap results VIIRS aerosol data also needed. If unavailable, aerosol parameters set to CERES default fill values and no data gap

results.– Current Terra and Aqua SSFs typically run 6-12 months after data date. NPP processing will be very

similar to Terra and Aqua.


CERES CARS Requirements Management

• CERES Level 3 and Level 4 requirements are being baselined in a CM-controlled document

• CERES Level 3 requirements have been reviewed by the SDS and will be provided to the SDS for inclusion in the SDS Requirements Specification

• Requirements change requires approval by Project Management and the Configuration Control Board

• Requirements are mapped into subsystem(s) for implementation

• Requirements are mapped into tests for verification and validation


Land PEATE CERES CARS

CERES RDRs

VIIRS Geo/Rad

VIIRS Aerosols

VIIRS Subsample code

CERES RDRs

Sub-sampling

VIIRS Geo/Rad

VIIRS Aerosols

Instrumentprocessing

ERBE-likeprocessing

Cloudsprocessing

Archive, Distribution

User Community

CERES Data Sources

SD3E

1- The Land PEATE receives CERES RDRs and VIIRS xDRs from the SD3E. 2- CERES RDRs are passed directly to the CERES CARS, where they are processed by the Instrument subsystem to produce inputs to the ERBE-like and Cloud subsystems. 3- The Land PEATE produces MODIS-like VIIRS radiance/geolocation and aerosol files.4- The Land PEATE sub-samples the MODIS-like VIIRS radiance/geolocation data using software provided by CERES.5- The Land PEATE sends the sub-sampled VIIRS files and the MODIS-like VIIRS aerosol files to the CERES CARS.6- The CERES CARS runs the downstream subsystems, beginning with Clouds, as data becomes available.7- Climate-quality data products are archived and made available to the user community along with Data Quality Summaries and user support.


CERES SDS Requirements

NPP CERES FM-5 Science Data Segment Requirements:

• The ASDC Ingest subsystem shall have the capability of ingesting, verify by Checksum, and archiving CERES RDRs, sub-sampled VIIRS radiance/geolocation files, and VIIRS aerosol data received from the Land PEATE.

• The CERES CARS Preprocessor of the Instrument Subsystem shall have the capability of producing level-0, attitude, and ephemeris files as expected by the processing software.

• The CERES DMT shall have the capability of generating and delivering to the ASDC the following Science Subsystem code:- Instrument (at launch)- ERBE-like (at launch)- Clouds and subsequent subsystem software (12+ months after launch)

• The CERES DMT shall provide software to the Land PEATE to sub-sample VIIRS radiance/geolocation files, and work with the Land PEATE to ensure that the sub-sampling code executes properly.


CERES SDS Requirements

NPP CERES FM-5 Science Data Segment Requirements (cont.):

• The CERES DMT shall have the capability of modifying and redelivering production code for all 12 CERES Subsystems to produce CERES NPP data products.

• The ASDC shall have the capability of generating, archiving, and disseminating climate-quality data products.

• The CERES DMT and Science Team shall have the capability of validating and writing quality summaries for all CERES data products.

• The ASDC shall have the capability of providing quality summaries and user support for data customers.


Requirements Verification

1. Baseline LaRC FM-5 L4 requirements by placing CERES CARS Requirements Specification Document on NX (CERES CARS, 09/2008)

2. Baseline SDS FM-5 L3 requirements by adding CERES to SDS Requirements Specification Document (SDS, 09/2008)

3. Map FM-5 L3 requirements to SDS Testing Scorecard and schedule (Vic Buczkowski, 10/2008)

4. Produce Requirements Traceability Matrix to map LaRC L4 requirements to CERES CARS Subsystems and test cases (CERES CARS, 10/2008)

5. Conduct FM-5 SDS L3 testing as documented in SDS Testing Scorecard (SDS/LaRC Test Team, (02/2009)

6. Conduct FM-5 LaRC L4 testing as documented in LaRC Requirements Traceability Matrix (LaRC Test Team, (mid-2009)


Instrument Subsystem Design Approach

Instrument Design Approachand

CERES DMT RDR Tests

Denise Cooper

Contributing Team Members:Ashley Alford

Dale WalikainenMark Timcoe


Instrument Design Approach

• Extensive reuse of existing software for instrument subsystem– Proven system used for current CERES processing– Majority of software written in Ada

• Newly developed code to be in C++– RDR Preprocessor

Creates 24-hr Level-0 and Toolkit compliant Ephemeris and Attitude data files

• Ada code will be updated to allow processing of FM-5 data

• PGE to process incoming NPP Level-0 data will be the first priority, other PGEs will be converted following successful testing of this PGE.


Instrument Design Approach

• PGE providing data to ERBE-like already in C, only changes will be to allow processing of FM-5 data.

• Initial version of all code will use the SDP Toolkit due to schedule constraints. Updates to remove dependency on the SDP Toolkit expected after launch.

• CERESlib updates to create a C-version of existing F90 Metadata creation routines.– Facilitate integration of existing Metadata with Ada, C and C++ software.


NPP Instrument Subsystem

CERES RDRs

L0 Data

Coeff Data

Geolocate &

Calibrate

BDSs

IESs

Pre-Es8Generator

Pre-ES8

To Convolution

To ERBE-Like

AttitudeData

Preprocessor

~133 Science & Diagnostic RDRswith Diary data

2 files needed to process(time span noon yesterday -

noon today)

14 files needed to process(time span hr 22 day before -

hr 00 next day)

Up to 3 files per day(Science, Calibration &

Diagnostic)

Up to 24 files per day(HDF4 Vdata containing

Geolocation valuesFiltered radiances)

Up to 6 files per dayBDS, BDSS, BDSD,

BDSM, BDSP & BDSG(HDF4 SDS & Vdatas containing

Geolocation valuesRaw data counts

Filtered radiancesInstrument H&S info)

1 file per day(Binary containing

Geolocated, Filtered radiances)

EphemerisData

AttitudeData

AttitudeData

AttitudeData

AttitudeData


Spectral ResponseFunction

ModtranData

UnfilteringCoefficients

Unfiltering

ERBE-like Spectral Corrections

No code changes needed to support NPP


DailyInversion

Pres8EID6

ES4

ES9

ES8

MonthlyAveraging

ERBE-like Inversion

From Instrument Subsystem

No code changes needed to support NPP

Unfiltering Coefficients

1 file per day(Binary containing

Geolocated, Filtered radiances)

1 file per day(HDF-EOS4 containing

GeolocatedFiltered & Unfiltered radiances &

fluxes)

1 file per day(Binary database

Info)

1 file per month(HDF4 containing

Regional, Zonal & GlobalGridded fluxes)

1 file per month(HDF4 containing

Global Average fluxes)


CERES DMT RDR Tests

• Objectives– Ensure the RDR preprocessor is able to read the CERES Science and

Diagnostic RDR HDF5 files and create the properly formatted Level-0, Ephemeris and Attitude data files that are compatible with the updated Ada and the converted C++ code.

• Description– Assume ingest of RDRs by ASDC; Alternative is to access RDRs stored

in specified location if not available from the ASDC– Test with selected RDRs on the development platform to provide output

that will be used by the Level-0 data processor to show that output Level-0, Ephemeris and Attitude data is as expected by this PGE.

– Test on the development platform with minimum of 28-hrs of RDR data through the Level-0 PGE, on to the Pre-ES8 generator to create the data used by ERBE-like.


CERES DMT RDR Tests

• Description (Cont’d)– Pre-delivery testing on production platform to ensure all systems work on

the production platform and generate scientifically equivalent data. Ensures all software works as expected on the production platform before

delivery Provides expected output used by CM to verify their test results

– CM testing using Instrument Subsystem Test Plan to verify that all necessary updates have been delivered and provide output equivalent to the provided expected output.

– ASDC SIT operational testing: e.g. Dec. 31, Jan. 1, Feb. 28/29, etc.

• Participants– Ashley Alford, Denise Cooper, Mark Timcoe, Land PEATE, LaRC

ASDC SIT


CERES DMT RDR Tests

RDRs

Science & Diagnostic

~133 per data day(Level-0 packets &

Diary data)

RDRPre-processor

ScienceLevel-0 File

CalibrationLevel-0 File

DiagnosticLevel-0 File

Total 10391 packetsPer data day

EphemerisData FilesAttitude

Data Files

Data files (Noon previous day -

Noon next day)12 Data files(2 hr data per file)


Instrument Schedule

• RDR Format Documentation Provided (Jun. 2008)• Unpacking of RDR Wrapper Implemented (Jul. 2008)• Conversion of RDR Contents to CERES Level 0 Format

Implemented (Sep. 2008)• Selection and Conversion of Attitude/Ephemeris Data

Implemented (Sep. 2008)• Informal Testing with Actual RDR (dependent on IDPS providing

early RDR) (Oct. 2008)• Instrument Subsystem Enhancements Integrated and Tested (Dec.

2008)• Instrument Subsystem Verification Testing Complete (Jan.

2009)• Delivery to ASDC through DMT CM for Formal Validation Tests

(by Mar. 13, 2009)


Clouds Subsystem Design Approach

CERES Clouds Subsystem Design Approachand

CERES DMT VIIRS Tests

Sunny Sun-Mack

Contributing Team Members:Yan Chen

Walt Miller


CERES SSF Processing(Subsystem 4)

Subsystem 4 - Determine Cloud Properties, TOA and Surface Fluxes, is made up of 3 different sets of code

• Clouds - VIIRS inputs will necessitate changes to support NPP

– Processes the imager data– Resulting pixel-level output written to temporary file

• Convolution - minor changes needed to support NPP– Merges the pixel-level data with the CERES footprint data– Resulting footprint-level output written to a temporary file

• Inversion - no change needed to support NPP– Reads the footprint level output file containing CERES radiance data and cloud properties– Computes unfiltered radiances, TOA fluxes, and surface fluxes– Generates the SSF data product which is archived and publicly released


NPP Impact on CERES Clouds and Convolution Design Approach

• VIIRS will replace MODIS as imager source– Ability to generate certain cloud properties changes– Some radiances, CO2 slicing, may be missing

• CERES IES (input from Instrument Subsystem) interface will remain the same

• New Point Spread Function (PSF) ancillary file will be generated to account for VIIRS imager resolution

• SSF interface change possible as CERES migrates to Edition3



• Newly developed VIIRS sub-sampling code at LaRC– VIIRS Sub-sampling Preprocessor

Written in C language Input: NPP_VIAE ( Radiances from Imaging “I” Channels ) NPP_VMAE ( Radiances and geolocations from Moderate-Resolution “M” Channels) NPP_VDNE ( Radiances and geolocations from Imaging Day-Night Band (channel) “DNB” ) Output: NPP_VIMD_SS ( Radiances and geolocations from all above three Imaging, Moderate and DNB, but sub-sampled and sub-channeled with certain channels being averaged )



• VIIRS sub-sampling code will run at the Land PEATE – The package to be delivered to Land PEATE by CERES CM at LaRC

VIIRS sub-sampling code Process Control Generator to produce process control file (PCF) Input data and Expected sub-sampled output Test Plan (PDF)

– VIIRS sub-setting code will be run at Land PEATE The output from VIIRS sub-sampling code, NPP_VIMD_SS, will be produced at Goddard Land PEATE Land PEATE will then deliver NPP_VIMD_SS to LaRC ASDC

• LaRC ASDC will ingest and archive NPP_VIMD_SS• NPP_VIMD_SS will be the Cloud Imagery Data (CID) as input to CERES Clouds Subsystem.


VIIRS Sub-setting Data Formats

1. Sub-setting NPP_VIAE Resolution: 375 meter Radiances from VIIRS Imaging Channels

Sub-setting Channels

I1 = 0.64 m

I2 = 0.865 m

I3 = 1.61 m

I4 = 3.74 m

I1 = 11.45 m

I1 = 0.64 m

I3 = 1.61 m

I4 = 3.74 m

I1 = 11.45 m

NPP_VIAE NPP_VIMD_SS


VIIRS Sub-setting Data Formats1. Sub-setting NPP_VIAE ( Cont’d)

M Pixels (NPP_VIMD_SS)I Pixels (NPP_VIAE)

Averaging 4 Imaging Pixels to 1 Moderate Resolution Pixel



1. Sub-setting NPP_VIAE ( Cont’d)

Sub-setting Samples

NPP_VIAE

NPP_VIMD_SS

Scan-lines x Pixels = 4608 x 6400


Every other pair of scan-lines

Every other pair of pixels



SDS Name (NPP_VIAE) Description Dimensions SDS Name (NPP_VIMD_SS) Dimensions

Radiance_Img_I1 Radiances for I1 in moderate resolution (6144,6400) Radiance_Img_I1_Avg (1536,1600)




Reflectance_Img_I1 Reflectance for I1 in imagery resolution (6144,6400) Reflectance_Img_I1_SubReflectance_Img_I1_Avg

(3072,3200) (1536,1600)

Reflectance_Img_I3 Reflectance for I3 in imagery resolution (6144,6400) Reflectance_Img_I3_SubReflectance_Img_I3_Avg

(3072,3200) (1536,1600)

BrightTemp_Img_I4 Brightness temperature for I4 at imagery resolution

(6144,6400) BrightTemp_Img_I4_Sub BrightTemp_Img_I4_Avg

(3072,3200) (1536,1600)

BrightTemp_Img_I5 Brightness temperature for I5 at imagery resolution

(6144,6400) BrightTemp_Img_I5_SubBrightTemp_Img_I5_Avg

(3072,3200) (1536,1600)

QF_VIIRS_I1_SDR_1 Quality control flag for I1 (6144,6400) QF_VIIRS_I1_SDR_1_Sub (3072,3200)

QF_VIIRS_I1_SDR_3 Scan quality control flag for I1 (192,4) QF_VIIRS_I1_SDR_3 (192,4)





QF_VIIRS_I5_SDR_1 Quality control flag for I5 6144,6400) QF_VIIRS_I5_SDR_1_Sub (3072,3200)


1. Sub-setting NPP_VIAE ( Cont’d): SDSs Kept



SDS Name (NPP_VIAE)

Description Dimensions SDS Name (NPP_VIMD_SS)

Dimensions

Radiance_Img_I2 Radiances for I2 in moderate resolution

(6144,6400)

Reflectance_Img_I2 Reflectance for I2 in imagery resolution

(6144,6400)

QF_VIIRS_I2_SDR_1 Quality control flag for I2 (6144,6400)

QF_VIIRS_I2_SDR_3 Scan Quality control flag for I2 (192,4)

1.Sub-setting NPP_VIAE ( Cont’d): The SDSs not being subset



NPP_VMAE

M1 = 0.412 m M3 = 0.488 m

M2 = 0.445 m M4 = 0.555 m

M5 = 0.672 m M7 = 0.865 m

M6 = 0.746 m M8 = 1.24 m

M10 = 1.61 m M9 = 1.378 m

M12 = 3.7 m M11 = 2.25 m

M13 = 4.05 m M14 = 8.55 m

M15 = 10.763 m

M16 = 12.013 m

2. Sub-setting NPP_VMAE Resolution: 750 meter Radiances and Geolocation from VIIRS Moderate Channels

Sub-setting ChannelsNPP_VIMD_SS

M3 = 0.488 m

M4 = 0.555 m

M7 = 0.865 m

M8 = 1.24 m

M9 = 1.378 m

M11 = 2.25 m

M14 = 8.55 m

M15 = 10 763 m

M16 = 12.013 m



1. Sub-setting NPP_VMAE ( Cont’d)

Sub-setting Samples

NPP_VMAE

NPP_VIMD_SS



Every other scan-line

Every other pixel



Latitude

Longitude

Solar Zenith Angle

Solar Azimuth Angle

Sensor Zenith Angle

Sensor Azimuth Angle

Satellite Range

Terrain Height

All Geolocation Info in NPP_VMAE is included in the Subset NPP_VIMD_SS

Scan Start Time

Scan Mid TimeNPP_VIMD_SS: SDS (64 bits float)

NPP_VMAE: Global Attributes

2. Sub-setting NPP_VMAE (Cont’d)

NPP_VMAE and NPP_VIMD_SS



• 2. Sub-setting NPP_VMAE (Cont’d): SDSs Kept

SDS Name Description Dimensions SDS Name Dimensions

Radiance_Mod_M11 Radiances for M11 in moderate resolution (3072,3200) Radiance_Mod_M11_Sub (1536,1600)







Reflectance_Mod_M11 Reflectance for M11 in moderate resolution (3072,3200) Reflectance_Mod_M11_Sub (1536,1600)






BrightTemp_Mod_M14 Brightness temp. for M14 at mod resolution (3072,3200) BrightTemp_Mod_M14_Sub (1536,1600)

BrightTemp_Mod_M15 Brightness temp for M15 at mod resolution (3072,3200) BrightTemp_Mod_M15_Sub (1536,1600)

BrightTemp_Mod_M16 Brightness temp for M16 at mod resolution (3072,3200) BrightTemp_Mod_M16_Sub (1536,1600)




QF_VIIRS_GEO_MOD_2 Quality control flag for geo location (3072,3200) QF_VIIRS_GEO_MOD_2_Sub (1536,1600)

QF_VIIRS_M11_SDR_1 Quality control flag for M11 (3072,3200) QF_VIIRS_M11_SDR_1_Sub (1536,1600)

QF_VIIRS_M11_SDR_3 Scan quality control flag for M11 (192,4) QF_VIIRS_M11_SDR_3 (192,4)

















2. Sub-setting NPP_VMAE (Cont’d): SDSs Kept


VIIRS Sub-setting Data Formats2. Sub-setting NPP_VMAE ( Cont’d): The SDSs not being subset


Radiance_Mod_M1 Radiances for M1 in moderate resolution (3072,3200)









Reflectance_Mod_M1 Reflectance for M1 in moderate resolution (3072,3200)





BrightTemp_Mod_M12 Brightness temperature for M12 at moderate resolution (3072,3200)

BrightTemp_Mod_M13 Brightness temperature for M12 at moderate resolution (3072,3200)



2. Sub-setting NPP_VMAE ( Cont’d): The SDSs not being subset


QF_VIIRS_M10_SDR_1 Quality control flag for M10 (3072,3200)

QF_VIIRS_M10_SDR_3 Scan quality control flag for M10 (192,4)















DNB = 0.7 m

Latitude

Longitude

Solar Zenith Angle

Solar Azimuth Angle

Sensor Zenith Angle

Sensor Azimuth Angle

Satellite Range

Terrain Height

Subset every other scan-line and every other pixel

3. Sub-setting NPP_VDNEResolution: 750 meterRadiances and Geolocation from VIIRS Day-Night Channel

Only One channel-->No Sub-setting Channel

Keep All Geolocation SDSs

Sub-setting Sample



4. File Size Summary

INPUT: NPP_VIAE ---> 460 MB / granule NPP_VMAE ---> 560 MB / granule NPP_VDNE ----> 330 MB / granule

OUTPUT NPP_VIMD ---> 400 MB / granule

~ 70% Data Reduction


Emissivity Maps x 4

IGBP

Water %

Elevation

Surface Maps

NPP CERES Cloud Subsystem

GMAOMOA

Clear Reflectance

History

CloudsProcessor

TRMM VIRS CID

MODIS Terra CID

MODIS Aqua CID

NPP VIIRS CID

Cloud Imager Data

(CID)

Algorithm Ancillaries

Snow/ice Model

BiDir Model

Directional Model

AngularModels

EIPD (Cookie Dough)

EQCG

EQCB

ECVS

ECV

Output

NPP VIIRS Subset: NPP_VIMD_SS To CERES Convolution


Convolution Subsystem Description

Cloud Pixel

Convolution

Inversion

X 24 hoursInstrument

IES

FQCI

SSFAI

SSFIEIPDCookie dough

FQCPSF

Convolve imager radiances, cloud properties, and aerosol into CERES footprint



• Objectives Ensure the VIIRS Sub-sampler processor is able to:

(1) Read

NPP_VIAE ( Imager resolution radiances),

NPP_VMAE ( Moderate resolution radiances and geolocations)

NPP_VDNE (Day / Night band radiances and geolocations)

(2) Subset

sub-channel

sub-sample

averaging of imagery resolution data

(3) Produce VIIRS sub-sampled output

NPP_VIMD_SS



• Description– If less than a couple of hours of NPP VIIRS data is available, then Land

PEATE pushes VIIRS NPP_VIAE, NPP_VMAE, and NPP_VDNE to a CERES/VIIRS designated computer

– For larger volumes of data, the Land PEATE sends all test NPP VIIRS data types to the ASDC for ingest, and then subsequently staged for access by the VIIRS Sub-sampler on a CERES/VIIRS computer

– Execute VIIRS Sub-sampler on local CERES/VIIRS computer to create and verify sub-sampled output using input VIIRS data from Land PEATE

– Load VIIRS Sub-sampler and selected VIIRS test data in the file space and computer designated for CERES processing at the Land PEATE, and then conduct testing at the Land PEATE and verify sub-sampled output

– CERES DMT delivers VIIRS Sub-sampler (tar files) and test plan to the CERES CM for delivery to the Land PEATE

– Land PEATE conducts VIIRS Sub-sampler testing creating sub-sample– Land PEATE verifies sub-sample and also send sub-sampled output to

either the ASDC or to a designated CERES/VIIRS computer for additional verification



NPP VIIRSSub-settingProcessor

NPP_VIAE

NPP_VDNE

NPP_VMAE NPP_VIMD_SS

• Required Input • Expected Output

• Participants– Sunny Sun-Mack, Yan Chen, Tammy Ayers, Walt Miller, CERES

Science Team members: Patrick Minnis, Norman Loeb, Tom Charlock, Dave Kratz and Dave Doelling, Land PEATE, and LaRC ASDC


CERES CARS Delivery Processto the Land PEATE

• Following unit, integration and verification testing by the CERES DMT, the VIIRS Sub-sampler must be delivered to the Land PEATE for final testing

• CERES DMT creates tar files containing source code, process control generator, selected NPP VIIRS input data and expected sub-sampled output

• CERES DMT updates Test Plan for testing at the Land PEATE

• CERES DMT delivers the above tar files along with the Test Plan to CERES CM

• CERES CM delivers the tar files and Test Plan to the Land PEATE’s designated machine. A “Delivery Notification” email is sent to the Land PEATE and all concerned


CERES CARS Delivery Processto the Land PEATE (cont.)

• Land PEATE conducts VIIRS Sub-sampler testing

• When the Land PEATE testing is successfully completed, the Land PEATE:– Sends an email to all concerned regarding the status– Promotes the delivery to their production environment– Makes the delivery (as tar files) available in a designated area for CERES

CM to retrieve– Sends a notification email to CERES CM that this action has taken place.

• Upon receiving this notification, CERES CM: – Retrieves the tarred delivery– Places it in the CERES CM repository

• Subsequent changes to CM controlled software are documented and tracked using a CERES Software Configuration Change Request (SCCR) and redelivered through CERES CM


NPP VIIRS Sub-sampler Schedule

• Data Format of VIIRS Subset for CERES Defined (June 1, 2008) Completed on Time.

• Data Product Catalog Pages Documenting VIIRS Subset for CERES Defined (August 1, 2008) Completed on Time

• Sub-sampler Code Complete and Running Locally at LaRC on Linux Machine (October 1, 2008)

• Goddard Land PEATE Accepts the Sub-sampler Code Delivery and Produces Sub-sampled Output from Input Data (December 1, 2008)

• Verify a Few Days of Sub-sampled Data Produced at GSFC (January 15, 2009)

• ASDC Ingests Sub-sampled VIIRS Data; Verified by CERES DMT (February 26, 2009)


CERES DMT Testing Approach• The CERES DM task was successfully appraised at CMMI Capability Level

3 /Equivalent Maturity Level 2

• SSAI is appraised at CMMI Capability Level 3/Equivalent Maturity Level 2

• CERES DMT uses a structured CMMI-based development process that includes unit and integration testing

• Functional (e.g., verification) tests are executed to verify requirements for all enhancements

• Regression tests are executed to verify existing functionality

• Delivery packages for CERES and VIIRS Sub-sampler to include:– Test Plan– Expected output– All inputs– Code and scripts

• Every CERES code delivery (to ASDC or Land PEATE) is under configuration management (CM) control

• Documented in CERES NPP DMS Software Management Plan Version 0


CERES CARS Testing Strategy

• Conversion of RDRs to Level 0 data will be tested by the DMT using defined formats and available test RDRs prior to delivery of code to the ASDC– Testing conducted in development environment

• Transmission of RDRs by the Land PEATE to the ASDC will be tested when:– CERES RDRs are available to the Land PEATE from the SD3E– Interface between ASDC and the Land PEATE has been established

• Final testing of conversion of RDRs to Level 0 data will involve:– Testing in the CERES operations test environment– Land PEATE sending all three RDR data types to the ASDC– ASDC ingesting data and acknowledging receipt to the Land PEATE– Instrument subsystem reading RDRs staged from ASDC archive and correctly

converting to Level 0 format


CERES CARS Testing Strategy

• VIIRS Sub-sampler will be tested by the DMT using test data provided by the Land PEATE prior to code delivery to the Land PEATE

– Testing conducted in development environment– Testing also conducted on designated Land PEATE computer

• Transmission of sub-samples by the Land PEATE will be tested when:– VIIRS Sub-sampler has been tested and delivered to the Land PEATE– VIIRS data is available to the Land PEATE from the SD3E or other source– Interface between ASDC and the Land PEATE has been established

• Final testing of Sub-sampler will involve:– Execution of Sub-sampler by the Land PEATE to produce sub-sampled data

product– Transmission of the sub-sampled data to the ASDC for ingest and archive– CERES DMT stages data from ASDC archive and verifies sub-sample


DMT NPP Schedule

10/1/08: VIIRS sub-sampling software complete and tested at LaRC

12/1/08: VIIRS sub-sampling software running at SDS Land PEATE, expected sub-sampled output has been verified by CERES DMT

1/15/09: Large set of VIIRS sub-sampled output produced at SDS Land PEATE and verified at LaRC by CERES DMT

3/13/09: Instrument subsystem code needed to support NCT3 complete and tested by DMT

5/15/09: Instrument subsystem code delivered to ASDC, tested, and promoted to operations

08/13/09: Instrument subsystem code needed to support NCT4 complete and tested by DMT Interim delivery to correct NCT3 problems and prepare for NCT4


11/19/09: Instrument subsystem code needed to support launch complete and tested by DMT



CERES CARS Build Schedule

CERES Data Management System

Projected Build promotion dates to ASDC Operations:

Build 1: NCT3 Functionality - May 2009

Build 2: NCT4 Functionality - October 2009

Build 3: Launch-Ready System - January 2010


ASDC Implementation Approach

ASDC Implementation Approach

Michael Little


ASDC Supports CERES PI in Ingest, Archive, Distribution, Production

• Ingesting and Archiving inputs for all Data Products– Use by Production in Producing Data Products– Use by Science Team in analysis and algorithm refinement– Long-term stewardship of data including migration across media changes

Moving from Tape archive with Disk Cache to Disk archive with Tape Backup

• Distribute CERES Data Products– External Customers order data products via ASDC Order Tool, ESDIS ECHO

ASDC Order Tool provides interim support ESDIS ECHO provides access to all EOS data products Subscription Services Plan to supply CLASS with subscription service as an offsite backup

– Internal users via file sharing Moving from Request/deliver model to online Fiberchannel/NFS

• Produce Data Products for CERES PI– Accept and test Production codes from DMT– Accept Production Requests from DMT– Run codes consistently to produce expected data products

Now by procedure; expect automation within one year


ASDC Evolution Approach

• Provide CERES Production Services at ASDC– Meet constraints of CERES PGEs

Avoid increasing workload on DM/Science workforce Support FORTRAN, Ada, and IDL languages Meet increasing memory and disk access requirements

– Use lessons learned to improve CERES production Improve I/O bandwidth limitations to allow multiple instances of some subsystems Reduce unplanned outages by increasing storage reliability Reduce dependence on active tape archive to improve labor intensive staging processes

– Increase production capacity for FM-5 and reprocessing Terra/Aqua– Replace current hardware at end-of-life

• Provide Access to CERES Data Products for Internal Users– Production– LaRC Science Community– CERES Data Management Team

• Improve Order Throughput for External Customers


64 x86 SMP256 GB

ASDC & SCF Integrated ArchitectureFiberchannel Connectivity

Fiberchannel Switch

64 x86 SMP256 GB

56 P6256GB

56 P6256GB

56 P6256GB

56 P6256GB

220TBDS4800

220TBDS4800

220TBDS4800

220TBDS4800

220TBDS4800

64 x86 SMP128 GB

16 x86 HS1264 GB

CommonSANhead

(NFS)(AFP)

(Samba)

ASDC JBOSSServer & TS-1& Ingest

ASDC SDP Production

64 x86 SMP256 GB

56 P6256GB

SCF Cluster(Access through

Sun Grid Engine)

220TBDS4800

220TBDS4800

SCF Disk Array

Inst

rum

ent

Inst

rum

ent

ERB

E-lik

e

SRB

/Pow

erSR

B/P

ower

SAR

B

14 4P6@8GB

SCF Interactive Processors

ASDC Disk Archive(R/W Production)

(R/O SCF)

Con

sodi

ne

TISA SO

FAC

loud

sC

loud

s

TISA

220TBDS4800

WW

WW

WW

64 x86 SMP128 GB

Cloud ModelingCluster (Xu)

64 x86 SMP128 GB

ECS DUEProcessing

DbServer

DbServer

Tesla

Tesla

3755

3755

GPU Experiment (YongHu)

SimServer

FCMonitor

LB1 LB1


220TBDS4800

ASDC & SCF TCP/IP Architecture

LaRCNet Switch

220TBDS4800

220TBDS4800

220TBDS4800

220TBDS4800

4x16 x86 SMP128 GB

SH1x3650

Common SANhead(NFS, AFP, Samba)

ASDC JBOSS Server & TS-1

ASD

C S

DP

Prod

uctio

n (S

GE)

SCF Cluster (SGE)

220TBDS4800220TB

DS4800SCF Disk Array (R/W)In

stru

men

tIn

stru

men

tER

BE-

like

SRB

/Pow

erSR

B/P

ower

SAR

B

14 4P6@8GBSCF Interactive Processors

ASDC Disk Archive (R/W Production)(R/O SCF)

SAR

B

TISA

SOFA

Clo

uds

Clo

uds

TISA

220TBDS4800

WW

WW

WW

64 x86 SMP128 GB

CloudModeling

Cluster

64 x86 SMP128 GB

ECS DUE Processing

ANGeDb

Servers

ANGeDb

Servers

Tesla

Tesla

3755

3755

GPU Experiment

SimServer

Load2Magneto

56 P6 256GB

56 P6 256GB

56 P6 256GB

56 P6 256GB

SH2x3650

192.168.16.x

FCMonitor

ASDC SGE Head

SCF SGE HeadLoad

1

LaRCNet 100Mbps EthernetPrivate Net (DS Mgt)Private Net (GPFS)Cluster & Mgt Net (SGE)Green Indicates SubSys Function

Legend

WWW ProxyServers

Cluster Mon(HW TBD)

64 x86 SMP256 GB

64 x86 SMP256 GB

64 x86 SMP256 GB

56 P6256GB

Zamboanga

Center Firewall

NISN WAN

IBM Directorx3650


I/F

To beResolved

Metadata

For OrderableData Products

ASDC Ingest Processes for NPP

ANGeDrop-Box

ASDCArchive

ANGeCore db

IMSdb

ASD

C In

gest

Pro

cess

or

Metadata

Location

& GUID

Subscription

Files

PDR

Files

ASDCDeliveryServer

Files

LocalProduction

PDRFiles

LandPEATE

PDRFiles

ECHOMetadataAdapter

NOAACLASS


NPP Impacts to the ASDC

Impact ResolutionIncreased storage capacity and performance needed for new CERES data set

Additional disks purchased with fiberchannel connectivity to production processors.Higher density tapes expand capacity of tape archives.

Increased processing capability needed for new CERES data set

Additional processors purchased and added to Linux cluster increasing processing capacity.Faster data access from new disk archive vs. tape archive.

Additional CERES data production requirements effect on staffing

Improvements in staging/ingest/archive reliability will offset staff workload for production.Staff experienced in processing CERES data products.

Operating procedures required for new data set and experience of staff

Job production using on existing Subsystem DMT Ops Manuals requiring minor updates for NPP.Staff experienced in processing CERES data products.

Ability of facility to accommodate additional hardware with respect to space, HVAC and power

ASDC has space/HVAC for expansion.Power upgrades 09/08/08

Increased data load impact on network throughput

New architecture will improve internal data production network throughput and access by SCF usersMay need additional bandwidth to Land PEATE for VIIRS

User access to current and future CERES data products

New architecture will speed ordering CERES data productsNew data products will be available through current ordering tools when released to public


Ingest/Archive Sizing EstimatesASDC can accommodate expected CERES data volumes from the Land PEATE

CERES RDRs: Science, Diagnostic, and Telemetry

File sizes are 1370.8 KB, 685.5 KB and 27.5 KB; excluding HDF overhead, about 700 KB/file

Estimate: (1370.8 + 700) KB/file * 131 files/day + 685.5 KB/file * 1 file/day+ 27.5 KB/file * 12 files/day

= 272.2903 MB/day

Sub-sampled VIIRS radiance and geolocation data:

Output File Size: Approximately 400 MB per 5-min file

Estimate: 400 MB/file * 288 files/day = 115,200.0 MB/day

VIIRS Aerosol data:

File Size: ~ 632 MB per 5-min file, as of June 1, 2008

57 MB per 5-min file if changing to 6 x 6 aggregation for both Land and Ocean 32 MB per 5-min file if changing to 8 x 8 aggregation for both Land and Ocean.

Estimate: ~12 files per hour or ~288 files/day * 632 MB/file = 182,016 MB/day

~12 files per hour or ~288 files/day * 57 MB/file = 16,416 MB/day

Note: The Land PEATE expects to switch to 6x6, or possibly 8x8, aggregation prior to launch

Total Volume - Large Aerosol Files = 297,488.29 GB/day 108,583.226 GB/yr or 108.58 TB/yr

Total Volume - Small Aerosol Files = 131.888.29 GB/day 48,139.226 GB/yr or 48.14 TB/yr


Security Compliance

• ASDC Approval to Operate Based on August, 2007 Review– System Security Plan: SC-010-M-LRC-1000

Approving Official: Steve Jurczyk, Deputy Center Director System Owner: John Kusterer, ASDC Manager System Security Manager: Michael Little, ASDC Systems Engineer

– Re-certification in progress based on CIO mandated schedule re-alignment– Re-certification based on ANGe/IBM installation to occur next spring

Include consolidation of SCF and ASDC Systems into single System

• IT Security Issues– Reconciliation of OCIO Mandated ODIN conversion with ASDC ITS needs

ODIN to take over support of workstations, desktops, etc ASDC to retain control/responsibility for servers

– Implementation of unannounced ITS measures by OCIO sometimes interferes with ingest and delivery over network

Agency CIO mandated transfer of all Center Firewalls to NISN is not fully understood


CERES Production Code Test & Evaluation

• Proven Processes Based on 10 Years of Experience in CERES Code Delivery,Acceptance, Testing, Evaluation and Production

• CERES DMT Develops Code for Production of Data Products– PGE(s) for each Sub-system (SS) tested and internally verified– DMT and ASDC develop test cases– Delivery to ASDC includes sample data, Operators Manual, code– Delivery Evaluation and Code Compilation by DMT Configuration Management comparing to

sample data products and using test cases– Delivery to ASDC for SS Integration and Test to evaluate producibility and completeness and

comparison to sample data products– ASDC and DMT work together to identify and resolve any problems through Software Trouble

Tickets– Promotion to Production status and ASDC CM when approved by Operations Readiness

Review (ORR)

• Data Product Production Validation Testing– Once code is in production, sample data products (ValRx) are produced with appropriate key

months– ValRx data products are evaluated by Science Team and approved prior to release into an

orderable status Unsuitable results may result in re-delivery

– Release also requires a Quality Summary by the SS Science Lead


ASDC Validation Testing Approach

Systems delivered to the ASDC undergo Validation testing in a controlled environment

DMT Subsystems

under CERES CM delivered to the ASDC

for Validation Testing

Delivered Subsystems

inspected for completeness

by ASDC CM, unique testing

scenarios identified

Subsystems validated in operations

testing environment for

all new requirements

and enhancements

Delivered subsystems regression tested for existing

functionality

Operations Readiness

Review (ORR) following successful Validation

Testing and/or mitigation of

existing issues

Problem reports generated and tracked for all identified

problems

Problem report resolutions incorporated into

configured software for redelivery (if needed) to the

ASDC for validation

DataProduction

andOperations

CERES CM ASDC CM ASDCValidation

Testing

ASDC CCBAnd ProjectManagement

ASDCRegression

Testing

ASDCOperations

CERES FM-5 Testing to Operations Life-Cycle

Problems Resolutions


NPP-Level Testing and Status

• Testing of the RDRs requires the SD3E to access data from the IDPS, the RDRs are then accessed by the Land PEATE and sent to ASDC

– ASDC will ingest the RDR data and acknowledge receipt

– ASDC will test data production when available after 05/09

• Testing of VIIRS data requires Land PEATE to run sub-sampler and send sub-sampled radiance/geolocation and aerosol data to the ASDC

– Land PEATE must be able to generate or access valid VIIRS data

– ASDC will ingest data and acknowledge receipt

• Ingest capability will be ready to support Early EEO

• Ingest and data production capability will be ready to support NCT3

• Interim delivery to address NCT3 problems to support NCT4 success

• All NPP-related functionality available at L-90 days for Launch


CERES ASDC RDR Tests• Objective

– Verify that the Land PEATE can access CERES RDRs from the SD3E and successfully transfer the data to the ASDC for ingest and archive

• Description– SD3E reads CERES RDR data from IDPS or through alternative source– Land PEATE accesses RDRs from the SD3E and transfers data to the ASDC– ASDC ingests RDRs and acknowledges receipt back to Land PEATE– ASDC archives data for staging and processing by Instrument Subsystem– ASDC will test data production when systems and RDRs available after 05/09– Additional reorder and data management functions will also be tested

• Input– CERES RDRs generated by IDPS or from Proxy data provided to Land

PEATE• Output

– CERES RDRs ingested and archived at ASDC– CERES Level 0 data for instrument subsystem processing available after 05/09

• Participants– ASDC, Land PEATE, SD3E, CERES DMT– IDPS when CERES RDR capability is available


CERES ASDC VIIRS Tests

• Objective– Verify execution of the VIIRS Sub-sampler by the Land PEATE and the

transfer of the sub-sampled radiance/geolocation and aerosol data to the ASDC• Description

– SD3E reads VIIRS data from IDPS or through alternative source– Land PEATE must be able to generate or access valid VIIRS data– Land PEATE executes VIIRS Sub-sampler and sends sub-sample to ASDC– ASDC ingests sub-sampled data and acknowledges receipt to the Land PEATE– ASDC archives data for staging and processing by Clouds subsystem – Sub-sampled and aerosol data examined by science and DMT staff– ASDC will test data production when systems are available after 05/09– Aerosol data ingest and archive tested when available from the Land PEATE

• Input– VIIRS data generated by IDPS or from Proxy data provided by Land PEATE

• Output– VIIRS sub-samples generated by the Land PEATE – VIIRS aerosol data product generated by the Land PEATE

• Participants– ASDC, Land PEATE, SD3E, CERES DMT, IDPS


Operational Support

• System Upgrades will Enhance Current Capabilities– CERES production system already runs at the ASDC– ANGe storage management upgrade should be operational this week– Hardware upgrades to be phased in with existing CERES Ingest, Archive,

Distribution and Production capabilities in Dec09, reducing risk to NPP needs

• Procedures and Documentation are available and tested– Operations staff with CERES processing experience already in place with defined,

proven procedures– ASDC staff already work closely with Land PEATE staff through interactions

supporting MODIS data transfers provided from the same data center (MODAPS)

• System availability will remain at 24x7– Current production model has minimized need for 24x7 staffing– System hardware and architecture evolution will increase system reliability and

reduce workforce requirements

• Improved system performance and additional CERES data products will lead to increased customer satisfaction


Documentation and Agreements• Most documentation and agreements already exist as part of CERES on

TRMM, Terra, and Aqua– http://science.larc.nasa.gov/ceres/docs.html– Documents may be updated if needed– New documents may be written explicitly for NPP, to meet special requirements

• Overall roles and responsibilities are documented in CERES Science and Data Products Working Agreement

– Working agreement between CERES IT and NPP Project Office– May need to be revisited when NPP transfers to NOAA leadership– Working Agreement in hands of NPP SEWG

• Operations Agreement (OA) being developed between ASDC and Land PEATE

– Based on existing MODAPS OA for MODIS data sent to ASDC– On-going relationship between MODAPS and ASDC for MODIS

• No OA planned for ADS/CLASS interface– Expect to use standard customer subscription agreement


ASDC Supports NPP Schedule08/31/08 Initial Operational Capability (IOC): CERES in ANGe09/15/08 Complete Installation/Configuration of IBM hardwareSep-Nov08 Test/Characterize IBM Installation11/13/08 Initial Delivery of CERES RDR to ASDC IngestNov08 Complete LaTIS conversion into ANGeNov/Dec08 Design/Develop/Test NPP Inst, VIIRS into ANGe11/26-12/23/08 Complete CERES RDR and VIIRS ICD12/01/08 IOC IBM ASDC+SCF Archive and Processing SystemJan/Feb09 Debug Aerosols/VIIRS Sub-sample into ANGe with Land

PEATE05/08/09 SDS Test2: Instrument RDRs into ANGe from Land PEATE05/15/09 Build 1 ORR: NCT3 Functionality06/09-15/09 NCT307/01/09-02/02/10 Design/Develop/Test Other NPP into ANGe10/15/09 Build 2 ORR: NCT4 Functionality/NCT3 Problem

Corrections12/02-09/09 SDS Test3/NCT401/15/10 Build 3 ORR: Launch-Ready System02/02/10 Instrument/VIIRS IOC03/02/10 MRR06/02/10 NPP Launch


Overall Schedule

• CERES Schedule is baselined

• CERES Schedule incorporated into NPP Schedule

• Schedule available as separate handout


Resource Assessment

• Required resources have already been assessed and hired

• Current experienced CERES staff leveraged for CERES FM-5

• CERES FM-5 staff augmented with key hires to address technical challenges– Major SSAI job fair resulted in hundreds of resumes

• Backfill positions created in current CERES support filled with new talent

• Resource estimate based on approximate lines of code to be generated

• Using industry productivity standards, CERES CARS has the resources in place needed to do the job

• Additional resources needed for NPP testing, processes, and working groups not included in industry standards

• Additional issues and requirements expected

• Ongoing CERES projects supported by SSAI can provide short-term support if critical needs arise


Risk Analysis

Raw Data Record (RDR)

Formats NotFully

Defined

Given that CERES Raw Data Record (RDR) formats are not fully

defined or understood in time, there is a possibility the code needed to support testing may

not be complete.

2 1 TechnicalSchedule

Erika Geier

M 05/29/08

The CERES teamis developing agiver/receiver listand this item is on

the list.

07/23/08

IPO/Raytheon andJanet Smith allKnow about this

risk.

Unavailability of CERES Raw Data

Record (RDR) Test

Data

Given that CERES Raw Data Record (RDR) test data from IDPS is not available prior to testing, there is a possibility the code needed to support testing may

not be complete.

3 1 TechnicalSchedule

ErikaGeier

W 07/23/08

We have requestedtest data; although,it doesn't soundlike we'll getmeaningful test

data.

Title Description L C Impact Owner Action Comments

Legend: L – Likelihood C – Consequence M – Mitigate W - Watch


CERES Issues/Concerns

• Need single sample of the Science, Diagnostic, Housekeeping/Telemetry RDRs to verify that we understand format no later than 10/15/08– CERES data content may be proxy, sample, or fill data– Prefer attitude/ephemeris not be fill data

• Desire 3 days (~ 28 consecutive hours) of CERES RDRs to run through preprocessor and Instrument Subsystem to aid in code development no later than 11/14/08