1 NetCDF4 Reformatting Toolkit: BUFR and GRIB Tailoring Preliminary Design Review April 14, 2008 Prepared By: Tom King 1, Walter Wolf 3, Lihang Zhou 1,

1

NetCDF4 Reformatting Toolkit: BUFR and GRIB Tailoring

Preliminary Design Review

April 14, 2008

Prepared By: Tom King1, Walter Wolf3, Lihang Zhou1, Yi Song2, Larisa Koval1,

1 PSGS, 2 IMSG,3 NOAA/NESDIS/STAR

2

Review Agenda

Introduction 9:00 am – 9:20 amWolf

Requirements 9:20 am – 10:00 amKing

Software Architecture 10:00 am – 10:45 am King

Quality Assurance 10:45 am – 11:05 amKing

Risks and Actions 11:05 am – 11:20 am Wolf

Summary and Conclusions 11:20 am – 11:30 am Wolf

3

Review Outline

Introduction Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions

4

Introduction

Presented by

Walter WolfNOAA/NESDIS/STAR

5

Introduction

Project Background» Terminology» IJPS» NPP/NPOESS» NDE

Project Objectives

Integrated Product Team

Project Plan

Entry and Exit Criteria

6

Project BackgroundTerminology

Raw Data Records (RDRs)

Sensor Data Records (SDRs)

Temperature Data Records (TDRs)

Environmental Data Records (EDRs)

Intermediate Products (IPs)

7


Raw Data Records (RDRs)» Full resolution, digital sensor data, time-referenced and

locatable in earth coordinates with absolute radiometric and geometric calibration coefficients appended, but not applied, to the data.

8


Sensor Data Records (SDRs)» Data record produced when an algorithm is used to

convert RDRs to geolocated, calibrated detected fluxes with associated ephemeris data. Calibration, ephemeris, and any other ancillary data necessary to convert the sensor units back to sensor raw data (counts) are included.

9


Temperature Data Records (TDRs)» Calibrated antenna temperature records from a

microwave instrument such as ATMS.

Environmental Data Records (EDRs)» Data records produced when an algorithm is used to

convert SDRs to geophysical parameters (including ancillary parameters, e.g., cloud clear radiation, etc.).

10


Intermediate Products (IPs)» The NPOESS products that are produced as an

intermediate step between the SDR and EDR. These are available from the IDPS upon request.

11

Project BackgroundNPP/NPOESS

• NPP and NPOESS, a joint Military/NOAA/NASA effort, is the next series of polar-orbiting satellites dedicated to among other things, operational meteorology. The objective of the NPOESS mission is to ensure continuity, improvement and availability of operational observations from an afternoon polar orbit (1:30 pm).

• Instrument packages on NPOESS:» CrIS, ATMS, VIIRS, OMPS, SEM, CERES

• NPP is the first of five missions with launch dates of ≈2011, ≈2013, ≈2016, ≈2018, ≈ 2020, respectively.

12

Project BackgroundNDE

Disseminate NPOESS Data Records to customers.

Generate and disseminate tailored NPOESS Data Records (versions of NPOESS Data Records in previously agreed alternative formats and views).

Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPOESS Data Records).

Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving.

Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA.

Provide software for NPOESS Data Record format translation and other data manipulations.

13

To build a software package that will tailor NPOESS and NDE products from netCDF4 into BUFR and GRIB2 formats in support of NDE’s overall tailoring efforts.

The NetCDF4 Reformatting Toolkit (N4RT) must be designed so it can easily be modified/expanded to incorporate the tailoring of new products.» Flexible» Extendable

The software must be able run within the NDE system architecture and operate within the NDE functional guidelines.

Output product formats and content must meet the needs of NOAA customers.

Project Objectives

14

Products to Reformat» ATMS Radiances » CrIS Radiances » Nadir Profile Ozone (OMPS) and OMPS Radiances » VIIRS Radiances » Snow Cover » Vegetation Index » Aerosol Optical Thickness » Sea Surface Temperature

Project Objectives Phase 1 Products

15

Integrated Product Team

IPT Lead: Walter Wolf (STAR)

IPT Backup Lead: AK Sharma (OSDPD)

NESDIS team:» STAR: Walter Wolf, Hank Drahos, Jaime Daniels, Yi Song, Thomas King,

Larisa Koval» OSDPD: Dave Benner, AK Sharma, Ricky Irving» OSD: Tom Schott, Jim Yoe» Data Center: Lei Shi (NCDC)

User team» Lead: Jim Heil (NWS), Stephen Lord (NWS /NCEP/EMC), John Derber

(NWS/NCEP/EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaard (JCSDA), Tony McNally (ECMWF), Fiona Hilton (UK-Met)

» Others: International NWP users, NWP FOs, Climate Users

Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP

16

Project Stakeholders

NOAA National Weather Service» Weather Forecast Offices» National Center for Environmental Prediction

Department of Defense» NRL» FNMOC» AFWA

Global NWP» EUMETSAT» UK Met» Meteo France» CMC

17

Project Plan

Year 1 – Design and Development (2008 – 2009)» Verify Requirements

– Work with customers to verify product requirements– Discuss with the current developers of similar translators to

determine what is required in their output files» Design the NetCDF4 reformatting toolkit;» Conduct PDR» Develop BUFR tables and GRIB formats with the product teams for

Phase 1 products» Work with NDE to determine the interface between the Level 1B

and the Level 2 NPP products and the reformatter» Conduct CDR

18

Project Schedule – Phase 1

19

Project Plan

Year 2 –Transition to Pre-Operations of Phase 1 Products (2009 – 2010)» Set up infrastructure to implement the readers and

writers for the data formats» Implement BUFR tables and GRIB formats for the Phase

1 products on the NDE hardware» Conduct Test Readiness Review for Phase 1 products» Transition and test system within the NDE environment» Conduct Code Review for Phase 1 products

20

Project Plan

Year 3 – Transition to Operations of Phase 1 Products (2010 – 2011)» Evaluate with NDE and OSDPD the implementation of

the Reformatting Toolkit within the NDE data handling system

» Conduct System Readiness Review for Phase 1 products

» Transition pre-operational Phase 1 product reformatting system to operations

21

Phase 1 Transition to Operations

22

Project Plan – Schedule

Schedule (Milestones)» Project begins – 07/01/08» Preliminary Design Review – 04/14/09 (10/21/08)» Critical Design Review – 09/25/09 (03/19/09)» Test Readiness Review – 06/09/09 (02/25/09)» Code Unit Test Review – 09/10/10 (01/29/10)» System Readiness Review – 01/31/11 (04/20/10)–Waive or shift to NDE

23

PDR Report

The PDR Report (PDRR) is a standard artifact of the STAR EPL process.» The PDR report will be produced after the PDR.» The report will be a critical artifact for the Critical

Design Review.

Guidelines for the PDRR are found in STAR EPL process assets

24

PDR Entry Criteria

Requirements Document

Review of NetCDF4 Reformatting Toolkit: BUFR and GRIB» Requirements» Software Architecture » Quality Assurance» Risks and Actions

25

PDR Exit Criteria

Preliminary Design Review Report» The PDR Report (PDRR), a standard artifact of the

STAR Enterprise Process Lifecycle (EPL), will be compiled before the CDR

» The report will contain:– Actions– Comments– PDR presentation

26

Review Objectives

Review the Requirements

Review Software Architecture

Review Quality Assurance

Identify risks and actions

27

Review Outline


28

Requirements

Presented by

Thomas KingNOAA/NESDIS/STAR

29

Requirements Overview

SPSRB Requirements were presented to the developers in a document entitled: “Level 1 Requirements for a NetCDF4 Reformatting Tool” (Version 1.5).

Text in yellow are those requirements derived from the original SPSRB requirements.

Product requirements have been added to those from the SPSRB and are presented here as well. These additional requirements were obtained in a series of meetings between the developers, EMC (the customer) and the heritage product teams.

Using all of this information a Requirements Allocation Document (RAD) has been generated for the Reformatting Toolkit project.

30

Original Phase I Product List

Prioritized Product BUFR GRIB2 Comments

ATMS Radiances X

CrIS Radiances X -Thinned radiances assumed and are okay- If concurrent delivery w/CrIS radiances, include cloud mask

Nadir Profile Ozone (OMPS) andOMPS Radiances

X Include OMPS radiances in nadir profile product

VIIRS Radiances X

Snow Cover X Incorporate into current snow product

Vegetation Index X

Aerosol Optical Thickness X

Sea Surface Temperature X

31

Original Phase I User/Heritage

Mapping

Prioritized Product EMC User Contacts

Heritage Product Contacts

Heritage Product

ATMS Radiances Dennis Keyser, John Derber

Dennis Keyser, John Derber

AMSU, AMSR-E

CrIS Radiances Jim Jung, Dennis Keyser

Jack Woollen, Simon Elliott IASI, AIRS

Nadir Profile Ozone (OMPS) andOMPS Radiances

Dennis Keyser Larry Flynn, Donna McNamara

SBUV, GOME

VIIRS Radiances Dennis Keyser, John Derber

Dennis Keyser, John Derber

AVHRR GAC

Snow Cover Michael Ek Sean Helfrich IMS Products

Vegetation Index Michael Ek Le Jiang, Felix Kogan AVHRR Veg.

Aerosol Optical Thickness Jeff McQeen Paul Haggerty MODIS Aerosols

Sea Surface Temperature Bert Katz, William Gemmill

Shasha Ignatov, John Sapper, Robert Grumbine

AVHRR derived SST (ACSPO)

32

Functional Requirements:Reformatting Toolkit

Software

Requirement: STAR shall deliver to NDE a reformatting toolkit capable of translating NESDIS NetCDF4 data products into NCEP-accepted data formats (i.e., BUFR and/or GRIB2).

» Requirement: The toolkit shall be capable of reformatting the NPP tailoring prioritized phase 1 product list.

33


Software

» Requirement: The toolkit shall provide its capabilities such that it may be run automatically within an operational system, especially within the NDE environment. – The Toolkit shall compile and run on the NDE IBM AIX P5 series

hardware.– The Toolkit shall interact with the NDE Data Handling System (DHS).– The Toolkit shall be able to read a Production Control File (PCF).– The Toolkit shall handle and return errors according to NDE/STAR

standard codes.– The Toolkit shall be able to write a PSF.

» Requirement: The toolkit shall consist of modular components that can be tested independently. – The code shall consist of a single compiled program that parses

arguments and logically assigns tasks to a family hierarchically structured tailoring subroutines.

– Data shall be stored in allocatable data structures.

34


Software

» Requirement: STAR shall include one update to the reformatting toolkit within its initial project plan.

» Requirement: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products.

» Requirement: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB2 in the reformatting toolkit.

» Requirement: STAR shall update the reformatting toolkit when NCEP updates its BUFR and GRIB2 libraries – Updates shall be made when there are updates to the versions of the

netCDF4 library being used by NDE.

» Requirement: NDE and OSDPD shall implement the updated reformatting toolkit into operations once STAR has updated it with BUFR and GRIB library updates from NCEP.

35


Software

» Requirement: STAR shall coordinate with the NDE Project before proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development.

» Requirement: The output from the toolkit shall be compared with the input.

» Requirement: The translation toolkit shall convert from the new format back into NetCDF4.

» Requirement: The reformatting software shall log each transaction’s control information, including: the calling application, the type of transaction requested, the start and end times, and completion status codes. – The Reformatting Toolkit software shall generate run logs and

return NDE/STAR standard (agreed upon) error codes to the DHS.

– Run times shall be monitored by the NDE DHS.

36

Functional Requirements:

Reformatting Toolkit Software

» Requirement: Applications running under either Linux or AIX Operating Systems shall be able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e.g., dissemination).

» Requirement: The toolkit parameters (e.g., how to use the service) shall be well documented. – Reformatting Toolkit Developers shall provide documentation in

the form of a tailored Delivered Algorithm Package (DAP).

» Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall be comprehensible by untrained operators. – Reformatting Toolkit shall use the standard set of error return

codes developed by NDE for code running within the DHS. The NDE system shall direct this information to operators.

37

Functional Requirements:

Reformatting Toolkit Software

» Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall include diagnostic details needed for troubleshooting. – All messaging shall be directed to a run log file. These messages

shall be documented in the Reformatting Toolkit tailored DAP.

» Requirement: STAR shall coordinate development of the reformatting toolkit Application Program Interface with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system. – The Reformatting Toolkit shall work with the NDE integration team

to integrate the Reformatting Toolkit software into the NDE DHS.

» Requirement: Toolkit code shall adhere to the STAR coding standards.

» Requirement: Performance shall be measured on a product level.

38

Program Requirements:Reformatting Toolkit

Project

Requirement: STAR shall provide monthly project status reports to OSDPD and OSD.

Requirement: Earned Value Management shall be performed on the project.

Requirement: STAR shall update the project plan on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration.

Requirement: The toolkit shall be implemented and tested six months before the NPP launch to ensure NDE readiness.

39

Product Requirements:CrIS Radiances

Requirement: The Reformatting Toolkit shall tailor the NUCAPS thinned CrIS Radiances from NetCDF4 into BUFR for EMC.» The Reformatting Toolkit developers shall work with EMC to

create a BUFR table for the NUCAPS thinned radiances based on AIRS and IASI.

» The table shall use delayed replication for storing the radiances.» BUFR messages shall be smaller than 50KB.» The BUFR format shall allow for the storage of negative

radiances.» The file shall contain the following data fields (see table next

slide):

40

Product Requirements:CrIS Radiances

Variable Name Variable Name Variable Name Variable Name

Satellite ID Latitude Height of Land Surface

Start Channel (per band)

ID of Originating Center

Longitude Satellite Height End Channel (per band)

Satellite Instrument Satellite Zenith Angle Land Fraction Calibration Quality Flags

Satellite Classification

Satellite Azimuth Land/Sea Qualifier Field of View Quality Flags

Year Solar Zenith Cloud Cover Geolocation Quality

Month Solar Azimuth Height of Cloud Top NUCAPS Quality

Day Ascending/Descending flag

Radiance Type Flags Channel Number

Hour Scan Line Number Scan-Level Quality Flags

Channel Radiance

Minute Field of Regard Type of Band

Second Field of View Starting Wavenumber (per band)

Location of Platform Orbit Number Ending Wavenumber (per band)

41

Product Requirements:ATMS Radiances

Requirement: The Reformatting Toolkit shall tailor the NPOESS ATMS Radiances from NetCDF4 into BUFR for EMC. » The ATMS BUFR file shall contain the TDR (Antenna Temperatures),

the associated Quality Flags, and the Geolocation data at native resolution (not resampled) data.

» The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is current provided for AMSU and MHS.

» BUFR messages shall be smaller than 50KB.» The file shall contain the following data fields (see table next slide):

42

Product Requirements:

ATMS Radiances

Variable Name Variable Name Variable Name

Satellite ID Latitude Bandwidths

ID of Originating Center

Longitude Antenna Temperatures

Satellite Instrument Satellite Zenith Angle ATMS Quality Flags 1-23

Satellite Classification

Satellite Azimuth

Year Solar Zenith

Month Solar Azimuth

Day Scan Line Number

Hour Field of View

Minute ATMS Channels

Second ATMS Central Frequencies

Location of Platform Polarization

43

Product Requirements:OMPS Ozone

Requirement: The Reformatting Toolkit shall tailor NPOESS OMPS Ozone products from NetCDF4 into BUFR for EMC.» The product shall contain OMPS Nadir Profile and Total Column (this

would be the version 8 ozone algorithm for both products).» The Reformatting Toolkit developers shall work with EMC to develop

an OMPS BUFR table based on that currently used for GOME and SBUV.

» BUFR messages shall be smaller than 50KB.

Note: The project originally planned for OMPS radiances to be included. John Derber (EMC) says that they are not needed.

44

Product Requirements:VIIRS SST

Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS SST products from NetCDF4 into BUFR for EMC.» Product shall contain Skin SST, Bulk SST, Quality Flags,

Cloud Mask, and geolocation data.» Reformatting Toolkit developers shall work with EMC to

create a BUFR table for the VIIRS SST product. » The VIIRS SST BUFR table shall be derived from that

currently being used for the AVHRR derived SST (from ACSPO - Advanced Clear-Sky Processor for Oceans).


45

Product Requirements:VIIRS Radiances

Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Radiances from netCDF4 into BUFR for EMC.» The product shall contain VIIRS radiances for 22 channels

with associated quality flags and geolocation. Coverage shall be global.

» The product shall contain the land and cloud mask if it doesn’t take too long for the IDPS to generate those EDRs.

» Reformatting Toolkit developers shall work with EMC to create a BUFR table for the VIIRS radiance product. This table shall be derived from that currently being used for the GAC AVHRR.


46

Product Requirements:Aerosol Optical Thickness

Requirement: The Reformatting Toolkit shall tailor NPOESS Aerosol Optical Thickness (AOT) from netCDF4 into BUFR for EMC.» The product shall contain the AOT, wavelength of AOT,

and Aerosol Size.» Reformatting Toolkit developers shall work with EMC to

develop the AOT BUFR table based on what has already been done for MODIS.


47

Product Requirements:VIIRS Snow Cover

Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Snow Cover products from netCDF4 to GRIB2 format for EMC.

After discussions with Ken Mitchell and Michael Ek, it was determined that EMC only wants NPOESS snow products that have been passed through the IMS (Interactive Multisensor Snow and Ice Mapping System).

According to Tom Schott the Reformatting Toolkit project is not currently funded to tailor products for IMS. This is anticipated in the future although IMS may want a format other than GRIB2.

The Reformatting Toolkit development team recommends that this requirement be removed from the current phase of this project.

48

Product Requirements: Vegetation Index

Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Vegetation Index products from netCDF4 to GRIB2 format for EMC.

After discussions with Ken Mitchell and Michael Ek, it was determined that EMC only wants a Green Vegetation Fraction that is like that currently being produced from AVHRR (Le Jiang, Felix Kogan).

The NPOESS VIIRS Vegetation EDR does not have GVF. GVF was present in the VIIRS Surface Type EDR, but was removed in the last version of the Common Data Format Control Books (Feb 2009). According to Andy Heidinger, that algorithm was based on the MODIS vegetation algorithms anyway.

The Reformatting Toolkit development team recommends that this requirement be removed for the current phase of this project.

49

Interface Requirements

Requirement: The Reformatting Toolkit shall receive the thinned CrIS radiances (~300 channels) files from NUCAPS as an input for generating the CrIS radiance BUFR files. These files shall be in NetCDF4 and shall already contain the thinned SDR and geolocation information.

Requirement: The Reformatting Toolkit shall receive the NPOESS ATMS TDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the ATMS radiance BUFR files.

Requirement: The Reformatting Toolkit shall receive the NPOESS OMPS Total Column Ozone EDR files and OMPS Nadir Profile IP files tailored into NetCDF4 as an input for generating the OMPS Ozone BUFR files.

50

Interface Requirements

Requirement: The Reformatting Toolkit shall receive the NPOESS VIIRS radiance SDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the VIIRS radiance BUFR files.» Requirement: The Reformatting Toolkit shall receive the NPOESS VIIRS IP

Cloud Mask Product to obtain the cloud and land mask for the VIIRS radiance BUFR files.

Requirement: The Reformatting Toolkit shall receive the NPOESS Aerosol Optical Thickness EDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the Aerosol Optical Thickness EDR BUFR files.

Requirement: The Reformatting Toolkit shall receive the NPOESS SST EDR files and associated Geolocation files tailored into netCDF4 as an input for generating the SST BUFR files.

51

Requirements Summary

The Reformatting Toolkit developers have met with NDE and all the users of the original Phase I NDE tailored products.

All Reformatting Toolkit project, functional, and product requirements have been captured and documented in the Reformatting Toolkit RAD.

As development continues the detailed product requirements shall continue to be refined.

NDVI and Snow Cover are recommended to be waived from the phase 1 requirements.

52

Review Outline


53

Software Architecture

Presented by

Thomas KingPSGS

54

Reformatting Toolkit Architecture

Hardware Environment» Development and Unit testing» Test Product Distribution» System Testing and Integration» Production

Data Files» Input Files» Static/Ancillary Files» Output Files» Run Files

Software Description» External Interfaces» System Level» Unit Level

55

Reformatting Toolkit Development Hardware

Reformatting Toolkit Development Hardware - Unit tests, and CrIS/ATMS simulation data generation will be conducted on the IASI development machine at NSOF. This machine is maintained under the ESPC maintenance contract.» IBM P570 (AIX 5.3)» 7 TB disk space» 16 CPU» 2 GB memory/CPU» IBM XL Version 7.0 C/C++» IBM XL Version 10.1 Fortran 77/90

56

Reformatting Toolkit Development Hardware

Additional Development Hardware - Reformatting Toolkit testing may be conducted at STAR on AIX. This machine was purchased with ground systems money. It will be located at NCWCP, and maintained by STAR IT.» IBM P570» 3 TB disk space» 16 CPU» 2 GB memory/CPU

Configuration management of Reformatting Toolkit code will be conducted in the STAR Collaborative Environment on STAR Linux hardware running IBM Clear Case.

57

Reformatting Toolkit Test Product Distribution

STAR Data Server - Reformatting Toolkit test products will be available on a distribution server at STAR (ftp2.orbit.nesdis.noaa.gov). » Linux» 3.2 TB disk space» Access via anonymous ftp» Products available on a near real time basis

This server will make available test BUFR and GRIB2 files containing simulated data, BUFR tables, and any additional resources.

58

Reformatting Toolkit Integration and

Production Hardware

NDE SADIE – The Reformatting Toolkit system testing and integration will be conducted on the SADIE integration platform working with NDE integration personnel. This hardware is located at NSOF and is maintained under the ESPC maintenance contract. » IBM P561 (AIX 5.3)» 50 TB disk space» 16 CPUs» 2 GB/CPU» IBM XL Version 9.0 C/C++» IBM XL Version 11.1 Fortran 77/90

59

Reformatting Toolkit Integration and

Production Hardware

NDE Test/Production Hardware - After successful system tests, NDE will check the code into configuration management and then the code will be promoted to the Test/Production machine by NDE system integration staff. This machine will be located at NSOF. It has not yet been acquired.» IBM P570 (AIX 5.3)» TBD Disk space» TBD CPUs» TBD GB/CPU

60

Reformatting Toolkit Data

The tables on the following slides show all the Reformatting Toolkit input and output. The input data files are those required by the Reformatting Toolkit software to produce the Phase I tailored products.

All input data must be netCDF4 format.

Ancillary/Static files, such as the BUFR tables are listed.» They will be delivered as part of the DAP» Described more thoroughly in the CDR

All output will be in either BUFR or GRIB2 format.

61

Reformatting Toolkit Input Data Files

File Format Source Description Purpose

CrIS Thinned SDR Radiances netCDF4 NUCAPS The NUCAPS CrIS thinned ~300 channel radiances for all FOVs.

CrIS BUFR

ATMS TDR netCDF4 NDE NPOESS ATMS full resolution file tailored into netCDF4 from the NPOESS HDF5.

ATMS BUFR

ATMS TDR Geo netCDF4 NDE NPOESS ATMS Geolocation file that is associated with the ATMS TDR.

ATMS BUFR

VIIRS M-Band SDR netCDF4 NDE NPOESS VIIRS Moderate resolution band SDR. VIIRS BUFR

VIIRS I-Band SDR netCDF4 NDE NPOESS VIIRS Imagery resolution band SDR VIIRS BUFR

VIIRS DNB SDR netCDF4 NDE NPOESS VIIRS Day/Night Band SDR. VIIRS BUFR

VIIRS M-Band SDR Geo netCDF4 NDE VIIRS Moderate resolution band SDR Geo file. VIIRS BUFR

VIIRS I-Band SDR Geo netCDF4 NDE VIIRS Imagery resolution band SDR Geo file. VIIRS BUFR

VIIRS DNB SDR Geo netCDF4 NDE VIIRS Day/Night Band SDR Geo file. VIIRS BUFR

VIIRS Cloud Mask IP netCDF4 netCDF4 NDE VIIRS granule files containing cloud data. VIIRS BUFR

VIIRS Cloud Mask IP Geo netCDF4 netCDF4 NDE VIIRS granule files containing the geolocation information for the cloud product.

VIIRS BUFR

62

Reformatting Toolkit Input Data Files


OMPS Nadir IP netCDF4 NDE NPOESS OMPS Nadir Profile ozone Intermediate Product file tailored into netCDF4. Geolocation information is currently in the file. NGAS documentation indicates that much of the content of this product is TBD.

OMPS BUFR

OMPS Total Column EDR netCDF4 NDE NPOESS OMPS Total Column ozone EDR file tailored into netCDF4.

OMPS BUFR

OMPS Total Column SDR Geo netCDF4 NDE NPOESS OMPS Total Column ozone SDR Geo file tailored into netCDF4. This same Geo file is the used with the OMPS Total Column EDR.

OMPS BUFR

Aerosol Optical Thickness EDR netCDF4 NDE NPOESS AOT EDR file tailored into netCDF4. AOT BUFR

Aerosol Optical Thickness EDR Geo netCDF4 NDE NPOESS AOT EDR Geo file tailored into netCDF4. AOT BUFR

Sea Surface Temperature EDR netCDF4 NDE NPOESS SST EDR file tailored into netCDF4. SST BUFR

Sea Surface Temperature EDR Geo netCDF4 NDE NPOESS Moderate Resolution Terrain-Corrected Geolocation file tailored into netCDF4. This is to be used with the NPOESS SST EDR.

SST BUFR

63

Reformatting Toolkit Ancillary/Static Data

Files


CrIS SDR BUFR Table ASCII N4RT CrIS thinned radiances (~300 channels, all FOVs) BUFR Template

ATMS SDR BUFR Table ASCII N4RT ATMS radiances full resolution BUFR Template

VIIRS SDR BUFR Table ASCII N4RT VIIRS radiances full resolution BUFR Template

OMPS EDR BUFR Table ASCII N4RT OMPS Nadir Profile and Total Column Ozone BUFR Template

AOT EDR BUFR Table ASCII N4RT Aerosol Optical Thickness and Particle Size BUFR Template

SST EDR BUFR Table ASCII N4RT Sea Surface Temperatures, Cloud Mask BUFR Template

64

Reformatting Toolkit Output Data Files

File Format Description Users

CrIS SDR BUFR CrIS thinned radiances (~300 channels, all FOVs) JCSDA, NCEP, EUMETSAT, UKMet, ECMWF

ATMS SDR BUFR ATMS radiances full resolution JCSDA, NCEP

VIIRS SDR BUFR VIIRS radiances full resolution JCSDA, NCEP

OMPS EDR BUFR OMPS Nadir Profile and Total Column Ozone JCSDA, NCEP

AOT EDR BUFR Aerosol Optical Thickness and Particle Size JCSDA, NCEP

SST EDR BUFR Sea Surface Temperatures, Cloud Mask JCSDA, NCEP

65

Reformatting Toolkit Run Data Files


PCF ASCII NUCAPS The PCF file containing all the required input parameters for the N4RT driver script.

Driver input

N4RT Run Log ASCII N4RT The N4RT log file containing all the standard error and standard output from a given run.

Diagnostic

N4RT Main Resource ASCII N4RT The internal control file required by the N4RT executable main program.

Executable input

PSF ASCII N4RT The Process Status File containing only the successfully generated output files.

Driver output list

66

Reformatting Toolkit Software:

External Interfaces The Reformatting Toolkit system will be run via the execution of a single driver

script that will be invoked, monitored, and managed by the NDE DHS Product Generation Manager.

Execution of the script will be event-driven (i.e. the arrival of a required input file whose type is predefined in the Reformatting Toolkit production rules given to NDE).

An invocation of Reformatting Toolkit by the NDE DHS will be used to do a single granule-to-granule file translation (as opposed to reformatting many files as once).

NDE will run the script in a working directory. All Reformatting Toolkit output will be produced in this same working directory.

The driver script will require an input PCF file containing parameters such as:» The input and output file names» Input and output product type IDs (e.g. ATMS radiances)» The type of conversion required (e.g. netCDF4 to BUFR)

67

Reformatting Toolkit Software:

External Interfaces

The driver script will run, parse the PCF content, run the compiled conversion code, handle program output and errors, direct required NDE error codes to the DHS, generate an output log, and generate a PSF.

If there are errors, NDE will save the contents of the run in a forensics repository.

NDE will manage and direct error status to the operators from the DHS system.

NDE will manage all distribution through the NDE DDS and the short term storage of tailored data on the SAN.

68

Reformatting Toolkit External

Interfaces to NDE

Reformatting Toolkit Driver

Script

SAN

NDE Product Generation Manager

Reformatting Toolkit External Interfaces

Product Generation

Specifications

Working Directory

Systems Configuration

s

Forensics Repository

Input Files &

PCF

InvocationProcess Req.

Rule SetsOutput Files &

PSF

BUFR & GRIB2

Output Files

PSF (N4RT output)

Return Code

Working Directory

Output

Input Files (netCDF4)

NDE DDS

PCF (N4RT input)DAP

Specifications

Data AreasConfigurations InfoN4RT SystemNDE Production Manager

NDE DHS Boundary

Input Files (netCDF4)

69

Reformatting Toolkit Software: System Level

The Reformatting Toolkit driver script will be a single Perl script that acts as a wrapper for the compiled Reformatting Toolkit code.

» There will be no hard coded paths in the script. All needed information regarding locations of files will come through the PCF.

» All system calls have their return values checked so the exits are graceful and informative.

» All standard out and standard error will be directed to a single log file.

» The driver script will translate the low-level program errors into the high-level numerical error codes expected by the PGM.

» The Perl script will generate an internal control file for the main Reformatting Toolkit program.

70

Reformatting Toolkit Software: System Level

The PCF content (nominal mode):» Input netCDF4 file» Output BUFR file» Output GRIB2 file» Direction of conversion (eg. NC to BF)» Input and output product types» Input BUFR table (if BUFR is an output)

The PCF content (test mode): » Input BUFR or input GRIB2» Output netCDF4 file» Direction of conversion (e.g. BF to NC)» Input and output product types» NC templates if (netCDF4 is an output type)

71

Reformatting Toolkit System

Level Data Flow

Reformatting Toolkit System Level Data Flow

PCF Return Value to PGM

Execution from PGM

N4RT Driver Script

Working directory

PSF

Working directory

N4RT MainConverter N4RT log

BUFR file

Resource

GRIB2 file

Working directory

netCDF4 NC Template

BUFR table

netCDF4

BUFR/GRIB2

Working directory

Nominal ModeTest Mode

72

Reformatting Toolkit Software: Unit Level

The Reformatting Toolkit main program will be a single main C++ program containing all the predefined data structures required by the code. Lower-levels will be in Fortran 90. Why this approach?» C++ main program will give the system flexibility for having to plug into future

types of architectures.» APIs for BUFR are only Fortran 77.» Much of the code being reused is already in Fortran 90.» Many users tend to request readers in Fortran 90.

In the Reformatting Toolkit main program, all data structures will be declared and a series of cases statements will direct the program flow based on the type of input files and the direction of conversion.

The Reformatting Toolkit subroutines at the next level down (and at all subsequent levels) will be in Fortran 90. This next level will manage:» Allocation, initialization, deallocation of data structures. » They are designed for overseeing specific product definitions and conversions.

73

Reformatting Toolkit Software: Unit Level

The lowest-level Reformatting Toolkit subroutines:» Perform the actual reading and writing of the specific file types.» Directly call the library APIs for BUFR, GRIB2, and netCDF4.

General Reformatting Toolkit compiled code characteristics:» The status of all functions are checked to allow for graceful and

informative exits.» No paths are hard coded in the compiled code.» The are no system calls from within the compiled code.» All code will be compiled as 64 bit to utilize the IBM architecture.

74

Reformatting Toolkit Unit

Level Data FlowReformatting Toolkit UNIT Level Data Flow

Prod N - NC2BF Prod N - BF2NCProd N - NC2GB Prod N - GB2NC

Allocate

Deallocate

Initialize

Prod N - Read BF

Prod N – Write NC

Allocate

Deallocate

Initialize

Prod N - Read NC

Prod N - Write GB

Allocate

Deallocate

Initialize

Prod N - Read GB

Prod N - Write NC

Allocate

Deallocate

Initialize

Prod N - Read NC

Prod N - Write BF

N4RT resource N4RT log file

BUFR GRIB2 NetCDF4 NetCDF4

N4RT Main

Nominal ModeTest Mode

75

Reformatting Toolkit Software: Libraries

BUFR Library » NCEP BUFRLIB library available on the NCEP website» Code is all Fortran 77» Compiled as 64 bit

GRIB2 Library» Available on the NCEP website» Compiled as 64 bit

netCDF4 Library version 4.0» Available from Unidata website» Compiled as 64 bit

Reformatting Toolkit developers will coordinate with NDE and the product teams to make sure that all parties are using compatible libraries.

76

Software Architecture Summary

Reformatting Toolkit software development, testing, and operation will be conducted on comparable IBM hardware at NSOF and STAR.

The Reformatting Toolkit code will run as a stand-alone unit within the NDE DHS.

The code will be modular to allow for easy reuse of code and expansion to accommodate new products.

The code will use only the official releases and NDE compatible versions of the netCDF4, BUFR, and GRIB2 libraries.

77

Review Outline


78

Quality Assurance

Presented by

Thomas KingPSGS

79

Quality AssuranceBackground

STAR is using the Capability Maturity Model Integrated (CMMI) to improve processes and practices for development and the transfer of research to operations.

The STAR Enterprise Life Cycle (EPL) merges the CMMI and SPSRB standards.

IASI and NUCAPS are both pathfinder projects (CMMI Level 3). The IASI system has been transitioned to operations successfully.

The Reformatting Toolkit algorithm development will follow the same process but tailored to the scale of the project.

80

The Reformatting Toolkit Preliminary Design Review (April 2009) » Will present the initial draft of the requirements and discuss a proposed design.» An Reformatting Toolkit Requirements Allocation Document (RAD) has been made

available to Reformatting Toolkit stakeholders. It will be updated throughout the lifecycle of the project.

The Reformatting Toolkit Critical Design Review (September 2009)» To finalize requirements and to verify that the chosen design is able to meet those

requirements.

An Reformatting Toolkit Test Readiness Review (June 2010) » Will present the unit test plan to demonstrate that the toolkit is ready to be run in the

Test Environment.

A code unit test review (August 2010) » Will be conducted to ensure that the Reformatting Toolkit software is able to fulfill

the functional software requirements.

The Reformatting Toolkit System Read Review (November 2010) » Has requested to be waived because Reformatting Toolkit will be run within the

NDE system.

Quality Assurance – ProjectImplemented Tailored STAR EPL

Reviews

81

CM Tool (IBM Rational ClearCase, Version 7.0 ) » Has been purchased and implemented in the Collaborative

Environment.

CM personnel have been identified.

CM training:» Administrator training completed. » Developers will be trained by the CM administrator.

Detailed CM Plan is under development.

Configuration Management

(CM)

82

STAR Coding Standards

Coding standards guidelines and quick references are available.

Provide a common list of abbreviations.

Adhere to the standards throughout the development life cycle.

Have checklists available for developers to keep track of the delivery status of the code.

83

Quality Assurance – Software

The Reformatting Toolkit software will be delivered incrementally as part of the series of prototype DAPs following the NDE Build Content Reviews (BCR). This will allow system testing of the code within the NDE DHS. The BCR schedule is as follows:» Feb 17 – April 17 (Initial integration)» April 20 – June 19 (Throttle and Load Balance)» June 22 – July 17 (Generate Hourly, Daily Products)» July 20 – August 14 (Generate Tailored and Regional Products)» August 17 – September 18 (Complex Production Rules)» September 21 – October 16 (Weather Event Based Production)» October 19 – November 14 (Production Monitoring)

84


All code development is being conducted on a platform that is nearly identical to the test and production target platforms using the same compilers and operating system.

Only the official releases of the NCEP BUFRLIB, GRIB2, and NetCDF4 libraries will be used in the software.

STAR code checking tools will be used to minimize coding bugs and to ensure that Reformatting Toolkit code meets STAR coding standards.

The status of all system calls and intrinsic functions are checked.

Unit tests will have the Reformatting Toolkit generate BUFR and GRIB2 products from NetCDF4 files. These BUFR and GRIB2 files will be directed back into the Reformatting Toolkit to generate new netCDF4 files.» The contents of the new and original netCDF4 files will be compared to demonstrate that

what went in matches with what came back out.» Customers will have access to test data products to verify that values appear reasonable

and that precision is not being lost in the format conversions.

85


An official (tailored) DAP will be delivered:» All Reformatting Toolkit code and system files» Test plans» Test data sets» Error messaging/handling» PCF format» Production rules» Product file specifications» Data flow diagrams» Estimates of resource usage» ATBD has been waived» Delivery memo

86

Quality Assurance – Products

Reformatting Toolkit developers will work with EMC, NRL, FNMOC, GMAO, EUMETSAT, and UK Met users to ensure that the proper BUFR descriptors are used in the BUFR tables and that, whenever possible, they are WMO approved (as opposed to local). » Chosen descriptors must work with NCEP BUFRLIB.

Reformatting Toolkit developers will work with EMC, heritage product developers, and NPOESS operational algorithm teams to ensure consistency with heritage products with respect to format and content.

Reformatting Toolkit developers will make BUFR tables and test products (BUFR and GRIB2) available to users before the operational products are made available. This will allow for preliminary product content validation.» CrIS and ATMS are currently being simulated at STAR.» For the other products such as VIIRS, we are working with the IPO (Joe Zajic),

the NPP Test Data Working Group (TDWG), and NDE to obtain simulated products.

87

Quality Assurance – Archive and Maintenance

Archive Plan» Reformatting toolkit will be integrated into NDE system and made

available to CLASS by NDE» Product archive requirements are addressed within product

development projects– NPOESS program office works with CLASS to archive xDRs– NOAA Unique Product (NUP) projects work with CLASS as

required

Long Term Maintenance Plan» The reformatting toolkit will be maintained by the OSDPD staff» STAR system developers will be available

88

Quality Assurance – Documentation and

Metadata

Documentation/Metadata Plan» The Documentation will include the SPRSB documents

and additional CMMI documents» No metadata is required for these products

89

Quality AssuranceSummary

Quality assurance plan will consist of: » Project reviews at which stakeholders are encouraged to participate.» Ongoing interaction with customers, heritage product developers,

operations, NDE, and the SPSRB.» Adhering to STAR/NDE software standards and use of standard

libraries only.» Software unit tests shall be presented in the TRR.» Documentation of the Reformatting Toolkit code operation, production

rules, and software tests will be in the DAP.» Documentation of requirements will be in the Reformatting Toolkit

RAD.» Early release of BUFR and GRIB2 products, tables, and additional

resources will allow for customer validation of products.

90

Review Outline


91

Risks and Actions

Presented by


92

Risk and Actions

Risk 1: NPOESS product formats and content are still changing, especially for VIIRS.

Risk Assessment: Low Impact: Changes in the NPOESS format may cause delays in the

development of the output product files. Likelihood: High Mitigation:

» Work through the NPOESS Data Format Working Group to obtain information on format and algorithm updates.

» Monitor the latest copies of the NPOESS Common Data Format Control Books (CDFCB) for any updates.

» Maintain contact with customers to inform them of any upstream product changes.

» Make the code design flexible so that changes in the upstream products translate into the minimum amount code revision.

Status: Open

93

Risk and Actions

Risk 2: The roles and responsibilities regarding who shall generate the set of required SPSRB documents for NDE has not yet been decided.

Risk Assessment: Low Impact: Documentation that has not been planned for may

cause delays and or cost overruns Likelihood: Moderate Mitigation:

» This issue as well as decisions regarding the content of the SPSRB documents is being worked by Ken Jensen and STAR, NDE, OSD, and OSDPD personnel. Reformatting Toolkit developers intend to participate in these meetings and discussions.

Status: Open

94

Risk and Actions

Risk 3: There are small variations in the types of platforms and the versions of the compilers.

Risk Assessment: Low Impact: Compiler differences may cause errors in the output

files. Likelihood: Low Mitigation:

» Reformatting Toolkit developers will work with NDE during system tests in the integration and production phase to ensure that those results match the results from the units. The BCR process (mentioned earlier) should help to resolve these issues early on in development.

Status: Open

95

Risk and ActionsSummary

There are currently 3 risks identified for the Reformatting Toolkit project.

The severity of all 3 risks is low.

These risks remain open at this time.

96

Review Outline


97

Summary and Conclusions

Presented by


98

The Project Mission and Schedule has been reviewed.

The Project Requirements have been reviewed» Recommend waiving the NDVI and Snow Cover products.

Software architecture, hardware, data, and interfaces have been reviewed.

Plans for quality assurance have been reviewed.

Risks and Actions have been reviewed.

Review Objectives Have Been Addressed

99

Current Status

A CrIS BUFR table has already been created working with Yi Song, Dennis Keyser, Jeff Ator, Jim Jung, Simon Elliott (EUMETSAT), and Nigel Atkinson (UKMet).

A test CrIS BUFR file was created from this table using simulated data and was delivered to EMC (mid-March 09).

The new CrIS BUFR table descriptors and test data file have been submitted to WMO by Jeff Ator (early April 09).

Test CrIS BUFR file has been made available to NCEP, NRL, FNMOC, and the global NWP centers

Work on the ATMS BUFR table is currently ongoing.

Reformatting Toolkit is currently working with Joe Zajic (IPO) to obtain access to simulated VIIRS products at NSOF.

100

Next Steps

Code Development Phase» ATMS BUFR coding and test file generation» VIIRS, OMPS, SST, and AOT BUFR table

development» Obtain VIIRS test data» Participate in the NDE BCR process

Critical Design Review (September 2009)

101

Open Discussion

The review is now open for free discussion

Documents

1 NetCDF4 Reformatting Toolkit: BUFR and GRIB Tailoring Preliminary Design Review April 14, 2008 Prepared By: Tom King 1, Walter Wolf 3, Lihang Zhou 1,