10/16/2008 HDF and HDF-EOS Workshop XII 1

Storing EOS data using HDF5 Archival Information Package

MuQun Yang (The HDF Group)

Choonghwan Lee (The HDF Group)

Ruth Duerr (NSIDC )

Definitions of some terms

• CDM(Common Data Model)• Unification of the data models of OPeNDAP,

netCDF and HDF5

• CF-Conventions• Convention of metadata providing spatial

properties, among climate research community…

• CLASS (Comprehensive Large Array-data Stewardship System)• Electronic library of NOAA environmental data

10/16/2008 HDF and HDF-EOS Workshop XII 2

Prerequisite

• METS(Metadata Encoding & Transmission Standard)• Standard for encoding structural metadata

• ISO-19115• International Schema for describing geographic

information

• File-level Metadata• Metadata about the individual file or granule

• Dataset-level Metadata• Metadata that applies to each and every

granule/file in the whole data set(product)

10/16/2008 HDF and HDF-EOS Workshop XII 3

10/16/2008 HDF and HDF-EOS Workshop XII 4

HDF5 Archive Information Package

Data file HDF5

METS

Primary Schema Extension Schema

|<mets>|---<dmdSec>----------------<MODS>|---<amdSec>--------------|--<techMD>| |--<rightsMD>| |--<sourceMD>|----<fileGrp>|----<structMap>

http://www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF5_AIP_White_Paper.pdf

HDF5 AIP Components

Metadata file

10/16/2008 HDF and HDF-EOS Workshop XII 55

NOAA SDS Program

NetCDF4/HDF5-data

NetCDF4 / HDF5 Data

METS

NSIDC/ ECS

HDF4-data

NCDC:CLASS

ISO-19115

HDF5-AIP

H4toH5

ECS to ISO-19115

NSIDC/ECS

Metadata

CDM/NetCDF4

ECS to METS

Enhanced H4toH5 conversion tool

• Convert HDF-EOS2 data to NetCDF4-compliant HDF5 data

• Official release (2.0) can be found at http://hdfgroup.org/h4toh5/

10/16/2008 HDF and HDF-EOS Workshop XII 6

$ ./h4toh5 –eos –nc4 input.he2 output.nc4$ ./h4toh5 –eos –nc4 input.he2 output.nc4

Challenges to do the conversion

• Retrieve geo-location information from HDF-EOS2 data

• Conform to NetCDF4 data model in the existing H4toH5 conversion tool

• ……

10/16/2008 HDF and HDF-EOS Workshop XII 7

• Grid lacks geolocation fields• Use predefined projections

• Geographic• Sinusoidal• Polar stereographic• …

• New converter creates geolocation fields• HDF-EOS2 API GDij2ll()

10/16/2008 HDF and HDF-EOS Workshop XII 8

Challenges: Handle EOS - Grid

Data [4][12]Lon[12]Data [4][8]Lon[4][8]

Geographic

Sinusoidal

• The size of geolocation fields can be different from data fields

• New converter has to handle geolocation fields correctly

10/16/2008 HDF and HDF-EOS Workshop XII 9

Challenges: Handle EOS - Swath

• Follow CF conventions• Create two variables: NewLongitude and

NewLatitude• Add to the data field an attribute coordinates=“NewLongitude NewLatitude”

• Keep the original Latitude and Longitude

10/16/2008 HDF and HDF-EOS Workshop XII 10

Challenges in conforming to NetCDF4

Data field has three columnsLongitude field has two columnsNew longitude has three columns

10/16/2008 HDF and HDF-EOS Workshop XII 11

Now some examples to show NetCDF4 files converted from EOS2

10/16/2008 HDF and HDF-EOS Workshop XII 12

A netCDF-4 file converted from EOS2 data at NSIDC

10/16/2008 HDF and HDF-EOS Workshop XII 13

A netCDF-4 file converted from EOS2 data at NSIDC

Deliverables and future work

• Deliverables1. Enhanced HDF4 to HDF5 conversion tool

http://hdfgroup.org/h4toh5/2. A validation tool to verify the correctness of the

conversion

Will be released soon!

• Potential future work

1. Reduce the output file size

2. Improve the performance

10/16/2008 HDF and HDF-EOS Workshop XII 14

10/16/2008 HDF and HDF-EOS Workshop XII 15

Acknowledgement

This work was supported under NOAA Scientific Stewardship Program grant number NA07OAR4310286. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NOAA.