View
26
Download
0
Category
Tags:
Preview:
DESCRIPTION
Storing EOS data using HDF5 Archival Information Package. MuQun Yang (The HDF Group) Choonghwan Lee (The HDF Group) Ruth Duerr (NSIDC ). Definitions of some terms. CDM(Common Data Model) Unification of the data models of OpenDAP , netCDF and HDF5 netCDF-4(network Common Data Form 4) - PowerPoint PPT Presentation
Citation preview
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 1
Storing EOS data using HDF5 Archival Information
PackageMuQun Yang (The HDF Group)
Choonghwan Lee (The HDF Group)Ruth Duerr (NSIDC )
2
Definitions of some terms• CDM(Common Data Model)
• Unification of the data models of OpenDAP, netCDF and HDF5
• netCDF-4(network Common Data Form 4)• Library manipulating array-oriented
scientific data• CF-Conventions
• Convention of metadata providing spatial properties, among climate research community…
• CLASS (Comprehensive Large Array-data Stewardship System)• Electronic library of NOAA environmental
data10/16/2008
HDF and HDF-EOS Workshop XII, Aurora, CO
3
Prerequisite• METS(Metadata Encoding & Transmission
Standard)• Standard for encoding structural metadata
digital library • ISO-19115
• Schema for describing geographic information(former FGDC)
• File-level Metadata• Metadata about the individual file or
granule• Dataset-level Metadata
• Metadata that applies to each and every granule/file in the whole data set(product)10/16/2008
HDF and HDF-EOS Workshop XII, Aurora, CO
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 4
HDF5 Archive Information Package
Data file HDF5
METS
Primary Schema Extension Schema
|<mets>|---<dmdSec>----------------<MODS>|---<amdSec>--------------|--<techMD>| |--<rightsMD>| |--<sourceMD>|----<fileGrp>|----<structMap>
http://www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF5_AIP_White_Paper.pdf
HDF5 AIP Components
Metadata file
10/16/2008HDF and HDF-EOS Workshop XII,
Aurora, CO 55
NOAA SDS Program
NetCDF4/HDF5-data
NetCDF4 / HDF5 Data
METS
NSIDC/ ECS
HDF4-data
NCDC:CLASS
ISO-19115
HDF5-AIP
H4toH5
ECS to ISO-19115
NSIDC/ECS Metadata
ECS Users
CDM/NetCDF4
ECS to METS
Enhanced H4toH5 conversion tool
• Convert HDF-EOS2 data to NetCDF4-compliant HDF5 data
• Official release (2.0) can be found at http://hdfgroup.org/h4toh5/
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 6
$ ./h4toh5 –eos –nc4 input.he2 output.nc4
7
Challenges to do the conversion
• Retrieve geo-location information from HDF-EOS2 data
• Conform to NetCDF4 data model in the existing H4toH5 conversion tool
• ……
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO
• Grid lacks geolocation fields• Use predefined projections
• Geographic• Sinusoidal• Polar stereographic• …
• New converter creates geolocation fields• HDF-EOS2 API GDij2ll()
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 8
Challenges: Handle EOS - Grid
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
Data [4][12]Lon[12]Data [4][8]Lon[4][8]
Geographic
Sinusoidal
• The size of geolocation fields can be different from data fields
• New converter has to handle geolocation fields correctly
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 9
. . .. . .. . .. . .. . .. . .. . .. . .. . .
Challenges: Handle EOS - Swath
• Follow CF conventions• Create two variables: NewLongitude and
NewLatitude• Add to the data field an attribute coordinates=“NewLongitude NewLatitude”
• Keep the original Latitude and Longitude
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 10
Challenges in conforming to NetCDF4
. . .. . .. . .. . .. . .. . .. . .. . .. . .
. .. .. .. .. .. .. .. .. .
. . .. . .. . .. . .. . .. . .. . .. . .. . .
Data field has three columnsLongitude field has two columnsNew longitude has three columns
1110/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO
Now some examples to show NetCDF4 files
converted from EOS2
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 12
A netCDF-4 file converted from EOS2 data at NSIDC
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 13
A netCDF-4 file converted from EOS2 data at NSIDC
Deliverables and future work• Deliverables
1. Enhanced HDF4 to HDF5 conversion toolhttp://hdfgroup.org/h4toh5/
2. A validation tool to verify the correctness of the conversionWill be released soon!
• Potential future work1. Reduce the output file size2. Improve the performance
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 14
10/16/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 15
Acknowledgement This work was supported under NOAA Scientific Stewardship Program grant number NA07OAR4310286. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NOAA.
Recommended