Embed Size (px)
Citation preview
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XV 1
Tools to Improve the Usability of NASA HDF Data
Kent Yang and Joe LeeThe HDF Group
April 17, 2012
www.hdfgroup.org
Tutorial Outline
Introduction
5 Tools
1. H4CF Conversion Toolkit (h4cf)
2. h4toh5 Conversion Tool (h4toh5)
3. HDF-EOS5 to netCDF-4 Converter (eos52nc4)
4. HDF-EOS5 Augmentation Tool (aug_eos5)
5. HDF-EOS2 Dumper (eos2dump)
Demo
HDF/HDF-EOS Workshop XV 2April 17, 2012
www.hdfgroup.org
Who Said Users Don’t Matter?
April 17, 2012 HDF/HDF-EOS Workshop XV 3
www.hdfgroup.org
NASAHDF/HDF-EOS
Data
Usability of NASA HDF Data
April 17, 2012 HDF/HDF-EOS Workshop XV 4
IDL
MATLAB
Panoply IDV
A school teacher’s viewPanoply IDV
A school teacher’s expectation
www.hdfgroup.org
The 5 Tools
April 17, 2012 HDF/HDF-EOS Workshop XV 5
Improve the usability of NASA Data
Supported by The HDF Group
Funded by NASA / Raytheon
http://hdfeos.org/software/tool.php
www.hdfgroup.org
h4cf
eos2 dump
aug_eos5
eos52nc4
h4toh5
hdf4
5 Tools that Improve Usability
April 17, 2012 HDF/HDF-EOS Workshop XV 6
eos2
eos5ASCII
NETCDF-4
NETCDF-3Panoply / IDV
MATLAB IDLNCL
www.hdfgroup.org
H4CF Conversion Toolkit
Follow CF conventionBoth Library and Toolshttp://hdfeos.org/software/h4cflib
April 17, 2012 HDF/HDF-EOS Workshop XV 7
hdf4
eos2
NETCDF-4
NETCDF-3 h4cfPanoply / IDV
www.hdfgroup.org
H4CF Conversion Toolkit
Two ways to build:
1. Use configure script.
2. Edit Makefile
template.
April 17, 2012 HDF/HDF-EOS Workshop XV 8
www.hdfgroup.org
H4CF Conversion Library
• Variables (datasets) list• Attributes (metadata) list
• All Vars & Attrs follow CF conventions• Develop your own CF application!
April 17, 2012 HDF/HDF-EOS Workshop XV 9
www.hdfgroup.org
H4CF Library APIs Example
// open the example HDF fileh4cf_open("geo.hdf");
// HDF file attributes are obtained in a C++ STL map objectconst map<string, string> file_attrs = h4cf_get_file_attrs();
// HDF file variables are obtained in a C++ STL list objectconst list<var*> pvars = h4cf_get_vars();
// close the HDF fileh4cf_close();
April 17, 2012 HDF/HDF-EOS Workshop XV 10
www.hdfgroup.org
H4CF Conversion Tools
2 Tools based on H4CF APIs:
cdldumper
h4tonccf
April 17, 2012 HDF/HDF-EOS Workshop XV 11
www.hdfgroup.org
cdldumper
• Usage: cdldumper filename.hdf
• Dump file in Common Data Language.• Useful for quick check and debugging purpose.
April 17, 2012 HDF/HDF-EOS Workshop XV 12
www.hdfgroup.org
cdldumper Demo
April 17, 2012 HDF/HDF-EOS Workshop XV 13
Number of attributes=1_FillValue=255
www.hdfgroup.org
cdldumper Demo
April 17, 2012 HDF/HDF-EOS Workshop XV 14
%./cdldumper -H AMSR_E_L3_5DaySnow_V09_20050126.hdf netCDF AMSR_E_L3_5DaySnow_V09_20050126.hdf {dimensions: XDim_Northern_Hemisphere = 721, XDim_Southern_…variables: ubyte Flags_NorthernPentad(YDim_Northern_Hemis… Flags_NorthernPentad:_FillValue = 255; Flags_NorthernPentad:origname = "/Northern… Flags_NorthernPentad:long_name = "/Northern… Flags_NorthernPentad:coordinates = "latitude…
Number of attributes=4_FillValue=255orig_namelong_namecoordinates
www.hdfgroup.org
h4tonccf
• Usage: h4tonccf filename.hdf [filename.nc]• generates either netcdf-3 or netcdf-4 file.
• follows netcdf classic model.
April 17, 2012 HDF/HDF-EOS Workshop XV 15
www.hdfgroup.org
h4tonccf Demo
April 17, 2012 HDF/HDF-EOS Workshop XV 16
www.hdfgroup.org
Don’t Follow CF Conventions Blindly
April 17, 2012 HDF/HDF-EOS Workshop XV 17
[1] CF: scale * value + offset[2] MODIS: scale * (value – offset)
Scale = 0.01Value = 5000Offset = -15000.0
[1] 0.01 * 5000 + (-15000) = -14950[2] 0.01 * (5000 – (-15000)) = 200
www.hdfgroup.org
-14950 Kelvin?
April 17, 2012 HDF/HDF-EOS Workshop XV 18
www.hdfgroup.org
MODIS Scale / Offset Correction
HDF/HDF-EOS Workshop XV 19April 18, 2012
www.hdfgroup.org
Converted File Size Comparison
April 17, 2012 HDF/HDF-EOS Workshop XV 20
72M
128M
656M
HDF-EOS2
Netcdf-4
Netcdf-3
9X
www.hdfgroup.org
Limitation – No Support Yet
April 17, 2012 HDF/HDF-EOS Workshop XV 21
MISR (SOM/LAMAZ)
MERRAAVHRRLISTO
QuikSCAT/SeaWindMOPITT L2
etc.
www.hdfgroup.org
h4toh5 Conversion Tool
Convert any hdf4 to hdf5.
http://hdfgroup.org/h4toh5
April 17, 2012 HDF/HDF-EOS Workshop XV 22
eos2
NETCDF-4
h4toh5
Usage: h4toh5 –eos –nc4 in.hdf out.nc
Static Binary w/ –eos & -nc4:
http://hdfeos.org/software/h4toh5/bin/
www.hdfgroup.org
h4tonccf vs. h4toh5
April 17, 2012 HDF/HDF-EOS Workshop XV 23
h4tonccf h4toh5w/ -eos –nc4
Handle non-eos2 hdf4
Yes No
FollowCF
conventions
Yes No
Outputnetcdf classic
Yes No
Outputnetcdf
enhanced
No Yes
www.hdfgroup.org
eos52nc4
• converts hdf-eos5 to netcdf-4.• follows netcdf enhanced model.• does not follow CF conventions. • Usage: eos52nc4 test.he5 test.nc• http://hdfeos.org/software/convert_hdfeos5.php
April 17, 2012 HDF/HDF-EOS Workshop XV 24
eos5
NETCDF-4eos52nc4
www.hdfgroup.org
aug_eos5
• augments hdf-eos5 to netcdf-4.• Both HDF-EOS5 and NetCDF4 can access it.• Usage: aug_eos5 [-i|-f map] test.h5
• modifies the input file directly. • does not follow CF conventions.• http://hdfeos.org/software/aug_hdfeos5.php
April 17, 2012 HDF/HDF-EOS Workshop XV 25
eos5
NETCDF-4aug_eos
5
www.hdfgroup.org
hdf5
netcdf4
Augmentation Concept
April 17, 2012 HDF/HDF-EOS Workshop XV 26
hdf-eos5HDF-EOS5
APINETCDF-4
API
aug_eos5
augmentation
hdf-eos5
HDF5 API
www.hdfgroup.org
Why NetCDF-4 Doesn’t Work
April 17, 2012 HDF/HDF-EOS Workshop XV 27
[nLevels][nTimes]
NETCDF-4 API
hdf-eos5
www.hdfgroup.org
What aug_eos5 Does
April 17, 2012 HDF/HDF-EOS Workshop XV 28
[nLevels][nTimes]
NETCDF-4 API
hdf-eos5
www.hdfgroup.org
Options in aug_eos5
1. Default option
• -i option
2.-f map_file option
April 17, 2012 HDF/HDF-EOS Workshop XV 29
www.hdfgroup.org
Default option in aug_eos5
Grid
•XDim, YDim: calculated
• Other dims: default fill value
Swath / ZA• All dims: default fill value
April 17, 2012 HDF/HDF-EOS Workshop XV 30
www.hdfgroup.org
Better -i option in aug_eos5
• Grid• XDim, YDim: calculated
•Other dims: 1,2,…,n
• Swath / ZA
•All dims: 1,2,…,n
April 17, 2012 HDF/HDF-EOS Workshop XV 31
www.hdfgroup.org
Best -f option in aug_eos5
• Needs an input map file.
• Input file specifies 3 rules:1. What to be filled in
2. How to fill in
3. Where to get the values• Most useful and flexible option
April 17, 2012 HDF/HDF-EOS Workshop XV 32
www.hdfgroup.org
aug_eos5 Input File Format
Each line in input file contains
1. How: 0,1,2
2. What: dim name3. Where: data file or existing variable in file.
4. (Optional) Multiple grid/swath/za case only:• grid/swath/za name:<name>
April 17, 2012 HDF/HDF-EOS Workshop XV 33
www.hdfgroup.org
aug_eos5 Input File Example
0 nTimes Time
1 nLevels data.txt
2 nCandidates
April 17, 2012 HDF/HDF-EOS Workshop XV 34
Copy the data
Read file data.txt
Fill 1,2,…, n
How What Where
95% cases – Copy the data
www.hdfgroup.org
Multiple Swath Case
swath name: BrO
0 nTimes Time
1 nLevels data1.txt
2 nCandidates
April 17, 2012 HDF/HDF-EOS Workshop XV 35
How What Where
swath name: BrO column
2 nTimes
1 nLevels data2.txt
Multiple
www.hdfgroup.org
aug_eos5 vs eos52nc4
April 17, 2012 HDF/HDF-EOS Workshop XV 36
aug_eos5 eos52nc4
hdfeos-5 Yes No
netcdf-4 Yes Yes
www.hdfgroup.org
HIRDLS/MLS Team Use aug_eos5
April 17, 2012 HDF/HDF-EOS Workshop XV 37
So You Can!
www.hdfgroup.org
eos2 dump
HDF-EOS2 Dumper Tool
Useful for tools that can’t handle some projections.
April 17, 2012 HDF/HDF-EOS Workshop XV 38
eos2
ASCII
MATLAB IDLNCL
http://hdfeos.org/zoo
www.hdfgroup.org
Important Tool for Zoo Maintenance
April 17, 2012 HDF/HDF-EOS Workshop XV 39
www.hdfgroup.org
Why Tools Need HDF-OES2 Dumper
April 17, 2012 HDF/HDF-EOS Workshop XV 40
Lat / Lonin
ASCII
MATLAB IDLNCL
HDF-EOS2dumper
LaRC MISR SOM Projection
HDF-EOS2
Lat / Lon
www.hdfgroup.org
HDF-EOS2 Dumper Options
Usage: eos2dump options filename.hdf
What: 1=lat / 2 =lonFormat: m=5 values per line
April 17, 2012 HDF/HDF-EOS Workshop XV 41
www.hdfgroup.org
eos2dump Latitude Example
eos2dump –c1 filename.hdf > latitude.txt
What: latFormat: one value per line (default)
April 17, 2012 HDF/HDF-EOS Workshop XV 42
9080706050403020100-10-20-30
www.hdfgroup.org
eos2dump Longitude Example
eos2dump –c2m filename.hdf > longitude.txt
What: lon
Format: 5 values per lineMATLAB cannot read it.
April 17, 2012 HDF/HDF-EOS Workshop XV 43
180 175 170 165 160155 150 145 140 135130 125 120 115 110
www.hdfgroup.org
Got Tools?
April 17, 2012 HDF/HDF-EOS Workshop XV 44
www.hdfgroup.org
h4cf
eos2 dump
aug_eos5
eos52nc4
h4toh5
hdf4
Enjoy the NASA HDF Data!
April 17, 2012 HDF/HDF-EOS Workshop XV 45
eos2
eos5ASCII
NETCDF-4
NETCDF-3Panoply / IDV
MATLAB IDLNCL
NASA HDFdata users
www.hdfgroup.org
… and observe the Earth!
April 17, 2012 HDF/HDF-EOS Workshop XV 46
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XV 47
Thank you !
April 17, 2012
www.hdfgroup.org
Acknowledgements
HDF/HDF-EOS Workshop XV 48April 17, 2012
This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, orrecommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.
www.hdfgroup.org
The HDF Group
Questions/comments?
HDF/HDF-EOS Workshop XV 49April 17, 2012
