HDF-EOS Tools

Page 1

HDF-EOS ToolsHDF-EOS Tools

Abe Taaheri, Raytheon IISESIP Meeting

Chapel Hill, NCJuly 9, 2013

Page 2

Outline

Page 3

Overview

• The HDF has been selected by the EOSDIS Project as the format of choice for standard product distribution of EOS data

• Under NASA contracts (ECS, EMD, EEB, EED) we developed HDF-EOS, and performed other development and maintenance activities

Page 4

Overview

– Libraries HDF-EOS 2 & 5 SDP & MTD Toolkits

– Tools HDF-EOS2 to GeoTIFF Converter (HEG) HDF-EOS plug-in for THG HDFView HDF-EOS2 to HDF-EOS5 conversion

– User support Helpdesk for Toolkit/HDF-EOS/HEG

Page 5

Tools

Why we need some HDF-EOS tools ?

Page 6

• HDF was selected for EOS data since

– Many of the HDF defined datatypes map well

to EOS datatypes Raster images, multi-dimensional arrays,

and text blocks

– HDF provides efficient data management Storage & performance

Why HDF?

Page 7

• Some other EOS datatypes do not map directly to HDF datatypes, particularly

@ geolocated datatypes

Therefore, some additions to traditional HDF are required to fully support these datatypes.

HDF-EOS Datatypes

Page 8

HDF-EOS Datatypes

To bridge the gap between the needs of EOS data products and the capabilities of HDF, 4 new EOS specific datatypes were introduced:

Point, Swath, and Grid Zonal Average (hdf-eos5)

Page 9

HDF-EOS Datatypes

• Point : Unorginzed data (spatial or temporal) that has associated geolocation information

• Swath: Time-ordered data such as satellite swaths (time-ordered series of scanlines),

or

profilers (time-ordered series of profiles).

• Grid: Data stored in a rectilinear array based on a well defined and explicitly supported

projection.

• ZA: Swath like datatype without geolocation mapping

Page 10

HDF4 File Contents – User View

Objects & Relationships

User Metadata

Object Data

Page 11

HDF4 File Contents – Format View

Vgroupname = variable_name

class = Var0.0

NDG

SDDSDNT

variable

name = variable_nameranktype

storagetype

data

Vdataname = attribute_name

class = Attr0.0

1 1

0…* 0…*

1

1

attributename = attribute_name

1 1

1 1

1

1 1

1 1

1 1 0...1

0...1

0...1

1

byte order,chunked storage,

compression, …

Object Data

HDF4 Mapping Project Update - Ruth Aydt

Page 12

HDF/HDF-EOS are Complicated!

• HDF is not efficient for Users since they need to learn

- how to use HDF & HDF-EOS libraries

- how to process the data correctly

• Licenses for commercial tools are expensive

The HDF Group & Raytheon have developed some tools to help users

Page 13

HDF & HDF-EOS Tools

For most users HDF readers or NetCDF/CF conversion applications are enough

- hdp, h5dump, eos2dump(for lat/lon), HDFView

- h4tonccf, h4cf, OpeNDAP Handlers

But some need more processing and tools appropriate for EOS data

• HDF-EOS to GeoTIFF Conversion Tool (HEG)

• HDF-EOS plug-in for HDFView

Page 14

Page 15

Name says HEG is:

A utility to convert EOSDIS data fromHDF-EOS2 to GeoTIFF format.

But in practice:

It can do more than just conversion to GeoTIFF

HEGHEG

Page 16

HEG FunctionalityHEG Functionality

• Conversion

– HDF-EOS (Swath or Grid ) to a single-band or multi-band GeoTIFF’s

– HDF-EOS Swath to HDF-EOS Grid --> GeoTIFF

OR

– ASTER Swath is written directly into GeoTIFF

– HDF-EOS Swath or Grid to generic Binary (with

metadata file)

Page 17


• Subsetting

– spatial: using Swath/Grid corner lat/lon or pixel numbers

– field: Grid, Swath, and Field selection

– band: Selection of data layers in 3-D and 4-D datasets ( 3-D to 6-D MISR SOM datasets )

Page 18


• Stitching

– (mosaicing) + subsetting + Reprojecting

• Reprojection

@To: UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN, Albers, CEA

• Subsampling

@Subsample stacks

Page 19


• Running from Command Line (batch jobs)

or • with Java-based GUI

• Metadata preservation/creation.

Page 20

Conversion GUI

HEG

Page 21

HEG

Stitch GUI

Page 22

BilinearNearest NeighborCubic Convolution

GeographicPolar StereographicUTMMercatorTransverse MercatorLambert AzimuthalLambert Conformal Conic

SinusoidalAlbersState PlaneCylindrical Eqyual-AreaNo-Reprojection

Conversion GUI

HEG

Lat-LonRows-Columns

Page 23

HEG

HEG Parameter File Used for running HEG on command line.

Page 24

Status log for HEG run

Page 25

HEG Data/Platform SupportHEG Data/Platform Support

• Currently supports

MODIS, MISR, ASTER, AIRS, AMSR-E products on TERRA and AQUA

– more than 210 products

• Operable on Sun, Windows, Linux, MAC

Page 26

Versions Available:

Stand-alone:

A downloadable desktop version with a User Interface

@Portable, written in Java.@Not dependent on COTS (eg. IDL).

HEG VersionsHEG Versions

Page 27

HEG VersionsHEG Versions

Integrated into ECS Data Pool:

Access through NASA archive online storage

@Reduces the transfer time of HDF-EOS data sets (if subset is requested)

@Provides the end-user with the exact file required by their application

Page 28

Examples (Examples (MISRMISR))

Mt. Etna Eruption: Mt. Etna Eruption: MISRMISR L1B Stitched Blocks L1B Stitched Blocks

Page 29

Examples (Examples (MODISMODIS))

MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5

2004 Hurricane Francis, 2004 Hurricane Francis, MODISMODIS

Page 30

Examples (Examples (ASTERASTER))

Two Swaths Stitched …..

….. and Subsetted

AST_L1B, 2003 California FiresAST_L1B, 2003 California FiresGeo. projection, Multi-Band Geo. projection, Multi-Band GeoTIFF (3 bands):GeoTIFF (3 bands):Visible Near Infrared (VNIR)Visible Near Infrared (VNIR)

Page 31

Examples ( Examples ( AMSR_E_L2_Ocean) swath

Swath Conversion to Geographic Grid/HDF-EOS output

Page 32

Future WorkFuture Work

Page 33

Page 34

HDFView HDFView

A Java-based browser from The HDF Group for HDF4 and HDF5 based data.

Page 35

HDFView with HDF-EOS Plug-inHDFView with HDF-EOS Plug-in

Page 36

HDFView with HDF-EOS Plug-inHDFView with HDF-EOS Plug-in

–Extends HDFView to browse HDF-EOS data in both versions (2 & 5).

–Is a plug-in module does not change HDFView functionality

–Identical look and feel when displaying HDF and HDF-EOS objects

–Access to complete breakdown of HDF-EOS objects.

e.g. grid info, projection info, dims, datafields, and attributes

Page 37

HDFView Plug-inHDFView Plug-in

What else HDF-EOS plug-in

adds to HDFView?

Page 38

Swath, Grid, and Point ObjectsSwath, Grid, and Point Objects

38

Plug-in

file

treeview

Swath/Grid

Lat/Lon

display

Page 39

HE2 to HE5 ConversionHE2 to HE5 Conversion

One click to convert HE2 to HE5 file format39

Page 40

AMSR-E SwathAMSR-E Swath

40

Getting Image of selected

pixels in a table Getting lat/lon for a selected pixel

in a table

Page 41

MODIS (HE2) and OMI (HE5) Grids MODIS (HE2) and OMI (HE5) Grids

41

Drawing

Coastlines

&

International

Borders

on

HE2 & HE5

Grids

Page 42

• Coastlines & International borders for

HE2 & HE5 swath

• Adding single-band/multi-band GeoTIFF writer

• Adding GeoTIFF reader with geolocating

capability

Future work

Page 43

AvailabilityAvailability

• Download, Documentation, Help:

HDFViewHDFView::http://newsroom.gsfc.nasa.gov/sdptoolkit/http://newsroom.gsfc.nasa.gov/sdptoolkit/

HDFView/HDFView_hdfeos_plugin.htmlHDFView/HDFView_hdfeos_plugin.html

HEGHEG: : http://newsroom.gsfc.nasa.gov/sdptoolkit/http://newsroom.gsfc.nasa.gov/sdptoolkit/

HEG/HEGHome.htmlHEG/HEGHome.html

Page 44

Help, CommentsHelp, Comments

• Questions, Comments, Requests:

[email protected]

[email protected]

http://hdfeos.org/forums/index.phpforums/index.php

Page 45

Thank You!Thank You!

Questions/comments?

Technology

HDF-EOS Tools