HDF-EOS APIs, tools, etc

1

HDF-EOS APIs, tools, etc.

Larry Klein, Abe Taaheri, and Cid Praderas

L-3 Communications

Government Services, Inc.

November 30, 2005

2

What is HDF-EOS ?• HDF (Hierarchical Data Format) is the format

chosen by NASA as the standard for the EOS program

• HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures– By providing a single interface to data structures

common to earth science– By providing a container for EOS inventory,

archive and product specific metadata– By establishing a consistent relationship between

geolocation and science data.

3

HDF-EOS Grid Structure

• Usage - Data which is organizedby regular geographic spacing, specified by projection parameters.

• Structure– Any number of 2-D to 8-D data arrays per structure, one

per data type (e.g. temperature) – Geolocation information contained in projection formula,

coupled by structural metadata.– Any number of Grid structures per file allowed.

4

HDF-EOS Swath Structure

• Data which is organized by time, or other track parameter.

• Spacing can be irregular.• Structure

– Geolocation information stored explicitly in Geolocation Field (2-D array)

– Data stored in 2-D or 3-D arrays– Time stored in 1-D or 2-D array, TAI is standard. – Geolocation/science data connected by structural

metadata

5

Swath Data Example

6

HDF-EOS Point Structure

• Data is specified temporally and/or spatially, but with no particular organization

• Structure– Tables used to store science data at a particular

Lat/Long/Height

– Up to eight levels of data allowed. Structural

metadata specifies relationship between levels.

7

Zonal Average Structure

• Generalized array structure with no geolocation linkage

• Data can be organized by time or track parameter

• Data spacing can be irregular

• Structure– Data stored in multidimensional arrays– Time stored in 1-D or 2-D array

8

HDF-EOS 2• HDF4 - based, storage format for EOS standard

products. • Used operationally by MODIS, MISR, ASTER,

Landsat, AIRS and other EOS instruments• Support for Grid/Point/Swath structures

• HDF4 and HDF-EOS 2 will be supported by NCSA/NASA for the fore-seeable future.

9

HDF-EOS 5

• Based on HDF5, a complete rewrite of HDF4 with a different user interface. – First released in 2000. – Format used by EOS Aura instruments

• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Supports same data structures– Added prefix ‘HE5_’ to HDF-EOS 2 functions. – Doesn’t preclude HDF5 functionality.

10

Top Level of HDF-EOS 5 File

STRUC. METADATASTRUC. METADATA

HDFEOSHDFEOS

SWATHSWATH GRIDGRID POINTPOINT

Root -- “/”Root -- “/”

ADDITIONALADDITIONAL

Global (File) AttributesGlobal (File) Attributes

HDFEOSINFORMATION HDFEOSINFORMATION

ARRAYARRAY

ECS METADATAECS METADATA

11

Basic HDF-EOS Functions

• Access - Open, create, close objects/files

• Definition - Set key parameters of data sets

• Input/Output - Read/Write data

• Inquiry - Return information about data

• Subset - Read data from specified region

12

File/Object Access

GRID:

• HE5_GDopen/he5_gdopen

• HE5_GDcreate/he5_gdcreate

• HE5_GDattach/he5_gdattach

• HE5_GDdetach/he5_gddetach

• HE5_GDclose/he5_gdclose

13

File/Object Access

POINT:

• HE5_PTopen/he5_ptopen

• HE5_PTcreate/he5_ptcreate

• HE5_PTattach/he5_ptattach

• HE5_PTdetach/he5_ptdetach

• HE5_PTclose/he5_ptclose

14

File/Object Access

SWATH:

• HE5_SWopen/he5_swopen

• HE5_SWcreate/he5_swcreate

• HE5_SWattach/he5_swattach

• HE5_SWdetach/he5_swdetach

• HE5_SWclose/he5_swclose

15

File/Object Access

ZA:

• HE5_ZAopen/he5_zaopen

• HE5_ZAcreate/he5_zacreate

• HE5_ZAattach/he5_zaattach

• HE5_ZAdetach/he5_zadetach

• HE5_ZAclose/he5_zaclose

16

Definition

GRID:

• HE5_GDdefdim/he5_gddefdim

• HE5_GDdefproj/he5_gddefproj

• HE5_GDdeforigin/he5_gddeforigin

• HE5_GDdefpixreg/he5_gd_defpreg

17

Definition

POINT:

• HE5_PTdeflevel/he5_ptdeflevel

• HE5_PTdeflinkage/he5_ptdeflinkage

18

Definition

SWATH:

• HE5_SWdefdim/he5_swdefdim

• HE5_SWdefdimmap/he5_swdefmap

• HE5_SWdefidxmap/he5_swdefimap

• HE5_SWdefgeofield/he5_swdefgfld

• HE5_SWdefdatafield/he5_swdefdfld

19

Definition

ZA:

• HE5_ZAdefdim/he5_zadefdim

• HE5_ZAdefine/he5_zadefine

• HE5_ZAdefcomp/he5_zadefcomp

• HE5_ZAdefchunk/he5_zadefchunk

20

Input/Output

GRID:

• HE5_GDwritefield/he5_gdwrfld

• HE5_GDreadfield/he5_gdrdfld

• HE5_GDwriteattr/he5_gdwrattr

• HE5_GDreadattr/he5_gdrdattr

21

Input/Output

POINT:

• HE5_PTwritelevel/he5_ptwritelevel

• HE5_PTreadlevel/he5_ptreadlevel

• HE5_PTwriteattr/he5_ptwrattr

• HE5_PTreadattr/he5_ptrdattr

22

Input/Output

SWATH:

• HE5_SWwritefield/he5_swwrfld

• HE5_SWreadfield/he5_swrdfld

• HE5_SWwriteattr/he5_swwrattr

• HE5_SWreadattr/he5_swrdattr

23

Input/Output

ZA:

• HE5_ZAwrite/he5_zawrite

• HE5_ZAread/he5_zaread

• HE5_ZAwriteattr/he5_zawrattr

• HE5_ZAreadattr/he5_zardattr

24

Inquiry

GRID:

• HE5_GDinqdims/he5_gdinqdims

• HE5_GDdiminfo/he5_gddiminfo

• HE5_GDprojinfo/he5_gdprojinfo

• HE5_GDorigininfo/he5_gdorigininfo

• HE5_GDpixreginfo/he5_gdpreginfo

25

Inquiry

POINT:

• HE5_PTlevelinfo/he5_ptlevelinfo

• HE5_PTinqattrs/he5_ptinqattrs

• HE5_PTattrinfo/he5_ptattrinfo

26

Inquiry

SWATH:

• HE5_SWinqdatafields/he5_swinwdflds

• HE5_SWinqgeofields/he5_swinqgflds

• HE5_SWinqattrs/he5_swinqattrs

• HE5_SWattrinfo/he5_swattrinfo

27

Inquiry

ZA:

• HE5_ZAinquire/he5_zainquire

• HE5_ZAinfo/he5_zainfo

• HE5_ZAinqattrs/he5_zainqattrs

• HE5_ZAattrinfo/he5_zaattrinfo

28

Programming Model

• Open the HDF-EOS5 file• Create or Attach to a specified

Grid/Swath/Point/ZA object• Define key object features and/or

perform necessary operations (with the data sets, attributes, etc)

• Detach from the Grid/Swath/Point/ZA object

• Close the HDF-EOS5 file

29

HDF-EOS Functionality

• Basic File I/O• Swath Interface• Grid Interface• Point Interface• Profile Interface• Generalized Array

Interface (Zonal Avg. (HDF5))

• Subsetting

• Fill Values• Compression (gzip,

szip, shuffling)• Chunking/Tiling• Aliases• Global (File), Group

& Local Attributes• Threadsafe Version

30

Applications

• HE5View (HDF5 - based browser)• EOSView (HDF4 - based browser)• HDFView (with HDF-EOS Plug-in), Java EOS

Browser (HDF-EOS 2 and 5 access)• heconvert (converts HDF4 - based Grid/Point/Swath

structures to HDF-EOS 5 equivalents) – Compliments HDF4 -> HDF5 conversion tool

– Not tested on all products

• HEG (HDF-EOS to GeoTIFF Converter/Subsetter)

31

HDFView• HDFView: A Java-based browser from NCSA for

HDF4 and HDF5 – based data.• HDF-EOS plug-in extends HDFView to browse HDF-

EOS data in both versions.– One reason: EOSView is becoming obsolete

• The extension is a plug-in module and does not change NCSA HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains.

• Identical look and feel when displaying HDF and HDF-EOS objects.

• Users can access a complete breakdown of HDF-EOS objects. e.g. grid information, projection information, dimensions, datafields, and attributes are displayed.

32

HDFView (Cont.)• Release 2.2 October 2005

- A separate HDF-EOS plug-in: Users install HDFView 2.2 and integrates HDF-EOS plug-in into it.- Ability to do "Save as" for HDF-EOS5: modify individual values in grid or swath data. Delete entire grids or swaths. Save the result. - Separate help window for HDF-EOS objects

• Release 2.3 January 2006 - Point Object viewing capability

- Geolocating pixels for swath and grid imagesFuture Plans:

- Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes.- Ability to create new grids/swaths

33

HDFView showing ASTER L1B swath

34

• A utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats.

• Two versions available:– A downloadable desktop version – Access through NASA archive online storage (Data

Pools)– Implemented also in prototypes for Stand-alone Data

Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs

• Details of internal file structures are transparent to users.

HDF-EOS to GeoTIFF Converter (HEG)

35

HEG Functionality• Subsetting (spatial, field, band)• Performs conversion to a single-band or multi-

band GeoTIFF’s• HDF-EOS Swath to HDF-EOS Grid and Swath• HDF-EOS Swath or Grid to GeoTIFF• HDF-EOS Swath or Grid to generic Binary (with

metadata file)• Stitching (mosaicing) + subsetting + Reprojecting• Running from Command Line (batch jobs) or with

Java-based GUI• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ,

GEO, SIN)• Subsampling (Subsample stacks)

36

HEG Functionality• Metadata preservation/creation.• Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E

products on TERRA and AQUA (>110 products), AURA/HDF-EOS5 support in progress

• Allows file selection from users local storage• Operable on Sun, SGI, Win, Linux (64-bit soon), MAC• Integrated into ECS Data Pool

– Reduces the transfer time of HDF-EOS data sets (if subset is requested)

– Provides the end-user with the exact file required by their application• User Interface

– Portable, written in Java.– Not dependent on COTS (eg. IDL).

37

Availability

• Access to libraries and applications: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html

• Email– [email protected] – [email protected]– [email protected]– [email protected]

38

Mt. Etna Eruption: MISR L1B Swaths: Stitched

39

Mt. Etna: Stitched, Subsetted ASTER L1B

40

MODIS L1B: Reprojected, Co-Registered with ASTER, La Plata Tornado

41

La Plata Tornado Damage: ASTER L1B reprojected/stitched/subsetted

42

MLS L2GP-H2O Product (2004, day 243), Geographic Projection, Full Globe

43

MLS L2GP-H2O Product, PS Projection, Northern Hemisphere

44

2004 Hurricane Francis, MODIS

(MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5)

45

AST_L1B, 2003 California FiresGeo. projection, Multi-Band GeoTIFF,

(3 bands: Visibile Near Infrared (VNIR))

Two Swaths Stitched …..

….. and Subsetted

Documents

HDF-EOS APIs, tools, etc