HDF5 Tools Updates

HDF5 Tools Update

Peter Cao - The HDF [email protected]

November 6, 2007

This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.

11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 2

Outline

Overview of current tools

New tools in HDF5

HDF-Java 2.4 release


Readers • h5dump, h5diff, h5ls• new tools: h5check, h5stat

HDF5 Command Line Tools

Writers•h5repack, h5repart, h5import, h5jam/h5unjam•new tool: h5copy, h5mkgrp

Converters•h4toh5, h5toh4, gif2h5, h52gif


Where to start

Tools located at bin/ directory

1. Use “-h” option for help

2. Online help http://www.hdfgroup.org/hdf5tools.html

3. Email to THG help desk [email protected]


h5dump

Dumps file content to

stdout in

• ASCII text• XML • Binary


h5dump -H SDS.h5

HDF5 "SDS.h5" { GROUP "/" { GROUP "Floats" { DATASET "FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } } } DATASET "IntArray" { DATATYPE H5T_STD_I32LE DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) } } }}


h5dump -d /Floats/FloatArray SDS.h5

HDF5 "SDS.h5" {DATASET "/Floats/FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 )

} DATA { (0,0): 0.01, 0.02, 0.03, (1,0): 0.1, 0.2, 0.3, (2,0): 1, 2, 3, (3,0): 10, 20, 30 }}}


h5dump -x SDS.h5


h5dump Binary Output

-b F, --binary=[MEMORY|FILE|LE|BE]

0000000 00000000 00000001 00000002 00000003

00000004 000000050000030 0000000a 0000000b 0000000c 0000000d

0000000e 0000000f0000060 00000014 00000015 00000016 00000017

00000018 000000190000110 0000001e 0000001f 00000020 00000021

00000022 000000230000140 00000028 00000029 0000002a 0000002b

0000002c 0000002d

h5dump -d /IntArray -o out_le.bin -b LE SDS.h5


h5diff

BetweenShow Differences

two objects or two files


h5diff SDS.h5 SDS2.h5

Dataset: </IntArray> and </IntArray>5 differences found


Report differences

Dataset: </IntArray> and </IntArray>position IntArray IntArray difference ------------------------------------------------------

[ 0 0 ] 0 10 10

[ 1 0 ] 10 100 90

[ 2 0 ] 20 200 180

[ 3 0 ] 30 300 270

[ 4 0 ] 40 400 360

5 differences found

h5diff SDS.h5 SDS2.h5 -r /IntArray


h5repack

Copies a file to a new file with • Compression filter• Data layout• Other user options


h5repack: new filters

-f {GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE}

For exampleh5repack -i SDS.h5 -o out.h5 -f GZIP=6


h5repack: data layout

-l {CHUNK| COMPA| CONTI}

For example

h5repack -i SDS.h5 -o out.h5 -l CHUNK=2x3


h5repack: latest version of file format

h5repack options File size in bytes

Original file 3,167,264

--latest 2,267,346

--latest –compact=9 2,162,254

--latest –compact=9 –ssize=50:dtype 299,882

The testing file was created with HDF5 1.6.5. The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of 32 bit integer


h5repart

Repartitions a file into a family of files

For example

h5repart -m 200m int16kx16k.h5 part200m%d.h5

977 MB

200 MB part200m0.h5

200 MB part200m1.h5

200 MB part200m2.h5

200 MB part200m3.h5

177 MB part200m4.h5


h5import

PATH work/First-set INPUT-CLASS TEXTFP RANK 3 DIMENSION-SIZES 5 2 4 OUTPUT-CLASS FP OUTPUT-SIZE 64 OUTPUT-ARCHITECTURE IEEE OUTPUT-BYTE-ORDER LE CHUNKED-DIMENSION-SIZES 2 2 2 MAXIMUM-DIMENSIONS 8 8 -1

GROUP "/" { GROUP "work" { DATASET "First-set" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) } DATA { (0,0,0): 1.01, 1.02, 1.03, 1.04, (0,1,0): 1.11, 1.12, 1.13, 1.14, (1,0,0): 1.21, 1.22, 1.23, 1.24, (1,1,0): 1.31, 1.32, 1.33, 1.34, (2,0,0): 1.41, 1.42, 1.43, 1.44, … } } }}}

Imports binary/ASCII data into an HDF5 file• h5import infile -c config_file [infile -c

config_file2 ...] -outfile outfile

For exampleh5import float5x4x2.txt -c First_set.conf -o First_set.h5


h5jam/h5unjam

• h5jamh5jam -u test_ub.txt -i test_ub.h5

• h5unjamh5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5

HDF5User Block

HDF5User Block

unjamJam


h5ls

/Floats Group/Floats/DoubleArray Dataset {10, 5}/Floats/FloatArray Dataset {4, 3}/Floats/subs Group/IntArray Dataset {5, 6}

• Lists selected information about file objects in the specified format

For Example, h5ls -r SDS2.h5


gif2h5 / h52gif

gif2h5 – converts a GIF file into HDF5gif2h5 apollo17_earth.gif apollo17_earth.h5

h52gif – converts an HDF5 file into GIFh52gif apollo17_earth.h5 apollo17_earth2.gif -i /apollo17_earth.gif/Image0 -p

"/apollo17_earth.gif/Global Palette"


h5toh4 / h4toh5

h5toh4 -- Converts an HDF5 file to an HDF4 file h4toh5 -- Converts an HDF4 file to an HDF5 file

http://www.hdfgroup.org/h4toh5/


New tools

h5stat

h5mkgrp

h5copy

h5check


h5copy

Copies an object within a file/cross files

/

FloatArray

FloatsIntArray

/

FloatArray


h5copy -f shallow

/

i1

floatsintegers

64-biti2 f32

f2f1

/

floats

64-bitf32

f2f1

/

floats

64-bitf32-f s

hallow


h5copy -f soft

/

-f soft

dset_SL

/f1/f1

f1

/

dset_SL

/f1/f1

f1

/

dset_SL

/f1/f1


h5copy -f ref

/

-f ref

d1

dset_ref

d2

1895

763

/

d1

dset_ref

d2

679

1287

/

dset_ref

0

0


h5mkgrp

Creates groups

/

info data

test1 test2

h5mkgrp test.h5 /info /data /data/test1 /data/test2


h5stat

Prints statistics about HDF5 file• Header size• Raw data size• Attributes, datatypes, and etc.

Helps • To troubleshoot overhead in HDF5 files

• To choose specific object’s properties and storage strategies


h5check

A validation tool that verifies if an HDF5 file is encoded according to the HDF5 File Format Specification


Why h5check?

• Verify if the file is compliant with the File Format

• As a verification tool required by the application of HDF5 File Format to be an ANSI standard

• Serves as a watch dog that the HDF5 library implementation is compliant with the File Format


Example: a compliant file

• % h5check example1.h5• VALIDATING example1.h5• FOUND super block signature• VALIDATING the super block at 0...• VALIDATING the object header at 928...• VALIDATING the btree at 384...• FOUND btree signature.• VALIDATING the local heap at 96...• FOUND local heap signature.• …• Result: File is in compliance.


Example: a non-compliant file

h5check invalid2.h5

• FOUND super block signature• VALIDATING the super block at 0...• VALIDATING the object header at 928...• VALIDATING the btree at 384...• FOUND btree signature.• VALIDATING the SNOD at 1248...• FOUND SNOD signature.• VALIDATING the object header at 976...• check_sym(at 1248): Errors from check_obj_header()• decode_validate_messages(): Failure in type->decode().• H5O_sdspace_decode(): Bad version number in simple dataspace

message.• VALIDATING the local heap at 96...• FOUND local heap signature.• Main(): Errors from check_obj_header().• decode_validate_messages(): Failure in type->decode().• H5O_attr_decode(): Can't decode attribute dataspace.• H5O_sdspace_decode(): Bad version number in simple dataspace

message.• …• Result: File is not in compliance.


Implementation Status

Basic file format components

1.6 - DONE1.8 - IN PROGRESS


HDF-Java 2.4

http://www.hdfgroup.org/hdf-java-html

ftp://ftp.hdfgroup.org/HDF5/hdf-java

HDF-Java 2.4 Released !!


HDF-Java 2.4 Release

• Build with HDF4 2r2 and HDF5 1.6.6• New platforms

Mac intelLinux 64-bit AMD Solaris 64-bit

• 36 new enhancements and 44 bugs fixed• Fixed memory leak• Test suite• Enhanced documentation


Memory leak in old version

while (true) { H5File f = new H5File

(fname, H5File.READ);

f.open(); f.close();}

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Technology

HDF5 Tools Updates