Upload
cora-copeland
View
215
Download
1
Embed Size (px)
Citation preview
9/19/2000 McGrath 9/19/2000 1
University of Illinois at Urbana-Champaign
HDFHDF
Transition from HDF4 to Transition from HDF4 to HDF5: IssuesHDF5: Issues
Robert E. McGrathRobert E. McGrath
NCSANCSA
University of Illinois at Urbana-ChampaignUniversity of Illinois at Urbana-Champaign
9/19/2000 McGrath 9/19/2000 2
University of Illinois at Urbana-Champaign
HDFHDF
Overall GoalOverall Goal
A discussion of compatibility, conversion, A discussion of compatibility, conversion, and interoperability issues for HDF4 and and interoperability issues for HDF4 and
HDF5, and HDFEOS.HDF5, and HDFEOS.
For more details see the white paper at:For more details see the white paper at:
http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdfhttp://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf
9/19/2000 McGrath 9/19/2000 3
University of Illinois at Urbana-Champaign
HDFHDF
Conclusions (Preview)Conclusions (Preview)
Transition from HDF4 to HDF5 is not Transition from HDF4 to HDF5 is not necessarily transparent or effortless.necessarily transparent or effortless.
9/19/2000 McGrath 9/19/2000 4
University of Illinois at Urbana-Champaign
HDFHDF
Conclusions: HDFConclusions: HDF
• Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be written or rewritten.
• Many tools will support both HDF4 and HDF5.
• Default translation from HDF4 to HDF5 will be supported as a utility and library.
9/19/2000 McGrath 9/19/2000 5
University of Illinois at Urbana-Champaign
HDFHDF
Conclusions: HDF-EOSConclusions: HDF-EOS
• Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be written or rewritten
• Translation of ‘pure’ HDF-EOS will be supported
• Translation of ‘hybrid’ HDF-EOS files is very difficult.
• Not possible to support V.2 and V.3 in a single program
9/19/2000 McGrath 9/19/2000 6
University of Illinois at Urbana-Champaign
HDFHDF
What Does Compatibility Mean?What Does Compatibility Mean?
9/19/2000 McGrath 9/19/2000 7
University of Illinois at Urbana-Champaign
HDFHDF
What does “compatibility” mean?What does “compatibility” mean?
• Ability to “do the same things” with new software– I.e., “compatibility” is relative to some purpose– Many dimensions and shades of compatibility– No golden solution sufficient for every use
• Assume HDF4 users want to continue doing what they are already doing, but move to HDF5
9/19/2000 McGrath 9/19/2000 8
University of Illinois at Urbana-Champaign
HDFHDF
Some Types of CompatibilitySome Types of Compatibility
CompatibilityIssue
Description
Conceptual The overall conceptual approach:the formal and informal models.
Format Storage layout and other formatdetails.
Software Compatibility of the software Programming model
The assumptions and approach to programming.
Features The set of features supported. Interfaces The programming interfaces.
9/19/2000 McGrath 9/19/2000 9
University of Illinois at Urbana-Champaign
HDFHDF
Compatibility of HDF and HDF-EOSCompatibility of HDF and HDF-EOS
Compatibility Issue HDF4/HDF5 HDFEOS2/HDFEOS3Conceptual Yes, H5 is a
supersetYes
Format No NoSoftware Programming model No? Yes Features Some Yes, V3 is a superset Interfaces No No
9/19/2000 McGrath 9/19/2000 10
University of Illinois at Urbana-Champaign
HDFHDF
InteroperationInteroperation
• My definition of “interoperation”: one program or tool may use HDF4 or HDF5 or both
9/19/2000 McGrath 9/19/2000 11
University of Illinois at Urbana-Champaign
HDFHDF
NCSA HDF4 and HDF5NCSA HDF4 and HDF5
9/19/2000 McGrath 9/19/2000 12
University of Illinois at Urbana-Champaign
HDFHDF
Single Combined HDF Library
dataset1 dataset2
Stored asHDF4 Objects
image1image2
Stored asHDF5 Objects
HDF4 file HDF5 file
HDF 4 Library HDF 5 Library
Interoperability: Any program may use either HDF4 or 5 Interoperability: Any program may use either HDF4 or 5 or both:or both:
Not Supported
9/19/2000 McGrath 9/19/2000 13
University of Illinois at Urbana-Champaign
HDFHDF
Application Specific Library
dataset1 dataset2
Stored asHDF4 Objects
image1image2
Stored asHDF5 Objects
HDF4 file HDF5 file
HDF 4 Library HDF 5 Library
Interoperability: Specific Application or tool may use Interoperability: Specific Application or tool may use either HDF4 or 5 or both:either HDF4 or 5 or both:
Up to Application and Tool Developers
Application specific Library
9/19/2000 McGrath 9/19/2000 14
University of Illinois at Urbana-Champaign
HDFHDF
HDF-EOS Version 2 and Version 3HDF-EOS Version 2 and Version 3
9/19/2000 McGrath 9/19/2000 15
University of Illinois at Urbana-Champaign
HDFHDF
Grid1 Grid2
Stored asHDF4 Objects
Grid3 Grid4
Stored asHDF5 Objects
HDF4 file HDF5 file
HDF 4 Library HDF 5 Library
HDFEOS Library
Interoperability: Interoperability: Any program may using HDF-EOS Objects with either Any program may using HDF-EOS Objects with either
HDF4 or HDF5 or both:HDF4 or HDF5 or both:Not supported
9/19/2000 McGrath 9/19/2000 16
University of Illinois at Urbana-Champaign
HDFHDF
HDFEOS V2 Library HDFEOS V3 Library
Grid1 Grid2
Stored asHDF4 Objects
Grid3 Grid4
Stored asHDF5 Objects
HDF4 file HDF5 file
HDF 4 Library HDF 5 Library
Interoperability: One application may use HDF-EOS Interoperability: One application may use HDF-EOS Objects with either HDF4 or HDF5 or both:Objects with either HDF4 or HDF5 or both:
Not supported
9/19/2000 McGrath 9/19/2000 17
University of Illinois at Urbana-Champaign
HDFHDF
TranslationTranslation
• Convert HDF4 objects (file) to HDF5 objects (file)
9/19/2000 McGrath 9/19/2000 18
University of Illinois at Urbana-Champaign
HDFHDF
Translation of HDF4 to HDF5Translation of HDF4 to HDF5
Stored asHDF4 Objects
Stored as AnalogousHDF5 Objects
/
HDF4 Library HDF5 Library
HDF4 File HDF5 File
Supported: NCSA mapping and h4toh5 tool and library
9/19/2000 McGrath 9/19/2000 19
University of Illinois at Urbana-Champaign
HDFHDF
Grid1 Grid2
Translation of HDF-EOS V2 to V3Translation of HDF-EOS V2 to V3
HDFEOSObjects
Stored asHDF4 Objects
Grid1 Grid2
SAMEHDFEOSObjects
Stored asHDF5 Objects
HDF4 file HDF5 file
HDF4 Library
/
HDF5 Library
HDF EOS V. 2.x Library HDF EOS V. 3 Library
Supported
9/19/2000 McGrath 9/19/2000 20
University of Illinois at Urbana-Champaign
HDFHDF
““Hybrid” HDF-EOS filesHybrid” HDF-EOS files
• HDF-EOS files may contain native HDF objects (images, arrays, etc.) as well as HDF-EOS objects (Grid, etc.)
• Almost all HDF-EOS files are ‘hybrids’.
• Very difficult to automatically translate hybrid files.
9/19/2000 McGrath 9/19/2000 21
University of Illinois at Urbana-Champaign
HDFHDF
““Hybrid” HDF-EOS FileHybrid” HDF-EOS File
NCSA HDF
HDFEOS
Science Application Code
Other
Components ofHDFEOS Objectsstored in HDFObjects
Other HDFObjects StoredDirectly ThroughHDF
One HDF file, objectsmixed together.
Very difficult to tell which
objects are part of HDF-EOS
9/19/2000 McGrath 9/19/2000 22
University of Illinois at Urbana-Champaign
HDFHDF
ConclusionsConclusions
Transition from HDF4 to HDF5 is not Transition from HDF4 to HDF5 is not necessarily transparent or effortless.necessarily transparent or effortless.
9/19/2000 McGrath 9/19/2000 23
University of Illinois at Urbana-Champaign
HDFHDF
Conclusions: HDFConclusions: HDF
• Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be rewritten.
• Many tools will support both HDF4 and HDF5.
• Default translation from HDF4 to HDF5 will be supported as a utility and library.
9/19/2000 McGrath 9/19/2000 24
University of Illinois at Urbana-Champaign
HDFHDF
Conclusions: HDF-EOSConclusions: HDF-EOS
• Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be rewritten
• Translation of ‘pure’ HDF-EOS will be supported
• Translation of ‘hybrid’ HDF-EOS files is very difficult.
• Not possible to support V.2 and V.3 in a single program
9/19/2000 McGrath 9/19/2000 25
University of Illinois at Urbana-Champaign
HDFHDF
For More InformationFor More Information
• NCSA White paper: “Transition From HDF4 to HDF5”: http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf
• “Guidelines for Mapping HDF4 to HDF5 Objects”, http://hdf.ncsa.uiuc.edu/HDF5/papers/H4-to-H5MappingGuidelines.pdf
• h4toh5 tool: Soon