View
146
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Source: http://hdfeos.org/workshops/ws04/presentations/Folk/Folk.ppt
Citation preview
- 1 -
University of Illinois at Urbana-Champaign
HDFHDF
Mike Folk
HDF-EOS Workshop IV
Sept. 19-21, 2000
HDF UpdateHDF Update
HDFHDF
- 2 -
University of Illinois at Urbana-Champaign
HDFHDF
TopicsTopics
• What is HDF?
• HDF community
• HDF4– past and future work
• HDF5– HDF5-HDF5 interoperability– Activities and growth– Tools and utilities– Focus for 2001
- 3 -
University of Illinois at Urbana-Champaign
HDFHDF
NCSA HDF MissionNCSA HDF Mission
To develop, promote, deploy, and support To develop, promote, deploy, and support open and free technologies that facilitate open and free technologies that facilitate scientific data storage, exchange, access, scientific data storage, exchange, access,
analysis and discovery. analysis and discovery.
- 4 -
University of Illinois at Urbana-Champaign
HDFHDF
What is HDF?What is HDF?
• Format and software for scientific data
• Stores images, arrays, tables, etc.
• Emphasis on storage and I/O efficiency
• Free and commercial software support
• Emphasis on standards
• Users from many engineering & scientific fields
- 5 -
University of Illinois at Urbana-Champaign
HDFHDF
HDF PeopleHDF People
• NCSA– QA, maintanance & support– Library development & performance– Tools development
• EOS & ASCI– Users, applications and tools– Performance
• Volunteers from all over– Users, applications & tools
- 6 -
University of Illinois at Urbana-Champaign
HDFHDF
Who is supporting HDF?Who is supporting HDF?
• NASA/ESDIS– Earth science applications, instrument data– All aspects of data management
• DOE/ASCI (Accelerated Strategic Computing Init.)– Simulations on massively parallel machines– Emphasis on parallel I/O performance, functionality
• NCSA– Grid, Vis, other R&D, user support
• Others– Applications, support, some R&D
- 7 -
University of Illinois at Urbana-Champaign
HDFHDF
HDF4HDF4
• User support, tools, maintenance, QA
• JHV (Java viewer) - two update releases
• HDF4 dumper (hdp) - many improvements
• Next HDF4 Release (Oct. 2000)– Bug fixes– Chunking for General Raster (GR) images– Unlimited dimensions for SDS– GIF <--> HDF4 conversion tools
- 8 -
University of Illinois at Urbana-Champaign
HDFHDF
HDF4 ArchivingHDF4 Archiving
• New File Format and Specification Manual– Facilitate long term maintenance of HDF4– Necessary for bug fixing, adding new features– Decreases learning curve for new developers– Available soon in
• PDF, HTML and PS formats
- 9 -
University of Illinois at Urbana-Champaign
HDFHDF
HDF5HDF5
- 10 -
University of Illinois at Urbana-Champaign
HDFHDF
The growth of HDF5The growth of HDF5
• Users in 27 countries
• Users include– Government agencies– National labs– Companies– Universities
- 11 -
University of Illinois at Urbana-Champaign
HDFHDF
The growth of HDF5The growth of HDF5
• Scientific fields– Astronomy, astrophysics, aerospace engineering
– Geophysics, remote sensing, meteorology,
– Oceanography, environmental Science
– Informational Science
– Medical Research (brain, cancer, biotech)
– Product model data
- 12 -
University of Illinois at Urbana-Champaign
HDFHDF
Facilitating interoperability Facilitating interoperability between HDF4 and HDF5 between HDF4 and HDF5
• HDF4 to HDF5 mapping specification– “Mapping HDF4 Objects to HDF5 Objects”
• http://hdf.ncsa.uiuc.edu/HDF5/papers
– Rules for mapping high level HDF4 objects to HDF5• How to describe HDF4 objects in HDF5• How to interpret HDF5 objects as HDF4 objects
• HDF4-to-HDF5 conversion software (later)• Space Research, Inc. Explorer 1.1
– reads both HDF4 and HDF5
- 13 -
University of Illinois at Urbana-Champaign
HDFHDF
HDF5 Activities in 2000HDF5 Activities in 2000
• HDF5 1.2.2 library release
• Fortran 90 & C++ API
• HDF5 Abstract Data Model
• XML Document Type Definition (DTD)
• HDF5 tools
• Support for users and application developers
• HDF5 Tutorial
• Searchable, printable documentation
- 15 -
University of Illinois at Urbana-Champaign
HDFHDF
Tools & UtilitiesTools & Utilities• NCSA
– Java wrapper for HDF5– H5View: Java browser/Editor for HDF5– H5gen: XML-to-HDF5 file generator (Java)– H5dump & H5ls
• Others– VisAD data adapter for HDF5 (Java toolkit)– HDF Inspector/Explorer– Open Data Explorer (IBM)– Ensight
- 16 -
University of Illinois at Urbana-Champaign
HDFHDF
H5ViewH5View
- 17 -
University of Illinois at Urbana-Champaign
HDFHDF
H5ViewH5View
• Java-based tool for browsing and editingJava-based tool for browsing and editing
• Display structure of fileDisplay structure of file
• Display content of objectsDisplay content of objects
• Create and delete objectsCreate and delete objects
• Modify values of objects and attributesModify values of objects and attributes
- 18 -
University of Illinois at Urbana-Champaign
HDFHDF
HDF5 XML DTDHDF5 XML DTD
• A flexible standard language that can describe an HDF5 file in precise detail– Datasets, dataspaces, datatypes – Groups and links, structure of the file– Values of the attributes and data
• http://hdf.ncsa.uiuc.edu/HDF5/XML
- 19 -
University of Illinois at Urbana-Champaign
HDFHDF
H5Gen: XMLH5Gen: XMLHDF5HDF5
• Reads an XML description of an HDF5 file
• Generates the corresponding HDF5 file
• Validates XML description vs. HDF5 DTD
- 20 -
University of Illinois at Urbana-Champaign
HDFHDF
Some uses of XMLSome uses of XMLView View
HDF5 files HDF5 files using a web using a web
browserbrowserXML XML
to to HTMLHTML
Java Java viewers,editors,viewers,editors,
other toolsother tools
XML to JavaXML to JavaJava to XMLJava to XML
Description Description in XML using in XML using HDF5 DTDHDF5 DTD
HDF5 HDF5 filefile
HDF5 HDF5 filefile
XML to HDF5 (H5gen)XML to HDF5 (H5gen)HDF5 to XMLHDF5 to XML
Generate, validate,Generate, validate,reconstruct HDF5 filesreconstruct HDF5 files
Data Data location location servicesservices
Catalog Catalog records records in XMLin XML
ArchiveArchiveArchiveArchiveXML XML
documentationdocumentation of metadata of metadata & structure& structure
- 21 -
University of Illinois at Urbana-Champaign
HDFHDF
Focus areas for 2001Focus areas for 2001
• Support Terra & Aqua• Get ready for Aura• Support HDF-EOS 3 • Enhance HDF5View• Refine XML DTD & design tools around it• HDF5 converters
– HDF5XML– HDF4-to-HDF5– Others (e.g. GIFHDF5)
- 22 -
University of Illinois at Urbana-Champaign
HDFHDF
Focus areas for 2001Focus areas for 2001
• Expand list of applications and users• Facilitate access to other tools & software• Get vendors on board
• Performance testing and tuning• Extend API with ease-of-use functions• Clusters and other new environments• Implement a thread-safe version• HDF5 advanced tutorial
- 23 -
University of Illinois at Urbana-Champaign
HDFHDF
Thank you!Thank you!
• HDF website– http://hdf.ncsa.uiuc.edu/
• HDF5 Information Center– http://hdf.ncsa.uiuc.edu/HDF5/
• HDF Helpdesk– [email protected]
• HDF users mailing list– [email protected]
HDFHDF
55