- 1 - HDFHDF
Mike Folk, Elena Pourmal , Bob McGrath
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
NOBUGS 2004HDF-EOS Workshop VIII
HDF Software ProcessHDF Software ProcessLessons Learned & Success FactorsLessons Learned & Success Factors
HDFHDF
- 2 - HDFHDF
OutlineOutline
• What is HDF? and Who is HDF?
• HDF “Architecture”
• Some statistics
• How do we measure success?
• How can we achieve success?
• Group practices
• Summing up – strengths, weaknesses, needs
- 3 - HDFHDF
What is HDF?What is HDF?Who is HDF?Who is HDF?
- 4 - HDFHDF
HDF in a nutshell – what it isHDF in a nutshell – what it is
• File format and I/O Libraries for storing, managing and archiving large complex scientific and other data
• Tools and utilities• Open source, free for any use (U of I license)
• Well maintained and supported
• From HDF group, NCSA Univ of Illinois• http://hdf.ncsa.uiuc.edu
- 5 - HDFHDF
HDF in a nutshell - featuresHDF in a nutshell - features
• General– simple and flexible data model
• Flexible– store data of diverse origins, sizes, types– supports complex data structures and types
• Portable– available for many operating systems and machines
• Scalable– works in high end computing environments– accommodates date of any size or multiplicity
• Efficient – fast access, including parallel i/o– Stores big data efficiently
- 6 - HDFHDF
HDF in a nutshell - usersHDF in a nutshell - users
• Apps in industry, academia, government– More than 200 distinct applications
• Large user base– E.g. NASA estimates 1.6 million users
• Underlying format for community standards– E.g. HDF-EOS, SAF, CGNS, NPOESS, NeXus
- 7 - HDFHDF
Example of HDF file: mixing and grouping Example of HDF file: mixing and grouping objectsobjects
Raster imageRaster image
palettepalette
3-D array 3-D array
2-D array2-D arrayRaster imageRaster image
lat | lon | temp----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6
TableTable
a
b c
z
x_foo_y
foo
1GB
Text : This file was create as a part of… see http://hdf.ncsa.uiuc.edu
- 8 - HDFHDF
HDF “Architecture”HDF “Architecture”
- 9 - HDFHDF
HDF “Architecture”HDF “Architecture”
File or other data sourceFile or other data source
HDF I/O libraryHDF I/O library– High-level, object-specific APIs.High-level, object-specific APIs.
– Low-level API for I/O to files, etc.Low-level API for I/O to files, etc.
Utilities and applications for Utilities and applications for managing, manipulating, managing, manipulating, viewing, & analyzing data.viewing, & analyzing data.
Low level Interface
HDF5 Applications Programming Interface
Tools & ApplicationsTools & Applications
FileFile
- 10 - HDFHDF
User’s controlled I/O and “storage”User’s controlled I/O and “storage”
• Data pipeline– Data transformation– Compression– Encryption– Storage layout
• Virtual file options– Stdio (normal file)– Split file– MPI-IO & other parallel– Network– Memory– custom
HDF I/O Library
HDF “File”HDF “File”
- 11 - HDFHDF
Supported languages and compilersSupported languages and compilers
• C• Wrappers:
– C++– Fortran90– Java
• Vendors’ compilers (SUN, IBM, HP, etc.)• PGI and Absoft (Fortran)• GNU C (e.g. gcc 3.3.2)
- 12 - HDFHDF
Supported Machines and OSSupported Machines and OS
• Solaris 2.7, 2.8 (32/64-bit)
• IRIX6.5 IRIX64-6.5
• HPUX 11.00
• AIX 5.1 (32/64-bit modes)
• OSF1
• FreeBSD
• Linux (SuSe, RH8, RH9) including 64-bit
• Altix (SGI Linux)
• IA-32 and IA-64
• Windows 2000, XP
• MAC OS X
• Crays (T3E, SV1, T90IEEE)
• DOE National Labs machines
• Linux Clusters
- 13 - HDFHDF
FileFile
Serial Parallel
Linux RH IRIX32 XP SV1
IA32 SGI Wintel Cray
Architecture in contextArchitecture in context
Low level Interface
HDF5 Applications Programming Interface
Tools & ApplicationsTools & ApplicationsC C++ F90 Java
- 15 - HDFHDFFileFile
Serial Parallel
Linux RH IRIX32 XP SV1
IA32 SGI Wintel Cray
Architecture in contextArchitecture in context
Low level Interface
HDF5 Applications Programming Interface
C C++ F90 Java
HDF-EOSHDF-EOS SAFSAF CGNSCGNS
Tools & ApplicationsTools & Applications
- 16 - HDFHDF
The testing challengeThe testing challenge
Machines × operating systems× compilers × languages
× serial and parallel× compression options× configuration options
× virtual file options× backward compatibility
= a large number
- 17 - HDFHDF
““Diversity makes our code better…”Diversity makes our code better…”
Todd Smith, Geospiza
- 18 - HDFHDF
Some statisticsSome statistics
- 19 - HDFHDF
HDF StatisticsHDF Statistics
• HDF Group– 15 FTE + 3-5 students– $2.1million annual budget
• HDF5 source code distribution – 2073 files– 917,186 Lines of code
• HDF Project – HDF5, HDF4, H4toH5, H5Lite, Java– 3,000,000 lines of code (estimate)
- 20 - HDFHDF
Docs33%
Examples1%
Tools4%
Libraries30%
Tools tests 4%
LibraryTests13%
Configure15%
HDF5 source distribution by categories HDF5 source distribution by categories (lines of code)(lines of code)
- 21 - HDFHDF
HDF5 staff investmentHDF5 staff investment
Test writing7%
Comm. with users 2%
Peer-to-peer comm. 12%
Code dev.33%
Porting/release testing 9%
Docs, design, consult 14%
Meetings, etc.9%
User's support14%
- 22 - HDFHDF
How do we measure success?How do we measure success?
- 23 - HDFHDF
How do we measure success?How do we measure success?
• Mission• Goals and objectives• Strong and continuing relationships with users• High quality software• Strong committed development team • Great working environment• Adequate funding
- 24 - HDFHDF
Mission, goals and objectivesMission, goals and objectives
• Mission– To develop, promote, deploy, and support open and
free technologies that facilitate scientific data exchange, access, analysis, archiving and discovery
• Goals (examples)– Innovate and evolve the technologies in concert with a
changing world of technologies – Maintain a high level of quality and reliability– Collaborate and build communities– Build a team
- 25 - HDFHDF
Mission, goals and objectivesMission, goals and objectives
• Objectives - how we reach the goal• Example:
– Goal• Maintain a high level of quality and reliability
– Objectives• Improve testing• Implement a program to insure excellent software
engineering practices• Develop and execute a plan to meet
quality/reliability standards
- 26 - HDFHDF
UsersUsers
• Number of users
• Happy users • Unhappy users • Users achieve their goals by using HDF
technologies
• Users coming back with new needs
• Financial support from users
- 27 - HDFHDF
SoftwareSoftware
• Technology that addresses users’ needs and demands (current and future)– E.g. big files, parallel access, multiple objects
• Usability– Number and types of applications– Appropriate APIs and data models– Available tools– Interoperability with other software
• E.g. IDL, MatLab, Mathematica
- 28 - HDFHDF
SoftwareSoftware
• Stability– Can data be shared?– Can software run on needed platforms
• Sustainability– Can read data written 15 years ago on obsolete platform– Is software available in 15 years?
• Acceptability– De facto standard
• Open standard for exchange of remote-sensed data• Over 3,000,000,000,000,000 bytes stored in HDF and HDF-EOS
- 29 - HDFHDF
How can we How can we achieveachieve success? success?
- 30 - HDFHDF
How can we achieve success? How can we achieve success?
• Maintain strong, responsible, and continuing relationships with users
• An approach to needs identification, software design, and software implementation based on sound principles of software engineering
• Effective technical processes for developing, testing, integrating and maintaining software
• Business and social processes based on sound group management principles
- 31 - HDFHDF
Stages of software development at Stages of software development at HDFHDF
• Getting started
• Creating an implementation approach
• Implementation and maintenance
• Relations with users and sponsors
• Group practices
- 32 - HDFHDF
Getting startedGetting started
• Discover a need• Identify a sponsor• Clarify the need, its role, and its importance• Enter task into the project plan
– Make initial estimate of time and resources for the task– Give it a priority– Identify task’s lead – Identify a person who will work on the task
- 33 - HDFHDF
Creating implementation approachCreating implementation approach
• Write up a needs/approach RFC (Request For Comment)– Actively solicit feedback from developers/sponsors– Revise until satisfied
• Write up a design/approach RFC– Get feedback from developers/sponsors– Revise until satisfied
• Revise project plan according to RFC results• Archive RFC
- 34 - HDFHDF
Implementation and maintenanceImplementation and maintenance
• Identify validation plan (need improvement)• Implement
– Library or tool– Tests– Documentation
• Ask sponsor and friendly users for feedback• Review results and repeat appropriate steps above as
needed• Clean up (documentation, Web, etc.) and announce • Support (debug, fix, add more tests, advertise)
- 35 - HDFHDF
Relations with users and sponsorsRelations with users and sponsors
• Who are our sponsors?– Organizations and communities with
institutional and financial commitment to HDF• NCSA, NASA, DOE ASCI, Boeing, …
– Agencies supporting R&D• NCSA, NASA, DOE, NSF, …
– Collaborators who make in-kind contributions• Cactus, PyTables, NeXUS, CGNS …
– HDF group members
- 36 - HDFHDF
Relations with users and sponsorsRelations with users and sponsors
• Each task is associated with a sponsor• Each task has a priority, which should be
confirmed with sponsor• Each task falls into one of these categories
– Research– R&D (research, possibly integrate into product)– Development
• Technology infusion• Library or tools enhancement
- 37 - HDFHDF
Group practicesGroup practices
- 38 - HDFHDF
Group practices - technicalGroup practices - technical
• Source code management: CVS
• Bug tracking: Bugzilla– Bugs entered by support staff and developers– Prioritized by staff– Easy bugs fixed “on the fly”
- 39 - HDFHDF
Group practices - technicalGroup practices - technical
• The testing challenge
• Code testing– Testing before code check-in– Regression testing– Remote testing– Different configurations testing– Backward compatibility testing
- 41 - HDFHDF
Daily test reportDaily test report
From: HDF group system admin <[email protected]>To: [email protected]: HDF5_Daily_Tests_FAILED!!!
*** HDF5 Tests on 041022 ***
============================= Watchers List=============================HDF5 Daily test features/platforms watchers and procedure---------------------------------------------------------
Procedure:The watcher will investigate and report the cause of failure by 11am.The developer who checked in the error code may report so by then too.The watcher or the developer should get thefailure fixed and report it by 3pm.
Platforms watchers:AIX 5.1 (copper) AlbertFreeBSD QuinceyHP-UX ElenaIA32 (tungsten) RaymondIA64 (tg-login) AlbertIRIX64-6.5 32,64-bit RaymondIRIX 6.5 RaymondLinux 2.4 PeterSolaris 2.7&8 32,64-bit ElenaWindows Kent
Features watchers:General Library QuinceyGeneral parallel Albertconfiguration Quincey, Jamesmpich RaymondFortran ElenaIntel compilers Elena + Kent (for windows) PGI compilers ElenaC++ Binh-Minh Thread-safety QuinceyTools Padro
---updated: 2004/10/01
============================= Tests Summary=============================****FAILED eirene: setenv CC icc setenv F9X ifc setenv CXX icc --enable-fortran --enable-cxx****
PASSED arabica: setenv CC /afs/ncsa/projects/hdf/packages/mpich_1.2.4/SunOS64_5.7/bin/mpicc setenv F9X /afs/ncsa/projects/hdf/packages/mpich_1.2.4/SunOS64_5.7/bin/mpif90 setenv ALL_LOCAL 1 --enable-fortran standardPASSED arabica: setenv CC mpicc setenv ALL_LOCAL 1 standardPASSED arabica: setenvN 2 CC cc -xarch=v9 setenvN 2 F9X f90 -xarch=v9 setenvN 2 CXX CC -xarch=v9 standard --with-szlib=/afs/ncsa/projects/hdf/packages/szip_new/SunOS_5.7-64bitPASSED arabica: standard --enable-cxx --enable-fortran --with-szlib=/afs/ncsa/projects/hdf/packages/szip_new/SunOS_5.7PASSED Cu12: --enable-parallelPASSED Cu12: --enable-parallel setenv CFLAGS -q64 setenv FFLAGS -q64 setenvN 3 AR ar -X 64 --enable-fortran --with-zlib=/afs/ncsa/projects/hdf/packages/zlib/AIX5.1-64bit --with-szlib=/afs/ncsa/projects/hdf/packages/szip_new/AIX5.1-64bit
- 42 - HDFHDF
Group practices - technicalGroup practices - technical
• Release levels– Development release– Official release– Past releases
- 43 - HDFHDF
Group practices - technicalGroup practices - technical
• Coding standards
• Maintaining platform-independence
• Maintaining time-independence
• Rules for changing APIs
• Documentation
• Rapid prototyping
- 44 - HDFHDF
Group practices – business and socialGroup practices – business and social
• Staff breakdown– User support– Documentation– QA– Software development– Testing– Team leadership– System administration
Basic library Basic library developmentdevelopmentBasic library Basic library developmentdevelopment
Support, Support, doc, QA, doc, QA,
maintenancemaintenance
Support, Support, doc, QA, doc, QA,
maintenancemaintenance
Tools and Tools and JavaJava
Tools and Tools and JavaJava
Parallel I/O, Parallel I/O, Grid, Grid,
big machinesbig machines
Parallel I/O, Parallel I/O, Grid, Grid,
big machinesbig machines
HDF ProjectHDF ProjectHDF ProjectHDF Project
• Team lead for each team• Most staff in two or more teams• Staff relationships
– Complement each other– Overlap each other– Keep each other honest
- 45 - HDFHDF
Group practices – business and socialGroup practices – business and social
• Accountability of everyone to the whole process• Help desk• Approaches to carrying out tasks
– Paying attention to technical proposals– Weekly HDf5 developer’s meetings– HDF seminars
• Management and administration– Performance reviews with emphasis on goals, development– Critical to success– That’s another talk
- 46 - HDFHDF
Summing upSumming upStrengths, weaknesses, needsStrengths, weaknesses, needs
- 47 - HDFHDF
StrengthsStrengths
• User support• Staff
– High quality, diverse staff with good morale– Staff commitment and enthusiasm
• Ability to address all aspects of product development– Emphasis on quality control – Fast bug fixing and frequent releases– Ability to focus on a single product over a long term
• High level of support from sponsors• Project’s visibility through NCSA, NASA, DOE, users
- 48 - HDFHDF
WeaknessesWeaknesses
• Software development team– Library expertise still concentrated among too few
developers– Team communication is challenging
• Processes– Release/maintenance take too much time and
resources– Configuration and porting are a huge time sink– We don’t do enough prototyping– Hard to keep up with new technologies– Parallel I/O hard to support
- 49 - HDFHDF
More weaknesses & challengesMore weaknesses & challenges
• Usability– Software too hard to use for casual users– Insufficient documentation– Insufficient tools for high level users– Insufficient interoperability with common tools and
formats
• Marketing– Marketing effort is inadequate– Need to connect better with users and potential users
• Viable long-term support
- 50 - HDFHDF
Most immediate needsMost immediate needs
• Configuration and build• Testing and prototyping• Marketing• Reporting
– Performance reports– General reports to users– HDF book
• Sustainable business model
- 51 - HDFHDF
Thank youThank you