Proprietary and Confidential. Copyright 2016, The HDF Group.Proprietary and Confidential. Copyright 2016, The HDF Group.

HDF Group Past, Present & Future

1

Outline

2

Who we are and where we come from

Mission and commitments to community

Future directions for HDF Group & HDF R&D

Who we are and where we come from

National Center for Supercomputing Applications - 19874

Big simulations

Desktop visualization

The birth of HDF 5

6

Version 1.0

7

NASA Earth Observing System

• 6,700 Data Products • 12 Data Archive Centers

• 16 terabytes per day • Distributed to 1.7 million

end users worldwide

Non-destructive testing - TRAPPIST

9

Shortcomings of HDF (1996)

Limits on object & file size (<2GB)

Limited number of objects (<20K)

Rigid data models

I/O performance

Code complexity

Accelerated Strategic Computing Initiative

11

How to maintain a nuclear stockpile without testing?

Data can be complicated

12

The birth of HDF5

13

Long-term data

preservation

14

HDF keys to success

15

NeedsHDF

Group

Software for the long term.

API, library, utilities, documentation

User support

Institutional support

Long term storage

Complex formatting

Complex uses

Open source durability

Spinning off

•Business model options•For-Profit or not?•How to make money?• Intellectual property

16

Who is the HDF Group?

Small company ~ 40 employeesfocus on High

Performance Computing and Scientific Data

Offices in Champaign, IL + 7 other locations

Our flagship platform –HDF5 – is at the heart of our

open source ecosystem.

Tens of thousands use HDF5 every day.

1500+ projects on Github

Work with industry, research instituitons,

government to improve HDF5,

build HDF5 solutions, provide support.

18

Mission and commitments to OSS and community

HDF Group Website

19

Software

20

HDF Support portal

21

HDF Forum

22

Blogs

23

Webinars

24

Technology Advisory Board

25

Get engaged

HDFql in Nov/Dec 2019Present a Webinar about your project

About your project or some aspect of HDF5 Write a blog

lori.cooper@hdfgroup.org Contact

27

Future directions for the HDF Group and HDF R&D

1998-2007 - Maturation

• Matured the HDF5 data model and customized I/O including parallel I/O

• Community standards NeXUS, HDF-EOS, etc. emerged

28

2008-2019 - Expanding to non-HPC communities and Exascale computing

VPIC VORPAL

0

5000

10000

15000

Hopper Intrepid Hopper Intrepid

I/O

Ba

nd

wid

th (

MB

/s)

Default

Tuned

0

100

200

300

400

500

600

Num

ber o

f uniqu

e us

ers

Top 15 libraries used on Edison (2014)

• Widespread use• Improved I/O performance• Growth of software ecosystem• New features to support particle accelerators, exascalearchitectures, observational data, etc.

29

Future directions

30

Continue…

31

Support legacy library versions

Adapt to OS changes, etc.

Improve library, tools, docs

Increase community involvement

Leverage new memory, computing and storage architectures

• Deep memory hierarchy

• Alternate storage and I/O

• Non-traditional HDF5 uses

32

Facilitate convergence of Big Data and HPC

• Combine data and compute services

• Large scale query

Ease of use, productivity

• New APIs for C++ community

• Working with tool developers

THANK YOU!Questions & Comments?

Proprietary and Confidential. Copyright 2016, The HDF Group.

35