Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
TheHDFGroup
www.hdfgroup.org
HDF5Update
High-DataRateMXMee9ngNSLS-IIatBNLMay26,2016
1
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Outline
2
• HDF5featuresrequestedandsponsoredbySynchrotronCommunity(2012-2016)• DirectchunkI/O(PSI,Dectris)• Dynamicallyloadedfilters(DESY)• SingleWrite/Mul9pleReaderorSWMR(DLS,ESRF,DESY)
• VirtualDataset(VDS)(DLS,DESY,XFEL)• Newrequirementsandfeatures• HDF5roadmapfor2016
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Directchunkwrite:H5DOwrite_chunk
5/25/16 3
www.hdfgroup.orgHDRMXNSLS-IIatBNL
PerformanceresultsforH5DOwrite_chunk
5/25/16 4
1 Speed in MB/s 2 Time in seconds
Test result on Lunux 2.6, x86_64 Each dataset contained 100 chunks, written by chunks
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Dynamicallyloadedfilters
• Problemswithusingcustomfilters• “Offtheshelf”toolsdonotworkwiththethird-partyfilters• h5dump,MATLABandIDL,etc.
• HDF5toolscannotreadfilecreatedby• h5py,PyTables,etc.
• Solu9on• ModifyHDF5sourcewithyourcodeanddistributeit
• Andwhatwillhappenifauserwantsfiltersfromthedifferentdistribu9ons????Oh….No….
• Usea1.8.11andlater• ProvidemaintainedlibraryofHDF5compressionfilters
55/25/16
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Approach
• Therearepredefineddefaultloca9onswheretheHDF5librarysearchesthesharedlibrariesorDLLswiththeHDF5filterfunc9ons.
• Thedefaultloca9onmaybeoverwrigenbyanenvironmentvariable.
• Encoding:Standardprogrammingmodeltoregistercompressionfilteronwriteopera9on
• Decoding:Transparentonread;nouserac9onrequired.
• Worksgreat,but….
65/25/16
www.hdfgroup.orgHDRMXNSLS-IIatBNL
User’sheadacheandpossiblesolu9ons
• WheretofindHDF5plugins?• Noofficialdistribu9onsite
• ExampleofHDF5filtersdistribu9onsonGithub! hgps://github.com/dectris/HDF5Plugin! hgps://github.com/Blosc/hdf5-blosc! hgps://github.com/nexusformat/HDF5-External-Filter-Plugins! hgps://svn.hdfgroup.org/hdf5_plugins/
• TheHDFGroupcanhelpwith• Maintainingcommonrepository• Tes9ngpluginwiththereleases• Distribu9ngsourceandbinariesforLinux,MacandWindows• Freeandlicensed(forfee)binarydistribu9ons
75/25/16
www.hdfgroup.orgHDRMXNSLS-IIatBNL
SWMR:Dataaccesstofilebeingwrigen
HDF5File
Writer Reader
…whichcanbereadbyareader…
withnoIPCnecessary.
Newdataelements…
…areaddedtoadatasetinthefile…
04/01/16
www.hdfgroup.orgHDRMXNSLS-IIatBNL
SWMR
9
• ReleasedinHDF51.10.0• Restrictedtoappend-dataonlyscenario• SWMRdoesn’tworkonNFS• Filesarenotcompa9blewithHDF51.8.*libraries
• Useh5format_converttool• ConvertsHDF5metadatainplace• Norawdataisrewrigen
www.hdfgroup.orgHDRMXNSLS-IIatBNL
VDS
10
• Datastoredinmul9plefilesanddatasetscanbeaccessedviaonedataset(VDS)usingstandardHDF5read/write
www.hdfgroup.orgHDRMXNSLS-IIatBNL
VDSExample
11
Seriesofimages
A
B
C
D
EF
a.h5
c.h5
e.h5
b.h5
d.h5
f.h5
A
C
E
B
D
F
t1 t2
t3 VDS.h5
111Imageat9met2
A
B
C
D
EF
A
B
C
D
EF
AADataset A
Dataset C
Dataset E
CC
Dataset B
DatasetD
Dataset FF
F
DD
EE
BB
M
n
kn
nk
k
n k n
n k
k
M
t2
t2
t2
t2
t2
t2
www.hdfgroup.orgHDRMXNSLS-IIatBNL
VDS
12
• VDSworkswithSWMR• FilewithVDScannotbeaccessedbyHDF51.8.*libraries
• Useh5repacktooltorewritedata(1.10.0-patch1)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Newrequirementsandfeatures?
13
• Tellusyourneeds:• Mul9-threadedcompressionfilters• H5DOread_chunkfunc9on• FullSWMRimplementa9on• Performance• Backward/forwardcompa9bility
• Otherrequests?
www.hdfgroup.orgHDRMXNSLS-IIatBNL
HDF5Roadmapfor2016
14
• May31-HDF51.10.0-patch1• H5repack,Windowsbuilds,FortranissuesonHPCsystems
• Mid-summerHDF51.10.1• Someinternalissuesfoundfor1.10.0
• December• HPCfeaturesthatdidn’tmakeitinto1.10.0release
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Ques9ons?
Thankyou!
15