Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Jörn Adamczewski-Musch
GSI / EEL 5 December 2017
DAQ storage for HADES (dabc plug-in)
LTSM gateway for MBS systems
Summary
5th December 2017 Jörn Adamczewski-Musch 2
DABC
5th December 2017 Jörn Adamczewski-Musch 3
TRB
TRB
TRB
TRB
TRB
TRB
TRB
TRB
TRB
TRB
Combiner
output
stream
http
DABC hld
Combiner
output
stream
http
DABC hld
Combiner
output
stream
http
DABC hld
Combiner
output
stream
http
EPICS run sync, GUI
/shm
/shm
/shm
/shm
QA monitoring HYDRA2
hld RFIO
/data0i
/lustre
storage interface
DABC
5th December 2017 Jörn Adamczewski-Musch 4
TRB
TRB
TRB
TRB
TRB
TRB
TRB
TRB
TRB
TRB
Combiner
output
stream
http
DABC hld
Combiner
output
stream
http
DABC hld
Combiner
output
stream
http
DABC hld
Combiner
output
stream
http
EPICS run sync, GUI
/shm
/shm
/shm
/shm
QA monitoring HYDRA2
hld LTSM
/data0i
/lustre
storage interface
Data Acquisition Backbone Core (http://dabc.gsi.de) Linux, C++, multithreading, modular, extensable plug-ins, multi-node DAQ
DABC LTSM plugin :
Implements dabc::FileInterface with fopen, fwrite, etc. methods (POSIX like)
Wraps and uses LTSM library by T.Stibor and tsm client by IBM
May write any file format (HADES .hld, MBS .lmd, ...)
generally configurable in dabc application by xml files
for HADES configurable by existing DAQ/EB scripts and setup files
5th December 2017 Jörn Adamczewski-Musch 5
5th December 2017 Jörn Adamczewski-Musch 6
mbs> con rfio lxg0546:2345 –disk
X86L-10:transport :Connected RFIO
server!
mbs> open file
/data.local1/myaccount/test_files_
first=10 size=1000 -auto -rfiofile
test.lmd
rfio client
rawDispRFIO64
rfio dispach
rawServRFIO64
rfio server
fork
write data
local harddisk
widely used at standalone mbs „traveling systems“
with local disk servers for several years.
5th December 2017 Jörn Adamczewski-Musch 7
mbs> con rfio x86l-59:2345 –ltsm
X86L-10:transport :Connected RFIO
server!
mbs> open file /lustre/hebe/mbs
first=10 size=1000 -auto –rfiofile
test.lmd
rfio client
rawDispLTSM64
rfio dispach
fork
write data
tape
rawServLTSM64
rfio server
TSM server
ltsm01_test
ltsm API
lustre
ltsmsync + copytool
benefit: MBS host does not
need TSM client library
(<- not available for LynxOS,
PPC Linux etc.)
new development
for MBS v6.3
Wrote data in parallel with HADES/dabc in September 2017:
(12 files x 3 MB/s), (8 files x 5 MB/s) from HADES DAQ pulser trigger
(1 x 75 MB/s) , (2 x 40 MB/s) (4 x 18 MB/s) self-generated data
10 GbE at HADES servers, but 1 TSM server only!
MBS with via RFIO/LTSM gateway (both on x86l-59) in August 2017: single file 40 MB/s at 5.5 kHz (without file: 70MB/s at 7 kHz)
1 GbE at MBS node, single TSM server
5th December 2017 Jörn Adamczewski-Musch 8
5th December 2017 Jörn Adamczewski-Musch 9
Data writing with LTSM has been implemented for GSI major DAQ systems (MBS, HADES/dabc)
Data access from batch farm possible via transparent /lustre file Data reading/copying possible via ltsmc (or easy wrapper script?) To do:
HADES needs 4 TSM servers at 10 GbE, optimization for 2018 beamtime
Test of MBS LTSM interface software rawDispLTSM64 / rawServLTSM64
Deployment and test of LTSM gateways for MBS ? dedicated gateway machine for each experiment ? ? common gateway servers provided by GSI IT department ?
5th December 2017 Jörn Adamczewski-Musch 10
5th December 2017 Jörn Adamczewski-Musch 11
The old gstore/RFIO system will no longer be supported by GSI
Writing to RFIO from HADES DAQ has failed already since 2016
New System: Ligtweight Tivoli Storage Manager (LTSM)
Development by T.Stibor, GSI-IT (https:/github.com/tstibor/ltsm)
Write to tape archive via sockets, like RFIO (but use IBM client directly)
Transparent synchronization between tape and lustre file system
„archive“ file name is lustre file path, file is retrieved on first lustre read access
DABC plug-in for LTSM has been developed in 2017
Successfully wrote data with HADES EB in September 2017:
(12 files x 3 MB/s), (8 files x 5 MB/s) from HADES DAQ pulser trigger
(1 x 75 MB/s) , (2 x 40 MB/s) (4 x 18 MB/s) self-generated data, 1 TSM server only!
Data Acquisition Backbone Core (http://dabc.gsi.de) Linux, C++, multithreading, modular, extensable plug-ins, multi-node DAQ
DABC hadaq plugin : UDP subevent receiver, event builder functionality
hld file output to RFIO and disk
EPICS interface via intermediate shared memory
writes same run info for ORACLE logbook
configurable by existing DAQ/EB scripts and setup files
QA monitoring events provided by „streamserver“ socket
HTTP server for EB monitoring and control
May optionally run pre-analysis processing (TDC calibration etc)
5th December 2017 Jörn Adamczewski-Musch 12
DABC performance (August 2014)
29 subsystems, 25 kHz pulser, 3 EB, RFIO and local disk file
=> 38 Mbyte/s per EB process, max CPU load 78%, no event loss
=> would achieve 608 Mbyte/s with all 16 EB processes (?)
typical pion beam data rate 2014: 9 Mbyte/s at <2kHz trigger rate)
netmem/evtbuild performance (August 2011)
28 subsystems, Au-beam trigger, 8 EB, RFIO and local disk file
=> typical 17 Mbyte/s per EB process without event loss
=> overall 135 Mbyte/s
5th December 2017 Jörn Adamczewski-Musch 13