Upload
mariah-ward
View
222
Download
1
Embed Size (px)
DESCRIPTION
LHCb Run2LHCb Run 3 Max. inst. luminosity4 x 10^322 x 10^33 Event-size (mean – zero-suppressed) [kB]~ 60 (L0 accepted) ~ 100 Event-building rate [MHz]1 40 # read-out boards~ link speed from detector [Gbit/s] output data-rate / read-out board [Gbit/s]4100 # detector-links / readout-boardup to 24up to 48 # farm-nodes~ # links 100 Gbit/s (from event-builder PCs)n/a final output rate to tape [kHz]
Citation preview
Online Triggers and DAQopenlab V
Niko Neufeld, CERN/PH
CERN openlab major review Oct. 2014
openlab V lightning DAQ N. Neufeld 2
Data acquisition and online challenges - recap
Online data filtering and processing(quasi-) realtime data reduction for high-rate detectors
High bandwidth networking for data acquisitionTerabit/s networks
Data-transfer and storage
openlab V lightning DAQ N. Neufeld 3
Evolution of the LHCb trigger-DAQLHCb Run2 LHCb Run 3
Max. inst. luminosity 4 x 10^32 2 x 10^33
Event-size (mean – zero-suppressed) [kB] ~ 60 (L0 accepted) ~ 100
Event-building rate [MHz] 1 40
# read-out boards ~ 330 400 - 500
link speed from detector [Gbit/s] 1.6 4.5
output data-rate / read-out board [Gbit/s] 4 100
# detector-links / readout-board up to 24 up to 48
# farm-nodes ~ 1600 1000 - 4000
# links 100 Gbit/s (from event-builder PCs) n/a 400 - 500
final output rate to tape [kHz] 12 20 - 100
openlab V lightning DAQ N. Neufeld 4
Detector front-end electronics
Eventbuilder network
Eventbuilder PCs + software LLT
Eventfilter Farm~ 80 subfarms
UX8
5BPo
int 8
surf
ace
subfarm switch
TFC500
6 x 100 Gbit/s
subfarm switch
Online storage
Clock & fast com
mands
8800Versatile Link
throttle from PCIe40
Clock & fast commands
6 x 100 Gbit/s
Online ArchitectureEC
S
5
Event-size [kB] Rate [kHz] Bandwidth [Gb/s] Year [CE]ALICE 20000 50 8000 2019ATLAS 4000 200 6400 2023
CMS 4000 1000 32000 2023LHCb 100 40000 32000 2019
Future data rates @ LHC (trigger stage 2)
openlab V lightning DAQ N. Neufeld
40000 kHz == collision rate LHCb abandons Level 1 for an all-software triggerO(100) Tbit/s networks required
openlab V lightning DAQ N. Neufeld 6
Project HTCC (High Throughput Computing Collaboration)
Partners: CERN, intelGoal apply intel technologies to Online computing challenges
Work-packages1. Interconnect for DAQ (Stormlake)2. Knights Landing for software triggers (HLT and L1.5)3. Reconfigurable Logic (Xeon/FPGA)
Use LHCb use-case as concrete example problems, but try to keep solutions and reports as widely applicable as possible
openlab V lightning DAQ N. Neufeld 7
Original “Use-cases” in Online (from openlab V white paper) 1) “Level-1” using (more/all) COTS hardware ✔2) “Data Acquisition” ✔3) “High Level Trigger” ✔ 4) “Controls”5) “Online data processing” ✔6) “Exporting data”
✔ == covered in HTCC
Additional material
Use-case #1 The first level trigger
openlab V lightning DAQ N. Neufeld 10
Calorimeter data
11
Finding Muons (2d view)
openlab V lightning DAQ N. Neufeld
Level 1 Trigger todayThe Level 1 Trigger is implemented in hardware: FPGAs and ASICs difficult / expensive to upgrade or change, maintenance by experts onlyDecision time: ~ a small number of microsecondsIt uses simple, hardware-friendly signatures looses interesting collisionsEach sub-detector has its own solution, only the uplink is standardized
openlab V lightning DAQ N. Neufeld12
openlab V lightning DAQ N. Neufeld 13
Level 1 challenge
Can we do this in software? Using GPGPUs / XeonPhis?We need low and near– deterministic latency Need an efficient interface to detector-hardware: CPU/FPGA hybrid?
Use-case #2 and #6Data Acquisition and Export fast data transport
Data Acquisition (generic example)
openlab V lightning DAQ N. Neufeld15
custom radiation- hard link from the detector
DAQ (“event-building”) links – some LAN (10/40/100 GbE / InfiniBand)
Links into compute-units: typically 1/10 Gbit/s
Detector
DAQ network
100 m rock
Readout Units
Compute Units
Every Readout Unit has a piece of the collision dataAll pieces must be brought together into a single compute unitThe Compute Unit runs the software filtering (High Level Trigger – HLT)
10000 x
~ 1000 x
~ 3000 x
openlab V lightning DAQ N. Neufeld 16
Data acquisition challenge
Transport large amount of data (multiple Terabit/s @ LHC) reliably and cost-effectivelyIntegrate the network closely and efficiently with compute resources (be they classical CPU or “many-core”)Multiple network technologies should seamlessly co-exist in the same integrated fabric (“the right link for the right task”), end-to-end solution from online processing to scientist laptop (e.g. light-sources)
Use-case #3 and #5Online Data processing
openlab V lightning DAQ N. Neufeld 18
Pattern finding - tracks
19
Same in 2 dimensions
openlab V lightning DAQ N. Neufeld
Can be much more complicated: lots of tracks / rings, curved / spiral trajectories, spurious measurements and various other imperfections
LHC High Level Trigger: Key Figures
Existing code base: 5 MLOC of mostly C++Almost all algorithms are single-threaded (only few exceptions)Currently processing time on a X5650 per event: several 10 ms / process (hyper-thread)Currently between 100k and 1 million events per second are filtered online in each of the 4 LHC experiments
openlab V lightning DAQ N. Neufeld20
21
Online Data processing challenge
Make the code-base ready for multi/many-core (this is not Online specific!)Optimize the online processing compute in terms of cost, power, cooling Find the best architecture integrating “standard servers”, many-core systems and a high-bandwidth network
openlab V lightning DAQ N. Neufeld