Upload
stash
View
50
Download
0
Embed Size (px)
DESCRIPTION
DAQ and Trigger upgrade. U. Marconi, INFN Bologna Firenze, March 2014. DAQ and Trigger TDR. TDR planned for June 2014 Sub-systems involved: Long distance optical fibres Readout boards (PCIe40) and Event Builder Farm The Event Filter Farm for the HLT Firmware, LLT, TFC. - PowerPoint PPT Presentation
Citation preview
1
DAQ and Trigger upgrade
U. Marconi, INFN BolognaFirenze, March 2014
DAQ and Trigger TDR• TDR planned for June 2014• Sub-systems involved:– Long distance optical fibres–Readout boards (PCIe40) and
Event Builder Farm– The Event Filter Farm for the HLT– Firmware, LLT, TFC
2
3
Readout System Review• Held on Feb 25th, 9:00 – 16:00• 4 reviewers– Christoph Schwick (PH/CMD – CMS DAQ)– Stefan Haas (PH/ESE)– Guido Haefeli (EPFL)– Jan Troska (PH/ESE)
• https://indico.cern.ch/event/297003/• All documents can be found in EDMS
https://edms.cern.ch/document/1357418/1
4
Trigger LHCb TB MeetingLHCb Technical Board Meeting 18/03/2014
Full software trigger,software LLT
R. Legac, Trigger Coordinator
5
Long distance fibres
The distance to cover with 850 nm OM multimode optical cables, from underground to the surface, is 300 m.
Locate the Event Builder Farm and the Event Filter Farm for the HLT outside the cavern.
• Minimum required bandwidth: 32 Tbit/s – # of 100 Gigabit/s links > 320, – # of 40 Gigabit/s links > 800, – # of 10 Gigabit/s links > 3200 – # of 5Gbit GBT links (4.5 Gbit effective):
> 10000– Current estimate O(15000), DAQ, Controls
and Timing system. Spares included
6
Long distance fibres (II)
• 144 fibres per cable. A total of 120 such cables.
• 3 patch panels (breakpoints) foreseen:expected attenuation ~ 3dB
• 4.8 Gbit/s signal produced on the detector by Versatile Link transmitters.
• VTTx to MiniPod for data acquisition.• MiniPod to VTRx for control, configuration.
MiniPod
7
Optical fibres studies
Measurement of BER vs. receive OMA(*) on different OM3 and OM4 fibres.(*) optical modulation amplitude
Transmission test at 4.8 Gb/s
The target value of 3 dB on OM3 using the Versatile Link and MP is reachable.
8
Optical linksOM3 vs OM4 A shortest path seems possible
LHCb DAQ today
2 x F10 E1200i
56 sub-farms
Readout boards: 313 x TELL1
9
Push-protocol with centralized flow-control
# links (UTP Cat 6) ~ 3000
Event-size (total – zero-suppressed)
65 kB
Read-out rate 1 MHz
# read-out boards 313
output bw / read-out board up to 4 Gbit/s (4 Ethernet links)
# farm-nodes 1500 (up to 2000)
max. input bw / farm-node 1 Gbit/s
# core-routers 2
# edge routers 56
DAQEvent builder
High speed network
HLT Event filter
10
PCIe40
PCIe40
PCIe40
PCIe40
PCIe40
PCIe40
PCIe40
PCIe40
16-lane PCIe-3 16-lane PCIe-3
EBF
EFF
Readout board review• Basic question to reviewers:
“We propose a change of baseline from ATCA/AMC to PCIe. Do the reviewers support the choice of baseline (PCIe) and back-up (ATCA)?”
• Answer:“Given the listed advantages the review committee endorses the choice of the PCIe based design as a baseline.”
11
12
PCIe Gen3 based readout A main FPGA manages the input streams and transmits data
to the event-builder server using PCIe Gen3. PCIe Gen3 throughput: 16-lane × 8 Gb/s/lane = 128 Gb/s The readout version of the board uses two de-serializers.
16-lane PCIe-3 edge connectorto the motherboard of the host PC
24 optical linksfrom the FEE DMA over 8-lane
PCIe-3 hard IP blocks
U. Marconi et al. The PCIe Gen3 readout for the LHCb
upgrade, CHEP2013
13
PCIe-3 test board• ALTERA development board, equipped with a Stratix V GX FPGA, model
5SGXEA7K2F40C2N
http://www.altera.com/literature/manual/rm_svgx_fpga_dev_board.pdf“The Stratix V GX FPGA development board is designed to fit entirely into a PC motherboard with a ×8 PCI Express slot that can accommodate a full height short form factor add-in card. This interface uses the Stratix V GX FPGA device's PCI Express hard IP block, saving logic resources for the user logic application”.
8-lane edge-connector
14
The PCIe-3 DMA test setup
GPU used to test 16-lane PCIe-3data transfer between the device and the host memory
The ALTERA development board8-lane PCIe-3
15
The DMA PCIe-3 effective bandwidth
DMA
over
8-la
ne
PCIe
-3 h
ard
IP b
lock
sAL
TERA
Str
atix
V
16
Test of PLX bridge
• Long-term test using GTX690 card / PLX 8747 bridge.
• Zero impact of using a bridge and two independent PCIe targets pushing data into a PC. Consistently around 110 Gb/s over long-term.
• No load balancing issues between the two competing links observed.
• Details on: https://lbonupgrade.cern.ch/wiki/index.php/I/O_performance_of_PC_servers#Upgrade_to_GTX690
10 h
110 Gb/s
PLXGPU
1GPU
2
PCIe 16
PLX
8-lane
16-lane
PCIe switch
Event builder node
17
PCIe40Event BuildingNetwork Interface data from
the detector~ 100 Gb/s
to theevent builderDual-port IB FDR – 110 Gb/s
from theevent builder
events that are beingbuilt on this machine
opportunity for doingpre-processing of full event here
to the HLT
empty
DDR340-50 GB/sHalf duplex
2x50 GB/sFull duplex
DDR340-50 GB/sHalf duplex
18
EVB performance
• Event Builder execution requires around 6 logical cores• 18 instances of the HLT software
CPU consumption Memory I/O bandwidth
• Event Builder performs stably at 400 Gb/s
PC sustains 100 Gb/s Event Builder todayIvy Bridge Intel dual CPU 12 cores
We currently see 50% free resources for opportunistic triggering on EB nodes
19
Fast Control
Costing (preliminary)
• Long-distance fibres:~ 1.6 MCHF 15000 fibres OM3, 300 m.– Excluding patch-cords to
detector and TELL40, cable-ducts, but including patch-panels, installation and testing
– No contingency, but several quotes.
• PCIe40:DAQ version: 5.8 kCHF ECS/TRG version: 7.9 kCHF– includes 15% contingency
• Readout network,Event-building (PCIe40) ~ 3.6 MCHF – Including event-builder PCs and event-
filter network, excluding farm-servers– Model based on InfiniBand FDR (2013
quotes)• Event-building (AMC40)
~ 9 MCHF – including event-filter network– Model based on 40G and 10G
Ethernet (on 2013 quotes)
20
21
Running conditions• Visible rate and mean visible interactions per crossing• At 2.×1033 , 27 MHz, 5.2 interaction per crossing
Mean visible interactions per crossing
22
LHCb upgrade: HLT farm• Trigger-less system at 40 MHz:
A selective, efficient and adaptable software trigger. • Average event size: 100 kB• Expected data flux: 4 TB/s• Total HLT trigger process latency: 14 ms
– Tracking time budget (VELO + Tracking + PV searches): 50% – Tracking finds 99% of offline tracks with pT >500 MeV/c
• Number of running trigger process required: 4.×105
• Number of core/CPU available in 2018: ~ 200– Intel tick-tock plan: 7nm technology available by 2018-19, the n.
of core accordingly scales as 12. × (32 nm/ 7 nm)2 = 250• Number of computing nodes required: ~ 1000
23
LLT
Readout reviewers’ opinion
… only the software option should bepursued if any.
24
LLT performance
Person-power
• Fibres: (CERN)– Tests and preparation of installation 0.5 py (person-year)– Installation & testing by external company 0.5 py for supervision and follow-up
• PCIe40: (Marseille, Bologna)– design, production, commissioning ~ 18 py
• LLT: (Marseille, Annecy, Clermont Fd., LAL) ~ 17 py • Firmware: Global framework, TFC, DAQ (Marseille, Bologna, CERN, Annecy) ~ 18
py (estimated)• DAQ: (CERN, Bologna)
– ~ 15 py (excluding ECS and high-level trigger infrastructure)• Overall person-power in involved institutes is sufficient, but does not allow for any
“luxuries”, as many people are also involved in the operation of LHCb during Run 2
25
N. Neufeld
Schedule TELL40
26
Schedule readout board
PCIe40 board plan: as INFN we should develop the PCB and prototypes, to qualify our production: be ready to produce the boards.
Sblocco 25 kE s.j.
27
Schedule DAQ
• Assume start of data-taking in 2020– System for SD-tests ready whenever needed minimal
investment• 2013 – 16: technology following (PC and network)• 2015 – 16: Large scale network IB and Ethernet tests• 2017: tender preparations • 2018: Acquisition of minimal system to be able to read out
every GBT– Acquisition of modular data-center
• 2019: Acquisition and Commissioning of full system– starting with network– farm as needed
N. Neufeld
28
Schedule: firmware, LLT, TFC
• All these are FPGA firmware projects• First versions of global firmware and TFC ready
now (for MiniDAQ test-systems)– then ongoing development
• LLT – Software 2014 - 2015– Hardware 2016 – 2017 (?)
29
Schedule long-distance fibres• Test installation 2014
– validate installation procedure and pre-select provider
• Long-term fibre test with AMC40/PCIe40 on long-distance 2014/2015
• Full installation in LS1.5 or during winter-shutdowns / to be finished before LS2
• Assumptions:– Installation can be done
without termination in UX (cables terminated on at least one end), if blown, fibres can be blown from bottom to top