CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 1
LHCb Readout System and Real-Time Event
Management
- Emphasis on the architectural and functional aspects not the particular technologies- Commissioning and status
“An Integrated Control System for the LHCb Experiment”, CMS1-1, Thursday 14.00
“Handling Online Information in the LHCb Experiment”, CMSP-8, Tuesday 16.40
Federico Alessio, CERN, in place of Richard Jacobsson, CERN,
on behalf of the LHCb Online Team
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Place in the World
2
Underground cavern at ~100m depth
LHC
LHC
in 50 institutes in 16 countries
700 people
~10m
~20m
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Challenge of an hadronic precision experiment Particle multiplicity Very large background Small ratio of interesting B-meson decays O(10−3 –10−9)
High statistics Design luminosity L~2x1032 cm-2s-1 (1/50 Atlas&CMS) 10 MHz visible interactions 100 kHz bb-event rate ~ 1012 bb-pair / year at LHCb 2 kHz event storage rate
LHCb – Single-arm forward spectrometer
3
p-beamp-beam
Efficient trigger for many B decay topologiesMuon system, ECAL+Preshower , HCAL, Vertex Locator
Efficient particle identificationRICH
Good decay time resolutionVertex Locator 5 mm from beam
Good mass resolutionTracker and Magnet
B
1 cm
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Trigger Architecture
4
Hig
h-Le
vel T
rigge
r
2 kHz
Level -0
L0 e, g
40 MHz
1 MHz
L0 had
L0 m
ECALAlley
Had.Alley
Global reconstruction30 kHz
HLT
1H
LT2
MuonAlley
Inclusive selectionsm, m+track, mm
Exclusive selections
Storage: Event size ~35kB
Level-0 Hardware Trigger 40 MHz 1 MHz Search for high-pT m, e, g, hadron candidates Latency 4 ms, i.e. pipelining 160 events
High Level Trigger Farm with O(1000) quadcores HLT1: Confirm L0 candidate with more complete info,
and add impact parameter and lifetime cuts HLT2: global event reconstruction + selections Processing time available O(milliseconds) Output rate 2 kHz
HLT needs to know how L0 is configured How to distribute to 1000 nodes simultaneously in
seconds when optimizing parameters during LHC fill?
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Evolution: Eliminating trigger levels
Readout Architecture RT2003
5
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 6
LHCb Readout System RT2009
SWITCH
High-Level Trigger farm
Detector
Timing & Fast
Control
SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH
READOUT NETWORK
LHC clock
Event Requests
Event building
Front-End
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
Readout Board
VELO ST OT RICH ECal HCal Muon
SWITCH
Mon. farm
CPU
CPU
CPU
CPU
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
L0 trigger
L0 Trigger
320 ROBs• 24 [email protected] Gb/s• 4 outputs@1 Gb/s
50 TB with 70 MB/s
3000 GbE ports35 GB/s
50 subfarms of ~40 nodes
Sh
ieldin
g w
all
5000 optical/analog linksO (4 Tb/s)
Offline See “Controlling a Large Trigger Farm Using Industrial Tools”, OPF2, Wednesday 11.00and “Management of the LHCb Readout Network”, OPF-3, Wednesday 11.00
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Design principle: A limited number of protocols and technologies
Readout network: GbE IPv4
Farm-to-storage: GbE TCP/IP
Overhead reduction <Event fragment> ~ 120 Bytes Overhead on IP/Eth is 58 Bytes
Pack fragments of several consecutive events = MultiEvent Packet (MEP) Protocol
Data Transfer Protocols
7
SWITCH
High-Level Trigger farm
Detector
Timing & Fast
Control
SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH
READOUT NETWORK
LHC clock
Event Requests
Event building
Front-End
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
Readout Board
VELO ST OT RICH ECal HCal Muon
SWITCH
Mon. farm
CPU
CPU
CPU
CPU
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
L0 trigger
L0 Trigger
Offline
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
SWITCH
High-Level Trigger farm
Detector
Timing & Fast
Control
SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH
READOUT NETWORK
LHC clock
Event Requests
Event building
Front-End
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
Readout Board
VELO ST OT RICH ECal HCal Muon
SWITCH
Mon. farm
CPU
CPU
CPU
CPU
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
L0 trigger
L0 Trigger
Offline
Calib
Readout control has two aspects: Control of data transfer
• MEP Packing • Destination assignment for event building and HLT• Load balancing• Partitioning for parallel activities
Management of event types and associated destinations/processing• Physics triggers• Calibration triggers• Luminosity triggers• Non-zero suppressed data• Luminosity scans (Vernier scan)
Driven and managed by the LHCb Timing and Fast Control System Responsible for distributing timing, trigger and synchronous and asynchronous information to entire
readout system FPGA based master: Readout Supervisor
• Also performs rate control and generates all types of auto-triggers and calibration sequences
Centralized Readout Control
8
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
TFC Master Info Flow
9
Readout Supervisor
LHC accelerator Beam Phase and Intensity Monitor
Subdetectors
HLT farm
L0 trigger
RS event bank
Multi Event Requests
Bunch currentsClock/orbit,
UTC, LHC Info
HW and run parameters
Run statistics
Detector status L0 Decision
RO ElectronicsTrigger Throttle
TFC
FE ElectronicsTFC
TFC
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Data Transfer ControlReadout network: Push protocol with passive pull mechanism
MEP Packing Control by Readout Supervisor:1. Trigger Type transmitted synchronously to all Readout Boards
Determines processing in ROB and synch check
2. IP Destination transmitted synchronously to all Readout Boards for each MEP Reception triggers closure and sending of last MEP Interleaving Trigger Type and Destination determines MEP packing Dynamic packing factor depending on event types Trigger type determines destination type (HLT, calibration, etc)
Farm Destination for next MEP chosen based on Credit scheme1. Farm nodes transmits Event Requests with a Credit(“declare as ready to receive”)
2. Readout Supervisor round-robin in Destination Table
3. Select destinations with positive Credit and decrement
Effectively, load balancing of readout network and HLT Farm Static load balancing of network by organization of Destination Tables in Readout Supervisor Dynamic load balancing of HLT farm nodes
Nodes request events when ready In all cases, event loss is minimized in case of failing/blocked/slow links or nodes
Ultimately, credit scheme regulate readout rate when low on credits
SWITCH
High-Level Trigger farm
Detector
Timing & Fast
Control
SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH
READOUT NETWORK
LHC clock
Event Requests
Event building
Front-End
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
Readout Board
VELO ST OT RICH ECal HCal Muon
SWITCH
Mon. farm
CPU
CPU
CPU
CPU
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
L0 trigger
L0 Trigger
Offline
Calib
10
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Event Management Readout Supervisor Data Bank appended to each event
Book-keeping Run Number regroups events taken with same configuration UTC Time to correlate event with Conditions DB Coarse quality bits from each sub-detector which may be used by HLT Trigger Type and Calibration Type determine type of processing Window of consecutive 25ns clock cycles which should be processed together (see later)
HLT needs to know how L0 is configured How to distribute to 1000 nodes simultaneously? Trigger Configuration Key distributed in RS data bank allows optimizing the trigger parameters in
real-time during LHC fill
Event Type
Run Number#D0
Step Number#D1
Trigger Configuration Key#D9
Bunch IDTAE
WindowTrigger
TypeCalType
FBX
TypeBunch Current
#D8
Detector StatusError Bits#D7
UTC Time (31 .. 0)#D6
UTC Time (63 .. 32)#D5
L0 Event ID (31 .. 0)#D4
L0 Event ID (63 .. 32)#D3
Orbit ID#D2
Event Type
Run Number#D0
Step Number#D1
Trigger Configuration Key#D9
Bunch IDTAE
WindowTrigger
TypeCalType
FBX
TypeBunch Current
#D8
Detector StatusError Bits#D7
UTC Time (31 .. 0)#D6
UTC Time (63 .. 32)#D5
L0 Event ID (31 .. 0)#D4
L0 Event ID (63 .. 32)#D3
Orbit ID#D2
11
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Detector Performance, Readout Performance and Data Quality Histogram collected from all systems Monitoring Farm spying on event streams at best effort
• Also produces histograms from an online reconstruction at best effort Histogram analysis Automatic checks and alarms Histogram inspected by Data Manager Shifter
Online Monitoring
12
SWITCH
High-Level Trigger farm
Detector
Timing & Fast
Control
SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH
READOUT NETWORK
LHC clock
Event Requests
Event building
Front-End
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
Readout Board
VELO ST OT RICH ECal HCal Muon
SWITCH
Mon. farm
CPU
CPU
CPU
CPU
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
L0 trigger
L0 Trigger
Offline
Calib
Histogram Handling
(ECS)
AutomaticHistogram
Analysis
Interactive Presenter
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
LHCb Data and Process Flow
13
Online 35 kB/evt@2kHz = 70 MB/s or 2 GB file(~60 kevts) / 30sOffline
Tie
r-0
/ 1
Tie
r-2
Storage(CASTOR)
Reconstruction
Simulation
Stripping
Analysis
HLT Bulk Stream Express Stream5 Hz
Storage(CASTOR)
Reconstruction
CalibrationAlignmentQ
C
Offline Control Room
20h/file, 20 kB/evt
Bookkeping
Data and Production ManagementData Quality Checking
Test Jobs
QC
Run Info
Dat
a Q
ual
ity
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Real-Time scheme to validate High Level Trigger, data flow and offline processing
= Be ready to receive, process and analyze 7 million events in the first hour of collisions
No-beam System Tests
14
SWITCH
HLT farm
TFC System
SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH
READOUT NETWORK
LHC clock
Event building
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
SWITCH
MON farm
CPU
CPU
CPU
CPU
Offline
LHCbMEP Requests
InjectorSimulated events
Detector
Front-End
Readout Board
VELO ST OT RICH ECal HCal Muon
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
Readout Board
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
FEElectronics
L0 trigger
L0 Trigger
MEP Request
Replacing detector with injection of 108 “accepted” simulated events real-time in Online system at HLT rate (2 kHz)
See poster “High-speed Data Injection for Data Flow Verification in LHCb, CMSP-28, Tuesday 16.40
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Commissioning LHCb
15
• Two years of intense work 2006 – 2008 with the aim to:
Operate the detector AND people as a unit with common tools Bring all components (sub-detectors and service systems) to operational state. Define, implement and validate the tools and procedures needed to run the detector as a whole Organise the activities to reach the ready state in time
Understand and calibrate the detector Test pulses, radioactive sources Cosmics LHC injection tests First days with beam
Operate with two shifters Operating the whole detector from one console Understandable high-level tools for diagnostics, alarms and data monitoring Homogeneity in the system Shifter training On-call Experts for all sub-systems and sub-detectors
Reach operational efficiency Starting (<10min) and restarting (<1 min) rapidly and smoothly
Crucial tool: Readout and processing of sets of consecutive 25ns clock cycles around “detector activity” trigger
• Time and space alignment• Leakage in preceding and subsequent clock cycles• Optimize signal over spill-over
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Commissioning with Cosmics
16
Challenge: LHCb geometry is NOT well suited for cosmics… “Horizontal” cosmics well below a Hz Still 1.6x106 good events (July – September 2008 ) recorded for the large sub-detectors
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
First Glimpse of LHC Protons
17
LHCb@LHC Sector Tests Aug-Sep 2008 Beam 2 dumped on injection line beam stopper (TED)
TED
TI8
LHC
Vertex Locator
Scintillator Pad Detector
Muon
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Experience with 150 Joules
18
RICH2 “photon blast”
• LHC Turn-On - September 10, 2008 An all too short honeymoon with LHC… Contrary to what we wish for the future, the splashes were Highly Desired Events!
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
Readout Architecture RT2014
19
See talk “A 40 MHz Trigger-free Readout Architecture for LHCb”, RTSA2, Tuesday 18.20
Upgrade plans already well underway: Another level of Natural Selection
L0 trigger
L0 Trigger
CERN
R. Jacobsson
CERN
16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009
LHCb has become an operational experiment “waiting” for beam
Readout system mature with advanced readout control and event management Reached a good compromise between use of COTS and custom electronics
Commissioning Still (too…) many experts in the control room…necessary ones or not…
Injection tests end of August 2008 gave the first ever LHC-induced tracks
Beam-collimator shots were obviously the high-light of 2008, “unfortunately”… LHCb very ready for the long run with LHC COLLISIONS 2009-10
Upgrade path is towards full 40 MHz trigger-free readout Becoming popular concept, cmp future acelerators Good topic for next years Real Time Conference
Conclusions
20