1

CERN

Real Time conference, Montreal May 18 – 23, 2003 Richard Jacobsson

Driving the LHCb Front-End

Readout

TFC Team:Arek Chlopik, IPJ, PolandZbigniew Guzik, IPJ, PolandRichard Jacobsson, CERNBeat Jost, CERN

2

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

LHCb Readout

Detector

Front-End electronics

L0 FE L0 FE L0 FE L0 FE

VELO

L0 FE L0 FE L0 FE

ST OT RICH ECAL HCAL MUON

SWITCH

L1 FE L1 FE L1 FE L1 FE L1 FE L1 FE L1 FE

SWITCH SWITCH SWITCH

SFC SFC SFC SFC SFC SFC

SWITCH SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH

CPU

READOUT NETWORK

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

Event building

CPU farm

TFC SYSTEM

L0 TRIGGER

LHC CLK

40 MHz

1 MHz

40 KHz

SORTER

Two levels of high-rate triggers

For more details see Niko Neufeld’s talk in session T3 (RT-088)

3

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

TFC Architecture (1)

Physics trigger

. . .

Throttle Switch TFC Switch

Clo

ck

L0 /

L1

ReadoutSupervisor 2

VELO FE ST FE OT FE RICH FE ECAL FE

Single TFC master :Readout Supervisor

Fan-out / Fan-in :Support partitioning

Front-End

Clo

ck

L0 /

L1

Local trigger(Optional)

Clo

ck

L0 /

L1

Clo

ck

ReadoutSupervisor 1

ReadoutSupervisor 3

ReadoutSupervisor 4

VELO FE

Event building network

Clo

ck

ReadoutSupervisor 1

4

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

TFC Architecture (2)

TFC components

- CERN Trigger, Timing and Control (TTC) distribution system. Common to all LHC experiments:

•TTCtx (electrical-optical converters)

•TTCoc (optical fan-outs)

•TTCrx (Receiver chips)

- Components endemic to LHCb:

•Readout Supervisors (“ODIN”)•TFC Switch (“THOR”)•Throttle Switches (“MUNIN”)•Throttle ORs

VELOL1 FE

TTCrx

VELOL1 FEVELOL1 FE

TTCrx

VELOL0 FE

TTCrx

VELOL0 FEVELOL0 FE

TTCrxVELOL0 FE

TTCrx

VELOL0 FEVELOL0 FE

TTCrx

Readout Supervisor

Readout Supervisor

Local trigger(optional)

L0 L1

Readout Supervisor

TFC switch L1 Throttle switchL0 Throttle switch

TTCtx TTCtx TTCtx TTCtx TTCtx

Thro

ttle

OR

TTC system

Clock receiverand fanoutLHC clock

L1

L0

Trigger splitterTrigger splitter

VELOL0 FE

TTCrx

VELOL0 FEVELOL0 FE

TTCrxVELOL0 FE

TTCrx

VELOL0 FEVELOL0 FE

TTCrx

VELOL1 FE

TTCrx

VELOL1 FEVELOL1 FE

TTCrx

TTCoc

VELOL1 FE

TTCrx

VELOL1 FEVELOL1 FE

TTCrx

VELOL0 FE

TTCrx

VELOL0 FEVELOL0 FE

TTCrxVELOL0 FE

TTCrx

VELOL0 FEVELOL0 FE

TTCrx

Thro

ttle

OR

VELOL0 FE

TTCrx

VELOL0 FEVELOL0 FE

TTCrxECALL0 FE

TTCrx

ECALL0 FEECALL0 FE

TTCrx

ECALL1 FE

TTCrx

ECALL1 FEECALL1 FE

TTCrx

Trigger splitterTrigger splitter

TTCoc TTCoc TTCoc

5

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

CERN TTC system 1

Developed in the CERN RD12 project: Timing, Trigger and Control distribution system based on fiber optics

Transmitting two channels multiplexed: A: Low latency 40 MHz signal B: Two types of broadcasts with Hamming

code protection:• Short broadcasts (8 bit data in 16 bit frame)

• Long broadcasts (16 bit data in 42 bit frame)

6

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

CERN TTC system 2 LHC:

Distribute LHC clock (40.08 MHz) and LHC orbit signal (11.246 kHz) to experiments over fiber with minimal jitter (~8ps RMS)

Experiments: Distribute clock, trigger and control commands to the detector readout over fiber

with minimal jitter

Prevessin LHC Control Room Experimental hall

several km

7

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

TTC in LHCb

Use of the TTC system in LHCb:

Channel A used to distribute (accept/reject signal)

• L0 trigger (40 MHz --> 1.1 MHz accept rate)

Channel B used to distribute short broadcasts with encoded:

• Bunch Counter Reset and L0 Event ID counter reset

• L1 trigger (1.1 MHz --> 40 kHz accept rate)

• Control commands (FE resets, calibration pulses)

Channel B used to distribute long broadcasts with:

• IP/Ethernet destination address for the data transmission over the network to the CPU farm

Broadcast order on channel B is handled according to a priority scheme

8

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

Readout Supervisor “Odin”

- Clock distribution

LHC clock

- L0 handling & distribution

L0

- Auto-trigger generator

Trigger generator

- Trigger controller Trigger controller

Throttles

- Command broadcast generatorCmd broadcast

generator- ODIN Front-End

Front-End

DAQ

- Exp. Control System interface

ECS interface

ECS

- L1 handling & distribution

L1

L1 broadcastgenerator

Experiment orchestra director - all mastership in a single module

Channel A/B

- TTC encoding

TTC Encoder

Designed with emphasis on:• Versatile - to support many different types of running modes

• Functions easily added and modified.

Credit Card PC

9

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

ODIN simulation

A lot of effort put on simulation. Specs have been simulated in behavioral model with a behavioral model of the

LHC machine, trigger system, and FE, using Visual HDL

Full Readout Supervisor with actual FPGA code and models of discrete logic has been simulated in the same simulation VisualHDL test bench

FPGA code also simulated at gate level in the same model

Q_IOI

Q_PIPE

Q_L0

Q_CMD

Q_L1

AFIFO

TFIFO

Q_GCS

Q_T1B

Q_RND

TTC

10

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

TFC Switch “THOR”

Crucial: Equal internal propagation delays. If skew too large, FE will suffer from timing alignment problems when using different Readout Supervisors.

Small jitter

Pool of Readout Supervisors

VELO

ST

OT

RICH

. . .

MULTIPLEXERS

DELAYS

ECSinterface

ECS

TT

C i

nfo

rma

tio

n

as T

TC

en

cod

ed e

lect

rica

l

Clock, trigger and command distribution and support partitioning

11

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

Throttle Switch “Munin”

Pool of Readout Supervisors

VELO

ST

OT

RICH

. . .

ECSinterface

ECS

Throttle feed-back from detector Front-End and trigger system to the appropriateReadout Supervisor

Th

rott

le s

ign

als

Throttle OR module is only a variation of the same board as 32:1

OR logic and history buffer

12

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

Conclusions TFC system architecture and use of TTC well establish

Different from the other LHC experiments with two levels of high-rate trigger

Emphasis on partitioning

Readout Supervisor

• All mastership in one module

• Provides a lot of flexibility and versatility

Switches

• Partitioning to support testing, calibrating, and debugging well integrated

The first prototype of the Readout Supervisor “ODIN” built and tested

Final prototype of TFC Switch built and tested

Final complete prototype of Readout Supervisor ready for production

General purpose test board “FREJA” is being designed

Self-checking scan of the functionality by producing stimuli and receiving the output of the

TFC system like a FE

13

CERN

Richard JacobssonReal Time conference, Montreal May 18 – 23, 2003

10U

20U

30U

40U

50U TTCmi

Readout Supervisors

(Optical tree couplers)

TTCtx’s

Switches

Status patch panel

GPS receiver