Design and commissioning of the ATLAS Muon Spectrometer RPC Read Out Driver TWEPP 2008 Topical Workshop on Electronics for Particle Physics Naxos (Greece)

Design and commissioning of the ATLAS Muon Spectrometer RPC Read Out Driver

TWEPP 2008 Topical Workshop on Electronics for Particle PhysicsNaxos (Greece) September 17th, 2008

Vincenzo IzzoIstituto Nazionale di Fisica Nucleare sez. di NAPOLI – ITALY

on behalf of the ATLAS collaboration

2

Summary

• The ATLAS Experiment

• The Trigger and Read-out system of the spectrometer

• The ROD board architecture

• The ROD in the DAQ commissioning

• Conclusions

3

The Large Hadron Collider and ATLAS

Proton-proton collider

ECM : 14 TeV

Interaction frequency: 40 MHz

Number of bunches: 2835

Luminosity at regime: 1034cm-2s-1

Search for Higgs boson

SUSY particles

CP violation study

Measure of transverse momentum with a resolution

pt / pt < 10 % at pt 1 TeV/c

4

The muon spectrometer

Located in the outer part of the apparatus with a cylindrical symmetry

It has an octagonal structure and is segmented in 32 sectors

Specialized detectors for precision measurements (MDT, Δx ~ 80 µm) and for trigger (RPC, Δt ~ 1 ns)

A toroidal magnetic field bends the trajectories of the charged particles

Designed to identify muons with a high (6 or 20 GeV/c) transverse momentum

L’utilizzo di costrutti

5

L1 Muon Trigger & DAQ

On detector Electronics execute the trigger algorithm at 40 MHz frequency and send information to the trigger processor, every 25 ns

On detector Electronics also elaborate and group the detector’s data. Data are grouped according to the same event number and bunch crossing parameters (Bunch Crossing ID and Level1Accept) and are stored in FIFO memories

ROD

Coincidence Matrix & PAD Logic

S-Link to

ROS(Read Out Systems)

8

RX/SL

CTP Intf

8

RX/SL

CTP Intf

L1TriggerProcessor

TTC

LHCL1A

L1A

L1A

On Detector On Detector electronicselectronics

Off Detector Off Detector electronicselectronics

In occurrence of the L1A signal, data are transferred on optical fiber from the FIFO memories to the OFF Detector electronics; here data are elaborated by the Read Out Driver (ROD) and then sent to the next acquisition levels (ROS)

FIFOsFIFOsFIFOs

TriggerTrigger

6

The ROD environment

Data arrive at the receiver boards RX-SL on optical fiber and are transmitted to the ROD via the custom backplane RODbus on a High Speed Serial Link (each RX-SL sends 48 bit@40MHz)

RODRXSL

RXSL

Timing (4) (LVDS)

Timing signals arriving at the ROD are distributed to the daughter boards over LVDS connections on the custom backplane RODbus

TTC

Data (48) & Timing (4) (LVDS)

After being elaborated and grouped in a frame, data are sent to the Read Out Buffer

MUON ROD FRAMEL1A = xxxx

SLink

PAD frame

PAD frame

32 ROD subsystems manage readout data of the whole spectrometer

7

The custom backplane RODbus

Plug-in on VME64x backplane

Central connector for ROD

Slot 2 and 4 for RX-SL

Slot 1 and 5 for µCTP Interface

Upper connectors for LVDS data (~2.2 Gbit/s)

Lower connectors TTL for controls (busy, reset, diagnostics)

Temperature sensor and ADC for power supply controls

The RODbuses have been successfully tested and are installed

µCTPI

ROD

µCTPI

RX/SL

8

The ROD board

• VME64x 6U board

• Based on XILINX VIRTEX2 FPGA technology

• FPGA internal clock: 80 MHz obtained from the board clock (40 MHz) by using a 2xDLL

• ARM7 Microcontroller

• RX SerDes : high speed serial link (~2.2 Gbit/s)

• Equipped with TTC receiver and SLink transmitter

VME FPGA RX SerDes ROD FPGA

Microcontroller

TTC Rx S-Link Transmitter

9

The ROD architecture

• The VME FPGA allows the communication with the VMEbus, with the microcontroller and with the receiver of the TTC

• The ROD FPGA is interfaced with the RODbus and with the RX-SL boards via the SerDes receivers

• The two FPGAs communicate via a custom serial protocol

• Data are sent to the Read Out Buffers through the S-Link transmitter

9

8

16

4

VME FPGA

ROD FPGA

TTCrq

S-Link

ck

SerDes32

ck

SerDes32

32

1

14

64

VM

Ebus

RO

Dbus

9

I2C

4 + 4 (LVDS)

uP

Power Supply analog

monitoring

RS232

2

10

9

8

16

4

VME FPGA

ROD FPGA

uP

TTCrq

S-Link

ck

SerDes32

ck

SerDes32

32

1

14

64

VM

Ebus

RO

Dbus

9

I2C

Power Supply analog

monitoring

RS232

2

4 + 4 (LVDS)

80MHz60MHz

40MHz

40MHz

80/2 MHz

40MHz

The ROD: clock domains

• The board is the meeting point of four asynchronous and independent clock signals:

• TTC (40 MHz, as ATLAS specification)

• S-Link (40 MHz, as ATLAS specification)

• Right SerDes (40 MHz, from right RX board)

• Left SerDes (40 MHz, from left RX board)

• FPGA Clock (80 MHz, in order to optimize performances)

• Microcontroller (as fast as possible)

• The ROD FPGA is the most complex part: the coexistence of all clock signals was made by partitioning the chip in clock domains

• To decouple the clock domains, FIFO memories have been used

11

Event Building: block diagram

FIF

O

Mux

FIF

O

Mux

FIF

O

Mux

FIF

O

Mux

FIF

O

Mux

FIF

O

S-Link interface

Event B

uilder Engine

TT

CIntf.

SerD

es Intf.

L1A, BCID

Trig. Type

L1A, BCID

Trigger TypeStrobeControls

TTC Clock

RX dataWrite EnableRX Clock


x2 DLL

40 MHz

Board Clock

40 MHz SLink Clock

Controls in

RX HeaderL1ABCIDDataRX Footer


Control Logic

Configuration Register File

Synchronous Serial Link

RODbus Interface

Serial data inSerial data out

Controls out

To S-Link

to/from VME FPGA

4k x 32

4k x 32

512 x 36

512 x 36

4k x 32

4k x 32


12

Event Building: clock domains

FIF

O

Mux

FIF

O

Mux

FIF

O

Mux

FIF

O

Mux

FIF

O

Mux

FIF

O

S-Link interface

Event B

uilder Engine

TT

CIntf.

SerD

es Intf.

L1A, BCID

Trig. Type

L1A, BCID

Trigger TypeStrobeControls

TTC Clock



x2 DLL

40 MHz

Board Clock

40 MHz SLink Clock

Controls




Control Logic

Configuration Register File

Synchronous Serial Link

RODbus Interface

Serial data inSerial data out

13

The Event Building algorithm (1)

L1A Fifo empty ?

yes

Write ROD header

no

Timeout ?

yes

no

Error handling procedure

yes

SerDes Fifos empty ?

Parse RX frame

no

continue


ROD HeaderL1A = xxxx

• The builder engine waits for a L1A signal to process data

• Then starts writing Header in the output FIFO (SLink, VME or both)

• The engine waits for data arriving from the RX boards, stored in SerDes FIFOs

• RX frames retrieved from SerDes FIFOs are parsed to find header and L1A

• If data are not available from RX boards within a programmable time window, an error handling procedure is started

14

The Event Building algorithm (2)

RX frames correct ?

yes

append RX frames




no Error handling procedure

write ROD footer

L1A Fifo empty ?

yes

no

ROD Footer

continue

• If the RX frame is correctly formatted and the embedded L1A matches the current one, it is appended to the ROD frame

• An error procedure is started elsewhere

• The ROD Frame is closed by a specific footer with keywords, word count, error flags and the elapsed time to build the frame

15

Event Building: the ROD Frame

ROD Frame Header

Left RX data

Right RX data

ROD Frame Footer

• The ROD Frame Header contains information about the run and the specific IDs of the event

ROD Muon Frame ROD HeaderROD Header

L1A IDL1A ID

Bunch Crossing IDBunch Crossing ID

Run numberRun number

Trigger typeTrigger type

………………

• The ROD Frame body contains data from the selected RX boards

• The ROD Frame Footer contains keywords, word count, error flags and the elapsed time to build the frame

Number of data wordsNumber of data words

Error wordsError words

Elapsed time to build the frameElapsed time to build the frame

16

Event Building: timing (1)


ROD HeaderL1A = xxxx


ROD Footer

• A typical ATLAS situation is shown, with RX Frames made of 80 words@32 bit

• Cursors show the time needed to assembly a ROD Frame

• The Event Builder Engine can manage an average L1A rate higher than 100 KHz

17

Event Building: timing (2)

• Timing information written in the ROD Frame Footer allows us to perform a real time analysis on the elapsed time to build the frame

• This has proved to be a very useful tool for fine tuning (average time, timeout occurrence…)

• Only 1 RX selected • 2 RX selected

18

Event Building: timeout errors

RX HeaderL1ABCIDData.....................

.......

.......

.......DataRX Footer

Timeout triggers here

Empty Fifo

SerDes Fifo Empty Timeout window

RX HeaderL1ABCIDData.......DataRX Footer

Skipped data

No data received in this time

frame

• A TIMEOUT error occurs when a SerDes FIFO is empty during the event building procedure

• After TIMEOUT error the Event Building stops and an error flag is asserted in the ROD Frame

• Event Builder realigns on next available Header

• The Timeout window is software programmable

19

Read-Out System

• A ROS unit is implemented with a 3.4 GHz PC housing 4 custom PCI-x cards (ROBIN)

• Each ROBIN implements 3 readout buffers

• ROS units contain 12 R/O buffers 150 units needed for ATLAS

All the RODs, of all the sub-detectors, send their data to the ROS

20

Commissioning the ROD

• The installation of the ROD boards (Hardware, cabling and signal testing) has been completed in March 2008

• The ROD boards have been successfully used in the commissioning phase with cosmic rays

• On August 2008, with random triggers, a Level1 rate of 100 KHz has been reached

• All the RODs have been used in the DAQ for the first LHC event seen in ATLAS, on September 10th

21

The ROD microcontroller

• The ARM7 microcontroller allows us to monitor the power supply on the ROD board, on the backplane and to access via I2C to all the TTCrx registers

• Microcontroller data can be read by RS232 and by VME

22

Future development

• An embedded Microprocessor, that can be downloaded in the ROD FPGA, is under development in our laboratories in Napoli

• The Xilinx MicroBlaze© embedded microprocessor has been specifically designed in order to control the Event Builder performances

Profiling of the status of the Event Builder FSM can be filled and monitored

Moreover, the Event building time can be monitored in real time

• The Microprocessor doubles the occupied resources leaving about 35% resources free

23

Conclusions

We presented the design of the processor that performs the Event Building of data of the RPC detectors of the muon spectrometer

The board architecture and some of the board features have been presented

The project was not limited to the design of a single board. Instead we built a whole environment and we are presently taking care of its integration in the ATLAS DAQ system

All the Read Out Driver boards have been installed in the ATLAS DAQ system and have been successfully used in the ATLAS DAQ commissioning phase

New monitoring features, based on Xilinx MicroBlaze© embedded microprocessor, are presently being developed

24

25

Backup slides

26

The trigger architecture

A selective but efficient trigger is needed for the ATLAS expected background: 1 Higgs event every 1013 produced events

Three trigger levels in order to reduce the rate from ~1GHz to ~200Hz

The first level must identify and flag (with the L1Accept signal) an interesting event and associate the bunch crossing ID

The latency is 2.5 μs: during this time data are stored in FIFO memories

Built with programmable logic devices (FPGA) and ASIC at the bunch crossing frequency (40 MHz)

27

EB

L2

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s

~10 ms

Calo MuTrChOther detectors

Read-Out SystemsL2P L2N

RoI

RoI data (~2%)

RoI requests

L2 accept (~3.5 kHz)

SFO

L1 accept (100 kHz)

40 MHz

40 MHz

100 kHz

~3.5 kHz

~ 200 Hz

160 GB/s

~ 300 MB/s

~3+6 GB/s

EFEFP

~ sec

EF accept (~0.2 kHz)

ROD ROD ROD

ROB ROB ROB

ARCHITECTURE(Functional elements)

SFI

EBN

Event Builder

EFN

DFM

L2SVROIB

Event Filter

Level 2

Sub-Farm OutputEvent Filter Processors

Event Filter Network

B.Gorini, The ATLAS Trigger and Data Acquisition - Architecture & Status, CHEP 06, Mumbai 13-19 Feb. 2006B.Gorini, The ATLAS Trigger and Data Acquisition - Architecture & Status, CHEP 06, Mumbai 13-19 Feb. 2006

28

Read Out System

ROS units contain 12 R/O buffers 150 units needed for ATLAS

A ROS unit is implemented with a 3.4 GHz PC housing 4 custom PCI-x cards (ROBIN)

Each ROBIN implements 3 readout buffers

B.Gorini, The ATLAS Trigger and Data Acquisition - Architecture & Status, CHEP 06, Mumbai 13-19 Feb. 2006B.Gorini, The ATLAS Trigger and Data Acquisition - Architecture & Status, CHEP 06, Mumbai 13-19 Feb. 2006

29

The timing of ATLAS with LHC

ATLAS is a synchronous system working at 40 MHz, the LHC interaction frequency.The clock signal is transmitted to the whole apparatus on a specific optical system (TTC)

In order to identify correctly an event, a number that identifies the bunch crossing (BCID) must be associated univocally to the event

The TTC transmits to the whole apparatus the 40 MHz clock signal, the BCID and the L1A signal, Level1Accept, generated by the Level 1 Trigger Processor to validate data

30

Event Building: syntax error

.........

........BCIDData.......DataRX Footer


RX HeaderL1ABCIDData.....................DataRX Footer

Skipped data


RX HeaderL1ABCIDData.......Data.........


Skipped data

• JUMBO Frame: frames greater than the maximum allowed length Maxlength. Realignment at next Header. Error Flag

• RX Frame incomplete: RX Frame Footer missing. Realignment at next Header. Error Flag

• RX Frame corrupted: RX Frame Header missing. Search for next valid Header

31

The ROD: the bandwidth

• Input data maximum length: 2 × (3+4(sector_logic) + 7 ( 4 + (8 × (3 + n_hit)))) = 706 16-bit words

• Input data maximum rate: 100 KHz

• Output data maximum allowed rate: 40 MHz

• Input data bandwidth: ~ 1.13 Gbit/s

• Output data bandwidth: 1.280 Gbit/s

• Input data typical length: 2 × (3 + 2 + 2 (4 + 4 ( 3 + 2 ))) < 120 16-bit words

• Input data maximum rate: 100 KHz

• Input data bandwidth: ~ 192 Mbit/s

• Green=PAD frame • Pink=RX frame• Blue=CM frame

32

Tuning the ROD in ATLAS

• Some of the default parameters had to be tuned in the ATLAS DAQ system

• Timeout window has been modified in order to span over eight orders of magnitude, from a 12.5 ns thick up to 2 s (used for “in lab” debug purposes)

• Run configuration parameters were added to the status word in the ROD footer, for monitoring and debug purposes

• ROD monitoring features allowed us to see a BCID misalignment, due to a wrong offset

33

The data path

9

8

16

4

VME FPGA

ROD FPGA

uP

TTCrq

S-Link

ck

SerDes32

ck

SerDes32

32

1

14

64

VM

Ebus

RO

Dbus

9

I2C

Power Supply analog

monitoring

RS232

2

4 + 4 (LVDS)




• TTC managed via I2C protocol by the microcontroller

• Timing signals (clock, L1A, synchronization) are received by the ROD FPGA and are distributed on the RODbus

• RX/SL frames arrive at the ROD FPGA via RODbus, through the SerDes, and are used to build the ROD frame

• Event monitoring and system status can be checked via VME

34

Event Building: clock multiplexers

DLL

DLL Feedback

40 MHz

80 MHz clock

TTC clock

SerDes clock

SerDes clock

Clock S

witching L

ogic

TTC domain

SerDes domain

SerDes domain

• The board can be tested and used “stand-alone”, even without TTC and RX/SL modules

• Clock multiplexers allow us to feed all the domains with the 80 MHz clock

• Xilinx FPGA modules (BUFGMUX) guarantee glitch-less operations

• Clock sources can be selected via software

• TTC and SerDes clock domains have been successfully tested @ 80 MHz, even if their working frequency in ATLAS is 40 MHz

35

The SerDes

• The chip-set serialize 48 bit on 8 serial channels

• Based on National Semiconductors DS90CR483-484 chip-set

• The bandwidth over each line is 280 Mbit/s

Documents

Design and commissioning of the ATLAS Muon Spectrometer RPC Read Out Driver TWEPP 2008 Topical Workshop on Electronics for Particle Physics Naxos (Greece)