ATL

-MU

ON

-PR

OC

-201

5-01

927

Nov

embe

r20

15

Level-1 Data Driver Card of the ATLAS New SmallWheel Upgrade

Panagiotis GkountoumisNational Technical University of Athens, Brookhaven National Laboratory

Email: panagiotis.gkountoumis@cern.chOn behalf of the ATLAS Muon Collaboration

Abstract—The Level-1 Data Driver Card (L1DDC) will be1

fabricated for the needs of the future upgrades of the ATLAS ex-2

periment at CERN. Specifically, these upgrades will be performed3

in the innermost stations of the muon spectrometer end-caps.4

The L1DDC board is a high speed aggregator board capable of5

communicating with a large number of front-end electronics. It6

collects the Level-1 along with monitoring data and transmits7

them to a network interface through a single bidirectional fibre8

link. Finally, the L1DDC board distributes trigger, time and9

configuration data coming from the network interface to the10

front-end boards.11

This paper describes the overall scheme of the data acquisition12

process and especially the L1DDC board for the upgrade of the13

New Small Wheel. Finally, the electronics layout on the chamber14

is also mentioned.15

I. INTRODUCTION16

The ATLAS New Small Wheel (NSW) upgrade is moti-17

vated by the high background radiation expected during Run-18

3 (2020) and ultimately luminosity of 7 × 1034 cm−2s−119

in HL-LHC (2025). The number of interactions per bunch-20

crossing (each 25 ns) will be increased up to 140 resulting21

in a dramatically large amount of produced data. In the22

ATLAS experiment [1] the present muon Small Wheels will be23

replaced by the NSW. The NSW is a set of precision tracking24

and trigger detectors able to work at high rates with excellent25

real-time spatial and time resolution. The new detectors consist26

of the resistive Micromegas (MM) and the small Strip Thin27

Gap Chambers (sTGC) [2].28

Furthermore, a radiation dose up to 1700Gy (inner radius)29

and a magnetic field up to 0.4T in the end cap region, create30

a hostile environment for the electronic components. To read31

out the high number of electronics channels and in order to32

survive in such a harsh environment new electronics must be33

fabricated and installed. In addition, correction mechanisms34

for Single Event Upsets (SEU - this is a change of state caused35

by a high-energy particle strike to a micro-electronic device)36

must be implemented to assure the integrity of the transmitted37

data.38

The L1DDC is an intermediate board that aggregates and39

transmits the Level-1 data (time, charge and strip address40

corresponding to a single hit) from multiple front-end (FE)41

boards to a network called Front End LInk eXchange (FELIX).42

This is achieved using a high speed serializer/deserializer43

Aplication Specific Integrated Circuit (ASIC) called GigaBit44

Transceiver (GBTX) [3] developed at CERN. In general, the45

L1DDC combines three distinct paths: Timing, Trigger and46

Control (TTC) [4] data, Data Acquisition and Slow Control47

information, into one bidirectional optical link at a rate of48

4.8Gbps, as shown in Figure 1.49

Fig. 1: On the left side the L1DDC, which resides on thedetector and is implemented with custom ASICs, combines thethree district paths (Timing and Trigger, DAQ, Slow Control)into a single bidirectional optical link.

II. CONNECTIVITY50

The L1DDC resides on the detector and interfaces with the51

FEs and for the MM case also with the Address in Real Time52

(ART) Data Driver Card (ADDC) [5] boards. The FE boards53

contain an ASIC called VMM [6] which provides trigger54

and tracking primitives for 64 channels. A second ASIC, the55

Read Out Controller (ROC) will aggregate, process and format56

the data generated by the VMM FE chips. The ROC will57

concentrate the Level-0 data streams from up to 8 VMMs,58

will filter the data based on the Level-1 Bunch Crossing ID59

(BCID) and transmit the data to the L1DDC through four serial60

streams, each capable of up to 320 Mbps data transmission.61

Moreover, ROC receives the TTC data and the Level-1 trigger62

from the L1DDC. For configuring and monitoring the VMMs63

a third ASIC called the Slow Control Adapter (SCA) will be64

used on the FEs. The ADDC boards collect directly from the65

VMMs the ART data which provide the strip information for66

the first hit in the MM and sTGC detectors.67

Because of the different characteristics of both detector tech-68

nologies, different FE boards will be fabricated for MM and69

sTGC detectors. For the MM detectors 8 FE (MMFE8) [7] will70

be connected to one L1DDC contrary to the sTGC detectors71

where only three FE will be connected to one L1DDC.72

On the other direction L1DDC interfaces with the off-73

detector electronics and especially with the FELIX interface.74

This connection is made through a CERN custom made optical75

transceiver called Versatile Transceiver (VTRx) as shown in76

Figure 2.77

Fig. 2: Schema of NSW electronics trigger and dataflow. TheL1DDC board is connected to the ROC and the SCA ASIC ofthe FE boards and also to the ADDC board. Depending on thedetector technology L1DDC is connected with eight FEs forthe MM and with three for the sTGC detectors. The L1DDCalso interfaces with the off-detector electronics and especiallywith the FELIX interface through a bidirectional fibre.

III. FUNCTIONALITY78

The GBTX ASIC and subsequently the L1DDC is capable79

of multiplexing a number of serial links (e-links) to a single80

fibre. One e-link, consists of three differential pairs (6 wires)81

being the clock (Clk+ and Clk-), the data in (Din+ and Din-)82

and the data out (Dout+ and Dout-). On the transmitting side83

data and clock have the same relative phase [3]. The GBTX84

ASIC can support up to 40 e-links divided in five groups called85

banks, as shown in Figure 3. Each bank can support up to eight86

e-links at 80Mbps, four e-links at 160Mbps or two e-links87

at 320Mbps. The transmitting data use the Double Data Rate88

(DDR) signaling.89

Fig. 3: GBTX e-link connectivity. There are five banks whichsupport up to 40 FE boards. In each bank eight FE boardscan be connected at 80Mbps, four at 160Mbps or two at320Mbps. An extra Slow Control e-link with a fixed rate at80Mbps is used for the connection to the ADDC board.

For the interconnection of the on-detector electronics the90

mini Serial Attached SCSI (Small Computer System Interface)91

(miniSAS) cables will be used. Specifically, for the MM92

detectors the L1DDC is connected to the FE board through93

a single miniSAS cable that carries two e-links. The one e-94

link is connected to the ROC ASIC and the other e-link to the95

SCA ASIC, as shown in Figure 2.96

Data rate simulations showed that for the inner portions of97

the MM detector the bit rate will exceed 320Mbps. For this98

reason, for the inner two FE boards, a special configuration99

scheme will be implemented. One e-link with 320Mbps and100

one e-link with 160Mbps are connected to the inner two FE101

boards, resulting in a sum of 480Mbps each. The next four FE102

boards have a bit rate of 320Mbps and finally the outer 2 FE103

boards, have a bit rate of 160Mbps. With this configuration the104

fifth spare bank of the GBTX is used for the communication105

with the SCA ASICs of each MMFE at the 80Mbps data rate.106

The GBTX ASIC has an extra Slow Control (SC) e-link with a107

fixed rate at 80Mbps for slow control information. This extra108

e-link will be used for the connection to the ADDC boards.109

The L1DDC provides configuration data, clocks and the110

Bunch Crossing Reset (BCR) signal to the ADDC board.111

The communication between the ADDC and L1DDC is done112

through one e-link for the configuration data and four extra113

differential pairs for the clocks and the BCR signals.114

A. L1DDC board description115

Because of the different FE characteristics, different L1DDC116

boards will be fabricated for MM and sTGC detectors. Both117

boards will use the same components, with the difference that118

in MM detectors nine connectors will be used (eight for the119

FEs and one for the ADDC) contrary to the sTGC detectors120

where only three connectors will be used. The size of the121

L1DDC board for the MM detectors will be 200mm in length,122

50mm in width and 18mm in height as shown in Figure 4.123

Contrary, the size of the L1DDC for the sTGC detectors will124

be 150mm in length, 50mm in width and 18mm in height125

as shown in Figure 5.126

(a) Top side of the board: The nine miniSAS connectors, the power connector,the shields of the FEAST DC-DC converters and the VTRX optical transceiverare visible.

(b) Bottom side of the board: The GBTX ASIC and the position pins are visible.The VTRx and the DC-DC converters will be cooled from the bottom side ofthe board.

Fig. 4: A 3D representation of the L1DDC board for the MMdetectors

As mentioned before, the miniSAS cables will be used for127

the connection of the on-detector electronics. The high flexi-128

bility of these cables which can support up to ten differential129

pairs and the small size of the 36 position connectors makes130

them suitable for the boards.131

(a) Top side of the board: the three miniSAS connectors, the shields of theFEAST DC-DC, the power connector and the VTRX transceiver are visible.

(b) Bottom side of the board: the GBTX ASIC is visible. The FEAST ASICand the VTRx will be attached to the cooling channel from the bottom sideof the board.

Fig. 5: A 3D representation of the L1DDC for the sTGCdetectors

B. ASIC description132

The GBTX is a full radiation tolerant ASIC fabricated using133

the 130 nm technology. Its power supply is 1.5V and its power134

consumption is 2.2W in full operation. E-links use Scalable135

Low-Voltage Signalling (SLVS) for 400mV (SLVS-400) [8].136

The SLVS is a differential standard with a swing of 200mV,137

centred on 0.2V. The GBTX ASIC has a Clock and Data138

Recovery (CDR) circuit which receives high speed serial data139

from the VTRx. It recovers and generates an appropriate high140

speed clock to correctly sample the incoming data stream. The141

serial data is then de-serialized and then decoded, with appro-142

priate error corrections, and finally DeSCRambled (DSCR).143

In the transmitter part the data are SCRambled (SCR), to144

obtain DC balance, and then encoded with a Forward Error145

Correction (FEC) code before being serialized and sent to the146

optical transceiver.147

The GBTX has registers for permanent storage that are148

called e-fuses. Initial configuration information is taken from149

the e-fuses, which can then be modified via the optical link150

itself or via an I2C slave interface. Finally, GBTX has a Joint151

Test Action Group (JTAG) interface for boundary scan [3].152

The VTRx optical transceiver consist of two radiation153

tolerant ASICs: the GigaBit TransImpendance Amplifier (GB-154

TIA) [9] [10] and the GigaBit Laser Diode (GBLD) [11].155

The GBTIA has a bit rate of 5Gbps (min) and a total156

jitter smaller than 40 ps. Its supply voltage is 2.5V and its157

power consumption is 250mW. The GBLD is also a radiation158

tolerant ASIC fabricated also in 130 nm. It has also a bit rate159

of 5Gb/s or more, supply voltage of 2.5V and its power160

consumption is about 325mW [11]. The VTRx is the largest161

component on the L1DDC board with a width of 45.3mm, a162

length of 14.5mm and a height of 10mm.163

According to the detector power distribution scheme the164

FEAST DC-DC converter [12] is used. This is also a custom165

DC-DC converter fabricated at CERN and has an input voltage166

range from 5V to 12V, 4A load capacity and achieves a 76%167

efficiency. It contains a radiation tolerant ASIC with total ion-168

izing dose up to 200Mrad (Si) and displacement damage up to169

5× 1014 n/cm2. The FEAST has been designed for operation170

in a strong magnetic field in excess of 40, 000Gauss (4T),171

and has been optimized for air-core inductors of 400−500 nH.172

It’s power consumption is about 1.5W and has an extremely173

low output noise. To power the L1DDC board with the two174

appropriate voltage levels (2.5V and 1.5V), two FEAST175

devices are used. The 1.5V analog voltage for the GBTX176

PLLs is provided from the same FEAST device after filtering.177

The overall power consumption of the L1DDC is estimated to178

be 6W.179

C. Frame format180

On the fiber side the GBTX transmits frames of 120 bits181

in the interval of 25 ns (BC clock), resulting in a line rate of182

4.8Gbps. The GBTx has the ability to transmit three different183

types of frames, the the GBT frame, the 8b/10b frame and the184

wide frame.185

In the GBT frame four bits are used for the frame header (H)186

and 32 are used for FEC. So, the data transmission is limited187

to 84 bits, corresponding to a user bandwidth of 3.36Gbps.188

From the 84 bits, the four are dedicated for Slow Control (SC)189

information (Internal Control (IC) and External Control (EC)190

fields) as outlined in Figure 6.191

Fig. 6: GBTX frame format

The FEC algorithm is built by interleaving two Reed-Solomon192

RS(15,11) encoded words with 4-bit symbols, each capable of193

correcting a double symbol error. This means that a sequence194

of up to 16 consecutive corrupted bits can be corrected. Finally,195

all configuration registers inside the GBTX ASIC are fully196

protected against SEUs with triple redundant registers [3].197

In the 8b/10b encoding option no error correction and only198

very limited error detection capability is possible. Further-199

more, this mode is only available in the transmitter part of200

the GBTX and as the 8b/10b coding is DC balanced no data201

scrambling is used in this mode.202

A wide frame mode format with only scrambling is available203

also for the transmitter direction. This is an alternative mode204

for data transmission where the forward error correction205

functionality is traded off for bandwidth. In this case the206

FEC field is not present and the space taken by the FEC207

code in the GBT frame is used to transmit data. As a208

consequence, the data field increases to 112 bits resulting in209

a total user bandwidth of 4.Gbps, representing an increase of210

(112−80)/80 = 40% of available bandwidth when compared211

with the GBT frame format. However this is done at the cost212

of having no SEU error protection on the transmitted data.213

Frame bits D[111:80] are scrambled separately to maintain214

DC balance of transmitted data. In the L1DDC case the GBT215

frame format is used.216

IV. ON DETECTOR PLACEMENT217

The location of the L1DDC on the MM detectors will be in218

the center of the both sides of the wedge as shown in Figure 7.219

This provides a way of equalizing the cable length as the FE220

boards are radially placed on both sides of the detector.221

Fig. 7: MMFE, ADDC and L1DDC placement on a MMwedge (four MM planes). The L1DDC are placed on the 4thplane of the wedge

In this case, a single L1DDC serves the eight FE boards222

for the one side of the plane. There are 16 FE boards in223

every plane resulting in 64 per wedge. This means that eight224

L1DDC boards are needed for every wedge. Also, there are225

two wedges in every sector and there are 16 sectors in every226

wheel. Summarizing, 512 L1DDC are needed for the MM227

detectors and 512 for the sTGCs detectors resulting in a total228

1024 of L1DDC boards [2].229

All the connectors will be placed on the top side of the230

board and the components that dissipate heat on the bottom231

side. These components with the help of a elastic thermal foam232

will be attached to a cooling channel in order to keep the heat233

at a low level.234

V. CONCLUSION235

The L1DDC board is the intermediate board responsible236

to collect the Level-1 data and to distribute the TTC and237

Level-1 trigger to the FE electronics. It is capable to handle238

a large amount of data and is fully compliant with the HL-239

LHC rates. In addition, L1DDC is a radiation tolerant board240

equipped with SEU mechanisms in order to assure the signal241

integrity. Its dimensions are relatively small in order to fit242

between two readout panels of the MM and sTGC chambers of243

the NSW detector for the upgrade of the ATLAS experiment.244

Finally, the L1DDC board must have a high reliability as245

Fig. 8: A cross section of two MM planes. In the upperpart of the picture, the L1DDC board placed on the detectoris illustrated. On the top side of the board the miniSASconnector, the power connector, the DC-DC converters andthe VTRX are placed contrary to the bottom side of the boardwere only the GBTx ASIC is placed. These ASICs are attachedto the cooling channel with the help of an elastic thermal foam.On the left side of the cooling channel is illustrated the spacerwith the high and low voltage cables. Finally, on the bottomside of the picture the FE board, placed between the two planesis also visible.

after the installation of the NSW it will not be accessible for246

replacement.247

ACKNOWLEDGMENTS248

The present work was co-funded by the European Union249

(European Social Fund ESF) and Greek national 872250

funds through the Operational Program Education and Life-251

long Learning of the National Strategic 873 Reference252

Framework (NSRF) 2007-2013, ARISTEIA-1893-ATLAS MI-253

CROMEGAS.254

REFERENCES255

[1] ATLAS Collaboration, The ATLAS Experiment at the CERN Large256

Hadron Collider, JINST 3 S08003, August 2008, p.4-18.257

[2] ATLAS Collaboration, New Small Wheel Technical Design Report, June258

2013, p.36-38, p.46-49.259

[3] P. Moreira, J. Christiansen and K. Wyllie, GBTx Manual, ver 0.8, May260

2015, p.9-10, p.14-21261

[4] S. Ask, D. Berge, P. Borrego-Amaral, D. Caracinha, N. Ellis, P. Farthouat,262

P. Glln, S. Haas, J. Haller, P. Klofver, A. Krasznahorkay, A. Messina,263

C. Ohm, T. Pauly, M. Perantoni, H. Pessoa Lima Junior, G. Schuler,264

D. Sherman, R. Spiwoks, T. Wengler, J.M. de Seixas and R. Torga265

Teixeiraah, The ATLAS central level-1 trigger logic and TTC system,266

JINST 3 P08002, August 2008, p.2-5.267

[5] V. Polychronakos, L. Yao, ADDC Design Report for February 2015 NSW268

Electronics design reviews, ver 1.1, February 2015.269

[6] G. de Geronimo, N. Ambiar, E. Vernon, N. Felt, J. Fried, G. Iakovidis,270

S. Li, J. Mead, J. Metcalfe, V. Polychronakos, VMM: A Front End ASIC271

for the Detectors of the New Small Wheels, ver 0.5, 2012.272

[7] Preliminary MMFE-8 Specification, ver 0.5, February 2015.273

[8] JEDEC SOLID STATE TECHNOLOGY ASSOCIATION Scalable Low-274

Voltage Signaling for 400 mV (SLVS-400), October 2001, p.1-6.275

[9] M. Menouni, P.Gui and P. Moreira, The GBTIA, a 5 Gbit/s Radiation-276

Hard Optical Receiver for the SLHC Upgrades.277

[10] GBT Project, GBTIA specifications, ver 1.7, May 2008.278

[11] GBT Project, GBLD Specifications, ver 1.0, May 2015, p.3-6.279

[12] Project DC-DC, FEAST Datasheet, ver 1.0, 2014, p.1-3280