Department of Particle Physics & Astrophysics ATLAS TDAQ upgrade proposal TDAQ week at CERN Michael Huffer, [email protected] November 19, 2008

Department of Particle Physics & Astrophysics

ATLAS ATLAS TDAQTDAQ upgrade upgrade proposalproposal

TDAQ week at CERN

Michael Huffer, [email protected]

November 19, 2008


2

OutlineOutline• DAQ support for next generation HEP experiments…

– “survey the requirements and capture their commonality”• One size does not fit all…

– generic building blocks• the (Reconfigurable) Cluster Element (RCE)• the Cluster Interconnect (CI)

– industry standard packaging• ATCA

• Packaged solutions– the RCE board– the CI board

• Applicability to ATLAS (the proposal):– motivation– scope– details

• ROM (Read-Out-Module)• CIM (Cluster-Interconnect-Module)• ROC (Read-Out-Crate)

– physical footprint, scaling & performance• Summary


3

Three building block conceptsThree building block concepts

• Computational elements– must be low-cost

• $$$• footprint• power

– must support a variety of computational models

– must have both flexible and performanent I/O

• Mechanism to connect together these elements– must be low-cost– must provide low-latency/high-bandwidth

I/O – must be based on a commodity (industry)

protocol– must support a variety of interconnect

topologies• hierarchical• peer-to-peer• fan-In & fan-Out

• Packaging solution for both element & interconnect– must provide High Availability– must allow scaling– must support different physical I/O

interfaces– preferably based on a commercial standard

• The Reconfigurable Cluster Element (RCE)– employs System-On-Chip

technology (SOC)

• The Cluster Interconnect (CI)– based on 10-GE Ethernet

switching • ATCA

– Advanced Telecommunication Computing Architecture

– crate based, serial backplane


4

(Reconfigurable) Cluster Element (RCE) (Reconfigurable) Cluster Element (RCE)

• Bundled software:– bootstrap loader– Open Source kernel

(RTEMS)• POSIX compliant

interfaces• standard I/P network

stack

– exception handling support

MGTs

Core

DSP tilesCombinatoric logic

Resources

Processor450 MHZ PPC-405

512 MByte RLD-II

Boot Options Memory Subsystem Configuration

data

128 MByte Flash

Data Exchange Interface (DEI)

instruction

reset & bootstrap

options

• Class libraries (C++) provide:– DEI support– configuration

interface

• Bundled software:– GNU cross-

development environment (C & C++)

– remote (network) GDB debugger

– network console


5

ResourcesResources

• Multi-Gigabit Transceivers (MGTs)– up to 24 channels of:

• SER/DES• input/output buffering• clock recovery• 8b/10b encoder/decoder• 64b/66b encoder/decoder

– each channel can operate up to 6.5 gb/s– channels may be bound together for greater

aggregate speed• Combinatoric logic

• gates• flip-flops (block RAM)• I/O pins

• DSP support– contains up 192 Multiple-Accumulate-Add (MAC) units


6

Derived configuration - Cluster Element (CE) Derived configuration - Cluster Element (CE)

Combinatoric logic

MGTs

Core

1.0/2/5/10.0 gb/s

PGP PGPPGP PGPPGPPGP PGPPGP

Ethernet MAC Ethernet MAC

MGTs

Combinatoric logic

E0

3.125 gb/s

E1


7

The Cluster Interconnect (CI)The Cluster Interconnect (CI)

• Based on two Fulcrum FM224s– 24 port 10-GE switch– is an ASIC (packaging in 1433-ball BGA)– XAUI interface (supports multiple speeds including 100-

BaseT, 1-GE & 2.5 gb/s)– less then 24 watts at full capacity– cut-through architecture (packet ingress/egress < 200

NS)– full Layer-2 functionality (VLAN, multiple spanning tree

etc..)– configuration can be managed or unmanaged

Management bus

RCE

10-GE L2 switch10-GE L2 switch 10-GE L2 switch

Q0 Q1

Q2 Q3


8

A A clustercluster of 12 elements of 12 elements

To back-end systems To back-end systems

ClusterInterconnect

Elements

From Front-End systems

switching fabric

switching fabric


9

Why ATCA as a packaging standard?Why ATCA as a packaging standard?

• An emerging telecom standard… • Its attractive features:

– backplane & packaging available as a commercial solution– generous form factor

• 8U x 1.2” pitch– hot swap capability– well-defined environmental monitoring & control– emphasis on High Availability– external power input is low voltage DC

• allows for rack aggregation of power

• Its very attractive features:– the concept of a Rear Transition Module (RTM)

• allows all cabling to be on rear (module removal without interruption of cable plant)

• allows separation of data interface from the mechanism used to process that data

– high speed serial backplane• protocol agnostic• provision for different interconnect topologies


10

RCE board + RTM (Block diagram)RCE board + RTM (Block diagram)

MFD RCE

flashmemory

RCE

slice0

slice1

slice2

slice3

slice7

slice6

slice5

slice4

MFD

Fiber-optic transceiversP2

Payload RTM

P3

P3

E1

E1

E0

E0

base

fabric


11

RCE board + RTMRCE board + RTM

Media Carrier with flash

Media Slice controller

RCE

Zone 1(power)

Zone 2

Zone 3

transceivers

RTM


12

Cluster Interconnect board + RTM (Block diagram)Cluster Interconnect board + RTM (Block diagram)

MFD CI

Q2

P2

1-GE

10-GE XFP

XFP

Q0

Q1 Q3

XFP

XFP

XFP

XFP

XFP

XFP

XFP

Payload RTM

P3

P3

10-GE1-GE

10-GE XFP

10-GE

(fabric)

(base)

base

fabric

(fabric)

(base)


13

Cluster Interconnect board + RTMCluster Interconnect board + RTM

CI

Zone 3

Zone 1

10 GE switch

XFP

1G Ethernet

RCE

XFPRTM


14

Typical (5 slot) ATCA crateTypical (5 slot) ATCA crate

fans

CI RTM

RCE RTM

CI board

Power suppliesRCE board

Shelf manager

Front

Back


15

MotivationMotivation• Start with the premise that ROD replacement is

inevitable…– detector volume will scale upwards with luminosity– modularity of Front-End-Electronics will change

• Replacement is an opportunity to address additional concerns…– longevity of existing RODS

• long-term maintenance & support– many different ROD flavors

• difficult to capture commonality & reduce duplicated effort– ROS Ingress/Egress imbalance

• capable of almost 2 Gbytes/sec input• capable of less than 400/800 Mbytes/sec output

– scalability• each added ROD requires (on average) adding one ROS/PC

– one ROD (on average) drives two ROLs– one ROS/PC can process (roughly) 2 ROLs worth of data

• physical separation adds mechanical & operational constraints


16

Scope & the Integrated Read-Out System Scope & the Integrated Read-Out System (IROS)(IROS)

• ROD crates– RODs– crate controller– L1 distribution & control– “back-plane” boards

• ROS/PC racks– ROS/PCs– ROBins

• ROLs (between ROS & ROD)• “wires” connecting these

components

• Proposal calls out for the replacement of the IROS…– Intrinsic modularity of the scheme allows replacing a subset

Level-2 (L2) trigger Event Builder (EB)

Integrated Read Out System (IROS)

Detector Front-End Electronics

• Upstream & downstream systems would remain the same…• Proposal is constructed out of three elements:

– ROM (Read-Out-Module)• combines functionality of ROD + ROS/PC

– CIM (Cluster-Interconnect-Module)– ROC (Read-Out-Crate)


17

Read-Out-Module (ROM)Read-Out-Module (ROM)

From detector FEE

ROC backplane

Cluster Elements

P3

ROM

Rear Transition Module

P2

10-GE switch

10-GE switch

L1 fanout switch management

(X4) 3.2 gb/s

10 gb/s

(X2) 10 gb/s

(X2) 2.5 gb/s

CIM


18

Read-Out-Crate (ROC)Read-Out-Crate (ROC)

from L1

CIM


10-GE switch

P3

Backplane


switch managementL1 fanout 10-GE

switch

Shelf Management

10-GE switch

10-GE switch

P3

ROMs

CIM

To monitoring & control from L1

To L2 & Event Building

switch managementL1 fanout

(X12) 10 gb/s

(X2) 10 gb/s

(X2) 2.5 gb/s


19

IROSIROS

USA15

UX15

SDX1

IROS

Switching fabric Switching fabric

L1 trigger

Event Builder farm L2 farm

(X384) 3.2 gb/s

(X24) 10 gb/s

TileCalLAr Pixel SCT TRT MDT CSC

Inner Tracker MuonCalorimeter

Cal RPC TGC TPI CTP


20

IROS plant footprintIROS plant footprint

Detector System

Detector Subsystem

Total RODs

Total ROLs

RODcrates

The ROS

Total ROMs

Total crates

Total CIMs

ROBIns PCs

Calorimeter

LArTileCal

192 762 16 255 68 48 4 8

32 64 4 24 8 8 1 2

Inner Detector

PixelSCTTRT

132 132 9 45 12 33 3 6

92 92 8 32 8 23 2 4

96 192 10 66 18 48 4 8

MuonMDTCSC

204 204 16 70 16 51 5 10

16 16 2 6 2 4 1 1

L1

CalorimeterMuon RPCMuon TGCMUCTPICTP

14 48 1 17 5 4 1 1

32 32 4 12 4 8 1 2

24 24 4 8 2 6 1 1

1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1

Totals 8361574

76 537145

235 25 45current proposed


21

Scaling & performanceScaling & performance

Phase

Input event rates (KHz)Data size (Mbytes)

Output rates (Gbytes/s)

IROS L2 EB Event ROI L2 EB

0 75 75 2 1.5 .0162 1.2 3

1 100 100 3 3.0 .0826 8.3 9

2 100 100 6 5.8 .4884 48.8 35• Proposal scales linearly with number of ROMs…

– 2 Gbytes/sec/ROM for L2 network– .5 Gbytes/sec/ROM for Event Building network

• For plug replacement example this implies an output capacity of…– 270 Gbytes/sec for L2– 118 Gbytes/sec for Event Building

• As a comparison current system has a total output capacity of…– 116 Gbytes/sec (8 NIC channels)

• Performance requirements as a function of luminosity upgrade phase– numbers derived from Andy’s upgrade talk (TDAQ week,

May 2008)– both ROI size & number change as a function of luminosity


22

SummarySummary• SLAC is positioning itself for a new generation of DAQ…

– strategy is based on the idea of modular building blocks• inexpensive computational elements (the RCE)• interconnect mechanism (the CI)• industry standard packaging (ATCA)

– architecture is now relatively mature• both demo boards (& corresponding RTMs) are functional • RTEMS ported & operating• network stack fully tested and functional

– performance and scaling meet expectations– costs have been established (engineering scales):

• ~$1K/RCE (goal is less then $750)• ~$1K/CI (goal is less then $750)

• This is an outside view looking in (presumptuous + sometimes useful)– Initiate discussion on the bigger picture– Separate proposal abstraction from its implementation

• common substrate ROD• integration of ROD + ROL + ROBin functionality

• Inherent modularity of this scheme allows piece-meal (adiabatic) replacement– can co-exist with current system

• Leverage recent industry innovation– System-On-chip (SOC)– High speed serial transmission– Low cost, small footprint, high-speed switching (10-GE)– Packaging standardization (serial backplanes and RTM)

Documents

Department of Particle Physics & Astrophysics ATLAS TDAQ upgrade proposal TDAQ week at CERN Michael Huffer, [email protected] November 19, 2008