Upton, New York WBS 6.7.1 & 6.7.3 Global Common Module & FELIX · 09/05/2018 · Global Trigger for HL-LHC Concentrates data for full event onto single processor for analysis at

Michael Begel

WBS 6.7.1 & 6.7.3Global Common Module &

FELIXMichael Begel & George Redlinger

Level-3 Managers & CAMsBrookhaven National Laboratory

U.S. ATLAS HL-LHC Upgrade Project DOE CD-1 Director’s Review Brookhaven National Laboratory

Upton, New York May 7-9, 2018

Michael Begel

US ATLAS HL-LHC CD-1, May 7-9, 2018

Technical Details: GCM

2

Michael Begel


Global Trigger for HL-LHC

3

● ATLAS planning major upgrade of triggerand data acquisition systems for HL-LHC

■ Global Trigger is crucial element○ new functionality for TDAQ

○ strong U.S. leadership in ATLAS● M. Begel (BNL)

Global Trigger (ATLAS L3)

● Hucheng Chen (BNL) Global Common Module

● W. Fisher (MSU)Firmware Interfaces

● C. Dudley (Oregon)Trigger Framework Firmware

● S. Majewski (Oregon)Trigger Signatures

WBS 6.7.1WBS 6.7.4WBS 6.8.3

WBS 6.7.3

WBS 6.7.4WBS 6.8.2

WBS 6.6.5

Michael Begel



4

>2300 incident optical fibers9.6 to 25.8 Gb/s link speeds

several link protocols

Michael Begel



● Concentrates data for full event onto single processor for analysis at 40 MHz■ approx. 50 Tb/s into Global Trigger■ exploits data aggregation and serial-to-time multiplexing■ features common hardware

5

GCM: Global Common ModuleMUX: GCM for data aggregation & time multiplexingGEP: GCM for event processing & trigger algorithms

Mu

ltip

lexi

ng

WBS 6.7.1

Michael Begel


● Features common hardware — Global Common Module (GCM)■ different functions implemented in firmware

rather than in hardware

■ two module configurations: MUX & GEP○ differ by choice of pin-compatible FPGA

● MUX: Xilinx Ultrascale+ VU9P● GEP: Xilinx Ultrascale+ VU13P

○ 24 of each configuration

■ each GCM features two nodes○ a node is a single FPGA○ nodes operate independently

● can communicate with each other

■ SoC for command, control, mon. & readout○ Xilinx ZYNQ Ultrascale+ MPSoC


6

Mezzanine

(optional)

Processor Processor

Command & Control & Readout

Proof of principle layout for costing purposes

Michael Begel


Hardware Deliverables

7

Michael Begel


● WBS 6.7.1 BNL firmware deliverables:■ Frame

○ module operation and low-level hardware interfaces■ Detector Down-Link

○ link protocols■ Unpacking & Processing

○ rearrange data to optimize trigger processing → unpacking/repacking○ aggregate data across multiple links○ serial-to-time multiplexing

■ MUX→GEP Interface & data transport

Firmware Deliverables

8

WBS 6.8.3

WBS 6.8.3

Michael Begel


Technical Challenges and R&D

9

● High-speed high-density optical-electrical modules■ Phase-I electronics rely on Broadcom/Avago miniPODs

○ 12 simplex links up to 14 Gb/s

■ HL-LHC requires similar density but with ~25 Gb/s○ technology starting to become available in industry○ R&D required for very high-density PCB configurations○ see risk discussion for additional details

● Data aggregation and time multiplexing■ trades latency for increased bandwidth

○ asynchronous with beam clock

■ maximize throughput while maintaining stability under adverse conditions○ noise bursts, intermittent links, etc

Michael Begel


Technical Details: FELIX

10

Michael Begel


Overview: FELIX in Phase-I

11

Ethernet

Ethernet

Adapted fromF.SchreuderSlides at TWEPP17

Michael Begel


Overview: FELIX in HL-LHC

12

Ethernet

Ethernet

Adapted fromF.SchreuderSlides at TWEPP17

Michael Begel


Deliverable Overview

13

• FELIX is a system of servers and software that interface the ATLAS sub-detectors to the DAQ system. The FE electronics are connected through custom I/O cards (housed in the servers) which make up the hardware deliverable for this WBS.

US Institutes involved: ANL, BNL

• An upgrade of FELIX for Phase1▪ Higher bandwidth (~ x2)▪ Higher trigger rate (~x10)▪ Interface to the new TTC system (LTI)▪ Possibility of subsystem-specific options

(e.g. tracker calibration)

• Key features▪ 16-lane PCIe Gen-4 (~200 Gb/s)▪ Will support up to 48 links ▪ Several input protocols will be supported

• ~575 FELIX cards needed for ATLAS▪ BNL will deliver 110 cards

Michael Begel



14

From the TDAQ TDR

The ITk alone takes up ~55% of the number of FELIX cards required for ATLAS.

An early R&D task is to demonstrate that FELIX meets the requirements for the ITk.

Michael Begel



• Hardware: 220 FELIX cards▪ BNL: schematic design and PCB layout

• Firmware modules▪ ANL: interfaces to TTC and busy system

▪ BNL: interfaces for different front-ends

• Production board testing: ANL and BNL

15

Block diagrams from Phase-I FELIX prototype

BNL

ANL

Michael Begel


Technical Challenges

• Primary challenges▪ Hardware design is more complicated than Phase-I due to higher PCIe bandwidth

and higher input data rate

▪ Holding FPGA cost down while keeping up with increased data routing capability

▪ TTC-PON is a new timing system from CERN to which we must interface.

16

Michael Begel


R&D Status

• R&D Accomplishments to date▪ Pixel IBL stave readout test (FE-I4B chip via VLDB cards and GBT links) with Phase-I

FELIX completed (Nov 2017)

17

Occupancy maps from threshold scan

Michael Begel


R&D Status

• R&D Accomplishments to date▪ Achieved strip module readout (HCC130, ABC130)

with FELIX. Calibration test and gain measurements performed. (Jan 2018)

18

Distribution of thresholds before and after calibration

ABC130Gain distribution

Michael Begel


R&D Status and Plans

• R&D Goals/Plans to Pre-Construction▪ CY18: continue ITk strips and pixels readout tests with Phase-I FELIX

o We are about 50% complete on our schedule for CY18

▪ CY19-20: design and evaluation test of FELIX demonstrator board, focusing on basic functionality of hardware, firmware, software and integration with ITk.

▪ CY21-22: two iterations of prototype boards

• Key decisions▪ FPGA selection: Dec 2022

19

Michael Begel


Project Management

20

Michael Begel


WBS and Management Structure

21

Michael Begel


L3 Project Management

22

● Management Structure■ Jinlong Zhang (ANL) deputy: Michael Begel (BNL)

■ 6.7.1 L3: Michael Begel■ 6.7.1 IC: Shaochun Tang

■ 6.7.3 L3: George Redlinger■ 6.7.3 IC: Kai Chen

Michael Begel


Bio Sketch of L3 Manager

● U.S. ATLAS HL-LHC Project■ deputy L2 manager for DAQ and Data Handling (WBS 6.7)■ L3 manager and CAM for Global Common Module (WBS 6.7.1)■ EVMS training completed in July 2016

● U.S. ATLAS■ Phase-1 Program Management Group■ Joint Integration Group

● International ATLAS■ co-convener of the TDAQ Global Trigger

○ Global Common Module & Data Aggregation are key deliverables

■ collaborator on ATLAS since 2008■ member of the extended Trigger/DAQ Steering Group■ manager for the Phase-I Upgrade Global Feature Extractor (gFEX)

in Level-1 Calorimeter Trigger

23

I have been working in Trigger/DAQ since 1989 with key leadership, operations, and production roles on E706, DZero, and ATLAS.

I have served on many ATLAS review panels as well as reviewing other experiments and programs.

Michael Begel


Bio Sketch of L3 Manager

• George Redlinger, staff physicist, BNL

• Years on ATLAS: 12

• On ATLAS I have worked mainly on physics analysis till Summer 2017

• On BNL-E949 (1999-2003) I was part of a team of 3 scientists responsible for trigger and DAQ. Supervised the BNL team comprising one EE and 3 technicians.

• For the KOPIO experiment (2003-2005) proposed at BNL, I was involved in DAQ design/R&D (bench tests of PCIe and Infiniband) and was the L2 manager for DAQ for the NSF proposal.

• I have also worked on the construction of the original CDF trigger, construction/installation/commissioning of the BNL-E787 low-mass drift chamber, and built a lead-scintillator photon veto system for E787

24

Michael Begel


● BNL’s Omega Group contributions to ATLAS:■ Original Construction

○ LAr Calorimeter & Muon Detector○ Electronics, Trigger, Software○ Technical Coordination○ Leadership & Management○ System Integration○ Installation & Commissioning○ Operations

■ Phase-I Upgrade○ LAr Calorimeter electronics○ Muon ASIC and electronics○ Calorimeter trigger electronics○ Data Acquisition electronics○ Leadership & Management○ System Integration○ Installation & Commissioning

Experience of Team

25

■ HL-LHC Upgrade○ ITk Silicon Strip Tracker○ LAr Calorimeter ASIC and electronics○ Trigger & Data Acquisition electronics○ Technical Coordination○ Leadership & Management○ System Integration○ Installation & Commissioning

Michael Begel


● Omega Group has a deep pool of talent in electronics■ synergistic development of electronics for LAr & Trigger/DAQ

○ common elements and interfaces ○ engineers learn about each others boards

● weekly meeting to discuss detailed status and technical challenges

● all boards internally reviewed by group prior to

BNL Instrumentation Division review ● mitigation against single-points-of-failure

■ work closely with BNL Muon and ITk Strip efforts

■ focus on overall system optimization for integrated detector readout○ gFEX used as data source & sink for FELIX integration○ FELIX integration with ITk Pixel and Strip ASICs

● Additional engineering available from Instrumentation Division■ collaborated on Original Construction and in both Phase-I and HL-LHC Upgrades

Personnel

26

TDAQ

Phase-IgFEX

Phase-IIGCM

TDAQ

Phase-IFELIX

Phase-IIFELIX

LAr

Phase-ILTDB

Phase-IIFEB2

Michael Begel


ES&H

27

● Safety is of the highest priority within the Project■ Work at BNL adheres strictly to our ES&H policy

■ Our ES&H officers:○ periodically inspect our laboratories and offices○ provide oversight and advice○ interface with CERN ES&H

● Main hazards for this deliverable■ Electrical Benchtop

○ low voltage, soldering irons, noise, etc■ All work done in compliance with safety policies at BNL or CERN

Position Name

HL-LHC Point of Contact Achim Franz

Lori Stiegler (acting)

Omega Group Gerrit van Nieuwenhuizen

https://www.bnl.gov/esh

https://www.bnl.gov/esh

Michael Begel


Cost and Schedule

28

Michael Begel


ATLAS L1Calo gFEX ModuleArchitecture:

ATCA blade3 VU9P & 1 ZU1935 miniPODs with 420 fibers250 W with Phase-I firmware

Maximum capability:input: 3.9 Tb/s on 312 fibersoutput: 1.4 Tb/s on 108 fibersinter-FPGA: 1.6 Tb/s (high-speed & low-speed)

Basis of Estimate

29

● GCM design based on Global Feature Extractor Module (gFEX)

■ ATLAS Phase-I L1 Calorimeter Trigger Module○ full calorimeter on a single module!○ capable of 3.9 Tb/s input rate

■ conceived, designed, developed, prototyped, and produced at BNL○ collaboration of BNL, Chicago, Indiana,

Lund, Oregon, Pittsburgh, Stockholm ○ Installation & Commissioning on-going!

■ controlled by U.S. ATLAS Phase-I Upgrade Project○ two L3 and CAMs: Module & Algorithms

25.6 Gb/s in PCB

Michael Begel


Basis of Estimate

30

• Material cost and schedule estimates are based on the experience of the Phase1 FELIX project

• M&S cost is dominated by the FELIX cards▪ Estimation of cost of components, PCBs and assembly come from a recent Bill of

Materials for the pre-production of the Phase1 FELIX card▪ Choice of FPGA and optical modules is still ongoing. Cost estimate is based on the

choices for the Phase1 FELIX

• Labor for hardware, firmware and integration testing based on adjusting Phase-I numbers

▪ See BOE for details

• Cost of travel to CERN based on recent experience

Michael Begel


Resource Optimization

31

● BNL is the most efficient and cost-effective institute for DAQ■ designed and produced the enabling technologies

○ same skilled engineers contributing to Global Trigger & FELIX○ no comparable commercial PCI or ATCA modules with large number

of high-speed optical links

■ talented group of experienced hardware & firmware designers○ Phase-I gFEX, FELIX, LTDB, ADDC and many other electronics modules○ synergistic development with other ATLAS electronics○ expertise in system integration ○ additional skilled labor available from BNL Instrumentation Division if needed

■ strong collaborations with key U.S. and International institutes

■ significant project management experience for similar deliverables

Michael Begel


● Labor estimated in analogy with Phase-I gFEX module■ 1.0 FTE/year engineering of GCM hardware■ 0.5 FTE/year engineering of GCM Frame firmware■ 0.5 FTE/year engineering of Data Aggregation firmware

Effort Estimate

32

BNL is sole institute contributing to WBS 6.7.1.

0.2 FTE/year of uncosted scientist (me) supported by Research Program forL3 Management

Michael Begel


Effort Estimate

33

Uncosted labor of 0.2 FTE/year (funded by the Research Program) forL3 project management

Labor estimated in analogy with Phase-I FELIX

Michael Begel


Closing Remarks

34

Michael Begel


Closing Remarks

35

● BNL is taking a leading role in designing significant parts of the hardware and firmware of Global Trigger and FELIX■ hardware based on existing work (gFEX & FELIX) for ATLAS Phase-I TDAQ Upgrade■ schedule resource loaded & risks assessed

● Areas of risk have been identified. We have confidence in the workflow and the estimates of costs and uncertainties.

Michael Begel


Backup

36

Michael Begel



37

WBS 6.7.1 — BNLConceptual block diagram of Global Trigger firmware and software for estimation of resource needs

Documents

Upton, New York WBS 6.7.1 & 6.7.3 Global Common Module & FELIX · 09/05/2018 · Global Trigger for HL-LHC Concentrates data for full event onto single processor for analysis at