LHC CMS Detector Upgrade Project Level-1 Trigger Update Wesley Smith, U. Wisconsin Project Management Group March 25, 2015 25 Mar. 2015, Wesley Smith PMG:

LHC CMSDetectorUpgrade

Project

PMG: Trigger Update 1

Level-1 Trigger Update

Wesley Smith, U. Wisconsin

Project Management Group

March 25, 2015

25 Mar. 2015, Wesley Smith


Project


Install fully paralleland higher bandwidthoptical path for CSC New CSC Muon Port

Cards mezzaninecards installed in cavern during LS1oPrerequisite done by

Operations Program

Alleviates bottleneck & sends all segments from each CSC (robustness to PU and collimated signals)

Build up new Track Finder in 2015 and commission in parallel, ready by 2016

Muon Trigger Transition


Old systemNew system

Cavern

Counting Room


Project

PMG: Trigger Update

EMU Trigger UpgradeCavern Counting Room

μTCA: Advanced Mezzanine Cards fromTelecommunications Computing Architecture(commercial telecommunications hardware)


MPC-EMUTFOptical Fibers*(Rice)

Muon Port CardMezzanine* (Rice)

Endcap MuonTrack-Finder(U. Florida)

EMUTF Infrastructure(U. Florida) Muon Sorter (Rice)

Trig

ge

r M

oth

erB

oa

rd

Da

ta M

oth

erB

oa

rd

Clo

ck&

Co

ntr

ol

Sector Processors

Clo

ck &

Co

ntr

ol

Mu

on

So

rte

r

*M&S on FY13 Ops. Prog.

VME

MTF7

(MP7)


Project


Muon Port Card (Ops)(Rice - M&S on Operations Program)


Use the existing MPC main board Backplane interface to TMB

remains unchanged

x 60 +20 spares

3 original optical links are still available

New mezzanine card with new FPGA and new links

Production done, tested, installed

(60 needed + 15 spares + 5 test setups = 80)

http://bonner-ntserver.rice.edu/cms/mpcs6.jpg


Project


EMUTF: Hardware Layout

MPC Mezzanine Upgrade

MTF7 μTCA Board

OpticalPatch Panel



Project

PMG: Trigger Update

Require three μTCA chasses to process 12 sectors + sort output

MTF7 occupies 2 μTCA slots, 12 units in system, each covers 60o sector

Muon Sorter also built on MTF7,1 unit in system use Boston U AMC13 for clocking and DAQ

Re-route and split optical signals using 12 passive optical patch panels

EMUTF Hardware Counts

6

SPSP SP SP SP

SPSP SP SP SP

SP SPMS

Chassis #1

Chassis #2

Chassis #3



Project

7

EMU Track-Finder(U. Florida)

25 Mar. 2015, Wesley Smith PMG: Trigger Update

Back: Core FPGA card ⇒

⇐ Front:Optics card

Muon Track Finder card: MTF713 needed + 4 spares + 3 test setups = 20Optimized for maximum input from muondetectors (84 input links, 28 output links)

Dual card w/large capacity for RAM (~1GB)to be used for pT assignment in track finding

• Development and testing complete, production started


Project

PMG: Trigger Update

MTF7 Hardware Prototypes @ CERN

8

Details of board hardware in supporting slides

Base Board [pT LUT mezzanine

not shown]Optical Board

Pass-through Backplane Board



Project


HW Prototypes: Testing @ Pt 5 Installed equipment:

Two μTCA chassis, MCH in each One complete MTF7 kit

[Base board + Optical board] One AMC13 Control computer PCIe connection to both chassis One EMU sector optical patch panel

Status: AMC13 clocking: OK PCIe access: OK MPC link tests:

o PRBS using IBERTo Both Endcaps – no errorso Signals going via 2-way and 4-way splitters

Confident in our ability operate the old trigger path with the new MPC mezzanine

Upgraded MPCs have participated in global runs since the last week of October, and in the extended cosmic run

Fulfilled “Do No Harm” Principle25 Mar. 2015, Wesley Smith


Project


Hardware Production Status

MPC Mezzanine Upgrade: 85 upgraded MPC mezzanines have been built and tested at Rice in the

summer of 2014 73 Upgraded MPCs are at CERN - 60 MPCs installed at P5, 13 spares

MTF7 Base Board: 12 base PCBs received, 24 more coming soon Initial 4 MTF7 base boards submitted for assembly, coming back next week Full quantity assembly after tests:

20 boards will contain FPGAs, the other 16 without FPGAs at this time

MTF7 Optical Board: 36 boards received from production 4 pcs initial assembly sent out [Mar 11th], coming back next week Full quantity assembly after tests

pT LUT module Production prototype under tests Read latency reduced as expected (2 BX instead of 6 BX in first prototype) Firmware under development now



Project


Commissioning Status Readout of trigger primitives from MPCs:

Using simplified logic for now One spy buffer Records 7 BXs on each L1A Once spy buffer full, stop recording,

read out Start recording again Trigger primitives decoded on the fly

in test software Valid primitives dumped into text file

Example of trigger primitive dump of cosmic muons:l1a: 0049 e: 023 m: 1 f: 5 hs: 076 wg: 004 q: 14 p: 09 lr: 1

l1a: 0053 e: 093 m: 4 f: 7 hs: 083 wg: 029 q: 13 p: 07 lr: 1l1a: 0063 e: 072 m: 3 f: 3 hs: 048 wg: 061 q: 15 p: 10 lr: 0l1a: 0070 e: 030 m: 3 f: 2 hs: 138 wg: 051 q: 15 p: 10 lr: 0 e: 030 m: 3 f: 2 hs: 084 wg: 049 q: 14 p: 08 lr: 0l1a: 007a e: 100 m: 3 f: 3 hs: 011 wg: 060 q: 12 p: 04 lr: 0l1a: 007b e: 107 m: 4 f: 2 hs: 015 wg: 021 q: 11 p: 02 lr: 0l1a: 0086 e: 051 m: 4 f: 7 hs: 126 wg: 051 q: 11 p: 03 lr: 1l1a: 0090 e: 121 m: 3 f: 5 hs: 098 wg: 060 q: 14 p: 08 lr: 0l1a: 0096 e: 030 m: 3 f: 6 hs: 141 wg: 055 q: 14 p: 08 lr: 0l1a: 009b e: 065 m: 2 f: 7 hs: 084 wg: 031 q: 12 p: 04 lr: 0l1a: 00a0 e: 093 m: 4 f: 0 hs: 032 wg: 007 q: 14 p: 08 lr: 0l1a: 00a6 e: 002 m: 4 f: 7 hs: 060 wg: 017 q: 11 p: 02 lr: 0l1a: 00ad e: 051 m: 4 f: 4 hs: 026 wg: 038 q: 12 p: 04 lr: 0 e: 051 m: 4 f: 4 hs: 026 wg: 035 q: 12 p: 04 lr: 0



Project

PMG: Trigger Update

Preliminary results, double-checking Will be used to generate “low luminosity” version of LUT

Algorithms



Project


Bitwise emulator(s): Signal emulation generated from Verilog source Human-written code following algorithm specification Compare HW vs Signal Emulation: HW is executing FW correctly Compare HW/Signal Emulation vs Human-written: FW is doing

what we think it is doing Human-written emulator implemented in CMSSW

[development branch]

Emulator updated to have option for 30 bit pT LUT Implementation currently in GitHub assumes full precision values

using at most 6 input parameters for each mode Including configuration parameter to select corridors These new features are undergoing debugging Expect to have code to generate corresponding binary files soon

Offline Software [Emulators]



Project


MTF7 hardware is different from CSCTF

New firmware needed for MTF7 [similar components] Loading 1 GB of memory content via PCI express New track finding algorithms better suited for occupancies of up to

18 hits per station, more robust against collimated muons Reading LUT memory content 4x per LHC clock cycle Post-LUT corrections (“tail clipping”) only has SW implementation Possibility of MVA evaluation – studying resource optimization Design, definition, monitoring of spy buffers Importing and inclusion of RPC hits in track finding / pT assignment

Readiness: taking cosmic runs with one sector with final pT LUT firmware [6 BX → 2 BX latency] will have full data

processing chain, will be ready to produce L1 muon objects

Firmware Status



Project


Data format + Packer / Unpacker: 1st version exists, awaiting integration

Run Control Framework: 0th version exists: no-op hooks based on legacy endcap

trigger run control framework

Online SW development is currently under-manned Have support from USCMS to hire SW expert for this task Hiring online SW expert was delayed [various issues] Rebooting search with high priority

Online Software



Project


Calo. Trigger Upgrade in Parallel: Split inputs from ECAL and HCAL

HCALEnergy(HTR)

ECALEnergy(TCC)

Regional Calo Trigger

Global Calo Trigger

EMcandidates

Regionenergies

HFEnergy(μHTR)

HCALEnergy(μHTR)

Layer 1Calo Trigger

Layer 2 Calo TriggerC

urre

nt L

1 T

rigge

r S

yste

m

Upg

rade

L1

Trig

ger

Sys

tem

oSLB

oRM

ECAL: optical Serial Link Board (OSLB) and optical Receiver Mezzanines (oRM) connect to Present and Upgrade Calorimeter Trigger (*Details of evolution in breakout).o All installed and operating in CMS Global Runs

HCAL: optical splitters drive both HTRs and μHTRs25 Mar. 2015, Wesley Smith

US

UK

US

UK

HCAL OpticalSplitters

2016: HCAL uHTR & assoc. Cal. Trigger Cards

(Cluster Finding)

(Object Finding)

2015: ECAL, Legacy RCT & associatedCal. Trigger Cards*


Project


Upgrade Calorimeter TriggerLayer 1 and 2 Hardware

US:

UK:


Vienna:

(Cluster Finding)

(Object Finding)


Project


Stage-1 Upgrade in 2015 (ops)


RCT 0 RCT 1 RCT 2 RCT 17…

oRSC 0 oRSC 1 oRSC 2 … oRSC 17

Existing RCT

18 crates

MP7 (new)Algorithm Card(s) (1-3)

Imperial HW + Core FW/SWUSCMS Algorithm FW/SW

CTP7 (new)Single Readout Card

In Situ RCT Test / Monitor USCMS HW/FW/SW

Global Trigger (existing)GT Inputs remain unchanged

DAQ – AMC 13 (new)

18 oRSCs (new)

Layer-1

Layer-2

USCMS


Project


oRSC Installation and Commissioning


Optical Regional Summary Cards are attached to eachRCT Jet Summary Card in the RCT racks and converts JSC output to optical fibers

18 cards (1 per/RCT crate). Each card outputs:3 copies of 2 fibers at 10 Gbps 1-3 MP7s1 copy of 2 fibers at 10 Gbps 1 CTP7 1 copy at 2Gbps legacy GCT

Preliminary connection/BERT tests in May 2014Final production completed summer 2014 Installation at CERN September 2014Exercised regularly and reliably for the last six months

All 18 oRSCs mounted on RCT crates

oRSC production board – U. Wisconsin


Project


Stage 1 Algorithm Firmware architecture“GCT in a chip” – J. Berryhill, FNAL team


18 10-Gbps optical fibers of RCT regions, egammas, feature bits

Products for GT and DAQ

Pipelined processing at 80 MHz with latency 10.5 bx


Project


Stage 1 Algorithm Firmware architecture“GCT in a chip” - FNAL


All algorithm features implementedAll features bit-level emulatedAll features board-tested


Project


Stage 1 trigger commissioning – FNAL, MIT, Rice


Remaining Hardware work:• Synchronization with ECAL and DAQ link commissioning in progress (w/Wisc.)Commissioning and validation:• MC patterns sent to RCT are sent through oRSCMP7 and compared with emulator• All objects have been bit-level validated with several types of MC samples • Now proceeding to more elaborate and automated pattern tests• Bit-level emulator performance validated against older emulator version for trigger

menu development the FW will deliver the expected physics performance


Project


Stage-1 Online software development


Online software has all components (RCT, oRSC, MP7) communicating with trigger supervisor for pattern testing, configuration and debugging, with appropriate GUI functionality.

To do:

DAQ functions

Automated config of MP7 via IPBUS through a Global Tag in a DB

Further online monitoring features


Project


Stage-2 Upgrade in 2016+


CTP+0 CTP+1 CTP+2 CTP+17… 18 CTP7s+h

MP7 Algorithm Card(s) (10+2)Imperial HW + FW/SW

USCMS Algorithm FW/SW

mTCA Global TriggerDAQ – AMC13

DAQ – AMC 13

Layer-1

Layer-2

CTP–0 CTP–1 CTP–2 CTP–17… 18 CTP7s–h

USCMS

Full flexibility for traditional or time-multiplexed architecture

Highest precision location of objects with dynamic clustering


Project


CTP7 Goals for Cal Trig Layer 1 Provide needed interconnect flexibility for all proposed

architectures and link speeds Maximize the power available to the Virtex-7 690T FPGA

for trigger data processing Exploit latest-available technologies

First two units delivered in Dec, 2013 Excellent results in checkout

Full Production Complete Mar ‘15, 2015 Excellent checkout results continued

o 24 total CTP7s shipped to CERN

True Triple-Link-Rate, Multi-Clock-Domain Cardo 6.4 and 4.8 Sync Rx, 10G Async Tx in same MGT quad


CTP7 Card Reminder – U. Wisconsin


Project


Production CTP7s


CTP7WithoutCXPs +heatsinks← FrontBack→

CTP7s← in Stage 2 Rack


Project


CTP7 Builds

Build Number

Time Frame

Series(PCB Rev)

Scheduled Qty

Reject/ Fail Yield Qty

1Q4/2013 Falcon

(A) 2 0 2

2Q2-Q3/

2014Raven

(B) 6 0 6

3 Q1/2015 Eagle (C) 8 0 8

4Q1-Q2/ 2015

Eagle (C) 40 0†

†Incremental result on build currently in-process



Project


CTP7 Production Steps (6)1. Solder Reflow (outside task at contract manufacturer)2. Incoming Inspection/Ohmmeter Test

• Check resistances on power rails and critical nets• Assign barcode at completion

3. Initial Mechanical Assembly• Install CXP Sockets, LED light pipes, JTAG mezzanine, front panel

4. Power On Test Suite• Program JTAG CPLD, MMC Microcontroller• Boot card in µTCA crate—verify IPMI functionality, DC supply rails,

ZYNQ booting, Ethernet connectivity, V7 load capability

5. Final Mechanical Assembly• Install heat sink, optical modules, PRIZM assembly, heat sink

cover

6. Functional Test Suite• Test MGT Links, AXI bridge, QDR memory, clock logic, ancillary I/O



Project


CTP7 Build 4 Status (As of 17-MAR-2015)

So

lder

Ref

low

Oh

mm

eter

Tes

t

Init

ial A

ssem

bly

Po

wer

-On

Su

ite

Fin

al A

ssem

bly

Fu

nct

ion

al S

uit

e

40

Un

its

Co

mp

lete

d T

hro

ug

h

Pro

ces

s S

tep

20

40 40

3028

4040

1 2 3 4 5 6Process Step



Project


CTP7 Cards at CERNCards at CERN from Builds 1-3: 12Completed Shipments from Build 4:

Shipment #1: 6 UnitsShipment #2: 6 Units

Scheduled Shipments:Shipment #3: 7 UnitsShipment #4: 7 Units

Estimate all shipments at CERN on/before 2nd Week of April, 2015



Project


Layer 1 Installation (S2E10)

ECAL/HCAL/HF Fibers enter back side of rack as individual LC connections

3U Optical Feed-thru panels pass MPO12 fibers internally through rack to front side

Rx connections made to CTP7 cards as in-line MPO12-MPO12 connections on front side

3 CTP7 uTCA Crates, each covering 120° of φ and all of η

1U Power Panel delivers 48V from existing RCT internal power supplies to crate Power Modules

CTP Crate #3

S2E10 Back Side

9×ECAL/HCAL/

HF96 LC to 8 MPO12

Patch Panel

S2E10 Front Side

2 × 7kW@48VPower Chassis

48V Power Panel

CTP Crate #1

48V Power Panel

CTP Crate #2

48V Power Panel

Fiber Feed-Thru

Fiber Feed-Thru9×ECAL/HCAL/

HF96 LC to 8 MPO12

Patch Panel

oRSC 10G Patch Panels



Project


All 4 Final CTP7 Crates installed and operating 1 for Stage 1 (rack 9) 3 for Stage 2 (rack 10)


CTP7 Crate Installation at CERN

oRSC

oRSC

LegacyRCT x9

Stg. 1Stg. 2

Stg. 2

Stg. 2

Stage 2


Project


Functionality Brings together 3 different subsystems: ECAL, HCAL, and HF Applies tower-level calibration in lookup tables Builds combined trigger tower words and streams them to Layer-2 for processing Captures inputs for DAQ readout Captures inputs and outputs at reduced rate for readout via Ethernet (for

validation) NB: All development in Vivado from the beginning

Input Features Extensive input diagnostics

o Link layer checks (reception of valid data, link speed, …)o Protocol layer checks (CRC and Hamming code verification)o Non-invasive eye-scans continuously running

Flexible FW-based input link alignment logico Allows link alignment on any combination of input linkso BC0 latency measuremento BC0 periodicity error counters

Input capture/playback o Data for 256 BXs (both 8-bit trigger tower energies and feature/fine-grain bits)o 256 BXs of data can be played out once per orbit, starting with BX=0. Or repeatedly

every 256 BXs


CTP7 2016 Trigger Firmware – IRelease V1.0 on March 9 - Milestone


Project


Layer-1 Processing Applies tower-level calibration in lookup tables

o η-corrected scaleo Run-time programmable over AXI bus

16-bit combined trigger tower word to Layer-2o 9-bit: E+H trigger tower sums, o 3-bit: Energy ratio: log(E/H), log(H/E)o 2-bits:ECAL and HCAL feature/fine-grain infoo 1-bit: E-over-H info, 1-bit: Denominator Zero Flag

Layer-1 DAQ Layer-1 implemented in 3 crates, each crate with one AMC13 with 5 Gbps output

bandwidth to DAQ (15 Gbps using 3 cards) Captures all input data (8-bit trigger tower energies + all feature/fine grain bits) at 100

kHz L1 rate within the maximum output bandwidth of 3 AMC13s:o In total, 3x 4.1 Gbps = 12.3 Gbpso Unpacked data + a few link diagnostic bits

DAQ chain fully integrated into CTP7 FW v1.0 AMC13 low-level driver core also integrated The same core that we used in October 2014 to successfully transfer data from CTP7

AMC13 FEROL (tests in B14) Core will be updated for recent BU improvements/updates/bug fixes25 Mar. 2015, Wesley Smith

CTP7 2016 Trigger Firmware – IIRelease V1.0 on March 9 - Milestone


Project


CTP7 FPGA Architecture

Uses a ZYNQ SoC device as the GbE endpoint and utility/support device, and a V7690T as the main processing device

Uses AXI as the interconnect interface (not IPbus)

AXI with an embedded processor (ARM or Microblaze) is a mainstream design flow in the Xilinx tool set

ARM processor (in ZYNQ) at head of hierarchy, AXI bridge extends connection into V7

Embedded Linux runs on the ARM Not intended to replace the control PCs Opens up new possibilities for operation and maintenance



Project


Integrated Eye Scan(ZYNQ-based Application Example)

Familiar Eye Scan diagrams available in IBERT

Scanning engine ported to CTP7 ZYNQ

Low-latency access to links via AXI interface

Parallel Eye Scans in ~same time as a single scan

Runs in parallel in CTP7 FW during data-taking



Project


Test Release Features: Linux, data access

o Peek/Poke Registers Dump/Fill Memories

Status: Done – used in May integration tests

Commissioning release Features: Client / Server TCP/IP software for RCT test & readout

o AXI memory device driver, ctp7Server, ctp7Client Status: Done – Used in July integration tests

Initial Operations and Low Luminosity Releases Features : Full support of Stage-1 use with Trigger Supervisor, Stage-2 testing Status: Command Line Interface version Done; Awaiting TS integration; Routine

use for both Stage-1 & 2

Mid/High Luminosity Releases Releases for operations in 2016 and beyond Status: In development Core: memsvc, rpcsvc done Application: CLI in develop. TS version to start in May


CTP7 Software Releases


Project


Tower level plots already available in DQM Currently using ECAL and HCAL TPGs read out from TCC/HTR Will retrofit the same code to use CTP7 DAQ when ready Occupancy, for hot-tower identification + possible inefficiencies Will add new bit-errors monitoring (TPG vs CTP7 compare)


Stage-2/Layer-1 DQM


Project

PMG: Trigger Update

Stage-2 Commissioning

39

System being built up at P5Current status - Layer 1: 22/36 CTP7s - Layer 2: 11* MP7s - Demux: 1 MP7 - μGT: 1 MP7

Layer 1

Three crates with 10 CTP7s

12 more CTP7s at CERN



Project


Muon Port Cards commissioned, operating

Endcap Muon track-Finder System in production

Endcap Muon track-Finder pre-production prototypes operating at CERN

Endcap Muon Trigger preliminary firmware and software operating at CERN

Stage-1 Calorimeter Trigger operating at CERN

Stage-1 Calorimeter Trigger firmware and software operating at CERN.

Stage-2 Calorimeter Trigger production complete

Stage-2 Calorimeter Trigger final installation and commissioning underway

Stage-2 Calorimeter Trigger operational firmware and preliminary software operating at CERN25 Mar. 2015, Wesley Smith

Trigger Upgrade Summary

Documents

LHC CMS Detector Upgrade Project Level-1 Trigger Update Wesley Smith, U. Wisconsin Project Management Group March 25, 2015 25 Mar. 2015, Wesley Smith PMG: