LHC CMS Detector Upgrade Project B04-3 Calorimeter Trigger Sridhara Dasu, University of Wisconsin DOE CD1 Review 26 August 2013 Sridhara Dasu, 26 August

LHC CMSDetectorUpgrade

Project

DOE CD1 Review - US CMS Upgrade - Trigger 1

B04-3 Calorimeter Trigger

Sridhara Dasu, University of Wisconsin

DOE CD1 Review

26 August 2013

Sridhara Dasu, 26 August 2013


Project


LHC Calendar 2012: Discovered the Higgs 2013-2014: Make upgrades to handle 14 TeV + 2 x 1034 cm–2 s–1

2015-2017: Discover new high energy physics phenomena using data at 14 TeV + study the Higgs + EWSB mechanism

2018-2019: Make upgrades to handle 14 TeV + 5 x 1034 cm–2 s–1

2020-2022: Program is dependent on what we find by 2018

CMS Plans 2013: Finalize trigger designs and prototyping 2014: Construct the trigger modules 2015: Early operation with pre-production modules (Stage-1) 2016-2017: Operations with fully upgraded trigger (Stage-2)

Rapid plan to ensure success in 2015-2017 run! Plans backed up by thorough prototyping that is already underway

The Grand Plan for Trigger Upgrade



Project

FPGA & Telecom technology advances Ubiquitous 10 Gbps links FPGAs with 72 rx/tx links with built in serdes

o Xilinx Vertex-7 platformo Gigantic coreso DSP units, BRAM and Embedded processors

Fiber optic componentso Avago mini-pods with 12x

Ripe time to upgrade from VME to uTCA

LHC long shutdown 2013-2014 Sufficient time for copper optical transition for inputs Ensures that upgraded electronics is ready for operation in 2016 Partial upgrade operation from 2015 itself

Sridhara Dasu, 26 August 2013DOE CD1 Review - US CMS Upgrade - Trigger 3

Technical Opportunity


Project


Calorimeter Trigger Layers 1 & 2


US:

UK:

Vienna:


Project


Algorithms

Algorithm layers First layer builds local clusters of energy deposits

o Determines energy and cluster type (EM or Hadronic) Second layer builds triggerable objects from clusters

o Electron/Photon, Tau, Jet, MET, …

Mapping to architectures Traditional architecture

o Algorithm layer-1 maps to physical layer-1 electronics– Key feature: nearest-neighbor sharing– Sufficient to build clusters locally

o Algorithm layer-2 maps to physical layer-2 electronics– All clusters available in one FPGA

Time multiplexing architectureo All primitive data passes through one FPGAo Potential for clever on-the-fly processing, if feasible/necessary



ProjectCurrent & Upgraded Calo Trigger


HCALEnergy

ECALEnergy

Regional Calo Trigger

Global Calo Trigger

EMcandidates

Regionenergies

HFEnergy

HCALEnergy

Layer 1Calo Trigger

Layer 2 Calo TriggerC

urre

nt L

1 T

rigge

r S

yste

m

Upg

rade

L1

Trig

ger

Sys

tem

HCAL Optical SplittersRisk mitigation

by operating current and

adiabatic stages of upgrade CT concurrently

CopperOptical

US

UK

US

UK

oSLB

oRM

oRSCCTP

MP7

HTR TCC mHTR mHTR


Project


Calorimeter Trigger ComponentsType Name Purpose

TCC Trigger Concentrator Card Existing ECAL trigger primitives (TP) (France)

oSLB Optical Serial Link Board Transmit ECAL TPs to RCT & CTP (Portugal / France)

mHTR HCAL Trigger & DAQ Board Create and transmit HCAL TPs

RCT Regional Calorimeter Trigger Existing trigger system

oRM Optical Receiver Modules Receive ECAL TPs on RCT(Portugal / France for Stage-1)

oRSC Optical Regional Summary Card Convert & transmit RCT output (US DOE NP for Stage-1)

CTP Compact Trigger Processor Large FPGA w/ optical & backplane(This Project)

CIOX Compact IO cross-point switch Intra-crate sharing switch(This Project)

MP7 Multipurpose Processor Large FPGA w/ only optical links (UK)


Project

DOE CD1 Review - US CMS Upgrade - Trigger

Evolution: Stage-1 Upgrade in 2015

Sridhara Dasu, 26 August 2013 8

Layer 2 Calo Trigger

HCALenergy

Regional Calo Trigger

Global Calo Trigger

HFenergy

oRSC

Layer 2Calo Trigger

oRM

ECALenergy

Layer 1 Processors

oSLB ECAL & HF ClustersHalf-tower position

4x4 E+H Clusters2x1 E+H Clusters

2-bit region ID

Layer 2 Processors

MuonJets & Sums

eGammaTau

Heavy IonSpare

RCT RO + Testing

GCT fallback remains

With just a fraction of final cards or even prototypes, derive many benefits

of full upgrade, incl. muon isolation

Uses reprogrammed RCT clusters with improved algorithms in Layer-2 Also, brings in all the power of finer grain HF using “Slice Test” of Layer-1 & 2

Improved PU subtraction, Isolation Calculation (e/γ/τ/μ) & Half-tower Position Resolution

US Items: oRSCs, oSLB/oRM commissioning CTP Layer 1 Processors RCT RO+Testing CTP

UK Items: MP7 Layer 2 Processors

36 mHTRs

2 CTP7s 9 CTP7s

Add to bring in finer grain EM clusters

2 CTP7s

4 MP7s


Project


Algorithm Studies

Stage-1 uses clusters from RCT+HF processed by new mTCA cards Adiabatic improvements to calorimeter trigger dataflow Algorithm stage takes advantage of larger FPGA resources Performance of the “revised” algorithms is very good

o eGammaTau studied by FNAL/Wisconsin o eGamma and muon isolation studied by UCSD o Muon isolation and b-jet tagging with muons studied by Ohio o Heavy Ion jet cleanup studied by MIT o Improved jet finding with PU subtraction by UIC

Further improvements using full ECAL granularity in Stage-1o Improved eGammaTau isolation & position (FNAL, Wisconsin)o Pile-up handling with lower LSB for ECAL (FNAL, Wisconsin)o Improved jet position and PU subtraction (Davis)



Project


Electron / Photon Trigger

Improved control on isolation Factor of 2-3 reduction in rate with small loss in efficiency



ProjectTau Trigger

Big improvement in efficiency with ~10X rate reduction!


Current tau trigger has large and negative PU dependence


ProjectJet Triggers


Single jet

Single jet thresholds similar to current Multi-jet trigger thresholds better in upgrade (PU sub.) Small improvement in efficiency turn-on

Quad jetHT = S PTJets


Project


Process(x2 improvement highlighted)

1.1 x 1034 cm–2 s–1 2.2 x 1034 cm–2 s–1

Current Upgrade Current Upgrade

W(en),H(bb) 57.7% 87.0% 37.5% 71.5%

W(mn),H(bb) 95.9% 100% 69.6% 97.9%

VBF H( ( )tt mt ) 42.6% 51.3% 19.4% 48.4%

VBF H( ( )tt et ) 24.4% 44.3% 14.0% 39.0%

VBF H( ( )tt tt ) 17.2% 53.7% 14.9% 50.1%

H(WW(eenn)) 91.4% 97.8% 74.2% 95.3%

H(WW(mmnn)) 99.9% 99.9% 89.3% 99.9%

H(WW(emnn)) 97.6% 99.4% 86.9% 99.3%

H(WW(menn)) 99.6% 99.5% 90.7% 99.7%

StopbWce, jets (600 – 450 GeV) 55.8% 68.2% 50.3% 64.8%

StopbWc ,m jets (600 – 450 GeV) 78.1% 81.6% 76.4% 84.5%

RPV Stopjets (200 GeV) 70.1% 99.9% 43.6% 99.9%

RPV Stopjets (300 GeV) 93.7% 99.9% 79.7% 99.9%

Physics Performance Summary

Ave

rage

Im

prov

emen

t: 1

7% (

Low

Lum

i) &

40%

(H

igh

Lum

i)



Project


Prototypes for all major cards exist & have passed tests

Upgrade is based upon common μTCA hardware platform also used by HCAL & other CMS systems

Upgrade trigger is replacing existing trigger hardware built by the same team → interfaces well understood

Same team wrote software & firmware for existing trigger system → scope & requirements well understood

Requirements are frozen in approved Trigger TDR


Technical Feasibility


Project


Wesley Smith CMS Trigger Project Manager 1994-2007, Trigger Coordinator 2007 – 2012 US CMS L2 Trigger Project Manager 1998 – present

Sridhara Dasu US CMS L3 Manager for Calorimeter Trigger, 1998 – present Author of original & upgrade cal. trig. Algorithms 1994 – present

Pam Klabbers – Operations & Testing CMS RCT Operations Manager (more than a decade on RCT project) CMS Deputy Trigger Technical Coordinator

Tom Gorski – Hardware & Firmware Over a decade of engineering on the CMS Calorimeter Trigger

Mathias Blake – Firmware New to project -- 5 years Applications Engineer at Xilinx, 2 years FPGA Engineer at DRS

Technologies, 3 years FPGA Design Engineer at Dolby Labs

Jes Tikalsky and Evan Friis – Software Providing prototype testing software (Friis has experience with current RCT)


Calo Trigger Team Experience


Project


Calorimeter Trigger WBS Detail


←MuonTrigger

M&S paid byCMS-France


Project


Upgraded HCAL Readout & Trigger Electronics (μHTR) is after split that sends data to old electronics (HTR)

μHTR supplies TP to upgrade trigger, HTR continues to send to present trigger

ECAL Trigger Concentrator Cards (TCC) are not upgraded, instead their mezzanines with trigger serial links are replaced (Lisbon):

Optical Serial Link Board oSLB replaces single copper link to current Regional Calorimeter Trigger (RCT – U. Wisc.) with two optical links (WBS 401.04.04.06)

One optical link to optical Receiver Module (oRM) on RCT (WBS 401.04.04.05)o Replaces copper Receiver Module

One optical link to input of Upgraded Calorimeter Trigger: CTP7 Card

574 oSLB’s & oRM’s with Fibers installed in 2014 French Contribution to Calorimeter Trigger (no US Upgrade Funds)

Presently oSLB & oRM in final prototype stage oSLB:

oRM:


HCAL & ECAL Trigger Primitives

OSLB Output / oRM Input: 4x channels @ 1.2 Gbps 4.8 Gbps


Project


Main processor card Virtex-7 690T

for processing ZYNQ

for TCP/IP + linux 60 10G optical

Input links 36 10G optical

Output links Function: Find ET

clusters & transmit to Layer-2

• Card Count = 46• 36 total +

8 spares + 2 test setups


401.04.04.02: CTP Card Concept

Virtex-7 VX690T FPGA

ZYNQXC7Z03

0EPP

1.5V

S

up

ply

2.5V

S

up

ply

3.3V

S

up

ply

1V

30A

S

up

ply CXP Module

12Tx + 12 Rx

CXP Module12Tx + 12Rx

CXP Module12Tx + 12 Rx

12XRx

12XRx

(CTP-6 BG View)


Project


CTP: Virtex-6 Prototype Exists


Back End FPGAXC6VHX250T/XC6VHX380T

Front End FPGAXC6VHX250T/XC6VHX380T

Avago AFBR-810B Tx Module

4X Avago AFBR-820B Rx Module

MMC Circuitry

JTAG/USB ConsoleInterface Mezzanine

Power Modules

Dual SDRAM for dedicated DAQ and TCP/IP buffering

12x Multi Gig Backplane Connections

Prototype with two V6 chips useful for most studies – limited to 6.4 Gbps IO & fewer (48) linksFinal V7 version offers more IO links and at higher 10 Gbps speed and larger logic


Project


CTP7 is simpler than CTP6 prototype Engineering effort for in-house design based on CTP6 work Procurement of PCBs is based on recent CTP6 purchase Good estimate of cost and schedule from prototype manufacture

Quotes available for all parts Virtex-7 690T is the major cost driver – with recent quote

o 3 690Ts donated by Xilinxo 5 690Ts purchased from Xilinxo Total of 8 FPGAs in-hand for prototypes

Optical modules based on recent purchase Miscellaneous parts also have quotes or estimated from

prototype purchases (power modules)


CTP: Basis of Estimate - Hardware


Project


Firmware and software efforts estimated from prototype testing + prior RCT experience

Firmware algorithms for testing provide the basis for the core firmware, which is the most complicatedo Tom Gorski and Mathias Blake

Trigger algorithm firmware work is based on prior experience managing GCT project (UK)o Wesley Smith was L1 project manager overseeing GCT

Software effort is based on current RCTo Sophistication of embedded Linux / TCP and shorter timeline for

implementation requires professional software effort– Jes Tikalsky

o Bulk of testing software from post-doc and graduate student– Pam Klabbers, Evan Friis and graduate students will develop software

Board and Firmware/Software done by the same group (Wisconsin) results in efficiencies


CTP: Basis of Estimate - Effort


Project


Function: For inter-board connections

Input/Output: 48x 4.8-6.4 Gbps

Prototype exists

Fully tested

Firmware is minimal and exists

CIOX cost well known

Ready for productionSridhara Dasu, 26 August 2013

401.04.04.03: Crosspoint IO Card– U. Wisconsin

Controller (MMC and link mgmt)

4X Avago AFBR-79EQDZ QSFP+ ModulePositions

4x4

Lan

e B

idir

ecti

on

al M

ult

i G

ig

Bac

kpla

ne

Co

nn

ecti

on

s

Backplane Rx/Tx

Redriver ICs (top and

bottom sides)

(2/crate x 3 crates = 6 + spares)


Project

DOE CD1 Review - US CMS Upgrade - Trigger 23Sridhara Dasu, 26 August 2013

401.04.04.04: CTP Infrastructure:Vadatech VT894 Crate Test Setup

BU

AM

C13

Vad

atec

h

MC

H UW

CT

P-6

UW

CT

P-6

TTC Downlink

UW

Au

x

(Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6)

U. Wisconsin designed backplane with dense card interconnectsmanufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog


Project


The dominant risks are: Availability of ALL trigger primitive inputs from ECAL, HCAL in

optical format Validation of large optical link plant in limited time Fully validated trigger algorithms in firmware Control and operations software

Risk mitigation strategy: Continue to provide fully operational current trigger system in

parallel with upgrade commissioning Partial operation of the upgrade systems provide tangible

benefits from 2015 onwards Firmware or Software issues are mitigated by provisioning

qualified additional temporary personnelo Ability to hire short term qualified staff at UW / FNAL / Elsewhere


CTP: Risk and Mitigation Strategy


Project


Goal (Original): Provide readout of original RCT descoped

during construction project Present readout through GCT input buffer

not workable with trigger evolution Uses connection to a single CTP6 prototype

or CTP7 card for DAQ readout

optical Receiver Summary Card (oRSC) paid by DOE Nuclear

VME Slave Interface Card Fits in current RCT Crates (1 per crate) 18 Cards in 2015 System Receives RCT Jet Sum Card Output

to GCT on Copper “SCSI” Data Cables Provides direct optical input to GCT

o Bypassing old optical conversion cards Planned use for Heavy Ion Triggers Prototype under test

Multiple optical outputs provide: Inputs for upgrade calorimeter trigger &

parallel operation of old & new trigger wherever ECAL & HCAL electronics notavailable in 2015


Calorimeter Trigger Risk MitigationoRSC Function:

Conversion to opticaloRSC Input:

RCT egt and jet clusters

80 MHz Parallel ECLoRSC Output:

6x copies at 6.4 Gbps optical


Project


Input/Output verified – low bit error rate – clear eyes


Prototypes Status (Optical Links)C

TP

6 to

CT

P6

(48x

6.4

Gbp

s lin

ks)

oRS

C t

o C

TP

6 (1

1x 6

.4 G

bps

links

)

oRSC to MP7(4x 6.4 Gbps links)


Project


12-links tested at 6.4 Gbps – no errors


HCAL TPG to CTP Integration Test


Project


Running Linux on CTP6 FPGA – PetaLinux

Startup screen via USB console

Root login, pinging controller PC at 192.168.1.2

Running on CTP-6 back end FPGA (`VHX250T)

Important step towards CTP-7 ZYNQ-based Linux



Project


mHTR to CTP6 integration test at Madison (done) Wisconsin/Minnesota : 12 links operated successfully

oRSC to CTP6 demonstrator at Madison (done) Test 2-3 oRSCs and interface before shipment to CERN

oRSC to CTP6 platform at Prevessin (done) 2 oRSCs emulate RCT + 1 CTP6 for RCT readout and algorithm tests Full platform for system operations and control software development

oRSC - MP7 Integration test at Prevessin (done) Wisconsin/Imperial July 2013 Milestone

CTP6 - oRSC-oRM/RCT - oSLB/TCC Integration Test at Prevessin Wisconsin/Lisbon/LLR October 2013 Milestone

CTP6 - oRSC-oRM/RCT - oSLB/TCC -uHTR Integration Test at 904 Wisconsin/Lisbon/LLR/Minnesota November 2013 Milestone

CTP7 - MP7 Integration test at Prevessin Wisconsin/Imperial March 2014 Milestone


Demonstrators & Critical Integration Tests


Project


2013 M&O + R&D Activity : oRSC construction, installation Firmware development for RCT readout and testing using CTP6 Lisbon / France : oSLB and oRM construction, installation Minnesota / Wisconsin : mHTR to CTP6 testing

2014 Ensure fully working legacy RCT system with oRSC + CTP6 RO CTP7, CIOX, … construction, validation Firmware development for CTP7 testing and 2015 operation

2015 Commission and operate Stage-1 system Firmware development and integration of Stage-2 system

2016 Commission and operate Stage-2 system


Adiabatic Upgrade Timeline


Project


CT Milestones


Activity ID Activity Name Target Completion

Milestone

T1020 L2 – Deliver Stage-1 Layer 1 to CERN 15-Nov-2014 15-May-2015

T1030 L2 – Deliver Stage-2 Layer 1 to CERN 10-Mar-2015 10-Sep-2015

T1040 L2 – Test operation at CERN 15-Aug-2015 15-Feb-2016

T1070 L2 – Test High-Luminosity Capability 28-Sep-2016 28-Mar-2017

T3310 L3 – Commission CTP7 for Initial Ops 15-Apr-2015 15-Jul-2015

T4000 L3 – Commission CIOX for Final Ops 04-Sep-2016 04-Dec-2016

T3320 L3 – Commission CTP7 for Final Ops 29-Sep-2016 29-Dec-2016

T4740 L4 – oSLB-oRM Optical Fibers Complete 15-Sep-2014

T4090 L4 – CIOX Software and Firmware Complete 10-Dec-2014

T4210 L4 – CIOX Production Complete 08-Jan-2015

T3710 L4 – CTP7 Production Complete 08-Jan-2015

T4510 L4 – CTP Infrastructure Production Complete 29-Jul-2015

T4300 L4 – CIOX Commissioning Complete 09-Sep-2015

T3980 L4 – CTP7 Test, Install and Commissioning Complete 30-Sep-2016

T3520 L4 – CTP7 Software and Firmware Complete 30-Sep-2016


Project


Calorimeter Trigger Prototypes are in good state Both CTP6 and CIOX validated CTP7 is in design stage and is based on CTP6 experience Crate infrastructure is also ready to go

Calorimeter Trigger Cost Basis is solid Hardware costs based on purchase orders or firm quotes Firmware and Software estimates are based on prior experience

o Experienced team with risk mitigation by ability to add people

Calorimeter Trigger Risk Analysis Cost and schedule risks with mitigation plans are presented

Calorimeter Trigger Team ready to get started CD1 approval will be much appreciated!


Conclusion

Documents

LHC CMS Detector Upgrade Project B04-3 Calorimeter Trigger Sridhara Dasu, University of Wisconsin DOE CD1 Review 26 August 2013 Sridhara Dasu, 26 August