LHC CMS Detector Upgrade Project P05 – Trigger Upgrade: 401.4 Wesley Smith, U. Wisconsin L2 Manager, WBS 401.4 August 26, 2013 26 August 2013, Wesley Smith

LHC CMSDetectorUpgrade

Project

CD-1 Review -- P05: Trigger Upgrade 1

P05 – Trigger Upgrade: 401.4

Wesley Smith, U. Wisconsin

L2 Manager, WBS 401.4

August 26, 2013

26 August 2013, Wesley Smith


Project

CD-1 Review -- P05: Trigger Upgrade 226 August 2013, Wesley Smith

L1 Trigger Upgrades & Inputs

ElectromagneticCalorimeter

ECAL

Hadron Calorimeter

HCAL

Endcap MuonDetectorsCSC, RPC

Not shown (off-detector)• Muon Trigger• Calorimeter Trigger

401.04


Project


Conceptual Design Trigger Overview Muon Trigger Calorimeter Trigger

Project Organization and Management ESH&Q Scope Schedule Risk Cost

Summary


Outline


Project


Conceptual Design



Project

LHC beams cross every 25 ns with ~ 23 interactions → 0.5 MB of data, but can only store ~ 1 kHz of 0.5 MB data → Trigger

Trigger rates are driven by the increase in luminosity, the center-of-mass energy, and by higher Pile-Up (PU)

Detector readout systems limit Level-1 trigger output rate to 100 kHz, latency to 4 μsec.

Mitigate by improving e/γ isolation, τ id, μ pT resolution, μ isolation, jets with PU subtraction and L1 menu sophistication

Increase system flexibility with high bandwidth optical links and large FPGAs, and standardize on μTCA telecom standard

Build and commission upgrade in parallel with current trigger system to safeguard physics, decouple from LHC schedule

Overall design: Two-layer calorimeter trigger with tower-level precision and PU subtraction Integrated muon trigger combining all CSC, DT and RPCs in track-findingCD-1 Review -- P05: Trigger Upgrade 5

L1 Trigger Upgrade – overview



Project

6

Physics priorities: Measure all Higgs BR precisely to confirm Standard Model or not ⇒

retain or improve current trigger capability critical Want to be able to answer question of naturalness - whether or not

there is new physics stabilizing Higgs masso SUSY remains a leading candidate, but if so, must have light stopso Also must be able to trigger on and search for all variants (e.g. RPV with

all hadronic final states) to draw a firm conclusion

Studied a set of benchmark physics channels Looked at performance of these channels in 2012 analyses with

and without L1 trigger upgrade at different luminositieso Based on a simplified trigger menu with a total rate < 100 kHz

Results for luminosities up to 2.2 x 1034 cm–2 s–1

Acceptance Improvements > x 2 in some channels for same rate Average Improvement 40% Results summarized in breakout sessions26 August 2013, Wesley SmithCD-1 Review -- P05: Trigger Upgrade

Upgrade Trigger Physics Performance


ProjectFinal Overall Upgrade Scheme


LHC CMSDetector Upgrade

Project

↑HCAL Upgrade↑

Trigger Upgrade

↓Trigger Upgrade↓

(Object Finding)

(Cluster Finding)


Project


Muon Trigger Transition


Install fully paralleland higher bandwidthoptical path for CSC New CSC Muon Port

Cards mezzaninecards installed in cavern during LS1o Operations Program

Alleviates bottleneck and send all segments from each CSC (robustness to PU and collimated signals)

Build up new Track Finder in 2015 and commission in parallel, ready by 2016

Old systemNew system


Project


401.04.03 EMU Trigger UpgradeCavern Counting Room

μTCA: Advanced Mezzanine Cards fromTelecommunications Computing Architecture(commercial telecommunications hardware)


401.04.03.03MPC-EMUTFOptical Fibers*(Rice)

401.04.03.02Muon Port CardMezzanine* (Rice)

401.04.03.04Endcap MuonTrack-Finder(U. Florida)

401.04.03.05EMUTF Infrastructure(U. Florida)

401.04.03.06: Muon Sorter (Rice)

Trig

ge

r M

oth

erB

oa

rd

Da

ta M

oth

erB

oa

rd

Clo

ck&

Co

ntr

ol

Sector Processors

Clo

ck &

Co

ntr

ol

Mu

on

So

rte

r

*M&S on FY13 Ops. Prog.

VME


Project

10

401.04.03.04: EMU Track-Finder(U. Florida)

26 August 2013, Wesley SmithCD-1 Review -- P05: Trigger Upgrade

Muon Track Finder processor: MTF7 13 needed + 4 spares + 3 test setups = 20 Optimized for maximum input from muon detectors

(84 input links, 24 output links) Dual card with large capacity for RAM (~1GB) to be used for pT

assignment in track finding

Current prototype based on Virtex 6 FPGAis undergoing final tests

Virtex 7 FPGA design is ~75% done, expected late 2013

Back: Core FPGA card with PT LUT mezzanine

Front: Optics card


Project


Calo. Trigger Upgrade in Parallel: Split inputs from ECAL and HCAL

HCALEnergy(HTR)

ECALEnergy(TCC)

Regional Calo Trigger

Global Calo Trigger

EMcandidates

Regionenergies

HFEnergy(μHTR)

HCALEnergy(μHTR)

Layer 1Calo Trigger

Layer 2 Calo TriggerC

urre

nt L

1 T

rigge

r S

yste

m

Upg

rade

L1

Trig

ger

Sys

tem

oSLB

oRM

ECAL: optical Serial Link Board (OSLB) and optical Receiver Mezzanines (oRM) connect to Present andUpgrade Calorimeter Trigger

HCAL: optical splitters drive both HTRs and μHTRs


US

UK

US

UK

HCAL OpticalSplitters

Stage HCAL uHTR & assoc. Cal. Trigger Cards

(Cluster Finding)

(Object Finding)


Project


Upgrade Calorimeter TriggerLayer 1 and 2 Hardware

US:

UK:


Vienna:

(Cluster Finding)

(Object Finding)


Project


401.04.04.02: Cal. Trig. ProcessorVirtex-6 Proto. Board – U. Wisconsin

Back End FPGAXC6VHX250T/XC6VHX380T

Front End FPGAXC6VHX250T/XC6VHX380T

Avago AFBR-810B Tx Module

4X Avago AFBR-820B Rx Module

MMC Circuitry

JTAG/USB ConsoleInterface Mezzanine

Power Modules

Dual SDRAM for dedicated DAQ and TCP/IP buffering

12x Multi Gig Backplane Connections


36 total + 8 spares + 2 test setups = 46

CTP6

48 optical inputs12 optical [email protected] Gbps


Project


Upgrade is based upon common μTCA hardware platform and components (Virtex-7 FPGA, multi-gigabit optical links) also used by other CMS systems

Many handles to facilitate testing – test-pattern injection, spy-buffer readout as well as test stands

Same team built existing trigger system and wrote its software & firmware → scope, requirements, interfaces well understood

Requirements are frozen in approved Trigger TDR

System is commissioned and operated in parallel with existing trigger

Working prototypes for all major cards exist & have passed tests Production quotes exist for all critical parts with confirmed delivery

times26 August 2013, Wesley Smith

Technical Feasibility


Project


Standardize within US CMS Trigger Project Use of a single FPGA type: Xilinx XC7VX690T

o Also used across CMS upgrade and other CERN projects

Negotiated US price at 50% of retail through CERN and US distributor (Avnet)

Standardize across US CMS Upgrade Project Use of a common crate/backplane infrastructure

o Use same μTCA architecture as is being used by other CMS subsystems such as HCAL.

o Built to similar specifications by common vendors.o Opportunities for pooling spares and cooperation on engineering. o This allows use of common components such as:

Use of standard DAQ/Trigger/Clock Interface: BU AMC13o Common connection to CMS data acquisition, trigger timing and control

systems. o Developed for HCAL project by Boston University


Value Engineering


Project


Project Organization



Project


401.04 Trigger Organization Chart

Muon Port Card & Sorter:L4 Manager: Paul Padley

CERN Integration:L4 Manager: Pam Klabbers

Muon Track-Finder:L4 Manager: Ivan Furic


Project


W.S. (U. Wisconsin) CMS Trigger Project Manager 1994-2007, Trigger Coordinator 2007 – 2012 US CMS L2 Trigger Project Manager 1998 – present

Darin Acosta (U. Florida) CMS Trigger Project Manager 2012-13 EMU Track-Finder, US CMS Trigger, 1998- present

Sridhara Dasu (U. Wisconsin) US CMS L3 Manager for Calorimeter Trigger, 1998 – present Author of original and upgrade cal. trig. Algorithms 1994 – present

Pam Klabbers (U. Wisconsin) CMS Regional Calorimeter Trigger Operations Manager

(more than a decade on RCT project) CMS Deputy Trigger Technical Coordinator

Ivan Furic (U. Florida) CMS CSC muon trigger maintenance and operations 2010-11 CMS CSCTF Upgrade leader

Paul Padley (Rice U.) US CMS EMU Project Manager 2006-2012 EMU Port Card, Clock Card, Sorter, 1998 – present


Trigger Mgm’t CMS Experience

Over 75 years of combined CMS Trigger ManagementExperience!


Project


Tom Gorski (U. Wisconsin) – Calorimeter Trigger Over a decade of engineering on the CMS Calorimeter Trigger

Alex Madorsky (U. Florida) – Muon Track-Finder Over a decade of engineering on CMS Trigger, EMU, CSCTF

Mike Matveev (Rice U.) – Muon Port Cards and Sorter Over a decade of engineering on CMS Trigger, EMU, Port

Cards, Sorter

Mathias Blake (U. Wisconsin) – Cal. Trig. Firmware New to project 5 years Applications Engineer at Xilinx, 2 years FPGA Engineer

at DRS Technologies, 3 years FPGA Design Engineer at Dolby Labs


CMS Trig. Engineering Exp.

Over 40 years of engineering experience, 30 on CMS Trigger


Project


ESH&Q



Project


Safety: follows procedures in CMS-doc-11587, FESHM L2 Manager (W.S.) responsible for applying ISM to trigger upgrade.

o Under direction of US CMS Project Management. Modules similar to others built before, of small size and no high voltage

Quality Assurance: follows procedures in CMS-doc-11584 Regularly evaluate achievement relative to performance requirements and

appropriately validate or update performance requirements and expectations to ensure quality.

QA: Equipment inspections and verifications; Software code inspections, verifications, and validations; Design reviews; Baseline change reviews; Work planning; and Self-assessments.

All modules have hardware identifiers which are tracked in a database logging QA data through all phases of construction, installation, operation and repair.

Graded Approach: Apply appropriate level of analysis, controls, and documentation commensurate

with the potential to have an environmental, safety, health, radiological, or quality impact.

Four ESH&Q Risk levels are defined and documented in CMS-doc-11584.


Trigger ESH&Q


Project


Project Scope



Project


Threshold: Demonstration of 98% agreement between the installed upgrade trigger

electronics at CERN and software emulation of this electronics through test pattern injection based on data taken after LS1, followed by demonstration of reduction of calorimeter and endcap muon trigger rates for electrons, photons, muons and taus with respect to the present system by a factor of two for a reduction of less than 15% in efficiency using the trigger emulator run on data taken after LS1. Incorporation of unganged ME1/1a data into the endcap muon trigger logic.

Objective: Demonstration of 99.5% agreement between the installed upgrade trigger

electronics at CERN and software emulation of this electronics through test pattern injection based on data taken after LS1 followed by demonstration of reduction of calorimeter and endcap muon trigger rates for electrons, photons, muons and taus with respect to the present system by a factor of two for a reduction of less than 10% in efficiency using the trigger emulator run on data taken after LS1. Calorimeter Trigger electron and photon position resolution improved. Incorporation of unganged ME1/1a data into the endcap muon trigger logic.


Key Performance Parameters


Project


None

Since both muon and calorimeter triggers have full self-test capabilities, both threshold and objective KPP are satisfied w/o requiring connected inputs or outputs.

Trigger electronics can store up sequences of test patterns and inject them into the front end of the trigger electronics at speed

Trigger electronics can receive its output, process and record this at speed for subsequent readout by DAQ.


External Dependencies


Project


Schedule



Project

26CD-1 Review -- P05: Trigger Upgrade

Schedule/Milestones


Targeting completion of L2 milestones 6 months earlier

401.04: 7 L2 Milestones, 8 L3 Milestones, 20 L4 Milestones

L2 Milestones:


Project


Critical path follows second phase of module production driven by start at CD3

> 2 years float to project completion


Schedule – Critical Path


Project


Risk Analysis



Project

CD-1 DOE Review -- Risk Management 29

16 risks:15 are threats and 1 opportunity: https://cms-docdb.cern.ch/cgi-bin/DocDB/ShowDocument?docid=11707

Lucas Taylor, 26-27 August 2013

401.04 Trigger Risk Register

Example risks

https://cms-docdb.cern.ch/cgi-bin/DocDB/ShowDocument?docid=11707


Project


Senior Engineer becomes unavailable (Low Risk) Hire new engineer, subcontract to consulting firm, use FNAL engineer

Funding is delayed (Low Risk) Commission with prototypes and/or fewer production boards

Software or Firmware does not meet requirements (Low Risk) Hire extra expert effort to recover schedule and help personnel

Boards are delayed (design, manufacture or testing) (Low Risk) Hire extra effort to speed up testing schedule

Vendor non-performance (Low Risk) Acquire spending authority to use alternative vendors (while original funds are being

unencumbered).

Input or output electronics (non-trigger) delayed (Low Risk) Built in capabilities of trigger electronics provide signals for their own inputs & outputs

Overall Risk Mitigation Strategy: Continue to provide fully operational current trigger system in parallel with upgrade

commissioning Partial operation of the upgrade systems provide tangible benefits from 2015

onwards26 August 2013, Wesley Smith

Overall Trigger Risks & Mitigation


Project


Cost



Project


C&S understood since prototypes for all major cards have been built and have passed tests

Have vendor quotes for all parts

C&S based on experience of the same team that built and wrote software and firmware for existing trigger system

Hardware is based upon common μTCA hardware platform also used by HCAL and other CMS systems

Same team wrote software and firmware for existing trigger system → scope and requirements well understood

Requirements are frozen in approved Trigger TDR

Schedule is robust because it always provides a working trigger at each stage with parallel operation of old and new trigger systems, commissioning and testing of new system during data-taking (with data) and evolution to final system while providing immediate availability of improved trigger in 2015


Quality of Estimate


Project


Base Cost = AY $K(No Contingency)


Using the Slide TemplateBase Cost by WBS

401.04.02 Trigger Management

2254%

401.04.03 MUON Trigger2,43949%

401.04.04 Calorimeter Trigger

2,36347%


Project

34CD-1 Review -- P05: Trigger Upgrade26 August 2013, Wesley Smith

Using the Slide TemplateM&S and Labor Base Costs

M & S Univ. Labor Total

Base Cost Base Cost Base Cost

401.04.02 Trigger Management 225 0 225

401.04.03 MUON Trigger 927 1,512 2,439

401.04.04 Calorimeter Trigger 1,257 1,106 2,363

Grand Total 2,409 2,618 5,027

Base Cost & Estimate Uncertainty in AY K$No FNAL Labor in Trigger Project


Project


Using the Slide TemplateLabor Resources by Fiscal Year

0

10

20

30

40

50

60

70

0

5

10

15

20

25

FY13 FY14 FY15 FY16 FY17 FY18 FY19

Cum

ulat

ive

FT

E

Ann

ual F

TE

Contributed Scientific

Technical

Engineering

Costed Scientific

Cumulative Total


Project


Using the Slide TemplateLabor Resources by Type

Costed Scientific.3 FTE

0%Engineering

16.6 FTE25%

Technical8.3 FTE

13%

Contributed Scientific40.9 FTE

62%


Project


Base Cost = AY $K(No Contingency)


Using the Slide TemplateCost Profile by Fiscal Year

FY13 FY14 FY15 FY16 FY17 FY18 FY19

Univ. Labor 0 804 904 910 0 0 0

M & S 0 1,027 1,145 237 0 0 0

FNAL Labor 0 0 0 0 0 0 0

Cumulative Total 0 1,831 3,880 5,027 5,027 5,027 5,027

0

1,000

2,000

3,000

4,000

5,000

6,000

0

500

1,000

1,500

2,000

2,500

Cu

mu

lativ

e C

ost

s K

$

An

nu

al C

ost

s K

$

Univ. Labor

M & S

FNAL Labor

Cumulative Total


Project


Trigger Design Maturity/Contingency

M&S: 2.4 M$ (48% of total cost)

Labor: 2.6 M$ (52% of total cost)

Fraction of the cost of the project divided by cost estimate type

Multiple Quotes / Travel72%

Preliminary Design

12%

Conceptual Design

10%

Novel Fabrication6%

Similar Procedure

100%


Project


Base Cost & Estimate Uncertainty in AY K$


Using the Slide TemplateBase Cost & Estimate Uncertainty

0

0

M & S Univ. Labor Total Estimate

Uncertainty

Base Cost

Estimate Uncertainty

Base Cost

Estimate Uncertainty

$% on base

costTotal

Estimate

401.04.02 Trigger Management 225 29 0 0 29 13% 254

401.04.03 MUON Trigger 927 416 1,512 605 1,021 42% 3,460

401.04.04 Calorimeter Trigger 1,257 250 1,106 442 693 29% 3,056

Grand Total 2,409 696 2,618 1,047 1,743 35% 6,770

No FNAL Labor in Trigger Project


Project


Summary



Project


Trigger Upgrades meet technical performance requirements

Scope and Specifications of Trigger Upgrade are sufficiently well-defined in TDR and CDR to support the C&S estimates

Risk managed by parallel operation of old and new trigger systems, commissioning and testing of new system during data-taking (with data)

Upgrade based upon common μTCA hardware platform and components used by other CMS systems

ES&H, QA plans, C&S based on experience with working prototypes

Management and Engineering teams are experienced with sufficient design skills, having designed and built original CMS trigger.

Trigger Upgrade not on CMS Project Critical Path

Ready for CD-1


Conclusions


Project


Additional Slides



ProjectL1 Upgrade TDR

Final presentation to LHCC was Tuesday June 11th

Public version of the TDR here:https://cds.cern.ch/record/1556311

Approved!

26 August 2013, Wesley SmithCD-1 Review -- P05: Trigger Upgrade 43


Project

CD-1 Review -- B04-1: Trigger Upgrade Overview 44

Process(x2 improvement highlighted)

1.1 x 1034 cm–2 s–1 2.2 x 1034 cm–2 s–1

Current Upgrade Current Upgrade

W(en),H(bb) 57.7% 87.0% 37.5% 71.5%

W(mn),H(bb) 95.9% 100% 69.6% 97.9%

VBF H( ( )tt mt ) 42.6% 51.3% 19.4% 48.4%

VBF H( ( )tt et ) 24.4% 44.3% 14.0% 39.0%

VBF H( ( )tt tt ) 17.2% 53.7% 14.9% 50.1%

H(WW(eenn)) 91.4% 97.8% 74.2% 95.3%

H(WW(mmnn)) 99.9% 99.9% 89.3% 99.9%

H(WW(emnn)) 97.6% 99.4% 86.9% 99.3%

H(WW(menn)) 99.6% 99.5% 90.7% 99.7%

StopbWce, jets (600 – 450 GeV) 55.8% 68.2% 50.3% 64.8%

StopbWc ,m jets (600 – 450 GeV) 78.1% 81.6% 76.4% 84.5%

RPV Stopjets (200 GeV) 70.1% 99.9% 43.6% 99.9%

RPV Stopjets (300 GeV) 93.7% 99.9% 79.7% 99.9%Wesley Smith, 26 August 2013

Physics Performance Summary(detail in breakout session talks)

Ave

rage

Impr

ovem

ent:

17%

(Lo

w L

umi)

& 4

0% (

Hig

h Lu

mi)


Project


CTP7 M&S: no HB/HE Backend until 2016 Delay purchase of 34 CTP7 cards until FY15 Use 12 CTP7 Cards w/prototypes as additional spares and for test

setups.

EMU CSCTF M&S: one endcap in 2015 and other in 2016 Commission and test one endcap thoroughly so 2nd can be integrated

quickly Delay spares and use prototypes for spares, test setups Compatible with Global Muon Trigger using different Endcap inputs

Stages and Content:


Trigger Staging


Project


Consideration of alternative designs has been performed within constraints of upgrading trigger system in a running experiment and operating both present and upgraded trigger system in parallel to ensure an understood and functional trigger system at any point during data-taking.

Provide incorporation of alternative designs and architectures in design itself so that as physics priorities and beam conditions evolve, algorithms and trigger methodology can evolve as well.

Moving towards providing all available detector information at input of trigger logic so that trigger decision is not impacted by upstream selection of information. Enables changes in trigger design to have as wide a range of

options as possible. 26 August 2013, Wesley Smith

Alternatives


Project


Schedule/Milestones – Level 3


Targeting completion of milestones 3 months earlier


Project


Schedule/Milestones – Level 4



Project


Response to Recommendationsfrom Previous Reviews



Project


Develop a plan prior to the CD-1 review for procurement of components that need to be installed prior to the end of LS1. Components are Muon Port Card Mezzanines and MPC-

EMUTF Optical Fibers that must be installed in collision hall during LS1.

Components moved to the Operations Program budget since are improvements needed when CMS starts up at end of LS1.

Incorporate the missing information identified within comment section into the CDR prior to the CD-1 review. This has been done and the CDR has been revised


Response to Recommendationsfrom May Review


Project


Prior to the upcoming CD-1 review, initiate discussions on obtaining CD-3a approval for production of components needed for the initial steps in the proposed staged installation plan. These components will be essential for maintaining current trigger performance, especially if the accelerator is unable to operate with 25 ns bunch spacing. CD-3a has been discussed with the DOE, and the Project

will not seek a CD-3a approval prior to the DOE CD-1 Review.


Response to Recommendationsfrom July Review

Documents

LHC CMS Detector Upgrade Project P05 – Trigger Upgrade: 401.4 Wesley Smith, U. Wisconsin L2 Manager, WBS 401.4 August 26, 2013 26 August 2013, Wesley Smith