New Report on the CMS experiment - INDICO-FNAL (Indico) · 2020. 1. 16. · Oliver Gutsche - 2020 January PUBLIC PAC Meeting - Report on the CMS experiment 14. January 2020 Group

Report on the CMS experiment

Oliver Gutsche for the Fermilab CMS group2020 January PUBLIC PAC Meeting

14. January 2020

Oliver Gutsche - 2020 January PUBLIC PAC Meeting - Report on the CMS experiment 14. January 2020

Outline

▪ CMS at Fermilab: Reminder

▪ Responses to recommendations from last PAC

▪ Brief status update - CMS experiment

▪ Brief status update - FNAL CMS group

2


CMS at Fermilab: Reminder

3

▪ One of the pillars of the CMS collaboration (2nd largest group after CERN)▪ Major contributions in physics analysis, Phase 1 and Phase 2 upgrades,

reconstruction/simulation, and computing/operations ▪ As a testimony of the essential contributions made by Fermilab over the years,

Fermilab scientists selected to serve as high level managers within CMS:๏ P. McBride (deputy spokesperson), B. Klima (Chair of Publication Committee)๏ Phase 2 Level-1 Trigger project: J. Berryhill (L1 manager)๏ Phase 2 HGCal project: J. Strait (L1 deputy manager, technical coordinator, L2 manager for engineering); Z.

Gecse (L2 manager for cassettes); J. Hirschauer (L3 manager for on-motherboard electronics); R. Lipton (L3 manager for silicon sensors)

๏ Phase 2 MTD project: J. Butler (L1 deputy project manager); S. Traczyk (technical coordinator); T. Liu (L3 manager for ETL front-end electronics); A. Apresyan (L3 manager for ETL engineering)

๏ DAQ project: R. Mommsen (L2 deputy project manager)๏ Physics, Performance, and Datasets Group: K. Maeshima (L2 manager for data quality monitoring and data

certification)๏ Software & Computing group: S. Lammel (L2 manager for Facilities)


CMS at Fermilab: additional management positions w.r.t July 2019

4

▪ Nick Smith selected as L2 manager for computing operations ▪ Don Lincoln selected as coordinator of QA/QC for the HGCal international

project▪ Daniel Elvira selected as convener of the newly created "Code

modernization and performance group”

▪ Oliver Gutsche selected as technical lead of the Portable Parallelization Solutions project of the DOE Center for Computational Excellence

▪ Nhan Tran asked to be the lead the AI project in Scientific Computing Division

▪ Joel Butler elected as DPF-Chair


Awards and Recognition: New w.r.t Summer 2019

▪ Nhan Tran, Early Career Award to expand particle collider research capabilities using artificial intelligence

▪ Jim Hirschauer, EPRA for leadership in CMS LHC Upgrade […] while spearheading ASIC development at Fermilab in the context of the HL-LHC CMS Upgrade Project

▪ Ron Lipton, EPRA for leadership of the Muon Scanner Project that won an R&D 100 award

▪ Steve Nahn, EPRA for outstanding leadership of the LHC CMS Detector Upgrade Project

▪ Fabio Ravera, 2019 IEEE Emilio Gatti and Franco Manfredi Best Ph.D. Thesis Award in Radiation Instrumentation

▪ Lindsey Gray, LDRD Graph Neural Networks for Accelerating Calorimetry and Event Reconstruction

▪ Nhan Tran, LDRD Accelerator Control with Artificial Intelligence

5


Responses to recommendations

from last PACs6


Recommendation from July 2019

“We recommend the Laboratory in coordination with DOE, support CMS/LHC in their request for recruiting new

personnel dedicated to supporting the HL-LHC and HPC software and computing developments, as well as

continuing to support research associates.”

7


Group Composition in FY20

8

▪ The department has followed up on the succession plan outlined in the past years

๏ Hired an associate scientist with focus on the Phase2 upgrade and detector R&D for future experiments• Specialization in silicon detectors.

๏ Hired an an associate scientist with focus on SW&C• Specialization in new architectures and new computing paradigms

▪ The lab has approved the hiring of 3 more RAs (as planned within the FY20 budget)

▪ Including RAs, the number of FTEs is still decreasing (46.5 in FY18, 41.2 budgeted in FY20)

๏ Continued support to RAs through a healthy research program is absolutely essential to the FNAL CMS program (analysis, upgrade, S&C)

▪ To compensate for anticipated additional reduction in personnel (retirements, transition of scientists to different experiments/institutions) and maintain the capability to deliver on our commitments, strategic hires of associate scientists is expected in the near future



“Given the crucial importance of the Tier-1 computing resources for the CMS physics program, the PAC

endorses the request for replacement of end-of-life equipment as soon as possible given budgetary

constraints. For the longer term, we recommend the development of a computing transition plan in line with DOE's strategic priorities, its international partners, the

needs of the experiment, and the timeline and resources needed for the required software developments.”

9


Quick update: USCMS Tier-1 performance in 2019

10

2019

▪ Fulfilling its duties to CMS, largest Tier-1 site serving the U.S. and most of the world๏ 2019 accomplishments:

• ramp-up of streaming PileUp to all sites for MC production

• increased HPC usage through HEPCloud• Transitioning tape system to new technology

▪ Performance CY19๏ Readiness for CY19 at 97.4%, down from 98.4% in

CY18• Explained by various factors, age of facility

contributing


History of Tier-1 Facility Investments

▪ 2016-2019๏ Significant delays in Tier-1 equipment investments๏ To provide resources requested by CMS, deferred retirement of

end-of-life hardware

11

Projected Needs


“Aging” Fermilab Tier-1 Facility, Poses Additional Risks

12

▪ 2020 Fermilab facility hardware age distribution in the “Deferred Retirements” Scenario

๏ Does not include additional $400k DOE program funds used to retire very old disk

▪ Fermilab will have to operate CPU and storage equipment that is much older than “warranty + 3 years”.

๏ Hardware under warranty is repaired/replaced by vendors. If older, this requires Fermilab effort

๏ We try to keep spares out of out each generation of OOW hardware

▪ Have been able to adapt to past budgets, by moving HW that violates our policy into the LPC analysis facility

๏ Adding more redundancy to compensate for higher unreliability ➜ more sysadmin effort needed to run facilities.

▪ The “Deferred Retirements” scenario does allow USCMS to keep up with CMS resource requests

๏ However at a higher risks to facility performance and potentially higher operations efforts


Needs for the Tier-1 - a solvable problem

13

▪ In 2021, w/ start of Run 3, Tier-1 equipment refurbishments can no longer be delayed, and for 2022 significant additional capacity needs to be procured for CPU and disk storage. ๏ However, the flat funding profile for FY20 and FY21 does not allow to fully

“catch-up”on the delayed Tier-1 hardware investments.

▪ A supplemental one-time injection of funds for the Tier-1 would reduce risks of running the Tier-1 with outdated equipment that cannot be operated cost effectively, and prevent delays in capacity increases required for the upcoming physics Run 3.


Plan: Rebalancing Upgrades ➜ Operations Program for Tier-1 in FY21

14

▪ Costs are estimated at about $2.9M above the currently planned budget level for Tier-1 equipment๏ To be fully effective the funds should become available during early FY21

at the latest, for hardware procurements during the 1st year of Run3 and commissioning for full use for the 2nd run year in 2022

▪ + $2.9M in FY21 would allow Fermilab to bring CPU and Disk servers back into “compliance”, in terms of useful lifetime of equipment, and allow for the capacity increase needed for Run 3: +30% CPU, +15% Disk, and +30% Tape๏ The plan would be to spend $7.375M in FY21 on Tier-1 upgrades, after

which the Tier-1 budget would be brought back to the level of $2.5M in the out-years


May Agency Review of USCMS Operations Program

15

▪ Recommendation from last ops review๏ “there is a tension between the needs of the US CMS HL-LHC upgrade project and those of the operations and

research components”๏ “some flexibility in the DOE funding profiles for operations, research, and HL-LHC upgrades would support

optimization of the overall US CMS program.”๏ “work directly with Fermilab management and the CMS HL-LHC upgrade project to develop coherent messages to

the agencies regarding budgetary needs, priorities, and overall CMS program optimization.”

▪ DOE seems open to rebalancing between operations and upgrade to help with this problem.๏ Worked out with upgrades ➜ feasible plan for FY21๏ Presented and discussed with lab management (CFO, CRO, research director)

and will be presented at the lab budget briefing to DOE in February



“We recommend the Laboratory maintain the LPC at a successfully functioning level with a long-term

perspective.”

16


Fermilab LHC Physics Center (LPC)

▪ LHC Physics Center: Established center of excellence for CMS. More than 500 users and 150 residents

๏ Nearly 800 people participate in LPC-organized workshop and events, fostering interactions with theorists and non-CMS members

๏ Co-led by C. Gerber (UIC) and S. Jindariani (Fermilab)

▪ Approximately 27% of all CMS papers have > 50% analyzers from LPC▪ The LPC has transformed the way CMS educates young students and postdocs.

๏ Record number of participants in CMS Data Analysis School and Tutorials ➜ almost 70 events per year in total๏ CMSDAS is currently ongoing (stop by to see how it works), participation increased over the last 2 years (now enables

to train ~70 students every year)

17

More than 90% of USCMS institutions have members affiliated with the LPC


Additional Needs for Research: plan for one-time supplement for LPC

18

▪ LHC Physics Center “Distinguished Researchers” program has been particularly hard hit by cuts

๏ it is a centerpiece in supporting the LPC efforts and the US CMS research community. ๏ allows junior and senior scientists to spend time at the LPC, this way providing effort to run the LPC program

• This for example includes the Data Analysis School and the Hands-on Tutorials, which are the training programs essential to prepare and maintain workforce expertise for students and postdoctoral to participate and contribute to CMS upgrades and operations.

▪ FY19 program was “resurrected” with additional funds being made available from DOE, and addition relief provided from Operations

▪ Expected FY20 and FY21 budget level is below what is needed to fully fund the next round of appointment cycles, to be called in summer 2020 and 2021.

๏ An injection of about $900K would allow that program to continue over the coming 3 years. • This one-time supplement will be of great importance for the preparations of start of Run3 and the first year of physics

data taking at Run 3 luminosities.

• It will secure the training and support for students and postdocs, and thus play an important role in providing the scientific personnel to help and support detector operations and the HL-LHC upgrade efforts that are ramping up significantly during those years and rely on well-trained scientific personnel.


“CMS plans to start using the new and unique Irradiation Test Area at MTEST in April 2019 to validate sensors for

the Outer Tracker. The availability of this facility is critical to meet the CMS testing and qualification schedule.”

Recommendation from January 2019

19


Irradiation Test Area

20

▪ Under construction in old MuCool Test Area experimental hall at end of Linac

๏ 400 MeV protons, 5e12 per pulse, ~40 pulses per minute, 12 hours per week to mitigate residual activation hazard

▪ Shielding cave built. Beamline and user infrastructure under construction.

๏ First silicon sample runs ~April until May 30 summer shutdown

๏ Schedule driver is shielding assessment๏ Other materials, cold box capabilities in Fall 2020

▪ ITA brings new valuable capabilities to the lab๏ 10 experiments are amongst the potential users, CMS is

one of them


Brief Status Update - CMS

21


CMS - Status of the experiment

22

Long Shutdown 2: 2019-2020:

▪ Many activities to be accomplished in 2 years

▪ On track๏ Accumulated only a

few weeks delay


LHC schedule update

23

▪ LS2 was extended by two months – such that the experimental caverns will be closed on May 1st 2021๏ CMS will make good use of that time to finish all LS2 work

▪ Extend Run 3 by one year – such that 2024 is included in Run 3 and LS3 starts in 2025;๏ CMS will have an updated schedule in April 2020

New LHC schedule

Technical Stop LHC commissioning pp physics ion physics

First 2 weeks

Second 2 weeks


CMS - Status of the experiment

944 collider data papers submitted as of Dec 28

24

New: CMS is writing AI papers that go through the collaboration-wide physics approval process, using collider data and are signed by the whole collaboration:

▪ HIG-18-027 (CSBS*), A deep neural network for simultaneous estimation of b jet energy and resolution (submitted 12/12/2019)

▪ EXO-19-011 (MLST**), A deep neural network to search for new long-lived particles decaying to jets (submitted 27/12/2019)

*Computing and Software for Big Science (CSBS) journal**Machine Learning: Science and Technology (MLST) journal


CMS - phase 2 detector upgrade status

25

All detector TDRs have been releasedNext:

● Trigger TDR● Software &

Computing TDR


CMS - Software & Computing update

▪ Computing activity continues during LS2 almost unchanged๏ Largest activities:

• legacy re-reconstruction pass of Run 2 data and new MC• TDR and Phase 2 upgrade simulation

๏ Example: Holiday production sustained 250k cores26

Production Plan Holiday Production Overview


Brief Status Update - FNAL

CMS group27


Fermilab CMS group - Recent physics highlights (1)

28

▪ The Fermilab group is finalizing a broad program based on the full Run 2 dataset aligned with P5 science drivers

๏ Standard Model• Searches for gluon fusion production of Higgs boson

decaying into bb and cc; • Measurement of the VBS production of W bosons

• Searches for triple gauge boson production ๏ SUSY

• Wino-like charginos and heavy neutralinos decaying into Wh & LSPs

• Higgsino-like charginos and heavy neutralinos in off mass shell gauge bosons

• RPV scenarios๏ Dark Matter

• Search for Dark Higgs scenarios

• Search for Self-Interacting DM 1908.01713

1905.04246

https://arxiv.org/abs/1908.01713



Fermilab CMS group - Recent physics highlights (2)

▪ The Fermilab group is finalizing a broad program based on the full Run 2 dataset (cont.)

๏ Exotica• Searches for semi-invisible jets to cover regions of phase

space with very boosted objects• Searches for long lived particles

• Searches for dijet resonances using events with 3 jets (1911.03761)

29

1911.03761▪ FNAL also continues its significant investment in the

development of advanced tools to enhance the sensitivity to challenging corners of the parameter and phase space๏ Innovative tagging of bosons, deployment of machine

learning, etc.




Fermilab CMS group - Upgrade activities

30


U.S. CMS - Phase 2 status update

31

▪ Received CD-1 approval end of December๏ Very positive CD-1 review at the end of October

▪ Two checkpoints in 2020๏ CD3a Review to approve long lead items: April 7

• Silicon Sensors, LYSO crystals, Outer Tracker Module Mechanics๏ Independent Project Review/CD-2: November 17

• CD-2 establishes Performance Baseline- Scope depends on Federal Budget situation and adjustments to recently updated

LHC Schedule, maybe be better to defer and also achieve CD-3, “Ready for Fabrication” in mid 2021

• IPR required annually, if IPR, then aimed at assessing readiness for CD2/3


Fermilab CMS group - Upgrade activities - Technical Focus

32

▪ Outer Tracker (Merkel, Canepa, Lipton, Spiegel, Berry)๏ Assemble 2250 Silicon Modules๏ Mount 1000 Modules onto Carbon Fiber Structure which comprises central section of inner 3 layers of CMS Tracker

▪ Calorimeter Endcap (Cheung, Gecse, Strait, Hirschauer, Freeman, Joshi, Lincoln)๏ Assemble 550 Cassettes๏ Design/Fabricate Concentrator ASIC for on-detector data handling (so called ECON)

▪ Trigger/DAQ (Berryhill, Tran, Mommsen, Cavanaugh)๏ Develop and software/firmware algorithms for Correlator Trigger๏ Procure and Deploy Data Logger

▪ MIP Timing Detector (Chlebana, Apreysan, Bauerdick, Liu, Butler, Tkaczyk, Gray)๏ Assemble 4700 Endcap Modules๏ Develop and Fabricate Endcap Readout ASIC

▪ Technical progress proceeds on all fronts๏ In particular, Fermilab Test Beam and soon-to-be Operational Irradiation Test Area are keys to final design validation๏ RAs are an integral part of the Fermilab groups working on the upgrade. They are essential for FNAL to deliver on the

lab’s commitment and, at the same time, benefit greatly from leading high profile instrumentation projects.


Fermilab CMS group - Software & Computing

33

U.S. CMS Software & Computing Operations Program

International CMS Software & Computing Project

▪ International CMS Software & Computing and the U.S. CMS Software & Computing Operations program are very interconnected, shown are areas where Fermilab CMS scientists are involved in the management

๏ Fermilab scientists (and computing professionals with guidance from scientists) are involved in leadership and execution of almost all aspects of CMS Software & Computing

▪ June 2019: Out-of-the-ordinary opportunity to brief DOE (Siegrist, Patwa, Chatterjee, Colby) about the breadth of the U.S. CMS Software & Computing R&D


Fermilab CMS group - Software & Computing activities

▪ CMS facilities and computing operations (Lammel, Smith)๏ Workflow operations, data management, facility operations

▪ Software development (Elvira, Mrenna, Pedro, Gray, Hall)๏ Core software (framework), Simulation software (Geant), Generators (Pythia), Reconstruction software (tracking,

HGCAL) ➜ new architectures (GPUs, FPGAs, … )

▪ Analysis frameworks and facilities R&D (Pedro, Gray, Gutsche, Smith, Cremonesi)๏ Columnar analysis, Coffea, Spark and analysis facility concepts

▪ Machine Learning R&D (Tran, Gray, Pedro, Liu, Cremonesi, Klijnsma, Herwig)๏ Development of deep learning applications for reconstruction and analysis optimization, Fast inference on FPGAs๏ Fully aligned with Fermilab AI efforts

▪ Facilities R&D (Gutsche, Jayatilaka)๏ HPC leadership class facility access R&D๏ Data management, organization and access R&D

▪ Computing model evolution (everyone!)๏ Evolution of the CMS computing model

▪ Management of International CMS and U.S. CMS (Lammel, Gutsche)๏ Facilities Management, Operations Program Management

34


Summary & Outlook

▪ Recommendations have been addressed ๏ Pending the approval for rebalancing from DOE (expected)

▪ CMS is on track to complete the LS 2 shutdown activities, making progress on phase 2 upgrades and continues to deliver excellent science while preparing software and computing for Run 3 and HL-LHC

▪ Fermilab is making major contributions to all these accomplishments

▪ Anticipated departure of personal in CMS department (retirements) of concern

๏ Refining succession plan to maintain our capabilities and deliver on our commitments (physics, phase 2 upgrades, software & computing, operations for Run 3 and HL-LHC)

35

Documents

New Report on the CMS experiment - INDICO-FNAL (Indico) · 2020. 1. 16. · Oliver Gutsche - 2020 January PUBLIC PAC Meeting - Report on the CMS experiment 14. January 2020 Group