M&O Scrutiny Group Reportto LHC Resource Review Boards

Page 1

11 – 13 October 2010

Bernd Löhr

DESY

Composition of the Scrutiny Group in 2010: unchanged from last year

Giovanni Batignani (INFN Pisa) George Ginther (FNAL,Rochester University)

Gabriele Cosmo (CERN) Bernd Löhr (DESY, Chair)

Paul Dauncey (Imperial College London) Gerhard Mallot (CERN)

Martyn Davenport (CERN) Emmanuel Tsesmelis (CERN)

Christinel Diaconu (Marseille) Michał Turała (INP PAN Cracow)

Scientific Secretary: Sascha Schmeling (CERN)

Page 2

Outline of the Presentation:

• Report on the five LHC experiments

• General topics of discussion with all experiments

ALICE

ALICE closing Report 2009:

Actual costs w/o power 4,887 kCHFBudget w/o power 4,687 kCHF

Power costs 513 kCHFPower costs NMS (budget) 153 kCHF Refund to NMS 79 kCHF Actual NMS power costs 74 kCHF

Year 2010 2011 2012 2013 2014

Costs w/o power 4,978 4,491 7,053 4,993 4,925

Costs w. NMS power 5,621 5,361 7,133 5,863 5,795

Costs full power 7,209 7,292 7,311 7,794 7,726

• On-line computer replacement delayed -> reduced request in 2011 but peak in 2012.

• Improvement of ventilation of the magnet and humidity control.

• Presently no detector opening foreseen during technical stop at end of 2010.

Page 3

ALICE request for 2011 M&O Aand projections to 2014 in kCHF:

ALICE budgets and costs including NMS power costs

-1,000

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

kC

HF

Budget

Costs

Contributions

Integr. Miss. Contrib.

Integr. Cash Balance

Integr. Balance

ALICE-History

The SG recommends the approval of the ALICE closing report for 2009 and the M&O A budget request for 2011.

ATLAS

ATLAS M&O A closing Report 2009:

Actual costs w/o power 13,472 kCHFBudget w/o power 13,610 kCHF

Power costs for Non-Member-States 936 kCHF

ATLAS request for 2011 M&O Aand projections to 2015 in kCHF:

Year 2010 2011 2012 2013 2014 2015

Costs w/o power 13,841 15,623 17,920 15,611 16,311 15,243

Costs w. NMS power 14,818 16,623 18,925 16,620 17,326 16,263

Costs w. full power 16,661 18,443 20,120 18,430 19,131 18,063

• Inner-Detector Cooling

• Magnet consolidation program

Major points of discussions:

Page 4

• IBL project

• On-line hardware replacement

ATLAS budgets and costs including NMS power costs

-3,000

-2,000

-1,000

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

11,000

12,000

13,000

14,000

15,000

16,000

17,000

18,000

19,000

20,000

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

kC

HF

Budget

Costs

Contributions

Integr. Miss. Contrib.

Integr. Cash Balance

Integr. Balance

ATLAS-History

ATLAS (cont. 1)

Inner Detector (ID) cooling compressors:

The existing compressors do not guarantee a reliable long-term operation.

Page 5

Two technical solutions are under study: thermo-siphon or compressor with magnetic bearing.SG requested approval by LHCC. So far LHCC encouraged the collaboration to continue studies and to find a solution.

M&O AID cooling consolidation

2010 2011 2012 2013 SUM

Planned in 2009 400 360 350 350 1460

Planned in 2010 420 380 780 20 1600

Magnet-system consolidation program:

This program started already in 2009. Main objectives are: magnet related improvements, spare compressor and turbo pumps, improved cryogenics control, improvements of helium system for faster recovery after dump.

Year 2009 2010 2011 2012 2013 2014 2015 SUM

Total budgetfor Magnet

2463 2338 1420 2277 1540 1178 1200 10547

Costs ofcons./upgrade

980 1108 297 1155 390 0 0 3930

ATLAS (cont. 2)

Page 6

Insertable B-Layer (IBL):

Initially replacement of radiation-damaged pixel detector layer, now partly an upgrade.The project foresees contributions from M&O A and M&O B.The M&O A part includes modifications of beam pipe, partly new Beryllium pipe, new tooling for removal and re-installation, power supplies and cabling. The SG has asked for approval of the project by the LHCC. This is outstanding.

An interim MoU is being signed by the involved Funding Agencies. A TDR has been submitted to the LHCC.

Contributions to the IBL project in kCHF

2009 2010 2011 2012 2013 2014 Sum

M&O A 180 940 1,050 995 900 4,065

M&O B

600 200 1100 1300 900 100 4,200

On-line hardware replacement:

Completion of DAQ system (3010 processors) was planned for 2010. Now stretched until 2013. delayed replacement costs.

M&O A 2010 2011 2012 2013 2014

DAQ replacements [kCHF]

1,140 2,275 3,315 2,587 3,825

2015

4,475

Page 7

ATLAS M&O B:

M&O B budget requests:

The SG’s recommendation concerning ATLAS;

The SG recommends the approval of the ATLAS closing reports for M&O A and M&O B of 2009.

The SG recommends the approval of the M&O B budget request for 2011. The recommendation of the 2011 budgets for the IBL and for the consolidation of the Inner Detector cooling are contingent on the approval of these projects by the LHCC.

Closing report for 2009: spent 6,246 kCHF to be compared to budgeted 6,976 kCHF.

ATLAS (cont. 3)

Cumulative cash balance for M&O B is now 2,154 kCHF

ATLAS M&O B 2010 2011 2012 2013 2014 2015

Inner Detector 2,875 3,585 3,835 3,265 2,395 2,245

Liquid Argon 1,321 1,186 1,176 996 996 796

Tile Calorimeter 605 724 673 669 668 505

Muons 700 640 720 570 570 570

Total 5,501 6,135 6,404 5,500 4,629 4,116

Page 8

CMS

CMS closing Report 2009:

Actual costs w/o power 13,596 kCHFBudget w/o power 10,408 kCHF

Power costs forNon-member states 196 kCHF

CMS request for 2011 M&O Aand projections to 2014 in kCHF:

Year 2010 2011 2012 2013 2014

Costs w/o power 11,911 12,553 14,760 12,576 10,752

Costs w. NMS power 12,742 13,384 15,591 13,407 11,583

Costs w. full power 13,711 14,353 16,560 14,376 12,552

• Large over-spend in 2009. • Repair of end-cap cooling (bushings) at end 2009 and beginning 2010.

CMS budgets and costs including NMS power cost

-2,000

-1,000

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

11,000

12,000

13,000

14,000

15,000

16,000

17,000

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

kC

HF

Budget

Costs

Contributions

Integr. Miss. Contrib.

Integr. Cash Balance

Integr. Balance

CMS-History

Major points of discussions :

Page 9

CMS (cont. 1)

Large over-spend in 2009:

• The long shutdown period in 2009 higher costs for manpower, specialized teams necessary.

• The new running schedule of the LHC with no shutdown in 2011 required preparations for the long shutdown in 2012 already in 2009.

Integrated savings up to 2008 are used to cover part the deficit in 2009.CMS will try to cover remaining 488 kCHF deficit by additional savings in 2010/2011.Possible items where savings might be possible have been indicated to the Scrutiny Group.

Repair of bushings:

• Unexpected failure of bushings in the cooling circuit of one end-cap at the end of 2009.

Repair of bushings from December 2009 to February 2010.Additional costs of 1,121 kCHF generated in 2010.These costs cannot be accommodated in 2010 budget, they are moved to the 2011 budget request.

On-line and off-line computing:

• Postpone on-line hardware replacement to 2012-2013. This saves 1.9 MCHF in 2011.• Reduce request for off-line computing from 4 FTEs to 3 FTEs.

The SG recommends the approval of the CMS closing report for M&O A of 2009 and the budget request for 2011.

Page 10

LHCb

LHCb closing report 2009:

Actual costs w/o power 2,387 kCHFBudget w/o power 2,386 kCHFPower costs forNon-Member States 84 kCHF

LHCb requests for 2011 M&O Aand projections to 2014 in kCHF:

Year 2010 2011 2012 2013 2014

Costs w/o power 2,512 2,545 2,617 2,627 2,607

Costs w. NMS power 2,591 2,672 2,658 2,711 2,734

Costs w. full power 3,112 3,515 2,917 3,227 3,577

The SG recommends the approval of the LHCb closing report for 2009 and the M&O A budget request for 2011.

LHCb budgets and costs including NMS power costs

LHCb foresees to complete the on-line farm by end of 2010LHCB on-line hardware: replace components only when broken, assume a typical lifetime of six years.

Almost constant spending profile for on-line hardware replacement

LHCb 2008 2009 2010 2011 2012 2013 2014

[kCHF] 155 300 200 300 350 350 350

-500

0

500

1,000

1,500

2,000

2,500

3,000

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

kC

HF

Budget

Costs

Contributions

Integr. Miss. Contrib.

Integr. Cash Balance

Integr. Balance

LHCb-History

Page 11

TOTEM

TOTEM closing report 2009:

Actual costs 410 kCHFBudget 449 kCHF

No power costs arise forthe TOTEM experiment

Taking into account the carry-over from previous years,The integrated cash balance is presently 6 kCHF.

TOTEM requests for 2011 M&O A and projections to 2014 in kCHF:

The SG recommends the approval of the TOTEM closing report for 2008 and the M&O A budget request for 2010.

TOTEM wants to install the T1 detector components during the technical stop at the end of 2010.This is a combined operation with CMS.TOTEM and CMS are currently estimating the costs and are negotiating with CERN management the related funding.

The Scrutiny Group recommends that TOTEM and CMS formalize their financial inter-dependence.

YEAR 2010 2011 2012 2013 2014

Total costs 448 440 409 437 430

Page 12

General points of discussion

New operational model of experiments:

The experiments have been asked to estimate the difference between ‘full running years’and ‘shutdown years’ for their M&O A requirements.

Running years: full power and gas consumption, full manpower needs for DAQ.

Shutdown years: additional manpower costs for detector maintenance, partially balanced by less gas and power consumption.

Shutdown years are in general only slightly more expensive considering regular maintenance,but necessary repairs may create a peak in expenditures.

Replacement of EVO:

Presently a commercial system, VIDYO, is under test by the IT-Division and the LHC experiments as an alternative to EVO.

Previous assumption was that only a one-time license fee will have to be paid for VIDYOwhich will be borne by CERN. Therefore the experiments planned for 50% of last yearsEVO costs for 2010 and nothing for the following years. This was the advice of the DRC.

Recent news from IT indicate that permanent costs will also arise for VIDYO, however, less that for EVO.

It is not yet decided to what extent the experiments will be charged if VIDYO will be chosen to replace EVO.

Page 13

Safety, a new level-2 item:

The experiments face considerable expenses for installation and training courses whichare related to personnel and radiation safety.

These costs are now collected under a new level-2 item, Safety, which is different from the existing one, Detector Safety Systems.

Safety cost [kCHF]

2011 2012 2013 2014

ALICE 77 85 35 35

ATLAS 830 850 880 950

CMS 100 100 100 100

LHCb 10 10 10 10

TOTEM 0 0 0 0

Page 14

Concerning the replacement rates of hardware this document states:

• Disks and fileservers typically after 3 years

• PCs typically after 4 years

• Network switches typically after 5 years

Already in 2008, the DRC questioned the validity of this model and the Scrutiny Group expressed the same opinion.

It was decided to review the model with all LHC experiments together after they gained some experience withreal data taking. The LHC experiments agree to this but insist that the revision will be done in a uniform process for all experiments.

The Scrutiny Group strongly suggests that requests for the 2012 budget, to be made in April 2011, should be basedon a revised model.

The DRC and the Scrutiny Group have agreed that a revision process will be started this year and a new model will be worked out by the end of February 2011.

On-line hardware replacement:

Agreement between LHC experiments and CERN management on replacement policy:

“Response from the LHC experiments to the RRB Scrutiny Group concerning the Maintenance and Operation of online systems (Trigger, Data Acquisition, Event Filter Farms, and Control Systems”

dated 6 September 2004.

Page 15

Accidents and unexpected urgent repairs:

The unexpected failure of the CMS end-cap cooling, resulting in the necessary immediate repair of the bushings,is a type of event which can happen again anytime, also in the other experiments.

The Scrutiny Group recommends the installation of a mechanism which addresses the financial aspect ofunexpected events which require immediate spending of a larger amount of money not foreseen in the budget.

Part of such a mechanism must be the early transmission of information to the CERN management, the Funding Agencies and the Scrutiny Group as well as a fast feedback from the Funding agencies and from the Scrutiny Group.

“Fluctuating budgets” vs. “flat budgets”:

Up to now: efforts to keep M&O A budgets at the same level over years.

Now new situation :

The real spending profiles for M&O A will be fluctuating considerably.

• DAQ systems are being completed periodic replacement costs;

• operating scenario of LHC with long shutdown only every third year;

• several large consolidation projects in experiments over the coming years.

Page 16

Funding Agencies (and experiments?) supposedly prefer “flat budget profile”.

The Scrutiny Group prefers fluctuating M&O A budget which reflect the real spending profile.

Flat budget means:

• money has to be accumulated over years for several different projects likely to be distributed over different level-2 items;

• for a given year this leads to “savings” which may necessitate “surplus refunds” (CERN-RRB-2006-026);

• in case of unexpected large expenses the savings must be protected against the Scrutiny Group and against the own collaboration;

• it makes it increasingly difficult for the Scrutiny Group to perform its task, follow the money flow, and give a well justified recommendation to the RRB.

The Scrutiny Group seeks advice from the RRBs regarding the future procedure.

Page 17

Composition of the Scrutiny Group:

Five members of the present Scrutiny Group reached the end of their 3 year’s appointment:

G.Batigniani (INFN Pisa) M.Davenport (CERN) C.Diaconu (Marseille ) G.Ginther (FNAL, Rochester) E.Tsesmelis (CERN)

The Scrutiny Group wants to thank these members for their invaluable contributions to its work.

The success of this year’s scrutiny process has been made possible by the fruitful cooperation of the Resource Coordinators and the management of the LHC experiment’s Collaborations, and the excellent material they have provided.

Additional Information

ATLAS Cost Increases, Cost Reductions

2010 2011: • external cryogenics - 1,190 kCHF• beam-pipe & vacuum + 850 kCHF• safety + 830 kCHF• on-line computing + 1,035 kCHF• magnet consolidation + 297 kCHF ______________________

+ 1,822 kCHF

total difference w/o power + 1,782 kCHF

2011 2012:

• beam-pipe & vacuum + 110 kCHF• on-line computing + 1,040 kCHF• magnet consolidation + 858 kCHF• evaporative cooling + 400 kCHF• general services + 250 kCHF• gas consumption - 250 kCHF ______________________

+ 2,408 kCHF

total difference w/o power + 2,297 kCHF

budgeted costincreases

budgeted costincreases

ATLAS New “Safety” Requests

2011:

• SX1 rad. protection buffer zone 230 kCHF• neutron shielding USA-UX 150 kCHF• neutron shielding endcap toroids 100 kCHF• ALARA tools 100 kCHF• ventilation consolidation 70 kCHF• C3F8 tank rad. protection 50 kCHF

Major contributions:

2012: Major contributions:

• neutron shielding USA-UX 200 kCHF• neutron shielding endcap toroids 120 kCHF• ALARA tools 150 kCHF• level 3 alarm processing unit 120 kCHF• fire protection (water mist ) 80 kCHF• ventilation consolidation 70 kCHF

ATLAS Safety

ATLAS Magnet Consolidation

FW/270810

ATLAS: Online Farm CPU (3 Full Year Cycle) Using new June 10 Profile

Available 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 20162004 100 200 200 200 200 200 100 100 100 100 100 100 100 2005 60 60 60 60 60 60 60 60 60 60 60 60 2006 346 346 346 346 346 346 346 346 346 346 346 2007 220 220 220 220 220 220 220 220 220 220 2008 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 2009 10 10 10 10 10 10 10 10 2010 400 400 400 400 400 400 400 2011 320 320 320 320 320 320 2012 320 320 320 320 320 2013 234 234 234 234

TOTAL 100 260 606 826 1,826 1,836 2,136 2,456 2,776 3,010 3,010 3,010 3,010

Of which replacements2004 - 5 10 105 105 105 5 5 5 5 100 5 5 2005 3 3 60 3 3 60 3 3 60 3 3 2006 17 17 346 17 17 346 17 17 346 17 2007 11 11 220 11 11 220 11 11 220 2008 50 50 525 525 50 525 525 50 2009 1 1 10 1 1 10 1 2010 20 20 400 20 20 400 2011 16 16 320 16 16 2012 16 16 320 16 2013 12 12 234

TOTAL - 5 13 125 193 515 296 639 936 728 1,082 1,268 962

Notes - Above replacements follow the agreed model as used to date - i.e. replacement in 3rd full year and 5% per annum in other years.Except the first full replacement of machines initially purchased in 2008 where 50% is deferred from 2011 to 2012

Revised costs assumed per CPU: 3.5 KCHF

Replacement cost [kCHF] 4 46 438 676 1,803 1,036 2,237 3,276 2,548 3,787 4,438 3,367ATLAS: Online Farm: CPUs02

1

#REF!

#REF!

ATLAS On-line Replacement

CMS Cost Development

Main changes in budgets are due to detector related costs and on-line computing costs.

Year 2009 2010 2011 2012__________________________________________________________________________________________

detector related budget 3,913 3,633 4,310 1) 3,590 kCHFactual costs 6,394 2) kCHF

on-line computing budget 2,400 3,243 2,843 5,891 kCHFactual costs 2,033 kCHF

1) included are 1,128 kCHF spent in 2009 for bushings repair (endcap cooling)2) savings in previous years is used to cover the overspend except 488 kCHF

M & O Cat. A Expenditure vs. Budget in 2009 Specific Items with over-spend

Item kCHFType Subsystem Budget Payments DifferenceExpenseM&O- A w/o PowerA.1. Detector related costsA.1.01 Magnet 30 36 (6)

A.1.02 Magnet controls 142 158 (16)Field Measurement 26A.1.03 Magnet power supply 40 43 (3)A.1.04 Gas systems 260 371 (111)Mixing and distrib. Completion/consolidat. 70Instrumentation 50A.1.05 Gas consumption 496 624 (128)CF4 without re- circulation 80Tuning of RPC re- circulation 50A.1.06 Cooling systems 220 228 (8)A.1.07 Cooling fluids(above –50°C) 220 336 (116)C6F14 plant refurbish. Replacing lost/contam fluid 180A.1.08 External cryogenics 375 385 (10)A.1.09 Cryogenic fluids (below –50°C) 90 112 (22)A.1.10 Moving/hydraulic systems 130 572 (442)Laser system 62Extra maintenance system 130VSL tuning of winches 40Additional tooling to replace blocktowers 100FSU manpower reinforcement 1 FTE 96A.1.11 Detector safety systems 180 845 (665)Remote sensing 52Overpressure seals 107Blast protection 120Fire protection completion 30Sniffer system SLA needed for L3 alarm 76DSS system completion/extension 55Leak detection system 50General safety & CMS specific training 30Dry air/ inertion system 70Cabling/hardware support 72A.1.12 Shutdown activities 410 820 (410)Field simulation 20Local cryogenics 32Extra Collab. Support (mechanics, cablers. Technicians) 256Specialist contractors (especially pipework) 98A.1.13 General Technical support 610 1,123 (513)Extra Collaboration workshop support 1 FTE 60Field Support Unit reinforcement 4 FTE 432A.1.14 UPS maintenance 80 91 (11)A.1.15 Electronics pool rentalsA.1.16 Beam pipe & vacuum 390 387 3A.1.17 Counting & control rooms 240 263 (23)

A.1. Detector related costs Total 3,913 6,394 (2,481)

A.2. Secretariat A.2.01 Secretarial assistance 225 225A.2.02 Economat 15 15A.2.04 Printing and publication 50 43 7

A.2. Secretariat Total 290 282 8A.3. Communications A.3.01 GSM phones; on- call service 20 20

A.3.02 Collaborative tools 100 366 (266)Cost of EVO Video- Conferencing tool 323

A.3. Communications Total 120 386 (266)A.4. On- line computing A.4.01 System management 796 723 73

A.4.02 Data storage, (temporary on disk) 52 45 7A.4.03 Detector controls 130 130A.4.04 Computers/processors/LANs 1,422 1,135 287A.4.05 Software licenses

A.4. On-line computing Total 2,400 2,033 367A.5. Test beams, calibration facilitiesA.5.01 General operation 60 59 1

A.5.02 Common electronics 15 15A.5.03 Electronics pool rentals 20 21 (1)A.5.04 Gas systems 10 10A.5.05 Gas consumption 10 9 1A.5.06 External cryogenics

A.5. Test beams, calibration facilities Total 115 114 1A.6. Laboratory operationsA.6.01 Assembly areas, clean rooms 20 20 ()

A.6.02 Workshops 250 257 (7)A.6.03 Laboratory instruments

A.6. Laboratory operations Total 270 277 (7)A.7. General services A.7.01 Cooling & ventilation 595 563 32

A.7.03 Power distribution system 60 124 (64)Electric infrastructure consolid (re- cabling, back- up, tfs) 90A.7.04 Heavy transport 340 441 (101)Extra crane driver & overtime 108A.7.05 Cranes 35 169 (134)Nacelle conformity & crane 28Nacelle/crane extra maintenance 82A.7.06 Cars 30 30 ()A.7.08 Survey 130 215 (85)Prolongation of Survey engineer & assistant 85A.7.09 Storage space 50 36 14A.7.10 Common desktop infrastructure 45 46 (1)A.7.11 Reviewing & Engineering 150 156 (6)A.7.12 Outreach 220 121 99

A.7. General services Total 1,655 1,901 (246)A.9. Core Computing Infrastructure & ServicesA.9.01 Central computing environment 458 440 18

A.9.02 Software process service 220 166 54A.9.03 User support 202 197 5A.9.04 Central production operations 695 585 110A.9.05 Hardware 70 20 50

A.9. Core Computing Infrastructure & Services Total 1,645 1,408 237

M&O-A w/o Power Total 10,408 12,796 (2,389)

Power 1,800 436 1,364

Expense Total 12,208 13,233 (1,025)

CMS Overspend in 2009

Cost break-down of the CMS end-cap cooling system bushing replacement. CHF Shutdown Activities : 401 k Mechanical team FSU (manpower) 296 Equipment management 14 Chemical analyses 15 Leak search instruments 31 Components and materials for repair 22 General stores items 23 Cars for repair teams 11k Cranes & lifts pre-emptive maintenance 22k Survey (manpower) 32k Heavy Transport 90k Additional Crane drivers & riggers Workshops 67 k Machinists and welders 56

(Collab) Specialist welding 11

Moving and Hydraulic systems : 117 k Maintenance of moving system 20

Maintenance of CMS nacelles 16 Hire of extra Nacelles 33 Hydraulics FSU (manpower) 48

Beampipe & vacuum 31k Support and protection 22 Ion pump Mag, shielding re-build 9 DSS expanded leak detection system : 149 k Controls interfaces 40 Concentrator trays 25 Installation, cabling &

commissioning (FSU, manpower) 84 Contract to ZEC service (Specialist pipework) 91 k Cooling repair QA/QC team 14k Cabling / recabling team 96 k (Collaboration : 8 persons) TOTAL 1121 k.

CMS Bushings Repair