openlab 2003 1

S.JarpCERN

CERN openlabTotal Cost of Ownership

11 November 2003

Sverre Jarp

openlab 2003 2

S.JarpCERN

Agenda Day 1 (IT Amphitheatre) Introductory talks: 09:00 – 09:30 Welcome. (W. von Rüden) 09:30 – 10:15 Setting the scene: A Global View of the plans for the LHC Computing

Grid – (L.Robertson) 10:15 – 10:45 Coffee break Part 2: 10:45 – 11:15 Review of calculated cost summary for 2003 (S. Jarp) 11:15 – 12:00 Estimated LCG Personnel and Materials costs (2004 – 2008) (B.Panzer) 12:30 – 14:00 Lunch Part 3 14:00 – 14:45 Planning the LCG Fabric at CERN (T. Cass) 14:45 – 15:30 External Testimony: Michael Levine (Pittsburgh Supercomputer Center) 15:30 – 16:00 Coffee break 16:00 – Discussions and conclusion of day 1 (All)

Day 2 (IT Amphitheatre MORNING, 513 1-024 afternoon) Discussions with HP: 08:45 – 10:45 One-on-one with HP Discussions with Intel: 11:00 – 13:00 One-on-one with Intel Discussions with IBM: 14:00 – 16:00 One-on-one with IBM

openlab 2003 3

S.JarpCERN

AIM Understand

Total Cost of Ownership “Linux fabric” in the Computer Centre

CPU servers Disk servers, Tape servers + Silos

To a “sufficient” level of detail

2003 (July/August) Budget situation Rather than actual sums across all cost centres

Personnel + Materials No direct measurement of quality

Nor the fraction of capacity actually in use at a given time

openlab 2003 4

S.JarpCERN

Cost of manpower Complicated issue

Approach chosen: Take IT salary budget for staff and fellows

Including direct benefits (pension fund, health care, etc.) Without counting management overhead

Neither inside IT nor CERN Actual number: 168.8 KCHF

Excluded: Associates + Students

In general, paid from the materials budget Associates: Some do not necessarily spend most of their

time at CERN Students: Avoid very short stays (Summer students)

openlab 2003 5

S.JarpCERN

IT budget categories In the TCO exercise the total IT budgets were

initially split into 9 categories: Computer Centre Services:

Cost of the Physics Fabric and closely related services, such as internal and external networking, etc.

Grid Development/Deployment Openlab Computing Infrastructure (non-physics oriented):

Windows, Mail, Web services, Phone exchange, etc. Product Support (mainly for Engineers):

Digital electronics, CAD, license office, Solaris support. etc. Controls for physics experiment

so-called “on-line computing” Database Services and Applications Communications Infrastructure for the LHC Accelerator Divisional Management

openlab 2003 6

S.JarpCERN

IT Division (total)Materials

(KCHF)FTE % of IT total

(Materials + FTEs)

(Basic) Computer Centre Services 12,486 79.45 40.4

Grid Development and Deployment 150 32.63 8.8

Openlab --- 3.75 1.0

Infrastructure (non-physics) 4,714 34.60 16.4

Product Support 1,954 17.73 7.7

Controls 686 18.95 6.1

Databases and Applications 1,662 15.65 6.7

Accelerator Communications 3,294 3.50 6.1

Divisional Management 796 21.35 6.9

Total 25,742 227.6 100.0

openlab 2003 7

S.JarpCERN

Box countCentral Linux Batch “white box” servers 1490

Central IDE Disk servers 250

Central PC Tape servers 60

Linux systems in the Computer Centre 1800

Linux systems outside the computer centre

1600

Central Windows servers 150

Central Solaris servers 50

IP devices on Campus: 28K

openlab 2003 8

S.JarpCERN

Capacity Numbers Other relevant capacity numbers are:

Total installed PC CPU capacity: ~1.2 Million SpecInt2000 (SI2K)

CPU capacity purchased by August: 400 * 1.6 KSI2K = 640 KSI2K

Total disk space: 200 TB Disk space acquired by August:

50 TB at required performance (actually: 65 TB)

Tape silos installed: 10 (with 6000 slots each) Tape drives:

50 STK9940B (+ 20 STK9840 with low usage)

openlab 2003 9

S.JarpCERN

Cost of Central Services Batch/Interactive services:

Acquisition costs for batch/interactive servers, software licenses and maintenance

Managed storage: Tape and disk servers (purchase and maintenance)

Computer Centre Infrastructure: Various contract staff contracts; Monitoring and

Supervision; Remedy Fabrics:

The budget line only covers the CC upgrade this year.

The two "Operations" lines: Cover costs associated with the various groups

(travel, associates/consultants, workstation purchases, etc.)

openlab 2003 10

S.JarpCERN

Cost of Central ServicesSUB-PPA Description Materials STAF+FELLSUM

KCHF FTE KCHFRCF-BATCH FIO Batch Services 616.0 2.97 1117.3RCF-INT FIO Interactive Services 203.0 0.75 329.6RCF-LINUX ADC Linux Support 50.0 3.69 672.9RCF-OPS Operations 710.0 2.25 1089.8RCF-STORE Managed Storage 1820.0 8.55 3263.2RCI-CAMP CS Campus Network 1795.0 9.65 3423.9RCI-CCI FIO Computer Centre Infrastructure1475.0 8.70 2943.6RCI-HELP US User Support and Quality Assurance260.0 6.35 1331.9RCI-LIC PS License Office 75.0 0.75 201.6RCI-OPS Operations 308.0 2.50 730.0RDC-CAMPM CS Campus NW management 0.0 0.00 0.0RDD-BCKUP FIO Backup service 586.0 0.80 721.0RGS-SCRTY Computing Security 87.0 5.76 1059.3TGR-FABR Fabrics (including CC upgrade)1620.0 13.08 3827.9RCF-XHOME ADC Home directories AFS 391.0 2.20 762.4

PS Physics-related (CVS, ASIS, SWTOOLS)0.0 3.50 590.8RDD-EXTN CS External Networking 1390.0 3.45 1972.4

FIO SERCO insourcing 0.0 8.00 1350.4Energy and Water 1100.0 0.00 1100.0Linux Fabric for Physics 12486.0 82.95 26488.0

openlab 2003 11

S.JarpCERN

Cost of FabricDescription Materials STAF+FELLSUM Frc-1 Materials-1 Frc-2 FTE-2 New Sum

KCHF FTE KCHF KCHF KCHF KCHFBatch Services 616.0 2.97 1117.3 1.00 616.0 1.00 501.34 1117.3Interactive Services 203.0 0.75 329.6 1.00 203.0 1.00 126.60 329.6Linux Support 50.0 3.69 672.9 0.50 25.0 0.50 311.44 336.4Operations 710.0 2.25 1089.8 0.90 639.0 0.90 341.82 980.8Managed Storage 1820.0 8.55 3263.2 1.00 1820.0 1.00 1443.24 3263.2Campus Network 1795.0 9.65 3423.9 0.15 269.3 0.10 162.89 432.1Computer Centre Infrastructure1475.0 8.70 2943.6 0.90 1327.5 0.90 1321.70 2649.2User Support and Quality Assurance260.0 6.35 1331.9 0.05 13.0 0.05 53.59 66.6License Office 75.0 0.75 201.6 0.10 7.5 0.10 12.66 20.2Operations 308.0 2.50 730.0 0.90 277.2 0.90 379.80 657.0Campus NW management 0.0 0.00 0.0 0.15 0.0 0.10 0.00 0.0Backup service 586.0 0.80 721.0 0.10 58.6 0.10 13.50 72.1Computing Security 87.0 5.76 1059.3 0.30 26.1 0.30 291.69 317.8Fabrics (including CC upgrade)1620.0 13.08 3827.9 1.00 1620.0 1.00 2207.90 3827.9Home directories AFS 391.0 2.20 762.4 1.00 391.0 1.00 371.36 762.4Physics-related (CVS, ASIS, SWTOOLS)0.0 3.50 590.8 0.50 0.0 0.50 295.40 295.4External Networking 1390.0 3.45 1972.4 0.90 1251.0 0.90 524.12 1775.1SERCO insourcing 0.0 8.00 1350.4 0.50 0.0 0.50 675.20 675.2Energy and Water 1100.0 0.00 1100.0 0.90 990.0 0.00 0.00 990.0Linux Fabric for Physics 12486.0 82.95 26488.0 9534.2 9034.3 18568.4

openlab 2003 12

S.JarpCERN

Costs beyond the fabric In many cases non-fabric-related costs

dominate Campus Network:

2991 KCHF

User Support and Quality Insurance: 1265 KCHF

Computer security: 741 KCHF

Etc.

In other words: The fabric benefits from large demands on other services

openlab 2003 13

S.JarpCERN

CPU part of fabricSUB-PPA Description New Sum Frc-3 Fractional Sum ((New Sum) * Frc-3)

KCHF KCHFRCF-BATCH FIO Batch Services 1117.3 1.00 1117.3RCF-INT FIO Interactive Services 329.6 1.00 329.6RCF-LINUX ADC Linux Support 336.4 0.75 252.3RCF-OPS Operations 980.8 0.75 735.6RCF-STORE Managed Storage 3263.2 0.00 0.0RCI-CAMP CS Campus Network 432.1 0.75 324.1RCI-CCI FIO Computer Centre Infrastructure 2649.2 0.75 1986.9RCI-HELP US User Support and Quality Assurance 66.6 0.75 50.0RCI-LIC PS License Office 20.2 0.75 15.2RCI-OPS Operations 657.0 0.75 492.8RDC-CAMPM CS Campus NW management 0.0 0.75 0.0RDD-BCKUP FIO Backup service 72.1 0.75 54.1RGS-SCRTY Computing Security 317.8 0.75 238.4TGR-FABR Fabrics (including CC upgrade) 3827.9 0.75 2870.9RCF-XHOME ADC Home directories AFS 762.4 0.75 571.8

PS Physics-related (CVS, ASIS, SWTOOLS)295.4 0.75 221.6RDD-EXTN CS External Networking 1775.1 0.00 0.0

FIO SERCO insourcing 675.3 0.75 506.5Energy and Water 990.0 0.75 742.5CPU part of fabric 18568.4 10509.3

openlab 2003 14

S.JarpCERN

Disk/tape part of fabricSUB-PPA Description New Sum Frc-3 Fractional Sum ((New Sum) * Frc-3)

KCHF KCHFRCF-BATCH FIO Batch Services 1117.3 0.00 0.0RCF-INT FIO Interactive Services 329.6 0.00 0.0RCF-LINUX ADC Linux Support 336.4 0.25 84.1RCF-OPS Operations 980.8 0.25 245.2RCF-STORE Managed Storage 3263.2 1.00 3263.2RCI-CAMP CS Campus Network 432.1 0.25 108.0RCI-CCI FIO Computer Centre Infrastructure 2649.2 0.25 662.3RCI-HELP US User Support and Quality Assurance 66.6 0.25 16.7RCI-LIC PS License Office 20.2 0.25 5.1RCI-OPS Operations 657.0 0.25 164.3RDC-CAMPM CS Campus NW management 0.0 0.25 0.0RDD-BCKUP FIO Backup service 72.1 0.25 18.0RGS-SCRTY Computing Security 317.8 0.25 79.5TGR-FABR Fabrics (including CC upgrade) 3827.9 0.25 957.0RCF-XHOME ADC Home directories AFS 762.4 0.25 190.6

PS Physics-related (CVS, ASIS, SWTOOLS)295.4 0.25 73.9RDD-EXTN CS External Networking 1775.1 0.00 0.0

FIO SERCO insourcing 675.3 0.25 168.8Energy and Water 990.0 0.25 247.5Disk/tape part of fabric 18568.4 6284.0

openlab 2003 15

S.JarpCERN

Summary All-in-all:

IT estimates (on average) that it has spent ~(1 + 1) MCHF on basic hardware across

CPU servers Disk servers + Tape servers

Our calculation shows: Linux fabric in total: 18.6 MCHF CPU part: 10.5 MCHF Disk/tape part: 6.3 MCHF External networking: 1.8 MCHF

Cost on top of basic hardware: 14.8 / 2 = 7.4

openlab 2003 16

S.JarpCERN

Backup

openlab 2003 17

S.JarpCERN

2003 (replacement) cost IT main entities

Acquisition (CHF)

Installation, Maintenance (CHF)

Quantity (per unit)

CERN total Acquisition (so far) 2003

CPU server 2,000 300 1600 SI2K 1.2 MSI2K 400

Disk server 10,000 2,000 1 TB (1.3) 200 TB 50

Tape cartridge

130 - 200 GB

Tape silo 150,000 + 6,000 slots 10 -

Tape drives (9940B/9840)

30,000/N.A. 45003000

30/20 MB/s 50 + 20 -

AFS home/project

80,000 1 TB 4 TB

AFS scratch 10,000 2,000 1 TB