PC Farms at CERN · 10-12 TB / month 1 month/year Manual Feed. 100 GB Cartridges. SONY DMS. CERN Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP DESY November 2, 1998 7-European

DESY November 2, 1998DESY November 2, 1998

CER

N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC Farms at CERN

Frédéric HemmerCERN-IT/PDP

DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 22CER

N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Disclaimer

! This will cover farms which imply an involvement of CERN’s computer center.

! There are other farms in strict online environments or “private” farms in building.


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Overview

! Off line farms• Linux farms• NT farms• Issues

! PC Technology & Performance! Online Farms & quasi online farms! Cost of ownership! Conclusions


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Linux Farms - Nomad

! Proof of concept in Summer 97! Straight NQS port! SHIFT SW client port! CERNLIB port! NOMAD observed a quasi linearity with

clock frequency compared to Alpha’s !!!• I.e. Alpha@266 MHz = PII@266 MHz

! Now 17 PC’s dual, 3 types of MB

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Cern Units

3Q97 4Q98 1Q98 2Q98 3Q98

NOMAD Installed Capacity


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Linux Farms - NA49

! NA49 already deployed privately a PC farm in their premises

! Request a new farm to be deployed in order to benefit from the computer center infrastructure (people and equipment …) in 1 H98

! Trivial deployment, running with NQS! Most PC’s are branded PC’s (HP)! Now completely off RISC for CPU! 18 DUALS @ 300->400 MHz


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

NA49 Analysis - data accessUnixUnixServerServerUnixUnixServerServerUnixUnixServerServer

CORECORETapeTape

ServersServers

HiPPIHiPPI

HPHPK260K260HPHPK260K260HPHPK260K260HPHPK260K260HPHPK260K260

FDDIFDDI

600 GB600 GB1 Run1 Run

PCPCPCPCPCPCPCPCPCPCPCPC

100BT

100BT

SGISGIChallengeChallenge

From experimentFrom experiment1010--12 TB / month12 TB / month

1 month/year1 month/yearManual FeedManual Feed

100 GB CartridgesSONY DMSSONY DMS 100 GB Cartridges


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Linux Farms (NA48)

! NA48 was using the QSW CS/2 (128 proc.)! CS/2 overload -> investigate PC’s in late

97! Installation of 12 Dual machines in 1Q98

and more ...


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Linux Issues

! EEPRO 100 B MP crashes! AFS support (MP)! NFS support (MP)! Commercial software! Manufacturer support for Linux! Very few Linux experts


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

NT offline Farms

! PCSF• Simulation facility but …

! COMPASS• Evaluating & benchmarking technology


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PCSF - Overview

! Configuration ! Applications! Data access! Specific work & solutions! Key issues! Conclusions


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PCSF - Goals

! Make PC+NT a standard option for Physics Data Processing, starting with simulation

! Establish a minimum management model for NT farm management

! Address scalability issues ! Gain Windows NT experience


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PCSF Milestones

! Joined RD47 in Autumn 96! Price inquiry issued in 12/96! Hardware delivered 4/97! Ready to use 6/97! RD47 report 10/97! Expansion 5/98


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PCSF Configuration (1)! Server running NT 4.0 Server SP3

• 1 dual capable Ppro @ 200 MHz, 96 MB, with 9 GB data disk (with mirroring). LSF central queues.

! Server running NT Terminal Server Beta 2• 1 dual Ppro @ 200 MHz, 128 MB, with 4 GB data disk.

Runs IIS 3.0 and is accessible from outside CERN. It also host the asp’s for Web access

! Servers running NT 4.0 Workstation SP3• 9 dual Ppro’s @ 200 MHz, 64 MB, 2*4GB • 25 dual PII’s @ 300 MHz, 128 MB, 2*4GB

All equipped with boot proms


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PCSF Configuration (2)

! Machines interconnected with 4 3com 3000 100BaseT switch

! Display/Keyboard/Mouse connected to a Raritan multiplexor

! PC Duo for remote admin access" There were problems with other products! All running LSF 3.0." LSF 3.2 does not work, support weak! Completely integrated with NICE


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Applications on PCSF

! ATLAS Dice simulation! NA45 1996 reconstruction! CMS reconstruction with Objectivity being

tested! LHCB simulation code ready! ATLAS reconstruction being ported! ATLAS/Marseille event filter prototype

scalability tests


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Data access

NT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PC

Network

Unix RFIOUnix RFIOServerServerUnix RFIOUnix RFIOServerServerUnix RFIOUnix RFIOServerServerUnix RFIOUnix RFIOServerServer

Unix TapeUnix TapeServerServer

stagexxxstagexxx commandscommands

RFIORFIO


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

ATLAS Level 3 DAQ

Processor FarmProcessor Farm

Event BuilderEvent Builder

# # ## # # # # ## # # # # ## # #

SFISFISFISFISFISFI

Readout BuffersReadout Buffers

1 GB/s1 GB/s

Storage (100 MB/s)Storage (100 MB/s)


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

ATLAS Event Filter

! Testbed for evaluating algorithms & sizing! Architecture & simulation studies! Monitoring, system management,

feedback, etc…! Interface prototypes (SFI, SFO)! Timescale : prototype -1 (I.e. end 98)! Status : sizing of an initial farm


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PCSF Usage

0

1000

2000

3000

4000

5000

6000

7000

8000

43 45 47 49 51 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41

Week #

NC

U h

ours

Idle

Used


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Specific work so far

! Installation (Remote Boot, Winstall, NICE replica’s, Install Server)

! User codes, CERNLIB, SHIFT! Job Starter! PC MGR! WNTS! Web Interface


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Installation! Disk cloning + change SID" Fastest method, but not very automated! Remote boot

• Remote boot install procedures with virtual disk• Use unattended setup, installs Winstall and other

things• Third party packages installed through Winstall

" boot prom support on some hardware


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Porting

! Usually porting code from Unix to NT is easy (NA45 code ported in 1 week)

! Usually porting production environment from Unix to NT is difficult (shell scripts)

! Porting build environment is difficult, better to use native tools (Dev Studio)" Mixing Unix and NT build environment,

revision control, etc.


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Jobstarter

! Initially inherited from Unix LSF CERN JobStarter

! Rewritten in C++, using PcMgrSvc for drive mapping

! Check execution preconditions! Clean up normal and abnormal job end! Kill popup dialog windows" Excel & Winzip in batch


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PcMgrSvc/Ctl

! Checks• Status of monitored processes/services• Amount of scratch space• Drive mapping(s)

! Map/Unmap drives! Sync. with time servers! Generate alarms on request! Gets all parameters from registry


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Web Interface

! As a solution to• Remote access from outside CERN• Access from non NT hosts

! Implemented as ASP’s with VB! Requires IIS on the server


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Web Interface - authentication


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Web Interface - Overview


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Web Interface - bjobs


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Web interface - bjobs result


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Windows NT Terminal Server


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Next Steps

! Finish and understand remote boot issues! Complete remote boot - remote install! AFS Integration! Build up resilience! Investigate how to use the new WfM, DMI,

PXE, ACPI, etc. initiatives! Investigate whether WSH is an alternative! Investigate NT’s I/O capabilities


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Key Issues

! AFS access! LSF support! Boot proms, equipment interoperability! CODE reintegration (Physics & CERNLIB)! Think Windows! Scalability & Management (home grown

solution vs. commercial apps.)! Remote & external access


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC with NT

! PC+NT has proven to work in batch environment, and is now an option for Physics Data Processing

! Farm management is less of a concern after have built a few tools (alternatives would be to use SMS or TNG), but some work is still needed

! Scalability has started to be addressed, but the relatively small number of nodes does not help here

! Considerable NT experience has been gained


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Issues so far

! Linux• EEPRO 100 B MP support• Commercial software• Manufacturer support• Very few local Linux experts

! NT• AFS access• LSF support

• Think Windows• Remote and external access

! PC• Interoperability (cards/MB combination• Remote Boot support


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC Technology evolution in 97

! Pentium Pro " Pentium II• 50 % raw performance increase• but 50 % cache performance reduction

! SEC " new motherboards! 440 FX " 440 LX (SDRAM, AGP)! Recent MB’s " embedded SCSI, E’net,

VGA! 100 Mbit E’net switches standard, 1000

Mbit arriving


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC Technology evolution in 98! Pentium II @300 MHz " Pentium Xeon @

450 MHz• MP support• 50 % cache performance increase

! Slot 2 " new motherboards! 440 LX " 440 BX, 440 NX (100 MHz, EDO)! Recent MB’s " No more available through

Intel, TYAN! 1000 Mbit/s E’net switches standard, >>

1000 Mbit/s arriving


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Racking evolution19981997


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

At the back ...


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Console multiplexors


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Fast Ethernet switches (Sep. 98)


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Fast Ethernet Switches (Oct. 98)


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

At the back of Fast Ethernet Switches (Oct. 98)


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Gigabit Ethernet Switches


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Network performance: Results! PC’s interconnected through 100 BaseT

3Com 3000 switch! Repeated with other H/W! Half duplex behavior! Block size does not matter! Linux uses less CPU than NT

" Good unidirectional performance" Disappointing CPU consumption on NT" Disappointing bi-directional performance


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC to PC Network performance

Linux Windows NT MaxMB/s CPU % MB/s CPU % MB/s

1->1 9->11 17 9->11 40 12.51<-> 1 4.6 10 5.3 44 12.5

7.5 21 5.2 44 12.512.1 10.5 25

1->3 11.7 55 12.53.9 20 4.23.9 20 4.23.9 20 4.2


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Network performance: issues

! Unexplained 0.5 MB/s observed with some eepro100 versions on PCRD hardware, but OK on PCSF

! Recent DEC E'net boards with chipset > 21140 give poor performance on Linux

! Surprising results PC/Alpha


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC/Alpha Network performance

Linux Alpha DUX MaxMB/s MB/s MB/s

1->1 11.1 11.1 12.51<-> 1 6.7 11.1 12.5

11.1 6.7 12.517.8 17.8 25


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC High Performance Networking

Gigabit Ethernet (10/98)! PII, 400 MHz, 440 BX,

100 MHz SDRAM, PCI 32/33, Tigon I

! 1500 bytes/packet: 28 MB/s, 40% CPU

! 9000 bytes/packet, 90 MB/s, 90% CPU

HiPPI (5/98)! PII, 300 MHz, 440LX,

SDRAM, Roadrunner to SGI O2000, 4 CPU, IRIX 6.4

! Transmit: 50 MB/s! Receive: 50 MB/s (53

MB/s with SMP)


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Disk performance! PC’s connected to SEAGATE ST19171W

using two Adaptec 2940 UW! NT needs a lot of tuning (default behavior

is to swap data out!)! Block size, BIOS settings, EDO/FPM does

not matter" Poor performance

" Windows NT even worse" Memory bandwidth is suspected


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Disk performance

# Streams Linux Windows/NT MaxMB/s CPU % MB/s CPU % MB/s

1 10.5 33 8.5 35 112 21 63 9.2 35 703 21 100 13.5 60 70

• Striping has no effect

••1 stream 2 stripes : 21 MB/s (22 max)1 stream 2 stripes : 21 MB/s (22 max)

••1 stream 3 stripes : 21 MB/s (33 max)1 stream 3 stripes : 21 MB/s (33 max)


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Disk performance: issues

! Memory bandwidth suspected! Need to test with LX/SDRAM, BX

SDRAM@100 Mhz! RISC PCI does not support variety of

boards! Combined disk/network performance even

worse : 5-6 MB/s on Linux


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Memory bandwidth (lmbench)

PCRD IBM DEC Prioris DEC PWS

Read MB/s 160 160 216 190Write MB/s 55 55 69 190


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Memory bandwidth (lmbench)

0

50

100

150

200

250

300

350

MB/s

Taho

e2

DK44

0LX

Thun

der2

Tige

r2

GA6

86DL

X

GA6

86(C

PU1)

GA6

86(C

PU2)

DEC

PWS4

33

SUN

Ultra

5

Thun

der1

00

N440

BX

Kaya

k XA

's

Com

paq

Prol

iant

160

0

Equipment

Mem readMem write


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Technology issues

! Technology evolves too fast (processors, chipsets, memory, motherboards, networking,...)• Changing environment/interoperability issues• Hard to maintain (obsolescence)• New NIC’s, drivers• Measurements valid only a few months" Difficult to establish stable environments

! Wide variety of solutions" Some combinations work, other not

! Local suppliers cannot help to solve problems


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC Performance summary! CPU performance fine! Network performance

• Some configurations do not work• Some configurations can saturate Fast Ethernet• Recent tests show excellent performance

! Memory performance• Now better than low-end RISC

! Disk Performance disappointing! Linux better than NT


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Online and quasi online farms

! NA48 Data Recording! NA45 Data Recording in Objectivity


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

NA48 Central Data Recording

Cisco 5505Cisco 5505

3Com

39003C

om 3900

FDDIFDDI

Fast EthernetFast Ethernet


XLNT GbitXLNT Gbit

FDDIFDDI

HiPPIHiPPI

GigaRouterGigaRouter

3Com 93003Com 9300Gigabit EthernetGigabit Ethernet

HiPPIHiPPI

CS/2CS/22.5 TB Disk space2.5 TB Disk space

SUN E450SUN E450500 GB Disk space500 GB Disk space

Event BuilderEvent BuilderOnline PC FarmOnline PC Farm

Sub detectorSub detectorVME cratesVME crates

7 KM7 KM

OfflineOfflinePC FarmPC Farm


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

NA 48 Data Recording in 98! May " September 1998! Raw Data on Tape

• 68 TB (1450 tapes, mainly 50 GB tapes)• 12.5 TB Selected Reconstructed Data• Total with 97 data : 96 TB

! Average Data Rate : 18 MB/s (peaks @ 23 MB/s)! CDR system can do 40-50 MB/s; limitation is CPU

Time available! Data recorded as files (4 million)


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

NA48 On Line Farm

! 11 Subdetector PC’s (dual PII-266, 128 MB)! 8 Event Building PC’s (dual PII-266, 128 MB, 18 GB SCSI)! 4 CDR routing PC’s (dual PII-266, 64 MB, FDDI)! All running Linux! Software event building in the interburst gap! Optional Software Filter (tags data)! Send data to computer center (local disk buffers : 144 GB , 2

hours)! On CS/2 : L3 Filtering and tape writing


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

NA48 Plans for 1999


HiPPIHiPPI

4 * SUN E4504 * SUN E4504.5 TB Disk space4.5 TB Disk space

EventEventBuilderBuilder

Sub detectorSub detectorVME cratesVME crates

7 KM7 KM

3Com

39003C

om 3900

HiPPIHiPPI3Com 93003Com 9300

Gigabit EthernetGigabit Ethernet


Cisco 5505Cisco 5505


On/OfflineOn/OfflinePC FarmPC Farm


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

NA45 Data Recording


HiPPIHiPPI

2 * SUN E4502 * SUN E450500 GB Disk space500 GB Disk space

Event BuilderEvent BuilderOn Line PC FarmOn Line PC Farm

Sub detector VME cratesSub detector VME crates

7 KM7 KM

3Com

39003C

om 3900

HiPPIHiPPI



SCISCI

3Com 39003Com 3900

3Com 93003Com 9300

NA48NA48

PCSFPCSF



N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

NA45 Raw Data recording in Objectivity

! October 98 ; November 98! Estimated bandwidth : 15 MB/s! Processes translate Raw Data format to Objectivity! Database files (1.5 GB) are closed, then written on tape! Steering done using a set of perl scripts on the disk

servers! On line filtering/reconstruction/calibration possible! Farm is running Windows NT! Reconstruction can use PCSF


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Current & Future Data rates at CERN

Year Experiments BandwidthMB/s

Raw DataTB/year

ProcessingSPECInt95

1990-2000

LEP 0.5 1 100

1997-2000

SPS 15-20 30-70 500

2000-2008

SPS 35 300 2000

2004- LHC 100-1000 3000 50000


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Summary

! On line PC farms are being used to record data at sensible rates (Linux)

! Off line PC farms are being used for reconstruction/filtering/analysis (Linux/NT)

! Still a lot to do on scalable farm management, global steering, CDR monitoring, etc..


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

PC Total Cost of Ownership

42%

51%

4%1%2%

HWMUXRackNetworkSysadm

• Software not included

• Install labor not included

• Assumes 3 years lifetime


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

DEC 8400 (12-Way) Cost of Ownership

• Software & SW maintenance not included

• Assumes 5 years lifetime

72.8%

0.1%13.8%0.9%12.4%

HWMUXHW MaintNetworkSysadm


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

General Conclusions (1)

! PC’s are now used for online, quasi online and offline environments

! The “offline” is now part of the online! The I/O is still done using RISC/Unix but

recent MP Xeon may change this …


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs


! PC technology is moving very fast• Good for performance• Not so for stability, interoperability• Not so for understanding issues

! The general management of large farms is not solved but …• Number of initiatives/standards/tools may

help us here : WfM, DMI, PXE, ACPI, SMS, TNG, etc.


N

CER

N --

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs

Euro

pean

Lab

orat

ory

for P

artic

le P

hysi

cs


! Linux vs. NT … the battle is over• Choose the one suitable to your application• NT can be used• Linux is usable (and offers more performance).

! PC real costs are usually not well understood

Documents

PC Farms at CERN · 10-12 TB / month 1 month/year Manual Feed. 100 GB Cartridges. SONY DMS. CERN Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP DESY November 2, 1998 7-European