View
214
Download
1
Tags:
Embed Size (px)
Citation preview
Evaluation of the 2-way Opteron 1U system
Klaus Schossmaier
CERN EP-AID
Computing Seminar
3 September 2003
Performance test of PCs based on AMD platforms
Computing Seminar, 3 Sep 2003
22-way Opteron 1U system
Welcome
The EP-AID group constitutes the core team of the Data Acquisition (DAQ) System for the LHC Experiment ALICE
Development of the DAQ software called DATE
Deployment of the DAQ system Computing fabric: ~500 nodes, GbE switches, ~400 optical links Permanent recording during Pb-Pb run: 1.25 GB/s during 1 month/year
Selecting products Evaluation of Linux platforms Intel: Pentium II, Pentium III, Xeon AMD: AthlonMP (32bit), Opteron (64 bit)
Computing Seminar, 3 Sep 2003
32-way Opteron 1U system
Overview
In March 2003 we received the Newisys 2100 evaluation system (“Beachhead”)
This 2-way 1U enterprise-class server is based on AMD Opteron processors (x86-64 architecture)
Our evaluation was done under NDA: Hardware, chassis, BIOS Linux operating systems Development tools Benchmarking Applications
Computing Seminar, 3 Sep 2003
42-way Opteron 1U system
Chassis & Periphery
Rack mountable Size: 42mm x 428mm x 725mm Robust construction
Easy to mount the rack module Easy to open the cover (no screws) Solid connectors
Not very hot or noisy! Good documentation
Labeling, e.g. MAC addresses Newisys “Getting Started Guide” Newisys support website
Computing Seminar, 3 Sep 2003
52-way Opteron 1U system
Hardware – Front View
2x hard disk bays:
- empty
- 36.7GB SCSI
hot swap
CD-ROM drive
Floppy drive
Operator panel
for the SP
4x 1GB DIMM
DDR266 SDRAM
4x 1GB DIMM
DDR266 SDRAM
2x Opteron 240, 1 MB L2 cache, 1388 MHz
2x Opteron 244, 1 MB L2 cache, 1793 MHz
Power supply
465W
(March – May)
(June – now)
Computing Seminar, 3 Sep 2003
62-way Opteron 1U system
Hardware – Rear View
PCI-X slot 64/133
full-length
PS/2 mouse
keyboard
SVGA video
Trident Blade 3D
8 MB
2x 10/100BaseT
for the SP
Service processor (SP)
2x 100/1000BaseT
BCM5703X chip
PCI-X slot 64/100
half-length
AMD-8131 PCI-X bridge AMD-8111 Southbridge
Computing Seminar, 3 Sep 2003
72-way Opteron 1U system
Booting & BIOS
Booting/Resetting/Shutdown is fast (~10 sec)
PhoenixBIOS 4.0 Release 6.0 Easy to enter the BIOS setup Well designed configuration menus Upgrade via the phlash16 flash-utility or the SP
System BIOS Revision 00.09.06 initial 00.09.08 bug fix 01.08.00 Opteron 1.8 GHz 01.09.00 Opteron “C” step
Computing Seminar, 3 Sep 2003
82-way Opteron 1U system
Linux Operating Systems
CERN Red Hat 7.3.2 Linux (2.4.18 kernel) Installation via boot floppy and NFS Everything works without any change in 32-bit mode
UnitedLinux 1.0 (2.4.19 kernel) Consortium of Conectiva, SCO, SuSE, Turbolinux Installation via CD-ROM (4 disks) using YaST Runs in 64-bit mode
Red Hat 8.0.95 = GinGin64 Installation via CD-ROM (6 disks) failed
Computing Seminar, 3 Sep 2003
92-way Opteron 1U system
Development Tools
GNU 2.96 C/C++ compiler default for Red Hat Linux installations only for 32-bit executables
GNU 3.2.1 C/C++ compiler default for UnitedLinux installations for 64-bit executables, 32-bit compatibility libraries
Portland Group Compiler Technology (PGI) Workstation 5.0-beta Fortan/C/C++ compiler commercial product for 64-bit executables
Other development tools (e.g. numerical libraries, debuggers, code migration, performance analyzers) exist, but not tested.
Computing Seminar, 3 Sep 2003
102-way Opteron 1U system
Benchmarking
CPU benchmarking Ubench 0.32, gcc 2.96, 32-bit mode Ubench 0.32, gcc 3.2.1, 64-bit mode bench/stress rootmarks, 64-bit mode
Memory bandwidth Stream 4.0, gcc 2.96, 32-bit mode Stream 4.0, gcc 3.2.1, 64-bit mode
Ethernet performance Iperf 1.2, gcc 2.96, 32-bit mode
Disk performance Bonnie, 64-bit mode
Computing Seminar, 3 Sep 2003
112-way Opteron 1U system
0
20000
40000
60000
80000
100000
120000
0 500 1000 1500 2000 2500 3000
MHz
ub
ench
/#C
PU
Pentium II Pentium III Opteron Athlon MP Xeon Itanium
Benchmark - CPU (Ubench)
64-bit
74955
97749
52122
66714
8806479798
1.0 GHz1.4 GHz
1.8 GHz
2.4 GHz
Computing Seminar, 3 Sep 2003
122-way Opteron 1U system
Benchmark - CPU (ROOT)
Done by Fons Rademakers (EP-SFT) Porting of ROOT and CINT was trivial
OO data analysis framework with ~900000 locs Porting to x86-64 under United Linux 1.0 in one afternoon Code was already 64-bit safe (no special x86-64 ifdef’s) No problem with gcc 3.2.1
Performance is very good
Opteron 1.4 GHz
1 MB cache, 8 GB RAM
Opteron 1.8 GHz
1 MB cache, 8 GB RAM
Itanium2 1.0 GHz
3 MB cache, 2 GB RAM
P4 Xeon 3.06 GHz
512 KB cache, 2 GB RAM
750 rootmarks
g++ 3.2.1 (-O2)
950 rootmarks
g++ 3.2.1 (-O2)
497 rootmarks
g++ 3.2.3 (-O2)
750 rootmarks
g++ 3.2.3 (-O2)
550 rootmarks
32-bit binary compiled on
P4 with g++ 3.2.2 (-O2)
1020 rootmarks
ecc 7.1 (-O)
Computing Seminar, 3 Sep 2003
132-way Opteron 1U system
Benchmark - Memory
1x Stream: 2x Stream: 4x Stream:
2x Opteron, 1.8 GHz, HyperTransport: 1006 – 1671 MB/s 975 – 1178 MB/s 924 – 1133 MB/s
2x Xeon, 2.4 GHz, 400 MHz FSB: 1202 – 1404 MB/s 561 – 785 MB/s 365 – 753 MB/s
4x DIMM
1GB DDR266
Avent Techn.
4x DIMM
1GB DDR266
Avent Techn.
Computing Seminar, 3 Sep 2003
142-way Opteron 1U system
Benchmark - Ethernet, Disk
Network benchmarking 2x Broadcom BCM5703X on motherboard 10/100/1000 Mbps Ethernet, auto-negotiation bcm5700 kernel module, 64-bit mode up to 105 MB/s, stable traffic during 4 days 20%+18% sys CPU as TCP/IP receiver 20%+2% sys CPU as TCP/IP sender
Disk benchmarking Seagate ST336753LC, 15kRPM LSI53C1030 dual channel Ultra320 SCSI controller 26.7 MB/s for writing a 6 GB file 36.2 MB/s for reading a 6 GB file
Computing Seminar, 3 Sep 2003
152-way Opteron 1U system
Service Processor (SP)
Dedicated SP on-board PowerPC based Own IP name/address Front panel Command line interface Web-server
Remote administration System status Boot/Reset/Shutdown Flash the BIOS
Computing Seminar, 3 Sep 2003
162-way Opteron 1U system
Applications - DAQ
Front-end computer / Event-Builder DATE v4.6 Red Hat 7.3.2 32-bit mode, binary compatible! Event-Building: 113 MB/s with 2 sources, 80% CPU Porting to 64-bit mode needs to be done
DAQ services NFS server Run control, message logger, database Online monitoring: performance, data quality
Computing Seminar, 3 Sep 2003
172-way Opteron 1U system
Applications – non DAQ
MySQL 4.0.13 for x86-64 64-bit mode, binary distribution UnitedLinux 1.0 ~3300 remote INSERT queries (80 chars) per sec
Web Servers HEP computing
Data analysis Simulation tools Visualizations
ALICE Performance
Data size
Hits ~1.4Gb
Digits ~1.1Gb
CPU time on 800MHz PIII
Hits ~24h
Digits ~15h
dNch/d ~ 8000
slice: 2o in
TPC
5000 PbPb events at 5.5 TeV/nucleon pair
were producedfor PPR
84’210 primary particles
ITS
PHOS
TRD
TOF
HMPID
Computing Seminar, 3 Sep 2003
182-way Opteron 1U system
Résumé
Merits Design: chassis, hardware, BIOS Execution: seamless 32-bit & 64-bit mode Benchmarking: CPU, memory, Ethernet, disk Applications: DAQ services, analysis, simulations
Improvements Linux operating systems: more in 64-bit mode Tool chain: more non-commercial tools
Questions Cost: processor and whole system Roadmap: AMD’s 64-bit processors
Thanks AMD: Pierre-Yves Ferrard, Francesco Torricelli, Bill Zipoy CERN ALICE: Pierre Vande Vyvre
Computing Seminar, 3 Sep 2003
192-way Opteron 1U system
References
Web pages www.amd.com/opteron www.x86-64.org www.hypertransport.org www.newisys.com
Technical Reports “Performance tests of PCs based on the AMD AthlonMP”,
ALICE-INT-2003-020 A report about the 2-way Opteron system is forthcoming
Further Talks 15 Sep 2003: “Linux platforms for the ALICE DAQ”
Access to our Opteron machine Please ask me for an account