Andrew Fecheyr LippensCAC: Actividad 215 October 2009

C L U S T E R C O N F I G U R AT I O N S

A cluster should be configured depending on the applications required to run on it. The type of cluster (high productivity, performance or availability) will define the specific needs of the cluster : storage, memory, level of redundancy, interconnection latency and bandwidth.

As a goal for the Budget Cluster we aim to build a teraflop cluster with acceptable availability using as little money as possible. The goal for the High Performance Cluster is to configure a machine able to enter the TOP500 list (based on the June 2009 ranking). We describe the Budget Cluster with more detail and discuss the different configuration choices. For the High Performance Cluster we include a brief description of the components.

Budget Cluster

For the budget cluster we choose inexpensive commodity computing nodes without any form of hardware redundancy. High availability requirements should come from the amount of computing nodes and the software’s ability to redirect workload from a failed node to online nodes. Another possibility is to have one or two of the computing nodes as hot or cold spare’s.

We do however configure the master and storage node with significant hardware redundancy, as failure of this node will bring down the whole cluster.

Master Node

For cost efficiency the master node will also provide the central storage for the cluster. A 2U rackserver with 8 hot swappable 3,5” slots was chosen. Because any downtime of the master node will result in downtime of the complete cluster, this node has a high level of hardware redundancy.

Supermicro provides a redundant power supply, registered ECC memory and a IPMI server management module with KVM-over-IP on a dedicated LAN port. The server also has a 3 year warranty with OnSite replacement within 4 hours of failure, ensuring that any downtime will be quickly resolved.

As for the storage component, a 8xSATA/SAS hardware raid solution by Areca was chosen. The 8 Western Digital 1TB drives can be used in RAID-6 or in RAID-5 with a hot spare bringing the total storage capacity to about 6TB with enough redundancy to cover two hard drive failures.

Andrew Fecheyr-Lippens! CAC Actividad 2

! 1 of 5

The high availability features, decent cpu power (8 x 2,4Ghz), fast and ample memory (24GB) and adequate fault-tolerant storage (6TB) bring the total cost of this node to 4337!. The Ahead-IT webshop provided a rough estimate of power usage for the chosen configuration: on full load it should require 337.24W (1,41A), on idle 231.85W (0,97A) and on average 274.79W (1.14A).

Component Brand Amount

Barebone Supermicro SC825TQ-R720LP 2U rackserver

Network 2 x Intel Gbit LAN 2 x 1 Gbit

Memory 6 x Kingston 4096MB DDR-3 1333Mhz Reg. ECC 24 GB

Processors 2 x Intel Xeon E5530 Nehalem 2.4Ghz Quad Core, 5.86GT/s, 8MB, 32/64 bit, VT, HT, 80W 8 x 2,4 Ghz

Hard Disks 8 x Western Digital 1000GB, SATA II, 32MB, 7200rpm, raid edition 8 TB

KVM Supermicro IPMI 2.0 with KVM over IP via dedicated LAN port

Power Supply Redundant 720 Watt High-efficiency 80+ Gold Level Power Supply

Hardware RAID Areca 1222, 8 x SATA/SAS hardware RAID, 256MB cache, PCI-E + Backup Battery

Warranty 3 Year 4 hrs EasyOnSite & Advanced Replacement

Extras Sliding rails, Power Cable (1,8m), Ethernet Cable (3m) and Nuts & Straps

Computing Node

The dutch/belgian server supplier Ahead-IT is widely know for it’s stunt prices for cheap commodity servers. This month they offer a Supermicro SCS216 1U rack server with an Intel Xeon processor starting at 300". We upgraded the default configuration to 4 GB of memory (+ 70") and QuadCore processors (+ 150"). We also added the extra pre-installation service so all the hardware is tested and the firmware updated (+ 30") bringing the total kost per node to 550!.

Component Brand Amount

Barebone Supermicro SCS216 1U rackserver (Chassis: CSE-811i-260)

Network 2 x Intel Gbit LAN 2 x 1 Gbit

Memory 2 x Kingston 2048 MB DDR2 667Mhz 4 GB

Processor Intel Xeon X3220, 2.4Ghz Quad Core, FSB1066, 8MB, 32/64 bit, VT, DBS, 105W 4 x 2,4 Ghz

Hard Disk 80GB Sata drive 80 GB

Power Supply 260 Watt Power Supply

Warranty 1 Year Advanced Replacement, upgradable to 3 year

Extras Sliding rails, Power Cable (1,8m), Ethernet Cable (3m) and Nuts & Straps

Andrew Fecheyr-Lippens! CAC Actividad 2

! 2 of 5

Theoretical Computing Power

According to Intel, every Quad Core Xeon X3220 has a theoretical power of 38,4 Gflops. (http://www.intel.com/support/processors/sb/cs-023143.htm)

The chart on the left shows the cluster cost vs computing power (in Gflops) for different numbers of computing nodes. The left-most datapoint corresponds with 1 computing node, the rightmost with 32 computing nodes. The increased costs at 11, 17 and 23 nodes corresponds with the need for different numbers of PDUs and KVM switches.

With 32 computing nodes the cluster has 128 processing cores, good for a Rpeak of 12228,8 Gflops. To achieve the goal of 1 Tflops Rmax , the cluster needs to have an efficiency of 81,78% or more.

Interconnect

Because of the budget constraints on this cluster we follow the same principle of choosing a commodity, widely used, interconnect: Gigabit ethernet. A HP ProCurve Switch (2510G-48) supplies us with 48 gigabit ethernet ports and the ability to create VLANs to isolate the master-computing node communication from external communication.

Power Distribution

The cluster requires 32 power outlets for the computing nodes, 2 outlets for the master node and 1 outlet for the network switch. To provide these 35 power plugs we added three Ingrasys iPoMan 1201/1202 power distribution units (PDU). Each PDU provides 12 remotely controlled power outlets with easy consumption monitoring and environment monitoring (temperature, humidity and in/out voltage and current levels). As every PDU is able to supply up to 32 Amps, the total cluster can use 96 Amps, more than enough for our budget cluster.

Rack Space

Component Amount x Units Total Space (U)

Master Node 2U 2

Computing Nodes 32 x 1U 32

PDUs 3 x 1U 3

Network switch 1U 1

KVM 2 x 1U 2

40

The cluster needs at least 40 rack units for all the hardware, so we include a 42U Triton Delta S 800 x 1000 Rack (RMA-42-A81-BAX-A1). If more is needed, Triton also offers a 45U Delta S Rack.

0

10000

20000

30000

0 325 650 975 1300

# of nodes: Total cost vs Computing power

Tota

l Clu

ste

r Co

st (!)

Gigaflops

Andrew Fecheyr-Lippens! CAC Actividad 2

! 3 of 5

Hardware management: KVM

To easily manage the 32 computing nodes, two stackable 16 port LEVEL ONE 1631 KVM switches are supplied. The master node has its own KVM-over-IP module so we don’t need an extra port. With this configuration every node can be controlled remotely over the network.

Total Cost

We sent a test order for all the needed hardware to built the budget cluster to Ahead-IT and received a quote back from them. The quote included a significant 2% quantity discount on the master node and the network switch, an extra 900" discount and a bonus free upgrade to 3 year EasyRMA service on the computing nodes. This levels the warranty on all nodes to three years and brings the grand total, including transport to Belgium, to 25.301,23 ! without VAT and 30.614,49 ! with 21% (Belgian) VAT. The quote is included as an appended pdf file at the end of this report.

High-Performance Cluster

For the High-Performance Cluster we chose to go with blade servers to minimize needed space. We also opted for InfiniBand interconnections to dramatically reduce the latency and overhead of the communication. Prices for these components come from ThinkMate and Colamco.

Master Node

The Master node supplies the cluster with 3TB of storage in dual RAID5 and has 96GB of fast DDR3 1333MHz memory.

Master and Storage Node $23,838.00

2 x Quad-Core Intel® Xeon® X5570 2.93GHz 6.4GT/s QPI 8MB Cache (95W)2 x Quad-Core Intel® Xeon® X5570 2.93GHz 6.4GT/s QPI 8MB Cache (95W)

Supermicro X8DTU-F - Intel® 5520 Chipset Mainboard - Dual Socket LGA1366 - Integrated IPMI 2.0 with Dedicated LANSupermicro X8DTU-F - Intel® 5520 Chipset Mainboard - Dual Socket LGA1366 - Integrated IPMI 2.0 with Dedicated LAN

12 x 8GB PC3-10600 1333MHz DDR3 ECC Registered12 x 8GB PC3-10600 1333MHz DDR3 ECC Registered

Supermicro SC216A-R900UB - 2U - 24 x 2.5" SAS - 900W RedundantSupermicro SC216A-R900UB - 2U - 24 x 2.5" SAS - 900W Redundant

24 x 300GB SATA 10000RPM - 2.5" - Western Digital VelociRaptor™ WD3000BLFS24 x 300GB SATA 10000RPM - 2.5" - Western Digital VelociRaptor™ WD3000BLFS

2 x 3ware 9650SE-12ML 12-Port (SATA) (MultiLane) (PCIe)2 x 3ware 9650SE-12ML 12-Port (SATA) (MultiLane) (PCIe)

2 x 3ware Battery Backup Unit for 3ware 9650SE Series2 x 3ware Battery Backup Unit for 3ware 9650SE Series

Supermicro 2-Port 4X 20Gb/s InfiniBand DDR cardSupermicro 2-Port 4X 20Gb/s InfiniBand DDR card

Three Year Warranty with Advanced Parts Replacement, 4h Response OnsiteThree Year Warranty with Advanced Parts Replacement, 4h Response Onsite

Blades

Blade Module Configuration $7,476.00

Supermicro SuperBlade Intel Xeon SBI-7426T-S3 Blade ModuleSupermicro SuperBlade Intel Xeon SBI-7426T-S3 Blade Module

2 x Quad-Core Intel® Xeon® X5570 2.93GHz 6.4GT/s QPI 8MB Cache (95W)2 x Quad-Core Intel® Xeon® X5570 2.93GHz 6.4GT/s QPI 8MB Cache (95W)

2 x Supermicro SNK-P0033P Heatsink2 x Supermicro SNK-P0033P Heatsink

6 x 4GB PC3-10600 1333MHz DDR3 ECC Registered VLP6 x 4GB PC3-10600 1333MHz DDR3 ECC Registered VLP

2 x 64GB Intel® X25-E 2.5" SATA Solid State Drive (Single Cell) (SLC)2 x 64GB Intel® X25-E 2.5" SATA Solid State Drive (Single Cell) (SLC)

Dual 4x DDR 20Gbps Infiniband or Dual 10GbE Ethernet (Mellanox ConnectX)Dual 4x DDR 20Gbps Infiniband or Dual 10GbE Ethernet (Mellanox ConnectX)

Three Year Warranty with Advanced Parts Replacement, 4h Response OnsiteThree Year Warranty with Advanced Parts Replacement, 4h Response Onsite

Andrew Fecheyr-Lippens! CAC Actividad 2

! 4 of 5

Each 7U DatacenterBlade (SBE-714D) offers space for 14 Processor Blades (SBI-7426T-S3). Every Blade module is configured with 2 Intel Xeon Quad-Core (Nehalem) processors and has a total of 24GB of memory. We also use RAID1 mirrored 64GB Intel X25-E SSD drives.

The Datacenter enclosure is configured with an Infiniband switch.

Enclosure Configuration $7,698.00

Supermicro DatacenterBlade Enclosure - SBE-714E-R42 4 x 1400W Redundant Power Supply - High-Speed InterconnectSupermicro DatacenterBlade Enclosure - SBE-714E-R42 4 x 1400W Redundant Power Supply - High-Speed Interconnect

Supermicro SuperBlade Infiniband SwitchSupermicro SuperBlade Infiniband Switch

Theoretical Computing Power

The cluster is composed of 6 Blade Enclosures (42U in total), with 84 Blade Modules, bringing the total amount of computing cores to 84 x 2 x 4 = 672. With each core good for a theoretical 46,88 Gflops the total Rpeak for this High-Performance Cluster is an amazing 7,875 Tflops. This is not enough to enter the last TOP500 list. Combining four of these racks should bring the Rpeak to 31,503 Tflops, which would give the cluster an honest chance to be included in the TOP500.

Interconnect

A Mellanox InfiniScale IV Switch with 36 ports provides the cluster with enough 40 Gbps InfiniBand connections in a 1U rack-mountable package. The cost for this core switch is $8,658. (link)

Hadware management: Included

The SuperBlade chassis management module (CMM) provides total remote control of individual server blades, power supplies, cooling fans, and networking switches remotely. System administrators enjoy the management ease and reassurance of continuous onboard instrumentation monitoring (temperature sensors, power status, voltages and fan speed). Remote power control capabilities to reboot and/or reset the server are available as well as remote access to the BIOS configuration and operating system console information vial SOL (Serial over LAN) or embedded KVM-over-IP capabilities.

Total Cost

When we add the core switch ($8,658) and a 45U rack ($1,000) the total amounts up to $707,668.

Andrew Fecheyr-Lippens! CAC Actividad 2

! 5 of 5

BedesignGordunakaai 629000 GentBE

Offerte 2090412 Client number 12222 VAT number : Date: 13/10/2009Amount Description Unit price VAT Netto

1.000 LEVEL ONE 1631 KVM-over-IP (optie) switch 16 port (cascade up to 128 ports) PS2/USB 1U Web management

1195.00 21.00 % 1195.00

2 x LEVEL ONE 1631 KVM-over-IP (optie) switch 16 port (cascade tot 128 poorten) PS2/USB 1U Web management 32 x Level One cable set voor KVM PS2/USB - 1.8 mtr 3.000 Ingrasys iPoMan 1201/1202, 12 port over IP, 16/32A, control power usage per port 431.50 21.00 % 1294.50

1 x Ingrasys iPoMan 1202, 12 port over IP, 32A, 2 inlets, controle stroomverbruik per poort 1 x Ahead-IT outlet stroomkabel 1.8M 1.000 HP ProCurve Switch 2510-48G port stackable 44 x 10/100/1000 + 4 x Dual Personality Ports 950.00 21.00 % -2.00 % 931.00

1.000 TRITON DELTA S81 RACK 800X1000 42U BLACK RMA-42-A81-BAX-A1 860.88 21.00 % 860.88

1.000 SNSC820: max 8 x SATA/SAS hot-swap // 48GB DDR-3 // redundant PSU 4336.68 21.00 % -2.00 % 4249.95

1 x 3 Jaar Ahead-IT Server Service met EasyRMA+, Advanced Replacement Service, 4 uurs EasyOnSite 9x5 service** & gratis levering

1 x Supermicro Slim DVD lezer 1 x Supermicro X8DTL-iF Server Moederbord met KVM-over-IP en software Intel ICH10R SATA RAID 1 x Supermicro SC825TQ-R720LP, hot-swappable, redundante voeding, 2U, rails incl. 1 x Areca 1222, 8 x SATA/SAS hardware RAID, 256MB cache, PCI-E 1 x Areca ARC-BBM Battery Backup Unit 1 x Supermicro IPMI 2.0 met KVM over IP standaard inclusief (via dedicated management LAN port) 6 x Kingston 4096MB DDR-3 1333Mhz Reg. ECC 8 x Western Digital 1000GB, SATA II, 32MB, 7200rpm, raid edition 2 x Intel Xeon E5530, Nehalem 2.40Ghz Quad Core, 5.86GT/s, 8MB, 32/64 bit, VT, HT, 80W 1 x EasyRMA+ met Advanced Replacement Service & 4 uurs EasyOnSite 9x5, 3 jaar ( gratis ) 1 x Ja, preinstallatie service Kleur heeft geen belang, Ja: kooimoeren, 10 stuks, Ja: set van 5 kabel straps, Netwerk Kabel 3M, Outlet stroomkabel 1.8M

voor bv. remote powerswitch 32.000 Supermicro SCS216 1U rackserver, upgradable: 1GB DDR-2, 80GB, Dual Core Pentium, free

delivery, sliding rails, dual Gbit LAN 550.00 21.00 % 17600.00

1 x Supermicro SCS216 1U rackserver, upgradable: 1GB DDR-2, 80GB, Dual Core Pentium, gratis levering, sliding rails, dual Gbit LAN

1 x 1GB geheugen naar 4GB: +70 EUR (u ontvangt 2 x 512MB los erbij) 1 x PROMO: Intel Xeon X3220, 2.4Ghz Quad Core, FSB1066, 8MB, 32/64 bit, VT, DBS, 105W: +150 EUR 1 x Ja, preinstallatie service Kleur heeft geen belang, Ja: kooimoeren, 10 stuks, Ja: set van 5 kabel straps, Outlet stroomkabel 1.8M voor bv. remote

powerswitch, Netwerk Kabel 3M -3.000 Discount SCS216 promo + free upgrade to 3 year EasyRMA+ 300.00 21.00 % -900.00

1.000 Transport rack 60.00 21.00 % 60.00

33.000 Recupel 0.363 euro 0.30 21.00 % 9.90

. OrderNR: 20090284 // Free offer request // Online/offline advance payment and delivery via UPS (standard, paying) Dear Sir, Dear Madam, / / This request is just a test configuration of a computer cluster we are considering. / Please do not

take any commercial action based on this request, it is of purely informational nature. / / Regards Fecheyr Lippens Andrew // Gordunakaai 62 // 9000 // Gent // Belgium // Amount without VAT VAT % VAT amount Total EUR

25301.23 21.0 5313.26 30614.49

23/10/2009 30614.49 Ahead-IT?s offers are free of obligation, errors/amendments aside, and without being binding. Signing this document means the customer agrees with the order of the products as described above. Upon placing the order, you automatically agree to our General Terms and Conditions, available 24/7 on: www.ahead-it.eu/nl/algemene-voorwaarden/ . * and ** and ** = see website.