Avoiding the $25 Million Server Data Center Power, Cooling, and Growth

8/14/2019 Avoiding the $25 Million Server Data Center Power, Cooling, and Growth

1/46

Avoiding the $25 Million ServerData Center Power, Cooling, and Growth

Prepared for:IBM Corporation

By:Sine Nomine Associates43596 Blacksmith Square May 1, 2007Ashburn, VA 20147


2/46

Avoiding the $25 Million Server

2007 Sine Nomine Associates ii

_________________________________________________________________________

Copyright 2007 Sine Nomine Associates

All rights reserved. Trademarks and brand names mentioned in this document are used only foridentification purposes. Sine Nomine Associates acknowledges the ownership of all product

trademarks by their respective manufacturers.

This report was developed by Sine Nomine Associates with IBM cooperation and funding. Thisreport utilizes publicly available information provided by various companies and sources,including IBM. The opinions are those of the report's author, and do not necessarily representIBM's position on these issues.


3/46


2007 Sine Nomine Associates iii

Document History

DateRevision/Reissue

Nature of Change

May 1, 2007 Version 01 New Release


4/46


2007 Sine Nomine Associates iv

Table of Contents

1 INTRODUCTION............................................................................................1

1.1 Constraints on Data Center Growth ......................................................................................... 1

1.2 The $25 Million Server ............................................................................................................... 2

1.3 Discrete Servers, Blade Servers, and System z ......................................................................... 2

1.4 Power and Cooling Impact of Virtualization............................................................................ 3

1.5 About Sine Nomine Associates................................................................................................... 4

2 DATA CENTER SCENARIO MODEL............................................................5

2.1 Fundamental Assumptions......................................................................................................... 5

2.2 Sources of Input Data ................................................................................................................. 5

2.3 Structure of the Model................................................................................................................ 6

2.3.1 Globals .......................................................................................................................................... 6

2.3.2 Platforms....................................................................................................................................... 7

2.3.3 Workloads..................................................................................................................................... 9

2.3.4 Scenarios..................................................................................................................................... 11

3 RESULTS IN DETAIL..................................................................................13

3.1 Workload Aggregation ............................................................................................................. 13

3.2 Constrained Server Density ..................................................................................................... 14

3.3 Total Cost of Ownership........................................................................................................... 16

3.3.1 Energy Cost................................................................................................................................. 18

4 CONCLUSIONS...........................................................................................21

4.1 Power and Cooling.................................................................................................................... 21

4.2 Total Cost of Ownership........................................................................................................... 21

4.3 Sine Nomine Associates' Recommendations........................................................................... 22

4.4 Avoiding the $25 Million Server .............................................................................................. 23

APPENDIX A: POWER/COOLING MODEL.......................................................24


5/46


2007 Sine Nomine Associates v

APPENDIX B: EXAMPLE CALCULATION........................................................39

BIBLIOGRAPHY ................................................................................................41

List of Tables

Table 1: Aggregate workload per server used in model................................................................ 14

Table 2: Constrained logical server density per enclosure ............................................................ 15

Table 3: Total logical servers in entire data center........................................................................ 20

List of Figures

Figure 1: Relative frequency of occurrence of server types in the model ..................................... 13

Figure 2: Logical server density per rack location unit ................................................................. 16

Figure 3: TCO per logical server for one, three, and five years.................................................... 17

Figure 4: Fifth-year incremental TCO for each scenario .............................................................. 18

Figure 5: Power utilization per rack location unit ......................................................................... 19

Figure 6: Annual power cost per logical server............................................................................. 20


6/46


2007 Sine Nomine Associates 1

1 Introduction

In this paper Sine Nomine Associates has analyzed the problem of functional density in a moderndata center, considering traditional rack-mount discrete servers, new-generation blade servers,

and virtual servers on both Intel and System z mainframe platforms.

This paper presents the results of deploying each of these designs in a hypothetical urban-areadata center, owned by a single large enterprise or shared by multiple medium-sized enterprises (aswould be the case in a hosting provider). This scenario was selected to illuminate an emergingconstraint on Fortune 500 IT: the fundamental operational limits of existing high-value datacenter space.

1.1 Constraints on Data Center Growth

Modern data centers have benefited from, and have been impeded by, recent technologicalchanges in computer architecture. Faster processors, denser mass storage, and inexpensivememory chips allow more and more work to be delivered from a single system. On the otherhand, as applications grow more complex, more resources are demanded from each application.Also, modern semiconductors1 consume power in direct proportion to their clock speed and totheir gate count. As both circuit complexity and clock speed increase, the result is a greater-than-linear increase in power consumption per server.

Additionally, rack density has increased dramatically in recent years. Servers that formerly wouldhave occupied two or even three rack units (2U or 3U) now fit into a single 1.75 inch rack unit(1U). Blade servers increase the density beyond 1 server per unit by orienting the circuit boardsvertically and cooling the system with carefully designed, highly efficient airflow components. Atypical blade server can pack 14 blades into 9 rack units, while another packs 16 blades into 10rack units. The density increases, but the power per server is largely dependent on processor andmemory, not board configuration, and therefore the required power per rackincreasessignificantly in today's environment.

Although some data centers are located in rural areas where real estate is relatively inexpensive,many are found in densely populated urban areas where real estate is extremely expensive orunavailable at any practical price. Nevertheless, urban data centers remain vital for organizations, because they already are in service and therefore incur only recurring costs rather than initialacquisition costs, and because they are close in space and time to other operational centers of thecompany, its customers and partners. In both urban and rural settings, there is usually a limit tohow much electrical power can be delivered to a given facility. In the rural area, it might (ormight not) be possible to "throw money" at the utility company to obtain a larger power feed, butin urban areas there is often simply no more capacity to be had at any price.

This is important because data centers are ultimately constrained not just by space, but also bypower. This leads to the problem of the "$25 Million Server."

1 This statement refers to metal-oxide semiconductor (MOS) chips, rather than bipolar junction chips. MOS(and CMOS) chips are used for almost all microprocessor and memory circuits, with bipolar chips nowbeing used only for ultra-high-speed functions such as multi-gigabit-per-second network transceivers.


7/46



1.2 The $25 Million Server

Data processing needs of companies typically grow over time, as the company grows in size andas the demands of the marketplace lead to increasingly complex and numerous systems. Given adata center of a certain size, with a limited amount of power and cooling available, the density ofservers steadily increases. Eventually, it simply becomes impossible to deploy one more server in

that data center. For a well-managed data center, under disciplined administration, this pointoccurs when the nominal power capacity (less a prudent safety factor) is reached. For companiesthat fall short of perfection, this point occurs when the already-overbooked power and coolingsystems finally fail, leading to overheated servers, tripped breakers or shutdowns, or brownouts.

Whether the breakpoint is a prudently chosen safety stop, or an empirically determined failurepoint, it is inescapable, and the company's only choice is to build a new data center. Thus, ineffect, that one additional server resulted in an expense of tens of millions of dollars, and hasbecome known in data center design parlance as "the $25 million server." The purpose of thispaper is to identify ways in which careful selection of server platform can push the $25 millionserver as far into the future as possible.

1.3 Discrete Servers, Blade Servers, and System z

This paper examines multiple configurations for each of three different platform types: discreteservers, blade servers, and mainframes, comparing their strengths and weaknesses.

As microcomputers2 began to appear in large-scale data centers, the natural tendency was toreshape a traditional PC mechanically so that it would fit into a standard 19 inch equipment rack.Early PC servers occupied 4 standard 1.75 inch rack units (vertically) each, or perhaps even more,but advancing technology has reduced typical servers to 2U or 1U sizes. Currently, a discretemicrocomputer server packs two or more processors into a 1U chassis.

To push for even higher density, server manufacturers took a cue from telecommunications

equipment makers and began offering blade servers. These use vertically oriented plug-in boards,with one server each, in an enclosure that is typically 7U to 10U in height. Sixteen compute nodesin a 10U chassis, for example, results in a density of 1.60 servers per rack unit, a 60%improvement over standard 1U discrete servers. Blades also offer the advantage of internalnetwork switches and (frequently) storage area network (SAN) switches, so fewer external switchports are needed for a given number of servers. Typically, a blade deployment needs only twonetwork and two SAN ports per blade chassis, not per blade, whereas discrete servers need two ofeach port per server.3

Both discrete and blade servers can be operated in either physical or virtual mode. Physical modededicates a compute node to each logical server, and provides maximum isolation at the expenseof a great deal of underutilized resources, since a typical modern CPU is much faster than all but

the most intensively compute-bound applications require. Virtualized microcomputer serversoffer strong isolation in software, with a hardware assist, and allow resource sharing and leveling

2 In this paper, the term "microcomputer" refers to any computer in which the central processing unit is amicroprocessor. All x86-based servers are considered microcomputers in this context.

3 This assumes N+N redundancy, the most common configuration.


8/46



between logical servers on a single physical platform. Virtualization also offers advantages inmanageability and in business continuity planning, because virtualization tools such as VMwareallow management of the virtual system images in a robust way.

Mainframes such as the IBM System z take consolidation with virtualization to its logical limit,concentrating dozens, hundreds, or even thousands of servers into a single physical machine.

Although it is theoretically possible to run a single system image on a mainframe, practicalexperience shows that this is the optimal solution in only a few very specialized situations.Mainframes offer full virtualization at the hardware level, and they are optimized to have loadleveling and resource sharing managed by the hypervisor, not the guest operating system orapplication.

Whereas a typical microcomputer has an extremely fast CPU and often a relatively modest I/Ocapability, the mainframe has processors that are only modestly faster than a microcomputer, butI/O capabilities that far outstrip those of an ordinary microcomputer. The fabrication budget andlarger physical size budget of a mainframe allow ultra-wide internal data paths with sophisticated"channel" controllers that offload almost all of the device management and data transfer workfrom the main CPU to a dedicated I/O processor. Mainframe CPUs spend much less of their time

servicing low-level device interrupts. Even so, the mainframe is not suited for extremelycompute-intensive tasks such as ray-trace rendering of 3D graphics.

Each of the three platforms has its benefits, and its drawbacks, and this paper looks at all three ina hypothetical enterprise data center with an aggregate workload that closely simulates workloadsSine Nomine Associates has observed in actual customer sites.

1.4 Power and Cooling Impact of Virtualization

System administrators often separate servers for reasons other than a need for additional hardwareresources. Organizational boundaries may require (or encourage) segregation of servers alongdepartmental or project lines, even though a single server could have been shared from a purely

technical standpoint. Servers may be dedicated to major customers or business partners due tocontractual constraints, or because these external organizations need administrative-level accessto certain applications. Some applications require separate servers because they are incompatiblewith other software that might logically coexist on the same server if not for that fact. Finally, forapplications with low average utilization and high peak utilization for brief intervals, it isnecessary to provision dedicated servers to meet the peak demand, rather than the averagedemand that dominates the server's workload.

The result of any of these situations is a data center full of dedicated servers that areunderutilized, albeit for very good business reasons, and this leads to low productivity of theenergy used by the systems in aggregate.

Virtualization allows underutilized physical servers to be consolidated in a way that maintains theisolation of their operating systems, application runtime environment, security context, networkbroadcast domain, and administrative authority. Load leveling in the virtual data center enablesplanners to share the peak-handling overhead among applications whose instantaneous workloadsare unlikely to peak at the same time. Having fewer servers in the data center proportionallyreduces the power consumption of the electronics and of the associated cooling machinery,resulting in substantial savings of recurring costs.


9/46



1.5 About Sine Nomine Associates

Sine Nomine Associates is an IT engineering consultancy experienced in multiple hardware andsoftware platforms, from microprocessor-based systems through UNIX workstations andminicomputers, and into large-scale mainframe systems. Sine Nomine Associates engineers workwith UNIX, Linux, Windows, Macintosh, and mainframe operating systems, taking a vendor-

neutral approach to solving customer problems.


10/46



2 Data Center Scenario Model

To analyze the power and cooling impact of virtualization and the various platforms on whichvirtual servers can be run, Sine Nomine Associates has created a detailed mathematical model of

a hypothetical data center. The model is presented in Appendix A of this paper. This sectionexplains that model in detail.

The basic question this model answers is, "Given that the data center cannot expand in space orpower, how can the enterprise expand to meet future needs, and what is the best platform to allowmaximum growth potential at the least cost?" The intent of the model is to determine how best toavoid the "$25 million server" problem.

The model will show how to avoid this scenario through server consolidation onto virtualizedhardware, including both microcomputers and System z mainframes.

2.1 Fundamental Assumptions

The model simulates an urban-area data center of a given size, with the presumption of no roomavailable onsite for further expansion and little or no additional electrical power available at thatsite. This may sound artificially constrained, but Sine Nomine Associates has worked with manycustomers in New York City, Los Angeles, Chicago, and Washington DC areas who face thisvery situation.

A single large enterprise with a company-owned data center is the focus of this study, thoughmost of the conclusions are applicable to a collocation facility shared by several medium-sizedenterprises. In all scenarios considered here, the model assumes that the company has broadlydeployed storage area network (SAN) technologies with efficient aggregation and managementcapabilities. The space, cost, and power required by the SAN arrays and management, and data

backup systems, are not included in this model, on the principle that the quantity of enterprisedata storage is not significantly affected by the choice of processing platform. This is the oneassumption that may not directly translate between a single large enterprise and multiple medium-sized enterprises, although one could presume multiple smaller SANs as well.

All of the servers in this model run Debian Linux, so that the cost of the operating system isequalized (at zero, in fact) between all platforms.4 Hypervisors (VMware for the Intel platforms,and z/VM for the mainframe) are included in the pricing calculations of this model, however.

2.2 Sources of Input Data

The model accepts as its inputs a number of data sources, including:

4 Since Windows can only run on Intel servers, there is no reason to compare Windows across Intel andmainframe platforms. If an enterprise runs a mixed Windows and Linux environment, this model can stillbe useful -- just assume that the Windows servers exist in a separate section of the data center whose squarefootage is not included in these calculations, and that the Windows environment is not subject to migrationfrom one platform to another.


11/46



Constants that represent the hardware characteristics of the various platforms andconfigurations being compared.

Constants that represent the capabilities and limitations of the hypothetical data center,such as its physical size, available power per unit area, and other parameters.

Constants for occupancy factors defining the efficiency with which the "white tile" spacein the data center can be utilized, amount of cooling energy required per watt ofcomputing energy, and similar factors.

Costing figures that capture the national average cost per kilowatt-hour of electricity, thecosts of hardware and software for the various platforms and configurations, and the costof data center construction per unit area and unit of available power.

Performance constants that allow qualitative differences between platforms to benormalized so that, at least at an estimate level, the capacity of different architectures canbe compared.

Constants that represent typical loading factors for various components of a typical serverunder various common application loads, and the relative frequency with which each ofthese server types is likely to occur in an enterprise data center.

Wherever possible, the input constants have been obtained from specific manufacturerspecification datasheets, well-respected published reports, or internal findings developed by SineNomine Associates in capacity planning and performance measurement projects. In a few cases,such as the total size of the hypothetical data center, an arbitrary constant must of necessity beselected, and these were chosen based on Sine Nomine Associates' extensive experience withmultiple large- and medium-sized customers ranging from Fortune 500 financial companies tomid-sized insurance and manufacturing enterprises.

2.3 Structure of the Model

The model was deliberately broken down into sections that are decoupled, as much as possible,from one another. This makes debugging easier, makes the model easier to understand, and keepsit as configuration-neutral as possible. The sections of the model include the following:

2.3.1 Globals

The Globals page provides constants, and wherever possible the numbers were generated fromobjectively verifiable external sources. Constants on this page include standard units per rack, thearbitrarily chosen size of the hypothetical data center, and electricity cost data (from the United

States Department of Energy).

There are occupancy constants that reflect the desired number of empty units (for futureexpansion) per rack, and the "IT yield" ratio which defines how much of the data center's "whitetile" area can actually be occupied by racks. This figure was obtained from an Uptime Institutereport based on statistics gathered from numerous actual data centers. The IT yield compensatesfor service aisles, building support columns, electrical and network wiring chases, cooling ducts,and in-room configuration workspace.


12/46



The Globals page also contains parameters for the non-platform infrastructure components, suchas network switches, storage area network (SAN) switches, and keyboard-video-mouse (KVM)switches. These items are presumed to be selected independently of the processing platform. Thechoice of platform will cause the number of needed Ethernet, SAN, and KVM ports to vary, butin this model it does not change the type of device selected for each of these roles. Theinfrastructure parameters also include a multiplier that determines the level of redundancy of each

infrastructure component (2X for Ethernet and SAN, 1X for KVM).

The Globals page is specifically constrained so that its data, whether calculated or constant,depends on no field from any other page. The intent is that the global constants are independentof platform and workload. The pricing data that appears on this page is per-unit pricing forspecific components, with no extended totals based on quantity anywhere on this page.

2.3.2 Platforms

The second page contains the hardware and costing details for each platform, without regard tothe workload. A few fields are calculated based on the global constants, but no fields on this pageare in any way dependent on the workload or scenario pages.

2.3.2.1 Intel Configurations

The Intel systems were all configured and priced to use Intel Xeon Woodcrest 5140 processorsrunning at 2.33 GHz clock speed. Other configurations of the same server brands can use faster orslower processors, but this particular configuration was selected because it was available acrossall of the servers being compared, and is a very common implementation for corporate datacenters. The basic configuration of each Intel system type is considered a typical build-out for astandalone Internet or intranet server, whereas the expanded system contains an additionalprocessor and much more RAM, so that it is more suitable for a virtualized platform.

The blade servers considered in this model consolidate up to 10 processor blades into a 7Uchassis, for a density of about 1.43 servers per rack unit. The non-blade Intel servers, of course,offer a density of exactly 1.00 servers per rack unit.

The pricing for the Intel systems was obtained from the manufacturer's online catalog and fromquotes obtained from their presales team.

2.3.2.2 Mainframe Configurations

In the case of the IBM System z mainframes, three configurations are considered, but in all casesthe assumption is that z/VM will be used as a hypervisor to virtualize the platform. It is almostuniversally agreed that the System z is not an economically viable platform for running a single

Linux instance, nor is it viable for most real-world applications running one server per logicalpartition (LPAR).5 Thus, this model does not consider those configurations.

5 Logical Partitions (LPARs) are a way to divide a single physical system into a relatively small number ofisolated sub-units that can be viewed by the operating system as if they were standalone systems. LPARs inthe mainframe environment are typically used for gross system partitioning (e.g., test vs. development vs.production) rather than fine-grained partitioning at the individual application level.


13/46



IBM does not publish list prices for mainframes, preferring to allow the customer to receive aclient-specific quotation from the IBM sales representatives. The prices reported here are acomposite of actual prices that have been paid by known customers, and published IBM prices forcomponents that do have a list price.

2.3.2.3 Infrastructure Differences

The quantities of network switch ports and SAN ports are differently calculated for the System zthan they are for the Intel systems. The Intel systems use a calculated value based on the numberof ports per infrastructure component, the redundancy multiplier, and the chassis size of thecomponents. This calculation, plus the mandated number of open slots per rack, iterates with thesize of the server enclosure to determine how many rack units are available for actual serverenclosures per rack. This, in turn, is used to calculate the number of servers that can be installedin a rack.

For the System z mainframe, most of those calculations are not applicable. The System z has allthe SAN and Ethernet ports it needs, and a single locally attached console replaces KVM. Thereis no need for a KVM-like switch function because there is only one physical machine in the"rack". Also, rack-level calculations do not apply because the mainframe occupies one full rack,period.

Note that neither the Intel systems nor the mainframes include upstream SAN and Ethernetswitching or routing. This has the net effect of giving a slight model bias to the Intel platform, butthe error is negligible because of the use of rack-local switches to fan-in the connections.

2.3.2.4 Normalizing Platforms to Common Specifications

Comparison of mainframe processors to microprocessors is difficult and widely debated amongcomputing experts, because the processors and their support components are radically different.

The differences are qualitative, not just quantitative. Microprocessors are typically categorizedinto CISC (complex instruction set computer) or RISC (reduced instruction set computer), withRISC systems having simpler binary operations that complete at blinding speed, whereas CISCmachines provide complex binary operations that take longer but do more. There is, of coursemuch overlap between these categories. Mainframe processors are, in a sense, an ultra-CISCdesign, with machine instructions carrying out very complex operations in microcode ornanocode.

A particular distinguishing feature of the mainframe environment is that virtualization ofhardware is much more developed and mature than even the best microprocessor-based systems.A larger portion of the mainframe instruction set is virtualized than the Intel instruction set, whichmeans virtual environments require less software emulation of hardware. Mainframeadministrators routinely run nested virtualized environments for testing and developmentsituations, something that is almost totally impractical on microcomputers.

Mainframes also possess ultra-high-speed interconnection mechanisms and massive-widthinternal data buses, allowing extremely fast transfers over virtual LANs. In fact, some transfersare actually memory-mapped so that the transfer never actually happens, and the data is simplyhanded off between sender and receiver by pointer manipulation.


14/46



As a result of these architectural differences, it is impossible to provide a single metric thatdirectly converts Intel gigahertz to mainframe MIPS specifications.6 However, based on extensivefield experience Sine Nomine Associates typically uses a figure of 6.0 MHz per MIPS for thelatest generation System z processors from IBM. Though this number is a simplification, it isnecessary in any kind of a capacity-planning model to provide some sort of normalizationmechanism. Sine Nomine Associates has assisted many customers with system integration in a

mixed mainframe/microcomputer environment, and this ratio has proven to be a reasonableapproximation for typical workloads.

In the Platforms page, the processing capacity of each system is converted to mainframe MIPSusing the above figure, and these MIPS numbers are carried forward to the scenario calculations.

2.3.3 Workloads

The effectiveness of processing platforms depends heavily on the nature of the workload, and anyresponsible expert will advise clients to carefully consider their specific workload when choosinga mix of platforms.

For the sake of this model, rather than choosing a single arbitrary workload "per server", theWorkloads page catalogs a total of eight unique workloads and their typical resourcerequirements, based on published benchmarks and monitoring data collected by Sine NomineAssociates on various projects. This allows the model to flexibly simulate detailed scenarios forcomparison, allowing much more precise analysis of relative configuration strengths. Theworkload types include:

Static web server -- This server type assumes a traditional publishing-type web site, withstatic and lightly dynamic content. Lightly dynamic content would consist of server-sideincludes (SSI), blogs, or content management systems written in Perl or PHP.

Database server -- The database server is presumed to run an SQL-compliant relationaldatabase manager as its primary function. The model assumes that the database servermost frequently processes simple or medium-complexity queries, and that large andcomplex queries are less common.

Domain name server (DNS) -- The DNS server is the quintessential lightly loadedmachine. DNS queries typically return only a small amount of data, and the data is read-only except for occasional changes. This is almost perfectly optimized for cacheperformance within the DNS server and within the operating system.

Ecommerce web server -- In a multi-tier application, this server would represent thecustomer-facing outer tier that processes HTTP queries from client browsers and

6 Originally, the acronym MIPS stood for Millions of Instructions per Second, representing the number ofbinary-level operations that a processor could do per second. This, however, becomes a rather amorphousconcept on modern high-complexity processors because, among other things, instructions take vastlydifferent amounts of time to complete. A full thread-context switch obviously takes much longer than asimple arithmetic operation, for example. In the mainframe context, then, MIPS has become a shorthandterminology that represents a relative, rather than absolute, processor speed. In essence, MIPS is a way tocompare one mainframe model to another when predicting capacity of a proposed upgrade.


15/46



provides the presentation-layer logic for the application. This may include portalcomponents being merged into the overall page, AJAX processing of dynamic-contentform fields, and adaptation to browser differences. This layer specifically excludes thecore business logic of the application as well as database services.

The relative frequency of this profile is higher than that of the business logic server (see

below) because in many intermediate-complexity situations this tier may function withouta separate business logic server, connecting directly to the database server.

File server -- A typical departmental file server performs very little processing, but agreat deal of network and disk I/O. The load imposed upon the file server is largelyindependent of the protocols in use, so this workload profile could represent a serveroffering NFS, CIFS (SMB), AFS, or some combination of these.

Business logic server -- This represents the second tier of a three-tier client/serverapplication, residing between the ecommerce web server and the database server. Thebusiness logic server in this profile is assumed to run a Java application in an enterprise-grade container such as IBM WebSphere, JBoss, or BEA WebLogic.

Management and/or monitoring server -- Servers that collect and analyze logmessages from other servers, and which are used as a host for monitoring and systemmanagement applications, perform a great deal of network I/O and a moderately-highamount of disk I/O. Typically, log aggregation uses efficient buffering algorithms toreduce the amount of time spent on disk writes, but they cannot control the characteristicsof the inbound network traffic as closely.

Email server -- The models final common server combination is the email server. In atypical enterprise, a small number of email relays receive mail from the Internet, and asmall number transmit outbound mail to the Internet. Within the company, however,there are usually a larger number of mailbox servers that hold user inboxes and archived

messages. The mailbox servers perform almost no processing, though the relay serversspend much of their time scanning for spam, malware, and other undesired content. In allcases, however, the I/O load of an email server is large. The profile shown in this modelis an aggregate assuming a few relay servers and a larger number of mailbox servers.

The individual workloads are aggregated into a weightedaverage workload, based on the relativefrequency of these servers in enterprises where Sine Nomine Associates has done other projects.For example, static web servers and file servers are quite common, often occurring at a rate ofone per department, whereas domain name servers (DNS) are relatively rare because one servercan reasonably support thousands of users.

The load factor for each workload profile is segregated according to specific resources, including

the main processor, main memory (RAM or working storage), disk subsystem, and networkinterface. The processor workload is further divided into application, kernel, and cryptographicloads, with the kernel load representing the time that the operating system spends managingphysical devices such as disk drives and network interfaces.

The workloads are normalized to a discrete Intel server with one processor running at 2.33 GHz,locally-controlled SATA disk storage, no cryptographic coprocessing, and ordinary Ethernetadapters.


16/46



Finally, each workload profile includes a ratio estimating how much of the network traffic will beserver-to-server rather than server-to-client. This is important in the virtualized environment,because much of the server-to-server traffic will be routed on virtual LANs that have extremelylow overhead. These ratios are based on Sine Nomine Associates' direct experience working withFortune 500 data center implementations around the world.

2.3.4 Scenarios

The final page of the model brings together a matrix of platforms and workload to produce themodel's scenario results. This page is the logical opposite of the Globals page, in that it consumesdata from all the other pages but does not feed back to any of them.

2.3.4.1 Primary Calculations

The scenario page initially calculates a "nominal" result for each configuration, based on itstheoretical density and capacity. The theoretical density is adjusted for infrastructure components(network, SAN, and KVM switches) but not for power limitations of the data center facility.

This calculation provides a number of servers that could fit into the data center based on theirown architectural limits. In the case of the virtualized environments, this is also limited by thebasic factors of the aggregate per-server workload.

2.3.4.2 Power Limitations

Real-world data centers do not have unlimited power, and it is quite common, especially in urbanareas, for data center density to be constrained not by rack capacity, but rather by power andcooling limitations.

This section of the page takes this into account. From the Globals page, the page obtains a figure

for watts per square foot, and then derates the average rack density based on the total amount ofpower available to the entire facility.

2.3.4.3 Platform Adjustments

The platform adjustments section encapsulates the qualitative differences among platforms,including:

On both Intel and mainframe platforms in this scenario, some of the usual low-level disk processing is offloaded because of the presence of the storage area network. Theoffloading factor is higher in the mainframe because of an additional level ofcoprocessing at the "channel" level, but both platforms receive a favorable adjustment

here, varying in degree only.

On the mainframe, built-in cryptographic coprocessors offload the vast majority of SSLencryption workload. The platform adjustments remove this from the overall CPU loadon the mainframe systems but not on the Intel systems, which lack dedicatedcryptographic coprocessors.


17/46



On a microcomputer, swapping to disk (due to overcommitted RAM resources) is a veryexpensive operation, but the mainframe's VDISK capability and high-level I/O channelprocessing make swapping much less expensive, proportionally. Thus, on a mainframe itis reasonable to overcommit memory at a much higher factor than on a microcomputer.

Mainframes use dedicated coprocessors for network and disk I/O (in fact, there is no such

thing as a "dumb" I/O adapter in a mainframe). The platform adjustments account for thisoffloading in the network and disk workload factors.

The microcomputers rely on frequent interrupts to the main processor to handle I/Odevices, whereas the mainframe uses a channel-oriented I/O interface that accepts high-level commands from the guest operating system. As a result, mainframes spend muchless of their time executing kernel-level code under Linux, and more of their time isavailable for application-level processing.

Finally, on both mainframe and microcomputer systems, virtualization improves thespeed of server-to-server network connections by allowing much of this traffic to berouted over virtual networks that operate (basically) at memory-bus speeds. The platform

adjustment examines the aggregate workload's ratio of server-to-server versus server-to-client traffic, and factors this against the number of logical servers per physical enclosurefor each platform, arriving at a figure that estimates the percentage of overall networktraffic that would be virtualized within the specific platforms.

2.3.4.4 Workload Calculations

The workload calculations are performed to determine, for the nominalcases, which of the load,power, and architectural constraints will limit each configuration's total capacity per unit. Theresult of this calculation is a limit of logical servers per chassis constrained by CPU, RAM, disk,cryptography, network, or architectural limits. The final result is the worst-case (minimum) ofthese values for each platform.

2.3.4.5 Adjusted Workload Calculations

The adjusted workload calculations work just like their unadjusted counterparts, except that theloading constraints take into account the platform adjustments explained previously. These arethe results, in the opinion of Sine Nomine Associates, which most accurately represent the real-world capacity of the data center, accounting for all the factors that are considered in this model.

It is the adjusted workload calculations which feed into the TCO calculations that follow.

2.3.4.6 Total Cost of Ownership (TCO) Calculations

The TCO calculations, in contrast to the complexity of the other parts of this model, are verystraightforward, tallying fixed costs of acquisition for the first year (only), and then adding costsof software support/maintenance and electricity for each year the systems remain in use. TCO iscomputed for one, three, and five year intervals. No cost-of-capital (net present value) orprojected growth rates are factored into these calculations, because these are numbers determinedlargely by corporate policy and accounting rather than by engineering factors.


18/46



3 Results in Detail

The calculations in the Sine Nomine Associates model yielded numbers which came as nosurprise to the SNA engineering team, who have worked extensively with both microcomputer

and mainframe platforms in customer data centers. As expected, processor resources constrain themainframe in all its configurations. This, combined with the notional limit of 8 mainframes perdata center, results in a total count of logical servers per data center that is much lower than ifmicrocomputers were used throughout. The lesson to be drawn from this is, quite simply, that onewould not want to replace the entire data center with mainframes for a typical aggregatedworkload. Rather, the mainframe would be applied selectively to replace microcomputer serversin those situations where its strengths are greatest. All of Sine Nomine Associates' customers whouse mainframe Linux are also using microcomputers where appropriate, deploying the proverbial"right tool for every job."

3.1 Workload Aggregation

The aggregated workload depends on the identified characteristics of each server category (webserver, database server, etc.) and on the frequency of occurrence of each type of server in atypical database. The following figure shows the frequency of occurrence used in this model.

Workload Frequency of Occurrence

25%

4%

7%

4%34%

4%

4%

18%

Static w eb server

Database server

DNS or LDAP server

e-Commerce w eb server

File server

Business logic server

Management and monitoring

server

E-mail server

Figure 1: Relative frequency of occurrence of server types in the model

The model is structured so that the notional "count" of servers in the aggregation process is notsignificant; only the ratio factors into the result. Note that this in itself is not a model output, butrather an input assumption presented here for reference only.

When the workload characteristics are aggregated, the result is as follows:


19/46



Aggregated WorkloadCharacteristics

Aggregated Workload in Notional Units

Weighted application

CPU load

5% 20.23 MIPS

Weighted workingstorage set

282.95 282.95 MB RAM

Weighted diskbandwidth

66% 258.21 unitless

Weighted crypto load 6% 23.71 unitless

Weighted networkbandwidth

86% 334.51 unitless

Weighted kernel CPUload

8% 29.64 unitless

Weighted server-to-

server net ratio

13%

Table 1: Aggregate workload per server used in model

The aggregated workload represents the amount of work assigned to each logical server in thedata center, for the model. The percentages represent a ratio to the nominal full-load capacity of atypical microcomputer with one Intel Xeon 5140 processor at 2.33 GHz and local disk storage.The memory is a working storage requirement in megabytes, and the processor utilization isconverted to equivalent mainframe MIPS.7 All of the workload characteristics are adjusted withinthe scenarios to account for multiple processors and architectural differences of the hardware.

The disk and network utilizations are considerably higher than processor utilization. Sine NomineAssociates has found this to be the case in almost every site where we have worked. There are, ofcourse, exceptions with compute-intensive applications such as finite element modeling, financialmarket models, and graphics rendering, but the quantity of such servers in a typical business issmall compared to the count of web, file, email, and database servers.

3.2 Constrained Server Density

The maximum server densities are calculated based on workload, physical space, infrastructurerequirements, and power and cooling load limits of the data center in which they reside. In thescenarios, the architecturally-adjusted workload factors are applied to these densities, resulting ina set of constrained server densities based on each load factor. The worst case for each platformthus establishes the final density limit for that platform in this data center, and those densities are

fed into the TCO models.

The table below shows the results of the constraint modeling, with the worst case for each platform scenario highlighted. Note that the discrete microcomputer servers tend to beconstrained by architectural limitations rather than other factors; this is reflective of their under-

7 See explanation in footnote 6 on page 9.


20/46



utilization in the typical environment. The virtualized microcomputers in this model wereconstrained by network bandwidth rather than by processor or memory limits.

DellPowerEdge

1950

DellPowerEdge

1950virtualized

DellPowerEdge

1955

DellPowerEdge

1955virtualized

IBM

System zBusinessClass S07

Small

IBM

System zBusinessClass S07Expanded

IBMSystem z

EnterpriseClass S18

Logical serversper chassisbased on CPUload

5.27 10.55 52.73 105.46 58.24 101.91 262.07

Logical serversper chassisbased on RAMcapacity

14.48 115.81 14.48 115.81 115.81 231.62 694.86

Logical serversper chassisbased on diskbandwidth

3.01 6.01 30.05 60.11 109.99 192.48 494.95

Logical serversper chassisbased on cryptoload

16.37 32.73 163.65 327.31 1197.87 2096.27 5390.41

Logical serversper chassisbased onnetworkbandwidth,excluding intra-chassis traffic

1.16 2.64 13.11 26.59 216.46 378.81 974.09

Logical serversper chassisbased onarchitecture

1 16 10 160 36000 36000 36000

Table 2: Constrained logical server density per enclosure

The above numbers are servers per chassis, not per rack. The table illustrates the constraintbehavior within each platform, and is not intended for direct comparison between platforms.

When examining server density per rack, the chart below shows again that the virtualized

environments have a substantial advantage over the discrete servers. The largest System z densitycalculated by the model was 262 logical servers per mainframe. A recent published conferencepresentation8 by a company using System z and Linux for server consolidation reports that the

8 The presentations were made at SHARE 107, Baltimore, Maryland, August 14-18, 2006, by a companylocated in the midwestern United States. The presentations are available from http://www.linuxvm.org/.


21/46



company is running over 300 servers on one mainframe, and Sine Nomine Associates hascustomers running systems of this scale and larger.

Number of Servers per Rack

28

73

40

106

58

101

262

0 50 100 150 200 250 300

Dell PowerEdge 1950

Dell PowerEdge 1950

virtualized

Dell PowerEdge 1955

Dell PowerEdge 1955

virtualized

IBM System z BC S07

Small

IBM System z BC S07

Expanded

IBM System z EC S18

Figure 2: Logical server density per rack location unit

The non-virtualized servers, of course, achieve a density that exactly equals the number ofphysical units that will fit into each rack, allowing for infrastructure components and independent

of the load profile. The density of the virtualized platforms depends on the aggregate load thatlimits how many logical servers will fit on each physical server.

3.3 Total Cost of Ownership

The initial cost of acquisition of the System z mainframe is somewhat higher per logical serverthan the cost of discrete or blade servers, providing a total cost of ownership (TCO) that is higherin the first year but which increases more slowly than that of the other alternatives over time. Foreach additional year, discrete and blade servers consume more power and cooling cost, whichdrives up their TCO until it eventually passes that of the mainframe. The following table showsthe TCO for each of the platform scenarios.

The figures cited here are for reference only and do not imply endorsement of this paper by SHARE or bythe company providing the presentations.


22/46



TCO per Logical Server

$0 $2 $4 $6 $8 $10 $12 $14

Dell PowerEdge 1950

Dell PowerEdge 1950

virtualized

Dell PowerEdge 1955

Dell PowerEdge 1955

virtualized

IBM System z BC S07

Small

IBM System z BC S07Expanded

IBM System z EC S18

Thousands

1 year TCO

3 year TCO

5 year TCO

Figure 3: TCO per logical server for one, three, and five years

Notice in Figure 3 that the System z TCO for one, three, and five years appears almost invariant,whereas there is a steady increase for all of the microcomputer systems. Only the fully virtualizedPowerEdge 1955 blade system approaches the low incremental TCO of the mainframe.

One interesting observation from the model is that the TCO per logical server on the System z isnot significantly affected by the size of the mainframe, provided the mainframe is appropriatelysized to the number of virtual servers being deployed. Customers planning System zconsolidation projects should, therefore, consider acquiring only the needed resources initially,and leveraging the System z's ability to be nondisruptively field-upgraded with additionalcapacity to meet growth requirements, ensuring that the available and needed resource levels arewell matched and minimizing long-term TCO.

The System z is highly optimized for virtualized operation, and therefore is not analyzed as adiscrete server. However, the analysis of virtualized versus non-virtualized operation on thediscrete and blade servers actually shows a slight increase in TCO for the virtualized

environments, due to hypervisor software costs that more than offset the power and coolingsavings. On the other hand, virtualization with appropriate management tools can significantlyreduce administrative costs, which is a factor not reflected in this model.

It is instructive to break out the fifth-year incremental TCO per logical server, as shown in thefollowing chart, to illustrate the annual cost savings of the System z once its acquisition cost hasbeen absorbed.


23/46



Fifth-Year Incremental Cost per Logical Server

$1,841

$713

$1,100

$419

$304

$175

$77

$0 $500 $1,000 $1,500 $2,000

Dell PowerEdge 1950

Dell PowerEdge 1950

virtualized

Dell PowerEdge 1955

Dell PowerEdge 1955

virtualized

IBM System z BC S07

Small

IBM System z BC S07

Expanded

IBM System z EC S18

Figure 4: Fifth-year incremental TCO for each scenario

Here the System z has a clear edge, because most of the annual TCO is power and coolingexpenses. The System z also benefits from the fact that z/VM support is a fixed annual priceregardless of the number of systems being supported, so its unit cost decreases as the number ofsupported servers increases.

Within the three System z scenarios, there is a clear trend showing the lowest incremental TCOfor the largest system with the most logical servers. The System z has a power consumption that

is not strongly dependent on the number of processor engines enabled, and a more powerfulconfiguration consumes only modestly more power than an entry-level configuration for the samesystem.

This situation explains, to a large extent, why the model imposes a "notional limit" of eightmainframes in the entire data center. In a practical situation, customers will deploy a few largermainframes rather than proliferating small- to mid-sized configurations in quantity, in order tomaximize the savings reflected in Figure 4 as well as saving labor costs and physical space.

3.3.1 Energy Cost

Power cost per logical server is another area of considerable difference among platforms, and the

difference in power consumption accounts for much of the difference in incremental TCO.

Recall that the energy utilization (and therefore cost) in this model includes not only the serveritself, but also any associated infrastructure components and the air conditioning equipment toremove all of the heat. Typically, every watt of power consumed by electronic equipment requiresan additional 0.7 watts of air conditioning power, according to Rob Snevely's book, EnterpriseData Center Design and Methodology.


24/46



Figure 5 compares the total power consumption per rack location unit, accounting for airconditioning energy as well as energy used by the electronics. With the data center's limitation of125 watts per square foot,9 the data center is nominally able to supply just under 4200 watts perrack location unit.10 Thus, the data center is not capable of being fully loaded with servers evenafter factoring in lost rack space due to IT yield. This requires a further derating of the datacenter, which is accounted for in the model as a limit on the data center's maximum totalserver

count.

Power per Rack Location by Platform

0 5 10 15 20 25 30 35 40 45

Dell PowerEdge 1950

Dell PowerEdge 1950

virtualized

Dell PowerEdge 1955

Dell PowerEdge 1955

virtualized

IBM System z BC S07

Small

IBM System z BC S07

Expanded

IBM System z EC S18

Platform

Power (Kilowatts)

Infrastructure power per rack

Server power per rack

Total power per rack

Figure 5: Power utilization per rack location unit

After that derating is applied, the model calculates the maximum total logical servers for theentire data center; the results are presented in Table 1. Note that the System z values are limitedby the "notional" constraint of eight mainframes per data center, as described in Section 3.3 andAppendix B. It would actually be possible to deploy several times this number of servers, butmost companies doing so would purchase a larger System z than the ones considered in thismodel, rather than replicating more than eight of these units.

9 Published sources often cite 100 W/sq-ft, but this is bumped up here to give the benefit of the doubt andassumes a high-quality physical facility.

10 Data center designers derate rack density from the theoretical concept of racks butted side-to-side, toaccount for air ducts, power distribution and UPS units, operator consoles, building support columns, etc.The resulting "IT yield" varies depending on data center design, but a recent article by the Uptime Institutecites a figure of 18% yield, which is used in this model.


25/46



DellPowerEdge

1950

DellPowerEdge

1950virtualized

DellPowerEdge

1955

DellPowerEdge

1955virtualized

IBM System zBusinessClass S07

Small

IBM System zBusinessClass S07Expanded

IBM Systemz EnterpriseClass S18

1372 3577 2320 6148 464 808 2096

Table 3: Total logical servers in entire data center

As shown in Figure 6, the annual power cost per logical server varies by orders of magnitude,with the traditional rack servers in non-virtualized mode faring the worst. For rack servers,virtualization reduces power per logical server by about 61%, and on blade servers virtualizationreduces power per logical server by about 62%. The blade servers benefit slightly from theimproved performance of virtualized networks within a physical server versus even high-speedphysical networks between servers.

Annual Power Cost per Logical Server

$920.70

$353.14

$550.06

$207.57

$127.23

$73.06

$32.80

$0 $100 $200 $300 $400 $500 $600 $700 $800 $900 $1,000

Dell Pow erEdge 1950


virtualized



virtualized

IBM System z BC S07

Small

IBM System z BC S07

Expanded

IBM System z EC S18

Figure 6: Annual power cost per logical server

Because the System z mainframe's power consumption is largely independent of its processorcount, the high-end mainframe's power consumption per logical server is almost 75% lower thanthat of the entry-level mainframe. Note, however, that all of the System z results are substantiallylower (at least 38%) than the best of the microcomputers.


26/46



4 Conclusions

4.1 Power and Cooling

Two conclusions leap from the analysis in this model, with regard to power and cooling:

Virtualization of underutilized servers, on any platform, will yield power and cooling costsavings that are substantial.

Among virtualization solutions, the System z offers greater potential power and coolingbenefits than any other available platform.

It is important to consider these conclusions together, rather than in isolation from one another.This model shows that mainframe consolidation tends to be constrained by available CPUresources, a conclusion that mirrors the practical experience of Sine Nomine Associates and its

customers. However, in applications that are well suited to its architectural features, the System zoffers spectacular power savings, reducing power and cooling costs to a negligible portion of totaland recurring costs.

4.2 Total Cost of Ownership

The model shows that the total cost of ownership (TCO) per logical server is very competitive between the mainframe and the microcomputer configurations. While in the model themainframes do not have a TCO advantage, it should be noted that this model intentionally omits(as out-of-scope) a number of "soft" factors that will affect the actual TCO at a customer site,specifically:

Managing virtual servers typically requires significantly less labor than managing anequivalent number of physical servers. This is true on both microcomputer andmainframe platforms, but the management tools on z/VM are more mature and plentifulthan the corresponding tools on the microcomputer platform. This is intended as nodisrespect to VMware, but simply reflects the fact that the mainframe environment hashad several decades to perfect virtualized management tools.

Power distribution units (PDUs), computer room air conditioners (CRACs), and thenetwork and SAN infrastructure components all require periodic maintenance andoccasional repairs, none of which are free. The mainframe requires significantly less ofeach of these resources, and thus one expects that the maintenance and repair costs woulddecrease proportionally. Even in an existing facility with pre-existing power and cooling

systems, one could argue that running these systems at a lower-than-maximum load levelwould prolong their useful life. It is certain that doing so provides greater capacityheadroom to handle temporary overloads from other sources.


27/46



4.3 Sine Nomine Associates' Recommendations

Based on the results of this model, and combined with practical experience at dozens of customerinstallations, Sine Nomine Associates offers the following recommendations for customersseeking to maximize the capability of existing data centers that are constrained by power, cooling,or space limitations:

1. Identify servers that are unitized for functional partitioning, security, or politicalreasons, rather than because of their resource requirements, and consolidate them

onto virtualized platforms. Virtualization is absolutely the best way to recaptureunderutilized hardware resources, and to reduce the number of physical servers that must be purchased, installed, powered, cooled, and maintained. In addition, a properlymanaged virtual data center reduces labor costs for deploying new servers and upgradingexisting servers, because of the ability to "clone" prebuilt virtual servers rapidly.

2. Use the System z mainframe as a part of the total virtualization environment. Manyapplications are constrained by I/O rather than CPU resources, and many commodityservices such as DNS, directory services, file and print sharing, and email can be

consolidated onto a single mainframe running virtualized Linux with little impact onclients, even if the clients are using Windows as their operating system. In addition, business applications written in Java exhibit excellent portability to both Linux andmainframe Linux platforms, and are excellent candidates for consolidation onto theSystem z.

3. On an attrition basis, begin a strategic migration to blades rather than discreteservers for applications where the mainframe is not the best alternative. The TCOcost differences between blades and discrete servers are not overwhelming, but neitherare they insignificant. When combining even a modest TCO advantage with the blades'improvements in manageability, the increased throughput of their internal SAN andEthernet switches, and the simplification of external cabling and infrastructure, it is

worthwhile to establish blades as the preferred choice for new hardware.

4. As a strategic direction, move toward storage area network (SAN) virtualizedstorage rather than local storage for applications and data. Virtualized storage is animportant foundation for gaining the most benefit from virtualized servers later, and itsavailability significantly improves the manageability of virtualized data centers. Note,however, that a virtualization initiative does not need to wait for full SAN penetration, asvirtualized servers can be migrated from local storage onto the SAN over time.

5. Leverage existing mainframe-native infrastructure such as databases rather thanmigrating these to microcomputer equivalents. The internal virtual networkingcapability of the System z is an enormous asset for making Oracle or DB2 databases,

CICS applications and application services, and Tivoli or equivalent backup/restoreservices available to consolidated applications running under mainframe Linux. If amainframe-native service is already available, the Linux guests can transparently utilizethat service in parallel with traditional mainframe applications.

6. Manage and monitor virtualized resources at the hypervisor level rather thanwithin the guest operating system. An operating system designed to run on discretesystems will not be aware of the capabilities of an underlying virtualization platform, andwill not allocate, schedule, and prioritize physical resources as well as the hypervisor can.


28/46



This argues for horizontal scaling, in which application functionality is divided amongvirtual servers of a smaller size with the hypervisor sharing resources between them.

4.4 Avoiding the $25 Million Server

The model demonstrates how power and cooling, even over physical space, limit the capacity ofdata centers with modern high-performance systems. Virtualization, regardless of the underlying physical platform, provides significant reduction in power per server, allowing many moreservers to be deployed.

Among virtualized environments, the IBM System z offers by far the best power performance.When deployed with workloads that are well suited to its architecture -- which includes many ofthe most common types of servers found in enterprise data centers -- the System z reduces theimpact of energy costs.

Ultimately, a data center's best configuration will involve selective deployment of multiple platforms, utilizing microcomputers for the most compute-intensive loads and mainframes forloads that are network and disk constrained. This synergy of technical strengths can extend thelife and capabilities of legacy data centers, overcoming power, space, and cooling constraints tosupport growth of the enterprise.


29/46



Appendix A: Power/Cooling Model

This Appendix contains the data from the model. Certain spreadsheet cells containing onlyintermediate calculations rather than input constants or output results are omitted here for the sake

of brevity, as they do not in themselves provide useful information.

Global Constants

Item Description Value Units Source Notes

Physical dimensions

Number of standard U perrack 42 rack units

Size of standard FibreChannel switch 1 rack units

Assumes QLogic SANBox 5602 orequivalent

Size of standard Ethernetswitch 1 rack units

Assumes Cisco Catalyst 4948 orequivalent

Size of KVM switch 2 rack unitsAssumes Raritan Dominion KX464or equivalent

White tile area of existing datacenter 8000 sq ft

Typical value for an enterprise data

center in an urban area is 7000-10000 sq ft based on SNAexperience

Infrastructure assumptions

Power consumption per FibreChannel switch 120 W


Power consumption perEthernet switch 300 W

Assumes Cisco Catalyst 4948 orequivalent

Power consumption per KVM

switch 180 W

Assumes Raritan Dominion KX464

or equivalent

Ports per Fibre Channelswitch 16 count


Ports per Ethernet switch 47 count

Assumes Cisco Catalyst 4948 orequivalent, allows one port foruplink


30/46



Servers per KVM switch 64 countAssumes Raritan Dominion KX464or equivalent

Performance comparisonassumptions

Maximum virtual servers perphysical server on x86platforms 16 count

Published limit for VMware ESX is128, but real-world installations inSNA's experience limit to 16 orless on a single physical server.Some of SNA's customers limit to 4per physical server as anadministrative limit.

Maximum virtual servers perphysical server on System zplatforms 36000 count

Theoretical limit in z/VM is 32,000in each of 3 scheduling queues,but a practical limit is 12,000 perqueue

Estimated x86 to System zperformance conversionfactor 6.0 MHz per MIPS

An estimated number based onpublished benchmarks for the z9systems by Linux users (notendorsed officially by IBM)

Minimum RAM per Linuxinstance 64.0 megabytes

Minimum memory for a reasonableboot environment, suitable forservers like DNS that use very fewmemory resources. Thisestablishes a limit on virtualservers on a physical platform.

Mainframe MIPS per IFLengine 355.0 MIPS

MIPS in this context are used inthe traditional S/390 sense, not theliteral million-instructions-per-second sense

Occupancy assumptions

Fibre Channel redundancyfactor 2 multiplier

Ethernet redundancy factor 2 multiplier

KVM redundancy factor 1 multiplier

Empty units per rack forgrowth allowance 4 rack units

Arbitrary assumption based onexperience in enterprise datacenters of customers

Ratio of infrastructure towhite-tile area 0.3 multiplier

Turner and Seader, "Dollars perkW plus Dollars per Square FootAre a Better Data Center CostModel than Dollars per SquareFoot Alone"


31/46



Rack area as fraction of whitetile area ("IT yield") 0.18 multiplier

Kenneth Brill, "2005-2010 HeatDensity Trends in Data Processing,Computer Systems, andTelecommunications Equipment"

Square feet per rack (white-tile area only), rounded up to0.5 sq ft multiple 33.5 sq ft/rack

Based on a rack footprint size of 24by 36 inches and the areapercentage ("IT yield") from the cellabove

Data center power/coolingdesign limit 125 W/sq ft

Published sources often cite 100W/sq-ft, but this is bumped up hereto give the benefit of the doubtand assuming a high-quality site

Nominal available power perrack limit 4187.5 W/rack

Costing assumptions

Electricity cost $0.0936 US$/kWH

U.S. Dept. of Energy, nationalaverage for commercial electricityfor 12 months ending December2006

Building cost per sq ft (spaceonly) $220

Turner and Seader, "Dollars perkW plus Dollars per Square FootAre a Better Data Center CostModel than Dollars per SquareFoot Alone"

Building cost per kVA ofusable UPS output, includingswitchgear, batteries, dieselgen., HVAC $22,000 US$/kW capacity

Turner, Seader, and Brill, "TierClassifications Define SiteInfrastructure Performance"

Base cost of System z BCwith 1 IFL and 64GB RAM $585,000 US$

SNA case study, not an officialprice quote from IBM

Cost of System z additionalIFL $95,000 US$ Published price

Base cost of z/VM per IFL $22,500 US$ Published price

Annual support cost of z/VM

(flat fee) $11,500 US$ Published price

Base cost of VMware ESX perprocessor $1,875

Pricing based on VM wareInfrastructure Standard Edition,published at $3750 per 2processors (normalized to per-processor here for calculationsimplicity)


32/46



Support cost for VMware peryear (flat fee) $11,500 US$

Pricing not published by VMware.Estimated at same as z/VMsupport cost

Cost of System z RAM $8,000 US$/GB

SNA case study, not an official

price quote from IBM

Ratio of HVAC to computerkW 0.7 multiplier

Rob Snevely, "Enterprise DataCenter Design and Methodology"(Sun Microsystems)

Cost of Fibre Channel switch $7,875Online price from Provantage.comas of 2007-04-04

Cost of Ethernet switch $7,652Online price from Buy.com as of2007-04-04

Cost of KVM switch $5,702

Online price from

SuperWarehouse.com as of 2007-04-04

Computed values fromconstants

Electricity per kWH per month(30 days) $67.39 US$/kW/month

Computed from the US DoE dataabove

Maximum racks in existingdata center 238


33/46

2007 Sine Nomine Associates

Platform Parameters

ParametersDell PowerEdge

1950Dell PowerEdge1950 virtualized

Dell PowerEdge1955

Dell PowerEdge1955 virtualized

IBM System zBC S07 Small

IB

Category 1U rack server

1U rack serveroptimized forvirtual servers Blade server

Blade serveroptimized forvirtual servers

Mainframeoptimized forvirtual servers

Maopvirt

Processor

Intel Xeon 5140dual-core 2.33GHz




IBM Multi-ChipModule (MCM)IFL engine

IBMMoIF

Configuredprocessors per

server 1 2 1 2 4Configured memoryper server 4 32 4 32 16

Size of chassis (rackunits) 1 1 7 7 42

Maximum serversper chassis 1 16 10 160 36000

Network ports perchassis (not allowingfor redundancy) 1 1 2 2 1

Fibre Channel ports

per chassis (notallowing forredundancy) 1 1 2 2 1

Power consumptionat specified config 670 670 4000 4000 5400


34/46




Dell PowerEdge1955



IB

Est. hardware cost

per chassis atspecified config $6,082 $20,421 $53,150 $196,340 $486,000

Est. hypervisor costper chassis atspecified config $0 $3,750 $0 $37,500 $90,000

ComputedParameters

Maximum chassisper rack (theoretical,allowing only

minimum possibleoverhead) 34 34 4 4 1

Theoretical serversper rack (notallowing foroverhead) 34 544 40 640 36000

Fibre Channelswitches per rack(redundancyincluded) 6 6 2 2 0

Ethernet switchesper rack (redundancyincluded) 2 2 2 2 0

KVM switches perrack 1 1 1 1 0


35/46




Dell PowerEdge1955



IB

Total overhead units

per rack 14 14 10 10 0

Actual net chassisper rack 28 28 4 4 1

Max servers per rackat best-case virtualdensity (whereapplicable) 28 448 40 640 36000

Est. infrastructurecost per rack forSAN, network, and

KVM $68,256 $68,256 $36,756 $36,756 $0

Estimated equivalentz9 MIPS per physicalprocessor 388 388 388 388 355

Estimated equivalentz9 MIPS per physicalserver 388 776 388 776 1420

Estimated equivalentz9 MIPS per chassis 388 776 3880 7760 1420

Estimated equivalentz9 MIPS per rack 10864 21728 15520 31040 1420


36/46




Dell PowerEdge1955



IB

Infrastructure power

per rack 1500 1500 1020 1020 0

Server power perrack 18760 18760 16000 16000 5400

Total power per rack 20260 20260 17020 17020 5400

Demanded power vs.available limit 484% 484% 406% 406% 129%

Configuration Notes

Max number ofserversrepresents a

reasonabletheoretical value,which will befurther limited byother calculationsin the model





Maserep

reathewhfurothin t

Processorselected foracross-the-boardavailability, foreasiercomparison





37/46




Dell PowerEdge1955



IB

Pricing NotesPriced online atdell.com

Priced online atdell.com

Priced atdell.com and byphone to Dellpre-sales

Priced atdell.com and byphone to Dellpre-sales

IBM does not

publish prices onmainframes; thepricing is basedon an estimate-driven model(see Globals forconstants)

IBM

pumapriondri(seco

QLogic 2462dual-channel FCHBA for SAN

access


access


access


access

Some valuesobtained fromSNA casestudies; detailsnot publishable

due to NDA.

SoobSNstuno

duDual-channelEthernet includedin base config

Dual-channelEthernet includedin base config



Added redundantP/S

Added redundantP/S

Doubly-redundant (2+2)power suppliesstandard

Doubly-redundant (2+2)power suppliesstandard

Minimumonboard DASD(36 GB, no RAID)selected




Removed Dellremotemanagementinterface

Removed Dellremotemanagementinterface

Dell remotemanagementinterface notused

Dell remotemanagementinterface notused


38/46




Dell PowerEdge1955



IB

Enclosure price

estimated at$5000, would besomewhat lowerif purchased inquantity

Enclosure price

estimated at$5000, would besomewhat lowerif purchased inquantity

Price abovereflects 10identical blades,but no quantitydiscount isreflected here

Price abovereflects 10identical blades,but no quantitydiscount isreflected here

Workload Characteristics

Workload scenarioassumptions

Static webserver

Databaseserver

DNS or LDAPserver

e-Commerceweb server File server

Businesslogic serve

CharacteristicsNet andDASD I/O

DASD I/O andmoderate CPU Low utilization

Net and DASDI/O, crypto

Net andDASD I/O

CPUintensive,moderate I/O

Frequency ofoccurrence (unitless) 30 4 8 4 40

Application CPU loadfactor (unitless) 0.05 0.25 0.03 0.1 0.02 0

Working storage set(megabytes) 256 1536 128 512 128 10


39/46


Workload scenarioassumptions

Static webserver

Databaseserver

DNS or LDAPserver

e-Commerceweb server File server

Businesslogic serve

Relative DASD loadfactor (unitless) 0.5 0.9 0.05 0.2 0.9 0

Relative crypto loadfactor (unitless) 0.02 0.02 0 0.2 0.1 0.

Relative bandwidthfactor (unitless) 1 0.75 0.1 0.5 1 0

Kernel CPU load factorfor I/O (unitless) 0.075 0.0825 0.0075 0.035 0.095 0.

Ratio of net I/O that isserver-to-server 0 0.9 0.1 0.5 0 0.

The aggregated workload values are reported in Table 1on page 14.

Scenario Results

DellPowerEdge

1950

DellPowerEdge

1950virtualized

DellPowerEdge

1955

DellPowerEdge

1955virtualized

IBM SystemBC S07 Sm

Platform Summary

Actual net chassis per rack 28 28 4 4

Max servers per rack at best-casevirtual density (where applicable) 28 448 40 640 360


40/46


DellPowerEdge

1950

DellPowerEdge

1950virtualized

DellPowerEdge

1955

DellPowerEdge

1955virtualized

IBM SystemBC S07 Sm

Estimated equivalent z9 MIPS perchassis 388 776 3880 7760 14

Total power per rack 20260 20260 17020 17020 54

Demanded power vs. availablelimit 484% 484% 406% 406% 12

Allowable racks in data centerbased on design limit 49 49 58 58 1

Allowable racks in data center,limited by notional limit 49 49 58 58

Platform Adjustments

RAM overcommit factor 1 1 1 1

Offloaded DASD support factor 0.5 0.5 0.5 0.5 0

Offloaded crypto support factor 0 0 0 0 0

Offloaded net support factor 0 0 0 0 0

CPU I/O handling offload factor 0 0 0 0

% of net I/O that is inside chassis 0% 12% 12% 13% 1

Workload Calculations

Logical servers per chassis basedon CPU load 5.273247496 10.54649499 52.73247496 105.4649499 19.298998

Logical servers per chassis basedon RAM capacity 14.47619048 115.8095238 14.47619048 115.8095238 57.90476


41/46


DellPowerEdge

1950

DellPowerEdge

1950virtualized

DellPowerEdge

1955

DellPowerEdge

1955virtualized

IBM SystemBC S07 Sm

Logical servers per chassis basedon DASD bandwidth 1.502666833 3.005333666 15.02666833 30.05333666 5.499450

Logical servers per chassis basedon crypto load 16.36525085 32.7305017 163.6525085 327.305017 59.893443

Logical servers per chassis basedon network bandwidth 1.159899653 2.319799306 11.59899653 23.19799306 28.299957

Adjusted Workload Calculations

Logical servers per chassis basedon CPU load 5.27 10.55 52.73 105.46 58

Logical servers per chassis basedon RAM capacity 14.48 115.81 14.48 115.81 115

Logical servers per chassis basedon DASD bandwidth 3.01 6.01 30.05 60.11 109

Logical servers per chassis basedon crypto load 16.37 32.73 163.65 327.31 1197

Logical servers per chassis basedon network bandwidth, excludingintra-chassis traffic 1.16 2.64 13.11 26.59 216

Logical servers per chassis basedon architecture 1 16 10 160 360

Logical servers per chassis basedon worst-case for platform 1.00 2.64 10.00 26.59 58


42/46


DellPowerEdge

1950

DellPowerEdge

1950virtualized

DellPowerEdge

1955

DellPowerEdge

1955virtualized

IBM SystemBC S07 Sm

Logical servers per rack,accounting for all limitations 28 73 40 106

Logical servers in data center,accounting for all limitations 1372 3577 2320 6148 4

Cost Calculations

Est. hardware cost per chassis atspecified config $6,082 $20,421 $53,150 $196,340 $486,0

Est. hypervisor cost per chassis atspecified config $0 $3,750 $0 $37,500 $90,0

Est. infrastructure cost per rack $68,256 $68,256 $36,756 $36,756

Acquisition cost per rack $238,552 $745,044 $249,356 $972,116 $576,0

Total acquisition cost $11,689,048 $36,507,156 $14,462,648 $56,382,728 $4,608,0

Power utilization for servers (kW) 919 919 928 928

Power utilization for HVAC (kW) 643 643 650 650

Total power utilization (kW) 1562 1562 1578 1578

Power cost per month (US$) $105,266 $105,266 $106,345 $106,345 $4,9

Power cost per year (US$) $1,263,196 $1,263,196 $1,276,135 $1,276,135 $59,0

Power cost per year per logicalserver $920.70 $353.14 $550.06 $207.57 $127

Hypervisor support cost per year(US$) $0 $11,500 $0 $11,500 $11,5

8/14/2019 Avoiding the $25 Million Server Data Center Power

Documents

Avoiding the $25 Million Server Data Center Power, Cooling, and Growth