PowerandCooling Phase1 Overview

7/30/2019 PowerandCooling Phase1 Overview

1/8

Special Study:

Power & Cooling Practices

and Planning at HPC

Data Centers

real tesng real data real results


2/8

About DICE

Avetecs HPC Research Division - DICE is

a geographically distributed test

environment that provides independent

product and technology testing, validation

and consulting services for new and

emerging HPC data management

solutions. The DICE team works with the

HPC community, including vendors, data

centers (government and industry) and the

research community to evaluate new and

emerging products and technology. DICE

is the foundation to enhance research

computing data and move technology

from discovery to deployment.

Seeking greater insight

This report reflects the research

and feedback from the HPC

community based on a

comprehensive survey

conducted in 2009. The final

conclusions and

recommendations contained

herein represent the views ofthe authors and should not be

interpreted as facts,

endorsements or official policy,

either expressed or implied, by

Avetec or IDC.

AUTHORS:

Roger Panton

Avetec, Inc.

Al Stutz

Avetec, Inc.

Michelle Parker

Marketing Impressions

Consulting & Research, LLC

COPYRIGHT Avetec 2009

through innovative leadership


3/8

HPC Data Center Power and Cooling Study

Copyright 2009 Advanced Virtual Engine Test Cell

All trademarks, service marks and copyrights on this page are owned by their respective owners. | 1

As the performance levels of HPC systems (as measured by the LINPACK Benchmark)have experienced exponential growth, a similar growth in power and cooling has occurred.With rising energy costs and aging infrastructure, power and cooling has become a majorconcern to HPC data centers around the world. A comprehensive survey study lead by

Avetec confirmed this growing issue and explored how data centers are coping. This studylead to a central deduction:As computer systems require more and more cooling and

power, the balance between computing performance and the ability for facilities andinfrastructure to accommodate new systems does not exist.

The Avetec HPC Research Division, the Data Intensive Computing Environment (DICEProgram) was contracted to evaluate cooling requirements using conventional air andemerging cooling technologies. As part of this study, Avetec subcontracted the IDC HPCGroup to conduct a worldwide survey of HPC data centers and vendors of HPC productsand services. Avetec and the IDC HPC Group developed a comprehensive survey toassess the current power and cooling situation, planning in place to address power andcooling requirements, and the forecasted solutions to be used by HPC data centers in thenext three to five years. This collaborative study evaluated the cooling requirementreduction in conventional air cooling which could be achieved using advanced andemerging cooling technologies such as liquid technology, increased use of outside cool airand cooled water. This groundbreaking study also addresses the most critical infrastructureissue facing HPC data centers today: The rapid increase in costly upgrades required tocool the new HPC systems.

KKEEYY NNEEEEDDSS VVAALLIIDDAATTEEDD BBYY TTHHIISS RREEPPOORRTT

1. High Performance Computing (HPC) is critical to the future of the United

States to maintain American competitiveness in science and technology.

2. Energy efficiency is critical to the future growth of HPC data centers.

3. The United States needs a capability where vendors and data centers can

openly test and evaluate emerging power and cooling technologies while

protecting intellectual property.

4. A strong HPC-community wide (government, industry and academia)

partnership needs to be established to research, develop and deploy

more energy efficient power and cooling solutions.

5. Since federal and state governments are the largest buyers of HPC

systems, they must take a leadership role in establishing policies andresearch directions that will allow for the growth and expanded

application of HPC solutions while encouraging innovation in power and

cooling solutions.

Recommendations addressing these needs are listed on pages 4 and 5.

Powering and Cooling Study:How HPC Data Centers Are Coping


4/8




THE GROWING ISSUE

Over the last ten years, high performance in supercomputing typically meant more power and speed per

second. What drove this quest for more performance? In March 2002, Japan took a major leap andbecame number 1 on the TOP500 list with the new Earth Simulator which had a benchmark of 35.86Teraflops (TFLOPS) (one trillion floating point operations per second). Before the Earth Simulator, thelargest and most powerful computer was a U.S. Department of Defense computer system thatdemonstrated 7.0 TFLOPS. The reaction in Congress was that the U.S. had fallen behind Japan and wouldfall behind the rest of the world in supercomputing. As a result of this concern, Congress set a policy toachieve a 1,000 TFLOPS, or one Petascale, computing system in the U.S. within the next 10 years.

June 2008 marked a new milestone in the HPC community when the first one-petaflop/s (PFlop) (onequadrillion floating point operations per second) HPC system made the TOP500 list. Known as theRoadrunner system (built by IBM for the U.S. Department of Energys Los Alamos National Laboratory), itachieved a performance of 1.026 PFlops and became the first supercomputer ever to reach this milestone.In November 2008, a second PFlop system became operational at Oak Ridge National Laboratory.

The significant accomplishments in compute power, however, have created other issues for the HPCcommunity one of which is that data centers were not designed to support the extreme density of thesemulti-core parallel systems. As systems have grown in floating point operations per second, so too havethe infrastructure investment costs required to accommodate these larger systems. And no data center isimmune to this growing challenge all federal agencies, universities and industry HPC data centers areexperiencing costly infrastructure upgrades. In addition, the steady increase in computing capability hasresulted in a substantial increase in electrical power consumption and cooling to operate these systems.Several studies show that heat dissipation requirements may double in the near future.

Growth Projections

The TOP500 list is useful in identifying the upper bound of computing capabilities of supercomputersystems. Looking at the performance development of computer systems in 1993 from the TOP500 list, asmeasured by the LINPACK Benchmark, the sum of the TOP500 equaled 1 TFLOPS. By 2004, the sumhad reached 1 PFlop, or a thousand fold increase in computing capability.

Figure 1

i


5/8




If the growth line in compute capability continues, the sum of the TOP500 computer systems is projected toreach 100 PFlops by 2012. See Figure 2 which projects performance development though 2019.

Figure 2ii

CONCLUSIONS & RECOMMENDATIONS

Avetec believes the HPC community is at a crossroad and that power and cooling is becoming a limiting

factor in HPC data centers ability to provide their user base the latest in computer system performance.Setting aside other issues that are impacting data center director decisions, like scalability of software,management of data that is being generated at a prolific rate, response to green initiatives, and softwarecost per node, the focus of this report is the impact of power and cooling on data centers.

Avetec believes some of the greatest assets the U.S. has are its ability to lead the world in thedevelopment of intellectual property and its leadership in science and technology. Future U.S.competitiveness will depend on utilizing the best toolset available and one essential tool is supercomputersystems that provide the computational power required to solve complex interdisciplinary problems inscience and engineering.

Avetec believes the explosion in data generation will also add to the power and cooling issue as theprimary amount of storage required will increase in order to meet the increased compute capability. Thereare promising developments in the storage area that could provide better performance to power usage. Aswith all new developments, the total cost of ownership will need to be competitive with current tape and

disk drives installation.

Finally, Avetec believes the HPC community cannot sustain the growth in power and cooling requirementsthat is quickly approaching the power consumption of a small to midsized city. Avetec believes thatdesigning and implementing solutions that address the growing power and cooling issue will requirecollaboration among the HPC community including data centers, vendors and the federal government.

The following recommendations are provided for consideration by HPC data centers, vendors andgovernment policy makers.


6/8




HPC Data Centers:

As a result of the trends in power and cooling requirements, HPC center directors are strongly consideringadding power and cooling requirements in their next procurement. Other initiatives to consider are listedbelow.

x Proactively work with vendors, researchers and the HPC community to identify and pursuebreakthrough power and cooling technologies.

x If not currently monitoring power and cooling consumption and costs, implement an approach andmethodology to measure and track consumption.

x Identify and implement methods to reduce consumption. Consider using new schedulingtechnologies that dynamically power up processors on demand.

x Install new air conditioning and power systems that reduce power consumption, e.g., direct current(DC) power, sealed cooling rows, exhaust heat to the atmosphere, etc.

x Establish comprehensive performance metrics to track and report results of new initiatives.

x Increase multi-core CPU utilization to achieve more results per kilowatt hour (KWh) consumed.

x Investigate enhancing cooling methodologies through the use of liquid technology.

Vendors:

x Develop technologies that will drive down power and cooling demands by using technologies thatdynamically power up processors on demand.

x Technology vendors need to design their system requirements to address end-to-end data centerinfrastructure in order to develop the most efficient center possible.

Government Policy Makers:

Create a commission to develop a complete list of policy and technology directions that the HPCcommunity can implement in the short term. The commission should also develop a list oflegislative level policies.

This increase in power and cooling leads to a policy question: What should the supercomputer communitysresponse be to restore the balance between compute system performance and the power and coolingrequired for these newer systems? The balance between increased power and cooling for increasedperformance on the compute side needs to be restored. The issue is how should the community respond?Should the community be proactive and try to influence the policy or reactive to what the policy settersdecide.

Taking a more proactive position, the following initiatives are suggested for consideration and discussion:

x The federal government should establish a timeframe and fund several research and technologyprograms with the overall goal of maintaining the current growth in supercomputer processing while

reducing power and cooling requirements. The suggested timeframe is within five years.

x The federal government should invest in new cooling technology research that improves coolingefficiencies at a lower cost.

x The federal government should invest in lower power and higher performance processors with alower cost.

x The federal government should invest in new material research for chip manufacturers with a goalof making significant improvement in the environmental range that chips can successfully operate(e.g., wider temperature and humidity range, more air contamination).


7/8




x The federal government could allow non-government HPC centers to depreciate new HPC systemsover a three to four year depreciation cycle, as this represents the normal life of an HPC system.

x The federal government should invest in research to enhance single processor performance.

x The federal government should invest in research to increase memory and storage performance.

x The federal government should invest in research to develop new programming models andlanguages that drastically enhance parallelism.

iTop500.14Nov.2008.PerformanceDevelopment.11Jun.2009iiTop500.14Nov.2008.ProjectedPerformanceDevelopment.11Jun.2009


8/8

Nationwide Test Environment for Data Management Innovations

DICE Program

Avetec HPC Research Division

Corporate Headquarters

4170 Allium Court

Springfield, OH 45505

T: 937-322-5000

www.diceprogram.org

Data Intensive Computing Environment

Documents

PowerandCooling Phase1 Overview