Reliability and Energy Efficiency – Not Mutually Exclusive William TschudiPrincipal InvestigatorLawrence Berkeley National Laboratory

AcknowledgementsCalifornia Energy Commission

Pacific Gas and Electric Company

Uptime Institute

Critical Facilities Roundtable

Rumsey Engineers

RMI

E Source

Power Supply Manufacturers Association

Industry Partners (Too many to name all)

Subcontractors:

EYP Mission Critical Facilities, Ecos Consulting, EPRI-PEAC

Energy Intensive High-tech Buildings

Cleanrooms

Healthcare

Data Centers

Laboratories

We Also Operate Data Centers

Overview

Why Research Data Centers?• Utilities were receiving requests for

unrealistic power densities

• A lot of misinformation was (is?) circulating

• Large continually operating base loads

• We saw large energy efficiency opportunities in other High-Tech buildings

• Technology improvements are transferable to other building types

Prior Data Center Research• Research “roadmap” for the CA Energy

Commission

• Participation in design charrette organized by Rocky Mountain Institute

• Energy benchmarking and case studies for 14 data centers

• Market assessment in CA

Overview

Current Data Center Research Activities• Benchmarking and Best

Practices

• Investigate UPS Efficiency Improvement

• Investigate Power SupplyImprovement

• Develop performance metrics – computing horsepower and energy use

• Demonstration Projects

• Technology Transfer

Benchmarking

Energy Benchmarking• Benchmark (measure) energy use in 6-10

additional data centers

– Energy intensity and end-use– System and component efficiency– Efficiency improvement suggestions

• Solicit additional benchmarks from other sources

• Identify and analyze better performing systems and document in “Best Practices” summary

• Develop self-benchmarking protocol

local distribution lines

to the building, 480 V

HVAC system

lights, office space, etc.

UPS

PDU computer racks

backup generators

computerequipment

uninterruptible

load

Electricity Flows In Data Centers

Computer Load Dens ity

010203040506070

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Fac ility

W/s

q.f

t.

16

Distribution of Computer Room Power Reported to Uptime Institute

0.00

0.20

0.40

0.60

0.80

1.00

0 20 40 60 80 100

Computer room UPS power (Watts/square foot)

Fra

ctio

n of

tot

al f

loor

are

a in

sam

ple

1999

2000

2001

Number of facilities Total floor area

Computer room power density

Million square feet W/square foot1999 35 1.55 22.92000 38 1.72 22.42001 48 1.86 25.3

Source: Uptime Institute, 2002.

Benchmarking

Both LBNL and Uptime Institute found average IT equipment loading at

~25 W/ft2

Benchmarking

IT Equipment Loading

(W/Sq.Ft. of electrically active floor space)

Current and Projected Load Intensity

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Facility

W/s

q.

ft.

Current Computer Load Projected Computer Load

BenchmarkingProjecting IT Load When Fully Loaded

H V A C (as a % o f to ta l load)

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5 6 7 8 9 10 11 12

D ata C en ter Id en tifier

% o

f to

tal

loa

d

Benchmarking

Data Center Benchmarking

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Facility

kW

/ t

on

Average 0.75

Benchmarking

Chiller Comparison

Total Average Efficiency kW/ton

0

0.5

1

1.5

2

2.5

3

Facility

kW

/to

n

Average 1.69

BenchmarkingTotal Chilled Water System Efficiency

0

20

40

60

80

100

0 20 40 60 80 100

Load Factor (%)

Eff

icie

nc

y (

%)

Benchmarking

UPS System Benchmarking

Standby Generation Loss• Several load sources

– Heaters– Battery chargers– Transfer switches– Fuel management systems

• Heaters alone (many operating hours) use more electricity than ever produced by the generator (few operating hours)

• Opportunity may be to reduce or eliminate heating, batteries, and chargers

Benchmarking

Standby Generator Heater

Benchmarking

No Impact on Reliability• Chiller and pumping efficiency

• Use of lighting controls

• Variable speed drives – pumps, chillers, fans

• Free cooling

• Improved UPS efficiency

BenchmarkingImproved Reliability and Energy Performance• Better match systems to their loads

• Improve humidity control and eliminate CRAC unit fighting

• Water side economizers

• Better control strategies – setpoints, cooling tower staging

• Better thermal stratification – high ceilings and properly sized underfloor

• Air management – delivering air where it’s needed

Benchmarking

General Recommendations• Benchmark to know where you stand

• Life-cycle cost analysis

• Facilities partnership with IT professionals

• Evaluate load spreading vs. compaction

Benchmarking

Available Benchmark Data?• Sources of other benchmark data?

• Please contact LBNL:

– Steve Greenberg: segreenberg@lbl.gov

– Bill Tschudi: wftschudi@lbl.gov

Benchmarking

Future Direction• Develop consensus on performance

benchmarks, collect data, quantify energy savings potential

• Incorporate other industry benchmark data

Power Dist98 - 99%

UPS88 - 92%

Power Supply68 - 72%

dc/dc78 - 85%

HVAC1,200W / 1 Ton (76%)

US Annual Energy Consumption of 30TW-h US Annual Energy Consumption of 30TW-h flows through this inefficient delivery pathflows through this inefficient delivery path

Data Center Power Delivery

Cost of Power DeliveryPower Path Efficiency

Power (kW)

MW-h

Cost

Load Cooling 3.1 113 $8,500

Loads Loads Systems (x100)

(not including dc/dc & ac/dc)

9.8

x 85% Dc/dc 2.1 109 $8,200$8,200

x 70% Power supply 5.1

x 90% UPS 1.9

x 98% Distribution 0.4

x 76% Delivery Cooling 3.0

Total = 40%Total = 40% Total 222 $16,700

Total efficiency Total efficiency ≈ 40%≈ 40%Cost of power delivery = $8,200 / 100Cost of power delivery = $8,200 / 100

Source: EPRI PEAC

Improvements in Cost of Power DeliveryPower Path Efficiency

Power (kW)

MW-h Cost ($k)

Load cooling 3.1 113 8.5

Loads Loads Systems (x100)

(not including dc/dc & ac/dc)

9.8

x 85% Dc/dc 2.1 109 109 6262

8.2 8.2 4.74.7

x 70% x 70% 85% 85% Power supply 5.1 5.1 2.1 2.1

x 90% x 90% 94% 94% UPS 1.9 1.9 0.9 0.9

x 98% Distribution 0.4 0.4 0.3 0.3

x 76% Delivery Cooling 3.0 3.0 1.7 1.7

Total = 40% 51% Total 222 222 175175

16.7 16.7 13.213.2

Annual cost reduced by $3,500 / 100Annual cost reduced by $3,500 / 100Source: EPRI PEAC

UPS Systems

UPS Activities• Determine the range of current UPS

efficiencies, highlight more efficient designs, and provide a means for comparing their total cost of ownership (TCO).

• Propose a new UPS labeling scheme that could be considered by Energy Star and other third -party efficiency labeling organizations.

• Conduct a scoping study to analyze the energy efficiency savings potential and performance of a complete DC power architecture for data centers.

Manufacturers data for efficiency versus load for current generation static and inertial UPS.

93.50%

92.80%

95.10%

93.50%

91.50%

92.00%

92.50%

93.00%

93.50%

94.00%

94.50%

95.00%

95.50%

Median 90th Percentile

100% Loading (Spec) 50% Loading (Spec)

Based on review of more than 100 static UPS models

UPS Systems

UPS Efficiency and Loading

32%

89%

64%

91%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Median 90thPercentile

Field UPS Loading (%)

Field UPS Efficiency (%)

Sample of 12 field measurements.

UPS Systems

UPS Measured Performance

Efficiency vs Load with High Efficiency Mode(HE) on and off

60646872768084889296

100

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Percent of Full Load

Eff

icie

ncy

(%)

Non Linear Load High Efficiency Mode Non Linear Load Double Conversion Mode

Measured Result

Manufacturer Spec

On average, existing high efficiency modes can make a 4 to 5 % difference in UPS efficiency.

Source: EPRI PEAC

UPS SystemsMeasuring UPS Efficiency - “High Efficiency” Option

UPS SystemsAnalyzing UPS Performance in “High Efficiency” Option

High Efficiency Mode 30% sag 10 cycle

-400-300-200-100

0100200300400

0 50 100 150 200

Time (ms)

Vo

lts

Input Voltage Output Voltage

Double Conversion Mode 30% sag 30 cycle

-400-300-200-100

0100200300400

0 50 100 150 200

Time (ms)

Vo

lts

Input Voltage Output Voltage

In “high efficiency” mode, there can be one cycle

(16.6 msec for 60 Hz) of voltage deviation on UPS output.

Power supplies downstream of UPS can ride through this.

Source: EPRI PEAC

Quoted efficiencies are in the 96% range and higher; we will be measuring actual performance in field/lab

tests

Caterpillar PentadynePiller

UPS Systems

Research of Inertial Units

Efficiency and Reliability

Coordinating with International labeling effort addressing quality & efficiency.

Possible UPS efficiency labeling criteria

UPS Systems

Labeling

UPS SystemsLabeling will consider non-energy,reliability issues

Power Supply Activities • Measure efficiencies of current server

power supplies

• Field testing to document achievable energy savings

• Recommend new efficiency levels to Server System Infrastructure Initiative (SSI) for consideration

• Assess other PS saving opportunities in DC applications

Power Supplies Can Be Much More Efficient – Without Affecting Reliability Performance

• The nation’s 3.1 billion power supplies waste about 3 to 4% of the entire U.S. electricity bill

• Worst designs are 30 to 50% efficient

• Designs already exist in the market that can bring power supply efficiencies to the range of 75 to 93% with big non-energy benefits

• More efficient designs will be developed if there is incentive.

A consistent test protocol using a standard loading guideline & test

report format will allow more

visibility on power supply efficiency

Server Power Supply Efficiency Test Report

77%75%

Value Units100~127/200~240

Volts

3.2/1.6 Amps50-60 Hz

152.9 Watts

202.9 WattsNote: All measurements were taken with input voltage at 115 V nominal and 60 Hz.

IRMS PFA 12.0/6.0 5.0/22.0 3.3/13.0 -12.0/0.25 5.0 SB/1.00.56 0.92 7.9% 20% Light 59 12.0/0.95 4.98/3.47 3.30/2.06 - 4.99/1.00 41 68%

0.66 0.94 7.5% 25% 71 12.0/1.22 4.98/4.46 3.30/2.64 - 4.99/1.00 51 72%

1.15 0.98 5.7% 50% Typical 130 11.9/2.57 4.98/9.41 3.24/5.58 - 4.97/1.00 100 77%

1.69 0.99 4.9% 75% 192 12.0/3.93 4.93/14.39 3.20/8.52 - 4.97/1.00 150 78%

2.25 0.99 3.3% 100% Full 258 12.0/5.27 4.88/19.34 3.28/11.42 - 4.97/1.00 200 78%

Tested by EPRI PEAC Corporation,Knoxville, TN.

TYPICAL EFFICIENCY (50% Load):

Combined Max. Output Power on 5V and 3.3V

Load (%)

Rated Specifications

Input Voltage

Input CurrentInput Frequency

6/14/2004

Serial Number

Rated Output Power

Efficiency %

ITHD

(%)

Input Watts

Test Date

Model Serial Number

Input AC Current Waveform (ITHD = 5.7% at 50% Load)

TPS1U

DPS-20PB -118 B Rev 04BZT0237025302

YearType

AVERAGE EFFICIENCY :

These tests were conducted as a part of California Energy Commision initiative to improve the efficiency of the server power supplies used in the Data Centers through the Public Interest Energy Research (PIER) program.

Output Watts

DC Terminal Voltage (V)/ DC Load Current (A)Fraction of Load

Manufacturer2Delta Electronics

2002

Input and Output Power

Input Watts

Output Watts

0

50

100

150

200

250

300

20% 25% 50% 75% 100%Loading (% of Rated Output Power)

Po

wer

(W

atts

)

Efficiency of the Power Supply

40%

45%

50%

55%

60%

65%

70%

75%

80%

85%

90%

0% 25% 50% 75% 100% 125%Loading (% of Rated Output Power)

Eff

icie

ncy

(%

)

Efficiency %

Input AC Current and Voltage Waveforms

-200

-150

-100

-50

0

50

100

150

200

0 0.005 0.01 0.015

Time (s)

Vol

tage

(V

)

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Cur

rent

(A

)

Voltage Current

Server Power Supply Efficiency Test Data: Current Market

48.6%

62.1%67.4%

72.5% 72.6% 70.9%62.2%

73.5%78.3% 80.0% 79.1% 77.8%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Source: EPRI PEAC

95th Percentile

Median

% power supply loading% power supply loading

effi

cien

cyef

fici

ency

Importance of a Flat Power Supply Efficiency Curve

500W 1U Server Power Supply Efficiency Data

0

20

40

60

80

100

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

% Loading

Eff

icie

ncy

(%

)

Typical Loading

Range for Server PSU

More design focus on efficiency when More design focus on efficiency when lightly loaded is neededlightly loaded is needed

Goals of the Power Supply Task• Long term:

– Move the market towards widespread adoption of energy efficient PS in data centers

– Create an energy efficiency labeling program such as ENERGY-STAR® for server power supplies

30

131

32 32

72

41

86

27 32

020406080

100120140

AC

DC

Loss

es

DC

/DC

Loss

es

Fan

s

Driv

es

PC

I C

ards

Pro

cess

ors

Mem

ory

Chi

pset

Electric ity Use in a Server

Based on a typical dual processor 450W 2U Server; Approximately 160W out of 450W (35%) is losses in the power conversion process

(Source: Brian Griffith: INTEL)

Power Supplies in IT Equipment

Recommended Power Supply Efficiency

50%55%60%65%70%75%80%85%

0 100 200 300 400 500 600 700PSU Watt Rating

Power Supplies in IT Equipment

Electronic Load Banks

Yokogawa Digital Power

Meter

Fluke 41 Power Harmonic Analyzer

Power Supply Load Test Fixtures Server Power

Supply

Computer Interface

Server Power Supply EfficiencyLab Test Setup

Overall Computing Efficiency

Performance Metrics

AC Power Input Versus Percent CPU Time

0

25

50

75

100

125

150W

atts

0

20

40

60

80

100

120

% C

PU

Tim

e

WattsProc Time %

Dell Power Edge 2400 (Web/SQL Server)

Very Low Processor Activity…

…does not relate to very

low power consumption

Most of the time the GHz processor is doing activities that

can be done by a MHz processor but the input power consumption is not changing much

Performance Metrics -limited activity includes:• An assessment of:

– Server activity profile based on application/server type

– Correlation of server activity and power consumption

• Work with Intel, AMD, IBM, HP, and potentially others to reach consensus on metrics for some applications.

• Develop efficiency guidelines based on performance metrics

Demonstrations

LBNL’s Role in Demonstrations• Scoping demonstrations of technologies

or strategies to improve energy efficiency in high-tech buildings

• Showcase new/emerging or under-utilized technologies or approaches

Currently Planned Demonstrations

• Heat removal from servers without fans

• DC powering rack of servers

• Air management improvements (PG&E)

Demonstrations

Conclusion• Many strategies to improve energy

performance in a data center can be implemented with no impact on reliability – in fact, many will actually improve reliability and lower capital cost.

• Major industry firms are interested in supporting efficiency (and reliability) improvement.

• If you want more efficient data centers – ask for them.

Thank You• Questions?

• Bill Tschudi, Principal InvestigatorLawrence Berkeley National Lab510.495.2417wftschudi@lbl.gov

• http://hightech.lbl.gov