Accelerating business innovation; a Technology Strategy Board programme Energy efficiency and research computing RUGIT Away Day,

www.gridcomputingnow.org

Accelerating business innovation;a Technology Strategy Board programme

Energy efficiency and research computingEnergy efficiency and research computingRUGIT Away Day, 24th Jan 2008RUGIT Away Day, 24th Jan 2008

Dave BerryDave BerryDeputy Director, NeSC &Deputy Director, NeSC &

Technology Lead, Grid Computing Now!Technology Lead, Grid Computing [email protected]@nesc.ac.uk



Contents

Background and goals

Desktop Grids for High Throughput Computing

Data centres and HPC centres

Ways forward



GCN! Aims

Accelerate the benefits to the UK economy of adopting modern computing technologies, i.e.:

The creation of scalable, secure, efficient ICT infrastructures,

For delivering IT services, linked to business processes through a service oriented architecture,

While achieving greater utilisation with reduced energy consumption and reduced costs



Knowledge Transfer Network

Part of the Technology Strategy Board’s Innovation Programme

Run by Intellect, NeSC and CNR

Activities

Web platform; user case studies; events; webinars; active sector and regional programme

Communities of Practice

Green IT (including MBE KTN, BCS, …)

Transport Modelling, Grids in Health, Public sector IT, …



Aims for this session

To expand our community of practice

Stimulate discussion

Share knowledge

Form relationships

Suggest a “best practices” document

Input to policy (Unis, RCs, government)

Want to learn as much as inform



Desktop Grids

High Throughput Computing

Many experimental scientists are more interested in number of jobs/month rather than instantaneous computing power

One approach: use “Spare” Cycles

No need for air conditioning, etc.

But don’t we want to switch off machines at night?

Claim: desktop grids can give computing power for less cost and less electricity

Central Manager

1600 Workstations

30 Workstations

Submit Nodes

Execute Nodes

master, startd, starter

master, schedd, shadow

master, collector, negotiator

Cardiff slides from James Osborne,High-Throughput Computing Week

NeSC, 27-30 November 2008

Power Consumption

Watts Consumed

0 0 5

100112

150

0

20

40

60

80

100

120

140

160

Off Hibernate Standby Idle Office Condor

Machine State

Wat

ts

Based on a P4 3GHz PC with 512MB RAM

Economic Viability

• Makes sound financial sense

• Hibernate saves £60 per year

• Condor = £30 per year (max)

• Dedicated = £150 per year

• Condor is 5 times cheaper


Saving of Hibernate = Cost of 100W Electricity (Idle State) for 16 Hours out of 24

Cost of Condor = Cost of 150W Electricity (Condor State) – Cost of 100W Electricity (Idle State)

Cost of Dedicated = Cost of 150W Electricity (Condor State) + Cost of 100W Electricity (Air Con)

Environmental Impact

• Makes sound environmental sense

• Hibernate saves 650Kg CO2 per year

• Condor = 325Kg CO2 per year (max)

• Dedicated = 1,625Kg CO2 per year

• Condor is 5 times greener





Across Campus

• Makes sound financial sense– Hibernate would save £600,000 per year

• Hibernate 16 out of 24 hours

• Makes sound environmental sense– Hibernate would save 6,500T CO2 per year

– Rainforest required = 52Km2 – Rainforest required = 40% area of Cardiff

Based on 10,000 P4 3GHz PCs with 512MB RAM






The best of both worlds?

What if we could power-off idle machines and wake them up when Condor has jobs to run?

JISC Low Carbon ICT project

http://projects.oucs.ox.ac.uk/lowcarbonict/

Develop an institution-wide wake-on-LAN service

Monitor energy consumption across the University

Write and implement a communications strategy

Towards Low Carbon ICT conference

Oxford, 19th March 2008



Data Centres and HPC CentresData Centres and HPC Centres

Need: 2xProcessing Need: 2xProcessing Capacity per annum; Capacity per annum; Target: 60% Energy Target: 60% Energy Reduction from 1990 Reduction from 1990 levels by 2050levels by 2050



The scale of the problem

Power consumption by data centres:

Estimated 1.5% of UK national electricity generation

Peak consumption of 8GWEstimated to rise from 46 TWH

in 2006 to 93 TWH in 2020 Comparable with consumption

by refrigeration…… or greenhouse gases emitted

by aviation

Technology Power Loss Chain:Fossil Fuel – CPU Used

95% 2.5%

2.5%

Transmission Losses

Transformer Losses

Data Centre

40% 25% 35%Cooling Losses

Power Infrastructure

IT Equipment

0.5% of Fossil Fuel Energy

65% 20% 15%Network Equipment

Storage Equipment

Servers

30% 45% 25%Power Supply

Other Components

CPU

20% 80%Idle Time Power

CPU Load Power

35% 65%Heat Exhausted

Electricity GeneratedFossil Fuel

Data Centre

Equipment

Servers

CPU

CPU Utilisation

Slide from BCS DCSG



Policies and measurement

BCS DCSG model

Open source model of energy-efficiency for data centres and servers

The Green Grid

Vendor consortium

EU code of conduct on data centres

Development of baseline measures

Voluntary contribution of data from subscribing organisations



Press Examples

Ultraspeed DC-based system in East London

Claims 30% power saving from use of DC

Extra 10% saving from diskless servers

http://pcworld.about.com/od/recyclin1/Data-center-claims-power-cuts.htm

Plan for green data farm in Lockerbie

Using renewable energy sources

Waste heat used to heat new “eco village”

http://www.redwasp.co.uk/newsitem.asp?id=280



GCN! Webinar

The Business Case and Methods for the Green Data Centre

Recording available on the web

Zahl Limbuwala

Chair, BCS Data Centre Specialist Group

Motivation and an introduction to the BCS model

Kate Craig Wood

Managing Director, Memset Ltd.

Practical steps to running a “carbon-neutral” data centre



Finance and administration

Who sees the power bills?

Who calculates the total cost of ownership?

Does this affect purchasing decisions?

Trade-off: cost of reliability vs. cost of downtime

Don’t overspecify

Staff requirements

Specialised HPC kit may need specialised staff

Other kit may not



Power engineering

Location

Near power stations

Or local generation (CHP?)

AC or DC?

Provisioning levels

Nameplate provisioning is inefficient

Cables

Oversize for lower resistance (can halve losses)

Route through cool underfloor area

Example benchmarks

2006 SC1425 Dual Xeon

2007 860 Single dual core Xeon

0 100 200 300 400 500

Dell 1U server power usage

LabelMax loadIdle

Watts

Slide from Kate Craig-Wood

Graded UPS usage

Latest switch-mode based generation vastly more efficient

96-96% vs. ~90% Most efficient when fully loaded (98%) Don't run at half-capacity 40KVA steps, rather than typical 500KVA steps

Slide from Kate Craig-Wood



Cooling

What target temperature?

Water or air?

Store heated water externally?

External air or recycled air?

Savings from fresh air cooling of 72% (Source: BCS)

Modular cooling

Cabinets or rooms?

Modelling and controlling heat flow

Waste heat

How to sell or reuse?



Fresh Air Cooling



State of the art?

“By careful design, matching the specification of the plant to the needs of the machine(s) we have brought the summer time cooling overhead down from 60 - 65% (very typical of most computer rooms) to 25 - 30%.

Further, by using freecool this drops to 7% in the winter.”



System utilisation

Server virtualisation

Load balancing

Note – this is typically done already for compute clusters

May still apply to other university systems

Choice of equipment

Multi-core, power management, etc.

Diskless servers



Ways forward

Best practice document(s)Perhaps proven + experimental?

Shared facilitiesShared data centres?Shared compute clusters (a la NGS)?

Training for IT staff?Case studies?Input to policy (Unis, RCs, government)

Benchmarks and Labelling

0 20 30 4010 50 60 70 80 90 100

300

200

100

400

Idle

Idle 0 10 20 30 40 50 60 70 80 90 100

190 243

Load

Power 267 286 304 320 334 347 360 371 381 390

Po

wer

(W

atts

)

Load (%) Sleep Peak PSU

500

600

Sleep

25

PSU

600

Peak

440

Power Report for <VENDOR> <DEVICE> <MODEL> Under <BENCHMARK>

Processor: 2 x 2.8GHz Quad CoreMemory: 4 x 2GB 667MHzDisks: 2 x 146GB SAS

IO Cards: 2 x 10W PCI-E 10GBEIO Cards: 2 x 18W PCI-E FCPSU: 2 x 600W

BCS Data Centre Specialist Group

Slide from BCS DCSG



More references

GCN! Webinar

http://tinyurl.com/2gtslj

BCS Data Centre Specialist Group

http://tinyurl.com/2dyy5t

EU Code of Conducthttp://tinyurl.com/2drxoh

HTC weekhttp://www.nesc.ac.uk/action/esi/contribution.cfm?Title=831

Technology Strategy Boardhttp://www.berr.gov.uk/innovation/technologystrategyboard

Documents

Accelerating business innovation; a Technology Strategy Board programme Energy efficiency and research computing RUGIT Away Day,