Energy Efficient Digital Networks Lawrence Berkeley National Laboratory September 22, 2009

Energy Efficient Digital Networks

Lawrence Berkeley National Laboratory

September 22, 2009

efficientnetworks.LBL.gov

EEDN PAC Meeting

Agenda

10:30 Welcome, Introductions, Project Overview

11:15 Energy Efficient Ethernet

11:35 Network Connectivity Proxying

12:15 Energy Efficiency Specs for Network Equipment

12:45 Break / Pick up lunch

1:00 Consumer Electronic Data/Network Links

1:15 Consumer Electronic Inter-device Power Control

1:45 Set-Top Boxes

2:00 Builder-Installed Miscellaneous

2:10 Energy Star Perspective

2:20 General Discussion

3:00 Next Steps (discuss Building Networks, if time)

3:30 Adjourn

Electronics as an End Use

• Electronics are an end use of electricity

“Devices whose primary function is Information(obtain, store, manage, present)”

–Includes both Information Technology (IT) and Consumer Electronics (CE)

–Much of this digitally networked already

• Conventional end uses (HVAC, lighting, appliances, …) all based in physics

• Electronics based in information

Network Structure

• Edge devices: PCs, servers - Displays, storage, phones, …

• Network equipment: switches, and routers

Networks and Energy

Network equipment ….

Routers, switches, modems, wireless APs, …

… vs networked equipment

PCs, printers, set-top boxes, …

How networks drive energy use

• Direct–Network interfaces (NICs)–Network products

• Induced in Networked products– Increased power levels– Increased time in higher power modes

(to maintain network presence)

Product

Network Int.

NetworkProduct

Link

Network electricity use in context

All Electricity: ~3,700 TWh

Buildings Electricity: ~2,700 TWhCommercialResidential

Telecom

Electronics: ~290 TWh

Networked: ~150 TWh ?

Network Eqt.: ~ 20 TWhOne central

baseload power plant (about 7 TWh/year)

• U.S. only• Annual figures circa 2008• All approximate

NOT to scale

Network electricity use in context, cont.

Buildings Electricity: ~2,700 TWh

CommercialResidential

Electronics

Networked~150 TWh ?

Net. Eqt.~ 20 TWh

• U.S. only• Annual figures circa 2008• All approximate

One central baseload power plant (about 7 TWh/year)

~290 TWh

This time to scale

Tel.

Some questions worth asking

• How much energy does all network equipment use? … telecom equipment? … edge devices?

• How much energy does network connectivity induce in edge devices?

• [ How much energy does IT avoid ]

• Where is all this headed?

• How much can we reasonably save in network eqt.? … in edge devices?

• What are research and implementation priorities?

Efficiency Approaches

Product Focus

Network Product Focus

Interface Focus

Protocol /Application

Focus

Need all approaches

Proxying Energy Star CE

Examples:

Finding Energy Savings Opportunities

Sample approaches

• Relax assumptions commonly made about networks–when feasible (rarely in core); mine wireless technology–these assumptions drive systems to peak performance

• average conditions require less energy

• many assumptions tied to latency

• Design for average condition, not just peak–rely on data about typical use

• Use Network to gather info about savings opportunities

• Use Network to enable edge device savings

Project Tasks- as proposed (% budget) -

• Information Technology Networks– Power-efficient Ethernet Links (13%)– Reducing Network-induced Consumption (17%)– Energy Efficiency Specs for Network Equipment (13%)

• Consumer Electronics Networks– Power-efficient Firewire Links (9%)– Consumer Electronics Inter-device Power Control (17%)– The Energy-efficient Set-top Box (24%)– Reducing Energy User of Hard-wired and Builder-

installed Equipment in New Homes (9%)

Project Partners(named in proposal)

• U.S. EPA Energy Star

• Cisco Systems

• Broadcom

• Force 10 Networks

• EFI

• University of South Florida

Market Connection

• Need pathway for widespread adoption of PIER-developed technologies; for EEDN primarily:– Industry standards– Energy Star specifications

• Many invited talks, including:– Internet Engineering Task Force tutorial– IEEE 802.3 (Ethernet) Committee tutorial– 2008 ACEEE Summer Study on EE in Bldgs.– Cisco Green Research Symposium– HP Labs Sustainability Innovation Workshop– CA Emerging Technologies Summit

Project Timeline and Status

• Summer 2005 - Original proposal

• January 2007 - Signed contract

• March 2010 - Scheduled end date (plan to extend)

• Work is about 2/3 complete

• Seek input on mid-course corrections for remaining tasks

• Considering follow-on projects to build on accomplishments of EEDN

IT-focused EEDN projects

• Energy Efficient Ethernet (EEE)– Reducing power consumption of

network links

• Network Connectivity “Proxying”– Reducing induced consumption of

networked devices

• Efficiency Specifications for Network Equipment (Specs)

– Providing market pull for more efficient network products

Agenda








1:45 Set-Top Boxes





3:30 Adjourn

EEE: Observations (1)

• Most links are mostly idle most of the time

• Actual traffic is bursty

• Most of time, full link capacity not needed

• Notebooks already dropped link rate in sleep

• Upper: file server link (Bennett, 2006)

• Lower: Snapshot of a typical 100 Mb/s Ethernet link (Singh)

0%

20%

40%

60%

80%

100%

0 1000 2000 3000 4000 5000 6000 7000Time (s)

Util

iza

tion


• Data networks are lightly utilized, and will stay that way, A. M. Odlyzko, Review of Network Economics, 2003

Network Utilization

AT&T switched voice 33%

Internet backbones 15%

Private line networks 3~5%

LANs 1%

Low utilization is norm in life — e.g. cars

• Average U.S. car ~12,000 miles/year = 1.5 miles/hour

• If capacity is 75 mph, this is 2% utilization


Source: METI, 2006Maximum throughput (Mbit/s)

1

10

100

1000

10000

100000

0.1 1 10 100 1000 10000 100000 1000000 10000000

Po

we

r co

nsu

mp

tion

(W

) Routers Throughput capacity is a function of links:

Energy cost is a function of capacity, not throughput

Measured power of variouscomputer NICs (averaged)Source: Christensen, 2005

EEE: Savings Opportunity (1)

• 1 Gbps (and lower) copper Ethernet links in the U.S. number in the 100s of millions - # increasing each year

– 2 NICs for each link– Each 1 Gbps NIC requires about 1 W

• Servers and many network links will migrate to 10 Gbps copper

– Each 10 Gbps NIC requires 5 W, maybe more

• With Audio/Video Bridging, Ethernet aims to penetrate the A/V market (# of possible links very large)

• New devices getting Ethernet (e.g. TVs)

EEE: Scope and Plan

• Review power consumption of several Ethernet technologies, technical approaches to changing speeds, and energy savings of these approaches

• Present these to IEEE, and if they take up the topic, work with them to create a standard

However….

• Even before we started, things changed:

• November, 2006 - IEEE 802.3 created the

Energy Efficient Ethernet Study Group

• We adapted our role as project advanced

EEE: Overall Timeline

• Sometime, 2004: Bruce and Ken Christensen come up with ALR idea

• July, 2005: Bruce and Ken propose ALR to IEEE 802

• November, 2006: 802.3 approves Call For Interest, renaming effort “Energy Efficient Ethernet”

• January, 2007: First EEE Study Group Meeting

• July, 2007: 802.3 approves move to Task Force status– Gets name IEEE 802.3az

• July, 2009: 802.3 approves first working draft

• August, 2009: First EEE NIC announced (Infineon)

• September, 2010: Anticipated FINAL approval

EEE: IEEE Timeline

Source: Mike Bennett, 802.3az TF Chair

EEE: Savings Opportunity (2)

• August 2009, Infineon announces first Gigabit EEE PHY

• Power drops 90% when littledata traffic

• Original proposal: Switch speeds quickly: << 2 seconds• This became: “Rapid PHY Selection” - switching in milliseconds• Later, proposal for “Low Power Idle” (LPI): promised

• Easier implementation• Greater power savings• Quicker transitions (switching in microseconds)

• After much discussion, EEE SG adopted LPI• Also added use of LLDP (Link Layer Discovery Protocol)

• Enables optimal timing of transitions to maximize potential savings of hardware beyond PHY

EEE: Technical Approach

Tq Tr TwTsTd

Quiet Quiet Quiet

Active

Refres

h

Refres

h

Wake

Sleep

Active

Low-PowerActive Active

EEE: LBNL Roles

• Chair of IEEE 802.3az Task Force (and EnergyEfficient Ethernet Study Group): Mike Bennett, LBNL Network Group

• Helping to guide process through key transitions

• Providing savings estimates

• Providing guidance on policy interest in EEE

• Educating energy community about EEE potential

• Posting materials to EEE web site

EEE: Next Steps

EEDN Project

• Prepare deliverables

Beyond EEDN

• Continue existing roles

• Monitor introduction of EEE components

• Provide policy guidance (e.g. role of LLDP)

• Help Energy Star incorporate as requirement

EEE: Summary

• Ethernet link utilization very low

• Energy can be made to track utilization– Savings in PHY 90%

• Standards process on track to realize this

• Products beginning to become available

• Policy in U.S. ready to react

• EEE could penetrate 100% of market

Agenda








1:45 Set-Top Boxes





3:30 Adjourn

Proxying: Observations (1)

LBNL Report: 1997

USF paper: 1998

Wake-on-LANintroduced: 1994

This is not a new topic:


Core Fact: Most PC energy use occurs when no one present

All time for year sorted by power level

Most of time when idle, could be asleep

PC savings potential is most of current consumption

Similar patterns apply to set-top boxes, printer, gameconsoles, …


• Network connectivity a key reason for systems to be on continuously

– Enterprise: Backups, IT admin access, remote access

– Home: Media sharing, communications, remote access

• Role of network connectivity in applications increasing

• Game consoles and set-top boxes getting PC-like functionality

Proxying: Scope and Plan

• Review –PC usages for network issues–Limitations of Wake-on-LAN (WOL)–Other relevant standards (e.g. DMTF, UPnP)–Savings potentials

• Develop proxy specification (in part from traces)

• Collect comments and refine

• “Sell” idea to industry, standards orgs., utilities, Energy Star, CEE, etc. (possibly including prototype)

• Desktop PC use nearly 70 TWh/year (U.S. only)• Idle time when no one present

easily half of this• Goals

– Enable large majority of PC users to use sleep without breaking their own or IT admin applications

• At least 80%. > 90% better. > 95% or > 98% even better.

– Enable both current and emerging common applications

– Enable standard to be used directly in (or easily adapted for) printers, set-top boxes, game consoles, etc.

• Savings from these also significant

Proxying: Savings

Proxying: Operation

LAN orInternet

PC

Proxy

1

3

42

1

2

3

4

PC awake; becomes idle

PC transfers network presence to proxy on going to sleep

Proxy responds to routine network traffic for sleeping PC

Proxy wakes up PC as needed

Proxy operation

Proxy can be internal (NIC), immediately adjacentswitch, or “third-party” device elsewhere on network

Proxy does: ARP, DHCP, TCP, ICMP, SNMP, SIP, ….

Proxying: Process

Standard• Ecma TC32-TG21

Prototypes• Microsoft Research “Somniloquy”• ???

Use Cases• In development

Trace Analysis• Intel Research Berkeley

TG21 participants

• Microsoft, Apple, Intel, AMD, Sony, Realtek, Oce,Hitachi, Lexmark, Terra Novum, LBNL

Proxying: Functionality

Components of the standard

• Basic Architecture

• Basic Frameworks (IPv4, IPv6, 802.3, 802.11)

• SNMP

• Teredo

• Remote wake

• UPnP

• mDNS/Bonjour

• Keepalives

Proxying: Timeline (partial)

• 1998: Ken Christensen publishes Proxy Server paper• 2001/2003: LBNL surveys find most desktop PCs on 24/7 in comm. bldgs.• 2003: Bruce and Ken begin discussions• September, 2004: Energy Star announces at IDF intention to address the

“Network Problem”• July, 2005: Bruce and Ken present proxying to IEEE 802• January, 2007: EEDN project officially commences• September, 2007: Ethernet Alliance publishes White Paper• December, 2007: Bruce presents proxying to IETF• January, 2008: Intel commits to helping• May, 2008: Initial discussions with Ecma International• September, 2008: Ecma creates TC32-TG21 on proxying• October, 2008: First TG21 phone meeting• January, 2009: First TG21 Face-to-Face meeting• June, 2009: Apple announces external proxying for mDNS/Bonjour• July, 2009: Energy Star computer spec V5.0 includes proxying• September, 2009: Fifth TG21 Face-to-Face meeting• November, 2009: Anticipated last F2F• November/09-March/10: Standard published• Shortly thereafter: Energy Star recognize Ecma standard

Proxying: LBNL’s role

• Pull idea from academic obscurity (work of Ken Christensen)

• Create interest in idea

• Work to put into Energy Star specification

• Work with Intel Research on trace analysis

• Identify best standards organization (Ecma International)

• Secure creation of Ecma TC32-TG21

• “Encourage” industry participation in TG21

• Define overall architecture of standard and several components

• Secretary for TG21

• Subcontract to Terra Novum (Tom Bolioli)

–TG21 contributor, convenor, Energy Star advisor

Proxying: Next Steps

EEDN project

• Help finish standard

• Create deliverables

Beyond EEDN –All working with industry and others

• Test internal proxying at LBNL and elsewhere

• Test external proxying at LBNL and elsewhere

• Refine Ecma standard based on testing

• Widely deploying initial proxy implemenations

• Create standard for communication with external proxy

• Create standard for monitoring proxy success

• Help extend proxying to printers, game consoles,and set-top boxes (and TVs and phones and …)

Proxying: Summary

• Most PC energy use occurs when no one present

• Network connectivity a key barrier to using sleep

• Technology can enable network connectivity in sleep

• Ecma standard will define much of this

• Energy Star ready to respond

• Additional steps needed to fully launch proxying

Agenda








1:45 Set-Top Boxes





3:30 Adjourn

• Equipment energy use changes little with load

• Utilization is very low

• Exaggerated estimates of network energy use

Specs: Observations on Energy and Network Equipment

367 W

376 W

Specs: Savings Opportunity

• Market differentiation exists in switch energy use

• Dialog with industry to better estimate savings

• Power use is a new design parameter–Previous design paradigm: reliability–Savings estimates vary from 25% to 75% –Achievable savings unknown

• Estimate the annual energy use of IP networks

• Estimate how energy use may change

• Focus specifications efforts on the products with the most potential impact

• Develop procedures and specifications–In collaboration with industry–In collaboration with Energy Star

Specs: Scope of Study

Annual Energy Use (2008)

Gig Switches

10/100Switches

Residential Network

EquipmentEnterprise Switches

Specs: Scope of Study

• Focus on network equipment that primarily carries IP traffic–LAN switches, routers–Home modems, routers–WiFi access points–Service provider equipment

• Lots of things not covered–POTS switching equipment –IP phones–Servers & desktop computer–Other end use and non-IP switching

equipment

Product

Network Int.

NetworkEquipment

Specs: Plan

Original Revised

All equipment togetherFocus on small equipment firstLarge equipment spread over time

Identify market interest, opportunities and barriers

Evaluate equipment power consumption and estimate typical power

Develop test procedures & specifications

Work with Energy Star on a program for network equipment

Specs: Small & Large Equipment

• Small equipment–Unmanaged switches < 9 ports–WiFi Access Points and Routers–Integrated home access devices–Optical network terminals

• Large equipment–Managed & modular switches–Dedicated security appliances

• In support of Energy Star Specifications Process–Small in late 2009–Large in 2010

Specs: Rough Energy Use Estimates

• Energy in USA: 19 TWh/yr (0.7% of US bldg total, 2008)

• Grew 16% between 2007 and 2008

• Forecast growth rate ~10% annually

Sources:Infonetics Market Data, 2003-2012FCC Broadband Market Data 2007-08Tolly Group Power MeasurementsLBNL Power MeasurementsAT&T Market EstimatesIndustry Data SheetsLBNL Market Research

Specs: Rough Energy Use Estimates

Customer Premises Equip (Small Equipment):

5.9 TWh

Switching Products:8.0 TWh

19 TWh Total

Specs: Products of Interest

• Small Network Equipment (SNE)–WiFi routers and access points–Cable modems–DSL integrated access devices–Cable integrated access devices–Unmanaged wired switches

• Large Network Equipment (LNE)–Managed switches–Modular core switches

• Line card vs box rating

Specs: SNE Market Interest and Barriers

• SNE dominated by ISP provided hardware

• Manufacturers of ISP provided SNE are not very interested–“Only if the service providers tell us to do it.”–Cost and reliability reign supreme–Integrated access devices are a moving target

• Other SNE manufacturers are interested–At least for marketing purposes–Some actual energy use reductions

Specs: LNE Market Interest and Barriers

• The “large equipment” market is interested–“But the real energy is in the data center (or PCs)”–Industry is addressing energy use

• ATIS network equipment test procedures, metrics

• Marketing flaunts green credentials

• Companies considering power in current redesigns

• Utilization is very low (throughput / capacity)–Need to test at realistic work loads

Specs: What to Test

Uplink utilization (10 days)

% Capacity

1%

0.5%

Modular switch with 144 GigE, 2x10GigE

• Most ports are inactive & unassigned–Incentivize power reduction when ports unassigned

Specs: What to Test

0.5 0.750.25Port Utilization

(ports assigned / total powers

N = 183 switches

• Throughput has small impact on power (<10% typical)–Incentivize stronger throughput vs power relationship

Specs: What to Test

Startup

Device under test: 6U modular switch with 96 GigE ports, 2x10 GigE (fiber), no PoE load, 320 Gbps fabricTested at LBNL, August 2009

Specs: Spec Development Progress

• Plan to use industry standard ATIS test procedures–Some modifications may be required

• Attended ATIS meeting on network equipment test procedure development–Procedures under development for most areas of interest–Working on a more formal relationship where we can

contribute to the process directly

• Internal procedures for testing network equipment–Provide a basis for our comments to ATIS

Specs: Next Steps

• Revise energy use estimates–Industry vetting –Values changing with time

• Complete market assessment

• Specification development–Relationship with ATIS–Work with Energy Star on modifications

Beyond EEDN:

• Continue Energy Star process

• Revisit procedures in light of data & technology

Specs: Summary

• Current network energy consumption ~ 19TWh annually

• Major consumers–Small network equipment–Enterprise switches

• Test procedure development is underway

• In a good position to assist Energy Star process

Documents

Energy Efficient Digital Networks Lawrence Berkeley National Laboratory September 22, 2009