1

Beyond 100GbE: How Datacenter Interconnects Drive

Demand for Higher Speed

Vijay Vusirikala, Bikash Koley and Paulie GermanoNetwork Architecture, Google

Outline

• Demand Drivers

• Google Views on

• Rate

• Reach

• Temperature range

• Lessons from 40G/100G

2

Global IP traffic growth

3

Internet Observatory Report

CAGR: 44%

MINT Study

CAGR: 50%

http://www.dtc.umn.edu/

mints/home.php

Cisco Study

CAGR: 40%

http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white_pa

per_c11-481360_ns827_Networking_Solutions_Whit

e_Paper.html

Global Internet Traffic is growing at 40%-50% year-over-year rate

Moore’s Law as REAL Driver for BW Demand

4Morris,Truskowski, IBM Systems Journal, Vol 42, No 2, 2003

~ 60% per year improvement in compute power/storage density

~ 60% per year increase in inter-compute BW demand

Warehouse-Scale Computer Interconnects

• Large number of identical compute systems

• Interconnected by a large number of identical switching gears

• Can be within single physical boundary or can span several physical boundaries

• Interconnect length varies between few meters to hundreds of kms

Interconnect Fabric Interconnect Fabric

Distance Between Compute Elements

Availa

ble

BW

What we would prefer

Today

Fiber-rich

Fiber-scarce

0.1

1

10

100

1000

10000

1995 2000 2005 2010 2015 2020

Year

"Moore's Law Traffic" Growth

Internet Traffic Growth

IEEE 802.3ae

IEEE 802.3z

Is Ethernet Speed Keeping Up?

6

IEEE 802.3ba

Tbit Ethernet

@ 2013?

Data

rate

(G

b/s

)

7

Beyond 100G: What data rate?

• 400Gbps? 1Tbps? Something “in-between”? How about all of the above?

• Current optical PMD specs are designed for absolute worst-case optical penalties, worst-case temperature and 15 year aging margin

• Significant capacity is untapped within the statistical variation of various penalties

Fiber Loss

Connector Loss

Dispersion Penalty

PMD Penalty

Jitter Penalty

Manufacturing Variation

EOL Margin

Back-to-back budget

IEEE 802.3

Link Spec

“Wasted

Lin

k

Ma

rgin

/

Ca

pa

city

“Other” Penalties

Rate Adaptive 100G+ Ethernet?

8

0

100

200

300

400

500

600

0 2 4 6 8 10 12 14 16 18

Bit

Rat

e (G

bps)

SNR (dB)

mQAM

OOK

• How to get there?

• High-order modulation

• Multi-carrier-Modulation/OFDM

• Ultra-dense WDM

• Combination of all the above

• There are existing standards within the IEEE802.3 family:

• IEEE 802.3ah 10PASS-TS: based on MCM-VDSL standard

• IEEE 802.3ah 2BASE-TL: based on SHDSL standard

• Needed when channels are close to physics-limit : We are getting there with 100Gbps+ Ethernet

• Shorter links ≡ Higher capacity

• Applications need to be aware of the available BW: MAC layer messaging

Is There a Business Case for Variable Rate?

9

0

0.05

0.1

0.15

0.2

0.25

0 10 20 30 40 50

Dis

trib

uti

on

Link Distance (km)

Example Link Distance Distribution

0

100

200

300

400

500

600

0 10 20 30 40 50

Po

ssib

le B

it R

tae

(Gb

ps)

Link Distance (km)

Example Adaptive Bit Rate Implementations

scheme-1

scheme-2

non-adaptive

• Cost of aggregate capacity for large number of “metro” links could be significantly lower with adaptive-rate Ethernet with a base-rate of 100G but capable of speeding up for shorter distance/ better link quality

400

500

600

700

800

900

1000

1100

0 100 200 300 400 500 600

Re

lati

ve C

ost

Max Adaptive Bit Rate with 100Gbps base rate (Gbps)

Relative Cost of 100T Capacity

Rate-Adaptive

Fixed-rate