100 Gigabit Ethernet Requirements & Implementation Fall 2006 Internet2 Member Meeting December 6, 2006 Serge Melle [email protected] 408-572-5200 Drew

100 Gigabit Ethernet Requirements & ImplementationFall 2006 Internet2 Member MeetingDecember 6, 2006

Serge [email protected]

Drew [email protected]

Fall 2006 Internet2 Member Meeting, December 6, 2006 2

Internet Backbone Growth

Industry consensus indicates a sustainable growth rate of 75% to 100% per year in aggregate traffic demand

Traffic increased more than 10,000x from 1990 to 2000 Traffic projected to increase an additional 1,000x from 2000 to 2010

[1] K. G. Coffman and A. M. Odlyzko, ‘Growth of the Internet’, Optical Fiber Telecommunications IV B: Systems and Impairments, I. P. Kaminow and T. Li, eds. Academic Press, 2002, pp. 17-56.


The Future Belongs to Tb/s Links! Carriers deployed Nx10 Gb/s networks several years ago

ECMP and LAG N now reaching hardware limit around 16 in some networks

Now evaluating deployment of (Nx) 40 Gb/s router networks Is this like putting out a 5-alarm fire with a garden hose?

Current Backbone growth rates, if sustained, will require IP link capacity to scale to > 1 Tb/s by 2010


Proposed Requirements for Higher Speed Ethernet

Protocol Extensible for Speed Ethernet tradition has been 10x scaling But at current growth rates, 100 Gb/s will be insufficient by 2010 Desirable to standardize method of extending available speed

without re-engineering the protocol stack

Incremental Growth Most organizations deploy new technologies with a 4-5 yr lifetime Pre-deploying based on the speed requirement 5 yrs in advance is

economically burdensome Assuming 5 yr window and 100% growth per year, ability to grow

link speed incrementally over 25 = 32x without a “forklift upgrade” seems highly desirable


Proposed Requirements (cont’d)

Hitless Growth Problematic to “take down” core router links for a substantial period

of time without customer service degradations SLAs may be compromised or require complicated temporary

workarounds if substantial down time is required for upgrade. Ideally, upgrade of the link capacity should therefore be hitless, or

at least only momentarily service-impacting.

Resiliency and Graceful Degradation Protocol should provide rapid recovery from failure of an individual

channel or component If the failure is such that full performance can not be provided,

degradation should only be proportional to the failed element(s).



Technology Reuse Highly desirable to leverage existing 10G PHYs, including

10GBASE-R, W, X, S, L, E, Z and LRM in order to foster ubiquity and avoid duplication of standards efforts

Deterministic Performance Latency/Delay Variation should be low for support of real-time

packet based services, e.g. Streaming video VOIP Gaming



WAN Manageability 100 GbE will be transported over wide area networks It should include features for low OpEx and should be:

Economical Reliable Operationally Manageable (e.g. simple fault isolation)

It should support equivalents for conventional transport network OAM mechanisms, e.g. Alarm Indication Signal (AIS) Forward Defect Indication (FDI) Backward Defect Indication (BDI) Tandem Connection Monitoring (TCM), etc.)

WAN Transportability Operation over WAN fiber optic networks Transport across regional, national and inter-continental networks The protocol should be resilient to intra-channel/intra-wavelength

propagation delay differences (skew)


Technological Approaches to 100 Gb/s Transport

Time Division Multiplexing(ie: Baud Rate)

10 Mbps

100 Mbps

1 Gbps

10 Gbps

100 Gbps

Space Division Multiplexing(ie: Parallel Optics)

1

4

12

8

Modulation(ie: Bits per Hz)

1 (e.g. NRZ)

2 (e.g. PAM-4, (D)QPSK)

4 (e.g. QAM-16)

8 (e.g. QAM-256)

Wavelength Division Multiplexing

(i.e. s)1 2 4 6 8 10

DWDMCWDM


Which Ethernet Application?

Ethernet is used today for many applications over different distances Distances > 100m primarily use

optical technologies

Performance for each application may be best advanced using a different approach

Telecom Application Class Translation Reach (km)

Very Short Reach (VSR) Intra-Room 0.1-0.3Short Reach 1 (SR-1) Intra-Campus 2Short Reach 2 (SR-2) Metro Access 10-15Intermediate Reach (IR) Metro Core 40Long Reach (LR) Regional 100Very Long Reach (VLR) Long-haul N x 100





(i.e. s)


Scaling Beyond 10Gb/s: TDM

• Too many problems!• 65nm CMOS will cap out long

before 100Gb/s• 100x shorter reach due to

dispersion (modal, chromatic, PMD, etc.)

• Bandwidth of copper backplane technology

• Fundamental R&D required to develop enabling technologies for low cost

100 Gb/s TDM unlikely to be a low-cost approach for any application in near future


10 Mbaud

100 Mbaud

1 Gbaud

10 Gbaud

100 Gbaud


1

4

12

8


1 (e.g. NRZ)


4 (e.g. QAM-16)

8 (e.g. QAM-256)


(i.e. s)1 2 4 6 8 10

DWDMCWDM


Scaling Beyond 10Gb/s: Modulation

Has never been applied to a high-volume optical standard and difficult for most applications of interest


1 (e.g. NRZ)


4 (e.g. QAM-16)

8 (e.g. QAM-256)


1

4

12

8


(i.e. s)1 2 4 6 8 10

DWDMCWDM


10 Mbps

100 Mbps

1 Gbps

10 Gbps

100 Gbps

• Digital Communication theory is well-established• Proven technology for copper technologies

1000BASE-T, DSL, Cable Modems, etc.• Limited use with optical technology• May be used in conjunction with other approaches


Scaling Beyond 10Gb/s: SDM

• OIF standards for Parallel Optical Interfaces• 10Gb/s VSR4 and 40Gb/s VSR5

• Slow adoption due to minimal market traction• Low volumes limits economic savings• Could be extended to 100 Gbps

• 12x 10 Gbps VCSELs

Most applicable to VSR applicationsSpace Division Multiplexing

(ie: Parallel Optics)

1

4

12

8


1 (e.g. NRZ)


4 (e.g. QAM-16)

8 (e.g. QAM-256)


(i.e. s)1 2 4 6 8 10

DWDMCWDM


10 Mbps

100 Mbps

1 Gbps

10 Gbps

100 Gbps


Scaling Beyond 10Gb/s: WDM

Extensive WDM technology development in past decade• Proven deployments in all telecom networks• Focus on cost reduction: CWDM, EMLs, etc.• 10GBASE-LX4 achieved success

• 4-color CWDM• SR applications

100 Gbps

Proven approach to reach Tb/s level bandwidth for even long reach applications


10 Mbps

100 Mbps

1 Gbps

10 Gbps

100 Gbps


1

4

12

8


1 (e.g. NRZ)


4 (e.g. QAM-16)

8 (e.g. QAM-256)


(i.e. s)1 2 4 6 8 10

DWDMCWDM


Drivers for a Super- (Multi-wavelength) Protocol

Per-channel bit rate growth historically and dramatically out-paced by Core Router interconnection demand growth

Requirement for WAN transportability strongly favors approach leveraging multiple wavelengths (Super-service


Won’t 802.3ad Link Aggregation (LAG) Solve the Scaling Problem?

LAG and ECMP rely on statistical flow distribution mechanisms

Provide fixed assignment of “conversations” to channels Unacceptable performance as individual flows reach Gb/s range A single 10 Gb/s flow will exhaust one LAG member yielding 1/N

blocking probability for all other flows VPN and security technologies make all flows appear as one

True deterministic ≥ 40G link technology required today Deterministic packet/fragment/word/byte distribution mechanism


Possible Channel Bonding Techniques

Traffic may be distributed over multiple links by a variety of techniques Bit/Octet/Word Distribution

Fixed units of the serial stream are assigned sequentially to lanes Small additional overhead allows re-alignment at the receiver Examples: 10GBASE-X, SONET/SDH/OTN Virtual Concatenation (VCAT)

Packet Distribution Sequence numbers added to packets to enable re-ordering at the receiver Large packets within the stream may induce excessive delay/delay variation to

smaller, latency-sensitive packets Examples: Multilink PPP, 802.3ah PME Aggregation Clause 61

Packet Distribution with Fragmentation Fragmentation bounds buffering requirements and delay associated with packet

size and packet size variation Overhead/link inefficiency is a function of the maximum fragment size chosen At 100 Gb/s and above, a fragment size can be chosen such that an effective

compromise between link efficiency and the QoS of individual, time-sensitive flows can be readily achieved

Examples: 802.3ah PME Aggregation, Multilink PPP


10Gigabit Ethernet Protocol

Reconciliation

PCS

PMA

PMD

Medium

XGMII

MDI

1

MAC

LAG

PHY

(Media Access Control)

(Link Aggregation Group)


Multilink Ethernet – N x 10G

PHY

Reconciliation

PCS

PMA

PMD

Medium

XGMII

MDI

Reconciliation

PCS

PMA

PMD

Medium

XGMII

MDI

1 N (ie: N = 10 for 100GbE)MultiLink Ethernet a.k.a. Aggregation at the Physical Layer (APL)

Reconciliation

PCS

PMA

PMD

Medium

XGMII

MDI

PHY

2

MAC

LAG

PHY

Multilink

MAC – Media Access Control

LAG – Link Aggregation Group


Multilink Ethernet Benefits

Ensures ordered delivery

Resilient and scalable Incremental hitless growth up to 32 channels

Minimal added latency

Line code independent, preserves all existing 10G PHYs

Orthogonal to and lower level than LAG

Scales into future as individual channel speeds increase


Multilink Ethernet Benefits (Cont.)

Concept well proven Packet fragmentation, distribution, collection and

reassembly similar to 802.3ah PME aggregation

Fits well with multi-port (4x, 5x, 10x, etc.) PHYs

Preserves existing interfaces (e.g. XGMII, XAUI)

Compatible with physical layer transport implementation over N x wavelengths


FPGAprovided by Xilinx

100GbE MAC with packet reordering,

implemented by UCSC

10 x 10Gb/s XFP boards, provided by

Finisar

Infinera DTN, provided by

Infinera

10x10Gb/s1310nm

10x10Gb/selectrical

10x11.1Gb/s15xxnm

New internet2 networkChicago – New York

Opt

ical

loop

bac

ks

2000km

Live 100 GbE Demo - Chicago to New York

* 100 GbE first demonstrated Nov 13 at SC06 between Tampa and Houston



Summary

100 GbE Requirements Protocol extensible for speed Hitless, incremental growth Resiliency and graceful degradation WAN transportability Technology reuse Deterministic performance Multi-channel operation

Multilink Ethernet meets the requirements

Technology proven over real networks

Thanks!

Serge [email protected]

Drew [email protected]

Documents

100 Gigabit Ethernet Requirements & Implementation Fall 2006 Internet2 Member Meeting December 6, 2006 Serge Melle [email protected] 408-572-5200 Drew