Spirent High Speed Ethernet Training · Spirent Communications PROPRIETARY AND CONFIDENTIAL 10 Key Factors driving the future of HSE in Data Center 3-tier 20:1 and 4:1 oversubscription

1 Spirent Communications PROPRIETARY AND CONFIDENTIAL

Spirent High Speed Ethernet Training

Spirent CIP Marketing

January 2015


Spirent HSE Training Sessions

Session 1 – HSE Market Overview

Session 2 – Spirent HSE Product Offerings

Session 3 – HSE Optics & Cabling Overview

Session 4 – HSE Layer 1 Technical Training


HSE Market Overview Discussion Topics

Ethernet Timeline

HSE Market Overview

HSE Data Center Market

HSE Service Provider Market


Ethernet timeline

1G is 16 years old

It’s ~ 70% of overall market share when

network + server ports are combined!

But it’s only ~5% of port revenue

The disconnect is due to the majority of 1G

ports being free LOM (LAN on MTB)

But those ports are losing utility to built in

WiFi!

10G is 12 years old, but makes

up 80% of port revenue today

10G is way beyond its development cycle

though!

40/100G is already 4 years old

and still <15% of revenue


The HSE Market Landscape

Overall HSE adaption is slowed by macroeconomic headwinds and structural

changes underway; 3.3% is the avg IT spend –only 11% spent on network

Service Providers:

100G demand will continue to grow from Ethernet and IP MPLS VPN deployments

Cloud Providers and Mega Data Centers:

Will likely shift from 10G to 25G at edge and move from 40G to 100G, or even proposed 50G for

uplinks

Enterprise Campus DCs (those forgoing Private Cloud/NaaS/IaaS):

Server access will likely migrate to 10GBase-T due to cost advantages of 10G copper LOM

Client access will move even farther away from wired closets to WiFi due to the bandwidth gains of

802.11ac

Core network links will move to 100G due to 802.11ac APs and 10G servers at edge

Wired edge uplinks will trend towards 40G for wired 10G servers

In reality, HSE is a tale of two very different markets between DCs & SPs


HSE Market Share

10G is still generating primary revenue

1G is declining but still significant

40G is just beginning to register but…

Even in 2015 1G & 10G will be 80% +

of total market

Ethernet port forecasts for 2015 have

gone down from $52B to $42B

As a consequence, NEM R&D spend

on “box testing” has been trending

down while overall R&D spend is

growing


HSE Market Forecasts

40G may ramp up over the next 3 years

100G sales are still low, but also could

accelerate from QSFP28 release

It’s 2015, what’s the holdup?

1st typical 10G Data Center uplinks were only hitting

4-6G avg in 2013

2nd a four x 10G LAG = 40G; bandwidth problem

solved and cheaper!

3rd 10G LOM costs/delays 40nm vs. 28nm silicon

4th 40G NIC card/QSFP optics cost/availability –

Intel X520-QDA1 just became available 2014Q2

5th a skip to 100G mentality is starting to surface

6th cable plants are mostly still Cat 5e/6, Cat 6A/7

will be needed for copper 10G

7th the ongoing DC topology design war is

stagnating adoption of newer switch models


HSE Market Disruptors

Cloud/IaaS/NaaS:

What percentage of enterprises move a

significant amount of infrastructure onto the

cloud?

IaaS/NaaS will reduce network equipment

sales via consolidation/virtualization

How much larger do Cloud Providers

and Data Centers get?

Companies like Google, Facebook, Amazon,

etc. are atypical network equipment

consumers

They are far more cost conscious and

demanding of product capabilities

They are very well resourced and can go it

alone if needed; i.e. Google’s alleged

“homebrewed 10Gbe switch”

As servers and network equipment

consolidate to fewer numbers of

buyers, vendors of such equipment

will be forced to follow suit.


HSE in the Data Center Landscape

The Access layer speed ultimately dictates the bandwidth needed at higher levels

of the network

10G adaption was slowed by the 2009 recession and expensive premiums for

initial 10G server fiber adapters and copper LOMs

Newer/cheaper 28nm based X540 10GBase-T LOM could speed adoption up

25G could also either split the market, derail or even push off upgrades at edge

Server sales growth is projected to slow the next few years slowing upgrades to 10G connectivity

802.11ac could impact overall access switch sales to Enterprises in favor of a purely wireless edge, but

should increase speed needs in core and WAN

Legacy 3-tier DC designs based on STP in the Enterprise Campus also hinder the

adoption of newer networking technologies like Leaf-Spine, TRILL, SPB, and SDN

Older gear w/ legacy protocol support is centered on 10/100/1000 & 10G SFP+ uplinks, w/ little to no

10GBase-T or 40G/100G options


Key Factors driving the future of HSE in Data Center

3-tier 20:1 and 4:1 oversubscription model

Spine and leaf architecture

10G server and switch adoption (edge)

The rise of mega Data Centers

Network topology design strategy

The BYD revolution


HSE in the Data Center Market


3-tier 20:1 and 4:1 Oversubscription Model

20:1 access oversubscription is standard

practice for network access layer

20:1 is harder to maintain with 10G

edge; 40G & 100G uplinks to the rescue!

What if 20:1 is too high for 10G at edge?

Enterprise Campus will likely have to reduce the

access rate 20:1 by at least 25% (15:1)

Data Centers allegedly already follow a 3:1 or 4:1

model

What about application demands?

Two of the most common server types (Web &

Database) have drastically different

recommended oversubscription models

Application-specific oversubscription

ratios:

Web servers 15:1

Application servers 6:1

Database servers 4:1


Spine and Leaf Topology: Drives East – West Traffic

Designed with TRILL/SPB/SDN control plane in mind (i.e. minimized CP)

Leaf-spine is currently 10G SFP+, but could go 40G or 100G or even 25G!

40 & 100G for Spine-core uplinks and 100G for cross-pod (larger DCs)


Spine and Leaf Topology: 40G or 100G?

Legacy STP design 10GBase-T example Cisco (10 ToR access switches)

Cisco 5500 max 56 10GBase-T + 28 SFP+ ports

840GB / 40GB = 21; 2 x 40G QSFP+ needed or 1 x 100G!

800GB / 200GB = 4; so 2 x 100GB up to core

New 2-tier design (w/ hash/ecmp) 10GBase-T/SFP+ example HP (4 sw leaf = convention):

480GB / 20G = 24; so 40G needed

160GB / 40GB = 4; no change needed here

3-tier design requires an upgrade to 100G,

where as Spine-Leaf only requires an upgrade to 40G.

Cisco advocates 100G and Arista advocates 40G,

seeing a connection?


10G Server and Switch Adoption

Cisco Nexus 5596T Switch

HP FlexFabric 5700 Switch Series

Ethernet switch market leaders – Cisco &

HP (70% combined) are already positioning

products for a 10G copper access future

Cisco 5500 series: 2U 32 + 24 x 10GBase-T 16 x

SFP+ or 4 x QSFP+

HP 5700 series: 1U 32 x 10GBase-T 16 x SFP+ 2 x

QSFP+

New 10GBase-T switches, cheaper 10G

LOM (Intel X540) might be the help needed

to get the edge from predominantly 1G to

10G

Intel 10GBase-T dual LOMs expensive w/ 40nm but

now getting a lot cheaper w/ 28nm silicon just releasing

1 x QSFP NIC + DAC = $700 + t/s


HSE on the Campus Client Edge

Bring Your own Device and changing workplace

trends are reducing wired edge traffic

WiFi ultimately reduces overall Ethernet wired

port shipments

IEEE 802.11ac a.k.a “the wired edge killer” stands in the

way

• Seen as the first real enabler of the purely wireless

client edge “dream”

• Supports speeds up to 10x faster than current WiFi

WiFi & Cellular data enabled devices reduce

bandwidth at edge –connect from anywhere

Silver lining – 802.11ac is forcing both speed

upgrades in the core and WAN

Most of that is to 10G now but should expand to 40 &

100G in the future as adoption takes off


2015 and beyond 10GBase-T surge?

Supermicro currently offers 17 Romley based MTBs w/ 28nm X540 controllers,

and they only started shipping in 2014!

Is Spirent’s 10GBase-T offering up to par if 10GBase-T surge occurs?

As a mitigation step Spirent could at least plan out a higher density copper 10G roadmap for the new

hardware just in case

PART NUM PRODUCT NAME

FX-10G-C2 HYPERMETRICS FX 10GBASE-T 2-PORTS



MX-10G-C2 HYPERMETRICS MX 10GBASE-T 2-PORTS




Mega Data Centers on the Rise

Mega Data Centers with largest rack

size = 50U+ will also drive 40 & 100G

adoption, allowing for more port density

per rack.

Connections from ToR to EoR/MoR will

need to increase bandwidth and

distance supported as well

100GBase CLR4 is under

consideration for this purpose

Proposed by Intel and Arista it offers longer

reach (2km) than SR4 (100m), but much less

costly than LR4 reach (10km)

Ultra small form factor QSFP28 optics proposed


Enterprise Campus Network Design Factors

Protocol choices can inhibit Ethernet

speed adoption

IEEE 802.1d a.k.a Spanning Tree Protocol is 29

years old!

Older software features keep older hardware in

place

Cisco vPC works with both 802.1D and 802.1ad

(LACP) and provides MLAG functionality

LAG/LACP allows for 10G SFP+ to remain in

the network as noted

3-tier M-LAGs vs. Spine-Leaf designs have

ramifications for 40 vs. 100G uplinks in DC In Aug, 2012 “What’s the best alternative to the

Spanning Tree Protocol” Webtorial, Cisco

advocated leaving STP in place as an L2

convergence protocol over M-LAG/TRILL/SPB

Cisco vPC works over STP


Enterprise Campus Designs Passed Over

A whole generation of pre-SDN STP replacements were largely

ignored:

Cisco Catalysts: Virtual Switching System (VSS/VSL)

Cisco Nexus: virtual Port Channels (vPC)

HP: Intelligent Resilient Framework (IRF/IRFL)

Avaya: Split-Multi Link Trunking (SMLT/RSMLT/IST) **Released in 2000**

Extreme: Multi System Link Aggregation (M-LAG)

Brocade: Multi Chassis Trunking (MCT)

Arista: Spline (M-LAG)

SDN offers promise to create traction, but this could be a ways off as

throughput performance is hindered by L2/L3 learning rates presently


HSE in the Data Center: Disruptors

Intel’s X540 10G LOM based on 28nm

silicon should ignite quicker adoption of

40G and 100G for uplinks but…

Server sales growth is a headwind:

x86 Server sales ramped up fast after the 2009

recession (1G LOM units; X540 not out then)

Low single digit growth is the forecast up to

2017 (2013 actually declined and 2014 was

flat!)

64-bit ARMs are emerging and could disrupt

x86 sales the way x86 disrupted mainframes

Microservers are also growing fast and typically

only offer 1G interfaces (moving backward)


The Emergence of 25G

Perfect for Data Centers because:

Cost friendly upgrade path from 10G at DC edge

25G aligns better to current serial I/O limitations

allowing the most optimal bandwidth and port

density switch configurations

25G lanes are already the mainstay of next generation 100G PMDs (see right) →

Overview of 25G lanes

XAUI (10G) was 4 lanes at 2.5G each

XLAUI & CAUI (40G/100G) moved to 10.1325G lanes

in 4 & 10 lane configurations respectively

802.3bj (100G backplane) started the 25G lane

revolution now part of 100GBase-SR4 PMD (CAUI4 -

CFP2, CFP4, QSFP28) all based on 4 x 25G lanes

opposed to 10 x 10G lanes (CFP/CXP)

25G Ethernet will start the single lane revolution for a

port speed; previously 10G, 40G & 100G were all

summations of lower speed lanes


HSE in the Service Provider Market


HSE in the Service Provider Landscape

Bandwidth growth is being driven by Ethernet and IP MPLS services growth, Asia

is leading the way

10M to 1G is the predominant Ethernet services speed, but 1G to 100G is

expected to surge in coming years

The implications of SDN & NFV are causing headwinds in the short term

AT&T and Verizon lead Ethernet services market in U.S.

100G port shipments to SPs doubled in 2014, and is expected to triple in 2015!

SPs expect 10G parity in pricing for 100G (i.e. $100G = 10 x $10G); currently it’s more like 15x cost

In a 2014 Infonetics survey, Cisco, Juniper, and ALU are seen as most trust

worthy vendors, with Huawei coming in fourth


Ethernet and IP MPLS VPN Markets

The combined global Ethernet services and

IP MPLS VPN services markets totaled

$62.6 billion in 2013, up 12% from the year

prior

Revenue from Ethernet services delivered

on 10GE and 100GE is forecast by

Infonetics to grow 300% between 2013 and

2018

The global carrier router and switch market,

including IP edge and core routers and

carrier Ethernet switches (CES), totaled $3.2

billion in 1Q14

Overall there is a much healthier market

share distribution


SDN and NFV are Causing Headwinds

Asia leads overall growth

10M → 1G is still the lion’s share!

1G → 100G should surge

Q1 2014 grew only 2% YoY, (down 13% QoQ) blamed on “hesitation” from implications of

SDN and NFV in the carrier market


AT&T and Verizon Lead Ethernet services market in U.S.

Carrier Ethernet forecasts are projected to grow steadily up through 2017

The bulk of this growth will come from Asia, NA, and EMEA

100G port shipments to carriers:

17,000 in 2013

~30,000 in 2014

Forecast 102,000 in 2015!


Service Providers expect 10G parity in pricing for 100G

Infonetics’ survey of 32 carriers around the

globe shows only 4% of the router and

Carrier Ethernet switch ports bought in 2013

were 100G

Should grow to 19% by 2016

75% of carriers have P-OTS or plan to by

2016

Most carriers also expect to pay 10Gbe

parity, i.e. 100Gbe costs 10x that of 10Gbe

Currently that premium is more like 15x cost

Same survey shows the % that NEMs made

it into the top 3 list in terms of quality and

reliability


© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name

“Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved.

Specifications subject to change without notice.

spirent.com




January 2015








Discussion Guide

Spirent HSE Offerings

Modular

Appliances

10/40 Gigabit

100 Gigabit

400 Gigabit

Spirent HSE Sales Strategy


Spirent Hardware Platforms : TestCenter Chassis

• Most competitive price per port

• Optimized for efficiency Lowest TCO

• Industry’s highest density dual speed 10G/40G card and soon the only eight port 100G card

Highest Density

• Proven performance, scales to stress the largest fabrics and multi-chassis architectures

Scale and Performance



Lower Capital Costs

Most competitive price per port

Leverage chassis license investment across more ports

Lower Operating Costs

Up to 67% less power per port

Smart shutdown of unused ports

Less space and cooling needed

Lower Time to Test

Up to 4x faster booting

Powerful GUI and wizards reduce configuration time

Intelligent results speed up troubleshooting

Lowest TCO



New 100G modules

8 port native CFP4

8 port native QSFP-28

No slot restrictions for any modules

The N11U supports 96 100G ports, 2x the competition

DX2 dual speed 25/100G (4x25G and 1x100G)

IEEE 25Gbe future support

FX2/MX2 dual speed 25/100G (4x25G and 1x100G)

Architecture Spirent’s TestCenter platform is designed from the ground up for the: highest density 100G, 400G, and Tb from a power, cooling, time stamping and time synchronization perspective

Highest Density


25GbE: Spirent’s plans

25G Ethernet

Module development underway

25GE PoC Demo System

Overview: • Support for 25GE Consortium spec

• QSFP28 interface

• To showcase Spirent’s 25GE

capabilities to the market

• Initial interoperability trials

25GE Engineering Modules

Overview: • Support for 25GE Consortium spec

• Native QSFP28 interfaces

• Builds on PoC

• Hardware and dev testing

Targeting: April Targeting: Mid-year


Data Density

DX2

Functional Depth

FX2

Multiplay Scale

MX2 Highest Functionality

Carrier Ethernet

Routers/Switches

Highest Performance

Application Aware

Routers/Gateways

Highest Density

Data Center

Switches

MODULES

http://www.juniper.net/us/en/products-services/switching/ex-series/options/xre200/


Spirent Appliances with HSE Support

Appliance

Test Module/Chassis

• High Scale/Performance

• ALL supported per stream stats

• The ‘Best in Class’

• Multi protocol/CPU Intensive tests

• Balance of functionality, port density

and per port cost

• Medium Scale/Performance

• ALL supported per stream stats

• Functional Testing

FX/FX2

MX/MX2

Available now

HSE on C50/C100 is deferred currently, but still planned


C100

APPLIANCE

C100-S2

24 Cores

128Gb RAM

100G support

Interface: CFP2/QSFP28

MX2 Functionality

2x100G


C50

APPLIANCE

Interface: CFP2

FX2 Functionality

1x100G

Interface: QSFP+

FX2 Functionality

1x40G or 4x10G

C50 Available: Q4, 2014

C50 Shown at the 2015 GSM!


100G

MODULES

DX2

Interface:

CFP2

CFP4

QSFP-28

Speed per Interface:

1x100G

2x40G

8x10G

Available: Now

FX2

Interface:

CFP2

CFP4

QSFP-28


1x100G

Available: Now

MX2

Interface:

CFP2

CFP4

QSFP-28


1x100G

Available: Now


100G

MODULES

FX

Interface: CFP2

Speed per Interface: 1x100G

Available: now

Interface: CFP, CXP (adaptor),

QSFP (adaptor)

Speed per Interface: 1x100G, 2x40G

Available: now

MX

Interface: CFP2

Speed per Interface: 1x100G

Available: now

Interface: CFP, CXP (adaptor),

QSFP (adaptor)

Speed per Interface: 1x100G, 2x40G

Available: now


MX/FX-100G-P2

MX/FX-100G-P2 modules come with CFP2 interfaces

CFP2 = C(100) form-factor pluggable version 2

• Supports 100G Only, supports L1 testing

• Roughly half the size and power of CFP

• Requires N11U or N4U chassis

• Currently supports ACC-6083 and ACC-6084 transceivers


DX2,FX2,MX2 100G-P4 (CFP2 Modules)

DX2-100GO-P4 100G only module

DX2-100G-P4 Tri-Speed 1x100G, 2x40G, 8x10G

• 2x40G and 8x10G must use CFP2 SR10 optics on Spirent end (ACC-6084)

• 2x40G custom cable needed (ACC-1025), 8x10G requires (ACC-1026)

FX2 and MX2 version of modules are 100G Only

All versions of modules will support:

• CFP2 to CFP4 adapter (ACC-6091A)

• CFP2 to QSFP28 adapter (ACC-6094A)

• Why only 2 x 40G or 8 x 10G?

• 25G electrical, 5GB per lane goes unused

• Or 20GB per interface hence 80GB available for 40G or 10G breakout!


Modules will be shown at OFC 2015; demo units avail as soon as Spring 2015

8-Port CFP4 and QSFP28 modules (Proteus)

Planned Higher Density 100G products (Q3 2015 targeted)

• 8 port native QSFP-28

• 8 port native CFP4

• DX2 streams/scale

• Tx = 8k

• Rx = 16k/4k (basic/analyzer)

• Future support for DX2 stats re-factoring

• Very cost competitive

• No tri-speed

• No line rate capture

• No future 25G/50G support

• No fX/mX versions

• No increased density

Sneak peak at new 8-port 100G modules


100G

Flexibility

Single module for multiple technologies

Native interface of CFP2

Pluggable and mixable adaptors for:

CFP4

QSFP-28

CXP (planned 2015)

CPAK (public announce pending)

Available on all DX2, FX2 & MX2 modules

Purchase 1 module and ability to support several

form factors


HyperMetrics MX/FX 100/40G Family

CFP Modules for both 100G and 40G operation

Each 40G CFP interface can be converted to two QSFP+ interfaces

• Choosing the right mode is done in the GUI

• By default when in 40G mode, the module assumes 2 QSFP interfaces per CFP interface

• If you are trying to use 40G CFP interfaces, it won’t work until the configuration is adjusted

CXP to CFP adapter also available

MX/FX-100G-Fx, MX/FX-40G-Fx

ACC-6069A 2-Port

QSFP+ to CFP adapter

ACC-6068A

CXP to CFP adapter


40/10G, 10G

MODULES

DX2

Interface:

QSFP+


1x40G

4x10G

Available:

Now

FX2

Interface:

QSFP+


1x40G

4x10G

Available:

Now

MX2

Interface:

QSFP+


1x40G

4x10G

Available:

Now


Warpath QSFP+, Model DX2-40G-Q24, DX2-40GO-Q24, DX2-10G-Q24

24 QSFP+ ports

• Highest 40G density available on the market

• Provides highest per chassis 10/40G density available on the market

DX2-10G-Q24

• QSFP+ 10G only module

DX2-40GO-Q24

• QSFP+ 40G only module

DX2-40G-Q24

• QSFP+ 10/40G module


400G Ethernet Test Solution

Industries most advanced 400G solution

• L1-L3 performance and functional testing

• 64 – 16k frame size support

400G statistics support

• Counts, rates, out of order, latency

• L1 debug stats: PPM clock adjust, BIP counts

Port Capture

• 400G wire rate packet capture www.youtube.com/watch?feature=player_embedded&v=Eh-bCREavvw


100/40/10G

BREAKOUT

CFP2 to 2x40G

Fiber fan-out MPO used with a

40GBASE-SR4 on DUT.

CFP2 SR10 Spirent End

CFP2 to 8x10G

Fiber fan-out MPO-LC used with a

10GBASE-SR on DUT.



DX2 / FX2 Interconnects Quoting and Selling

P/N Description Supported HW Comments

ACC-6085A COPPER DAC QSFP+ 40GBASE-CR4 3M

dX2-40G-Q8

dX2-40GO-Q8

fX2-40G-Qx

fX2-40GO-Qx

QSFP+ 40GE only

ACC-6087A COPPER DAC Breakout QSFP+ to 4x10G SFP+ 3M

dX2-40G-Q8

dX2-10G-Q8

fX2-40G-Qx

fX2-10G-Qx

10GE only to SPF+ DUT

ACC-6089A OPTICAL TRANSCEIVER QSFP+ 40G-SR AND 4X10G-SR MMF All fX2 and dX2 Use with 1021A for 40GE or

1016A/1017A for 4x 10GE

ACC-1016A OPTICAL FIBER BREAKOUT MPO TO 4 LC PAIRS MM 3M

dX2-40G-Q8

dX2-10G-Q8

fX2-40G-Qx

fX2-10G-Qx

Use with 6089A for 10GE operation to

SFP+ DUT. Short Range only

ACC-1017A OPTICAL FIBER BREAKOUT MPO TO 4 LC PAIRS MM 10M

dX2-40G-Q8

dX2-10G-Q8

fX2-40G-Qx

fX2-10G-Qx


SFP+ DUT. Short Range only

ACC-1021A OPTICAL FIBER RIBBON MPO TO MPO CROSSOVER MM 5M

dX2-40G-Q8

dX2-40GO-Q8

fX2-40G-Qx

fX2-40GO-Qx


QSFP+ DUT. Short Range only


DX2 STATS

EVOLVING

Adding the following Statistics:

Advanced Sequencing

• Expected Sequence Number

• Sequence Run Length

• Received Packet Count

• Lost Packet Count

• In Order Frames

• Re-Ordered Frames

• Duplicate Frames

• Late Frames

Jitter

• Previous Jitter

• Maximum Jitter

• Total Jitter

To These DX2

Module/Modes

• DX2-100G-P4 in 100G mode

• DX2-100G-P4 in 2x40G mode

• DX2-40G-Q8 in 40G mode

• DX2-40G-Q24 in 40G mode

Oh by the way…

Increased routing scale 10x on the

100G modules


Spirent’s HSE Portfolio Overview

Spirent is the established test leader in 4 x 25G PMD which is the cornerstone of

future HSE development

Spirent is working on its 3rd generation of 100G cards based on 4 x 25G lanes

Ixia released their first 100G board based on 4 x 25G in late Q4 2014

Spirent has unmatched flexibility: CFP2/CFP4/QSFP28/CPAK on one card!

Spirent has higher 10/40/100G density per module –and significantly higher

density per chassis due to no slot limitations!


Spirent’s HSE Sales & Marketing Strategy

Spirent must continue grow it’s SP advantage –throw salt in the wound

Alternative switch vendors: OEMs, White box, smaller players like Arista and HP

will become crucial opportunities in coming quarters as 100G QSFP28 and even

25/50G emerge

Spirent has appliances: C1/C50/C100 that are perfectly suited to entering lower margin players, or previously hard to

penetrate accounts with support, or planned support for 40/100G

Spirent STC appliances must plan support for QSFP28 and coming 25/50G speeds

Watch the 10GBase-T x86 server and access switch market carefully

Watch the Ethernet Services leader board for Carriers leading in L2/L3 VPNs

Return to our L1 Ethernet roots:

Increase our Ethernet PG&A advantages across platforms: scale, resolution, density, analysis

Re-train the field again on those advantages!


Spirent’s HSE Sales & Marketing Strategy

Data Centers are changing: SDN, NFV…

Adoption of SDN & Overlays is Spirent’s ally

Cisco controls the Control Plane (CP) – the

legacy one!

Ixia’s only advantage over Spirent is that

same legacy CP: STP, EIGRP, OSPF etc.

But times are changing (see chart) →

Alternative CPs and overlays could

accelerate the industry’s pull away from

Cisco’s/Ixia’s DC leadership advantage

Modern DCs will likely put the emphasis

back on speeds & feeds; Spirent’s specialty!

50%

55%

60%

65%

70%

1Q13 2Q13 3Q13 4Q13 1Q14 2Q14

Cisco % MS Ether Switch past 1.25 yrs

% MS EtherSwitch

In 3Q10, Cisco's peak market

share was 83%





spirent.com




January 2015








Form Factors – It’s 2004 all over again!

Do you remember XENPAK, XPAK, X2 and XFP?


Standards & Standards Bodies

High Speed Ethernet (HSE)

Defined by IEEE in 802.3ba

Adds to 802.3 the necessary changes for 40G and 100G Ethernet

Form factors defined by MSAs

CFP for CFP, CFP2, CFP4

SFF for QSFP+, QSFP28

Electrical and component requirements defined by OFC

Exception

10 x 10 MSA – defines special PMDs for HSE

http://www.google.com/url?sa=i&source=images&cd=&cad=rja&docid=NZk3-2_k7bWBMM&tbnid=CYynpNcjLuNW6M:&ved=0CAgQjRwwAA&url=http://www.dbdump.org/news/assets_c/2009/08/IEEE-logo-34.html&ei=ZSTCUe3aNcetigKXkIGACA&psig=AFQjCNGvfXNfPlkMNanDFymACHatHmkurg&ust=1371764197950126

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=bcmGdLCccqYohM&tbnid=QHy705PR8d-YoM:&ved=0CAUQjRw&url=http://www.10x10msa.org/&ei=5yTCUcDdOKHgiwL3s4C4Dg&bvm=bv.48175248,d.cGE&psig=AFQjCNE6vwz3XpVI6pSYdexwVWEZh1fNJg&ust=1371764325367806


HSE Form Factors Summary

CFP (today) – supports 100G and 40G

CFP2 (today) – supports 100G,40G and 10G

CFP4 (today) supports 100G

CXP (today) – supports 100G and InfiniBand 12x QDR

QSFP+ (today; not shown) – supports 40G and 4 x 10G

QSFP28 (1H’15) – supports 100G,50G and 25G

CPAK (today; not shown) – supports 100G


Form Factors (CFP)

CFP, CFP2,CFP4 are approx half

the physical size from previous

form factor

CFP4 smallest form factor shown far right

and below width = 21mm

Source: CFP4 MSA


Form Factors – the wildcards

CPAK

Cisco decided industry not moving quickly enough

Acquired the technology and designed and built its

own optics form factor!

Size is very similar to CFP2

Planning to support all standard PMDs

Huawei has also expressed interest in

designing a form-factor

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=YfVCZMNhpilH2M&tbnid=VI_hsBZbk0UxIM:&ved=0CAUQjRw&url=http://www.cisco.com/en/US/prod/collateral/optical/ps5724/ps2006/white_paper_c11-727398_031813_ns581_Networking_Solutions_White_Paper.html&ei=E0XCUebyF6LliAL944GwDQ&bvm=bv.48175248,d.cGE&psig=AFQjCNGLpEOONvETDKvaCgtNMsSZhjsN6Q&ust=1371772425253106


PMD vs. Form Factor

Form factor tells you the size and packaging of the transceiver

CFP, CFP2, CXP, QSFP+

PMD tells you the medium of the “wire” and how the bits are put on “the wire”

100GBASE-LR4, 40GBASE-SR4, 100GBASE-SR10, 40GBASE-LR4, 40GBASE-CR4, etc.

Two transceivers with different form factors but the same PMD will

interoperate – the “wire” and how the bits are put on the wire need to be the

same, not the transceiver packaging

CFP 40GBASE-SR4 talks to QSFP+ 40GBASE-SR4

CFP 100GBASE-SR10 talks to CXP 100GBASE-SR10

CFP2 100GBASE-LR4 talks to CFP 100GBASE-LR4


Cable Assemblies

Fully integrated transceiver pair with “cable”

Cable can be copper (most common) or fiber (AOC)

Copper cable assemblies

Known as DACs (direct attached copper / cable) or Twinax

Copper can be active or passive (passive most common)

Common lengths are 1m, 3m, 5m, 7m

Can be 40GBASE-CR4 compliant or not

Fiber cable assemblies

Known as AOCs (active optical cables)

Lengths are usually in the 5s or 10s of meters (5m, 10m, 15m, 50m, 100m)

More common for 100G speeds

No IEEE PMD, but usually identified as SR optic


Copper Cable Assemblies – making them work

Running high-speed signals over copper wires requires tuning

The tuning varies by the length and gauge of the cable

Tuning is accomplished by pre-shaping the signal as it’s transmitted

Tuning must be done in the lab using an apparatus and high-speed scope

These are the “pre-emphasis” settings in the STC GUI


Fiber types

Single mode (SM) vs. Multi-mode (MM)

Single mode used for LR / ER

Multi mode used for SR

Quality of fiber

MM – OM3, OM4

SM – OS1, OS2

Higher quality (OM4 and OS2) capable of longer reach

Fiber “assembly”

Built for specific PMDs

Number of strands and terminating connectors


Fiber Assemblies

“DWDM” PMDs typically use SMF

E.g., 100GBASE-LR4, 40GBASE-LR4, 10 x 10 MSA

Pro: Ability to leverage existing fiber infrastructure

Con: More costly to design/build the optics

Short reach PMDs typically built for ribbon MMF

E.g., 40GBASE-SR4, 100GBASE-SR10

Each physical lane runs over its own fiber

Ribbon fiber terminated with MPO/MTP connector

Pro: Optics are cheaper to design/build

Con: Ribbon fiber not always readily deployed


MPO Terminated Ribbon Fiber Assemblies

MPO = “Multi-fiber Push On” assembly

• Also called MTP by Corning

Wide variety of high density cabling options

• MPO to MPO

• MPO cassette for patch panels

• MPO breakout into SC, LC, etc

40GBASE-SR4

• 12 fiber MPO cable, uses 8 fibers

100GBASE-SR10

• 24 fiber MPO cable, uses 20 fibers

May require new ribbon fiber infrastructure

24-fiber “stacked” MPO Cable Courtesy of Brocade


QSFP+ Overview

QSFP+ = quad SFP+

Equivalent to 4 SFP+ interfaces in a smaller QSFP+

package

Can run 40G Ethernet as well as 4 x 10G Ethernet

4 x 10G operation connected to a SFP+ DUT

requires “breakout” cable options


QSFP+ 40G Optics and Availability

CFP

Used for 1st Gen 40G solutions

Required for reaches of ER and greater

QSFP+

Current Gen 40G

40GBASE-SR4

40GBASE-LR4

4 x 10GBASE-SR compliant

4 x 10GBASE-LR compliant (alpha / beta 3Q/4Q

CFP – First Gen 40G

QSFP+ – Current Gen


CFP2 and CFP4 Optic Availability

CFP2

LR4 and SR10 production

CFP4

CFP4 requires yet another leap in technology to bring

power/heat down from CFP2

Concern CFP4 SR4 and LR4 Main PMD’s

New chips being designed to cope with above

QSFP28 / zQSFP is competing design – DACs in alpha /

beta today

CFP – In Production Today

CFP2 – Production Today

CFP4 – Production Q1 2015


Beyond Pluggable Optics

Arista fixed optics blade

Uses MXP optics with MPO connector fiber breakouts

Software configurable to 10G/40G/100G

Data Center only application


QSFP+ 40GBASE-CR4 DAC Cable

QSFP+ to DUT

QSFP+ to Tester

ACC-6085A Copper DAC QSFP+ 3M

Note: Some DUTs require 40GBASE-CR4 Auto-Negotiation with 40G

DACs dX2-40G is not capable of AN while fX2-40G is


40GBase-CR4

QSFP+ SFF-8436 (Direct Attach Copper – DAC)

Length

(meters)

AWG

0.5 30

1 30

2 30

3 30

4 28

5 26

6 24

7 24

Panduit offers different lengths/AWG Typical QSFP+ DAC

Elpheus - 0.5m costs $49

and 7m costs $159

Pin Locations


40GBase-CR4

MDI Direct Attach Copper (DAC) IEC 61076-3-113

Spirent does not support this

connector and it is not popular

for 40GBase-CR4. Same

cable used for Infiniband and

10GBase-CX4

Pin Locations

Typical MDI DAC


QSFP+ 40G-SR4 Optical Test Port Interface

MPO Connector plugs to

front of a 40G QSFP+

Optical Xcvr in QSFP+

Cage

QSFP+ 40G Optical Xcvr;

40GBASE-SR4

Some SR4 models work for

BOTH 40G and for 4 x 10G

ACC-6089A OPTICAL

TRANSCEIVER QSFP+ 40G-

SR AND 4X10G-SR MMF ACC-1021A OPTICAL FIBER

RIBBON MPO TO MPO

CROSSOVER MM 5M


QSFP+ 40G-LR4 Optical Test Port Interface

• QSFP+ 40G Optical

Xcvr;

• 40GBASE-LR4

• These LR4 optics DO

NOT breakout into 4x

10GBASE-LR

• A different optic is

required for 4 x

10GBASE-LR breakout

• 40GBASE-LR4

expected 2H’13

ACC-6077A OPTICAL

TRANSCEIVER QSFP+

40GBASE-LR4 SMF

(expected 2H’13)

LC SMF fiber connectors

plug into QSFP+ cage


QSFP+ 10G Copper Test Port Interface;

QSFP+ to 4x SFP+ DAC Breakout Cable

4x SFP+ QSFP+

ACC-6087A COPPER DAC

BREAKOUT QSFP+ TO 4X10G

SFP+ 3M

10G fanout in our GUI similar to ACC-6069A

CFP -> QSFP+ adapter representation

STC GUI shows 4 x 10G ports

Requires speed mode switch for 10G

QSFP+ port = one port group

Each 40G port is a group

All 4 10G ports off a QSFP+ are a group


QSFP+ 10GE Optical Test Port Interface; QSFP+ Xcvr plus 4:1

MPO Breakout Fiber (MPO to LC)

MPO Connector plugs to

front of a QSFP+ Optical

Xcvr in dX2 or fX2

4 pair of LC fiber connectors

plug into 4 10G SFP+ Optical

Xcvrs in DUT

QSFP+ 4x10GE Optical Xcvr; 10GBASE-SR

Some 40GE-SR4 xcvrs work for BOTH 40G

and for 4 x 10G

This optic DOES NOT support LR

operation

10GBASE-LR support expected end of

2013

ACC-6089A OPTICAL

TRANSCEIVER QSFP+

40G-SR AND 4X10G-

SR MMF

ACC-1016A OPTICAL FIBER BREAKOUT

MPO TO 4 LC PAIRS MM 3M

ACC-1017A OPTICAL FIBER BREAKOUT

MPO TO 4 LC PAIRS MM 10M

*Each port group

must run in either

10G or 40G


DX2-100G-P4 CFP2 Tri-Speed Support

ACC-6084A OPTICAL

TRANSCEIVER CFP2

100GBASE-SR10 MMF

MUST be used for 2x40 or

8x10G

Fiber fan-out MPO used with

a 40GBASE-SR4 on DUT.


(ACC-1025)

Fiber fan-out MPO-LC used with a 10GBASE-SR

on DUT. CFP2 SR10 Spirent End (ACC-1026)


100GBase-CR10 MDI SFF-8642 (CXP)

Molex offers 0.5, 1, 2, 3, 4, 5, 6,

7, 8, 9, 10 and 11 meter lengths

(all 30 AWG)

Elpheus – 0.5m costs $136 while

6M cost $429

Spirent supports this cable on

100G-F modules when you have

ACC-6068A

Pin Locations (84 pins)


100GBase-CR4 QSFP28 SFF-8665

Length

(meters)

AWG

0.5 30

1 30

2 28

3 26

4 25

5 24

LEONI offers different lengths/AWG

QSFP28 DAC from Mellanox

Schematic Pin Locations


100GBase-CR4

CFP4 – No Examples Found on Market (12/2014)

Length

(meters)

AWG

0.5 30

1 30

2 28

3 26

4 25

5 24

Lengths/AWG

Schematic Pin Locations

Example DAC


CFP2 100GBASE-LR4 Using LC SMF

1 pair of LC SMF fiber

connectors plugs into CFP2

in DUT

The other pair plugs into

CFP2 in tester

• CFP2 100G SMF optical

Xcvr;

• Availability now

• Interoperates with CFP

100GBASE-LR4

ACC-6083A OPTICAL

TRANSCEIVER CFP2

100GBASE-LR4 SMF


CFP2 100GBASE-SR10 Using MPO Ribbon MMF

MPO Connector plugs to front of a

CFP2 Optical Xcvr in MX/FX-100G-P2

MPO-terminated 24-fiber ribbon

• CFP2 100G MMF optical Xcvr;

• Availability Now

• Interops with CFP and CXP

100GBASE-SR10

ACC-6084A OPTICAL

TRANSCEIVER CFP2

100GBASE-SR10 MMF

ACC-1022A OPTICAL FIBER RIBBON

MPO TO MPO 24-FIBER CROSSOVER

MM 3M


Qualifying Optics for use with Spirent TestCenter

Spirent has a rigorous process for qualifying optics we re-sell

This provides a list of known working optics that is consistent from release to release

There are 14 tests performed and a general rule is a qualified optic must pass all tests

This procedure tests that all functions related to optics work (hotswap, error-free traffic, break/restore

link, read optics info, etc.)

Helps to eliminate variables when trouble-shooting issues

The active list of qualified optics can be found here:

http://iprep.spirent.com/iprepna/stc/Sales%20Training/Transceiver-List.pdf

Other HSE optics are tested and are known to generally work (link TBD)

http://iprep.spirent.com/iprepna/stc/Sales Training/Transceiver-List.pdf







spirent.com




January 2015






Session 4 – HSE Layer 1 Technical Training Part 1 & 2


HSE Physical Layer Technical Training Part 1


HSE L1 Technical Training Part 1 Discussion Topics

Ethernet Review

IEEE 802.3ba nomenclature

HSE PCS Sublayer


Ethernet Review IPGs & PPS

10MB

IPG = 9.6us

~14,881 pps (64 byte pkts)

100MB

IPG = .96us

~148,810 pps

1GB

IPG = .096us

~1,488,095 pps

10GB

IPG = 9.6ns

~14,880,952 pps

40GB

IPG = 2.4ns

~59,523,809 pps

100GB

IPG = .96ns

~148,809,523 pps

pps =

bits / (preamble + SFD + frame gap + frame size)

Ex. 1,000,000,000 / (56 + 8 + 96 + 512)

=1,488,095.23


Ethernet Frame format

Ethertype

2 bytes

Protocol Data:

46 - 1500 bytes

Time

7 octets 1 octet 6 octets

Destination

address Preamble SFD

Source

address

Length/

type Data Pad

Frame check

sequence (CRC)

6 octets 2 octets 46-1500 octets 4 octets

Min Frame = 512 bits (64 octets)

max Frame = 1518 octets

Length

2 bytes

Protocol Data:

40 - 1496 bytes

DSAP:

1 byte

SSAP:

1 byte

Control

1 to 2 bytes

Ethernet II

data format

IEEE 802.3 data

format defined

by IEEE/ ANSI/ISO

standards

This is not

Novell’s 802.3

raw frame


Ethernet MAC Parameters 10G/40G/100G

slotTime = 2x duration an electronic pulse

requires to travel max distance between two nodes

interPacketGap = minimum idle time between

packet transmissions

attemptLimit = maximum retransmission retries

backoffLimit = longest retransmission delay

interval

jamSize = length of jam signal

maxBasicFrameSize = largest allowed frame

size (untagged)

maxEnvelopeFrameSize = value chosen for

802.3as “Ethernet Frame Expansion” request

supporting new max frame size

minFrameSize = smallest allowed frame size

burstLimit = longest duration station can hold

transmission medium for continuously

ipgStretchRatio = number of bits in a frame

requiring one octet of ipg extension (stretch)

Source: IEEE 802.3ba


Ethernet Preamble, SFD & FCS Field

Preamble

The preamble begins a frame transmission

When generated by a MAC shall consist of 7 octets with the following bit values:

• 10101010 10101010 10101010 10101010 10101010 10101010 10101010

Start of Frame Delimiter (SFD)

The SFD (Start Frame Delimiter) is one octet and indicates the start of a frame by immediately

following the preamble. The bit sequence for SFD is:

• 10101011

Frame Check Sequence (FCS)

Cyclic Redundancy Check algorithm used at transmit and receive to generate a CRC value for

storing in the Ethernet FCS field

CRC generating polynomial (used in conjunction with data polynomial M(x) to calc CRC value):

G(x) = x32 + x26 + x23 + x22 + x16 + x12 + x11 + x10 + x8 + x7 + x5 + x4 + x2 + x + 1

Computed as a function of the contents of the source address, destination address, length, LLC

data and pad(that is all fields except the preamble, SFD, FCS, and extension)


Jumbo Ethernet Frame format


10G SR 64B/66B Line Code Review

64B/66B line coding offers a 3.125% overhead (comparably low to other schemes)

Continues to be used for 40 & 100G Ethernet

PCS encodes 64 bits of data (received as 2 x 32bit words) with a 2 bit “sync header”;

header’s built from data in the two 32 bit words

Sync header is mostly used for alignment and to reduce BER

Data + sync header results in 66 bit vector, ready for lane handoff (electrical to optical)

Decode is essentially the reverse process

Sync header has only two valid values (see 802.3ae Clause 49):

01 – data only

10 – data and control or control only

source: Marek Hajduczenia, PhD ZTE


High Speed Ethernet Introduction

HSE taskforce was formed in 2008 and amended 802.3ba for speeds of 40 and

100 Gb/s Ethernet preserving key attributes of 10Gbe

40G Ethernet uses (Roman XL):

XLGMII – 40 Gigabit Media Independent Interface

XLAUI – 40G Attachment Unit Interface

100G Ethernet uses (Roman C):

CGMII – 100 Gigabit Media Independent Interface

CAUI – 100G Attachment Unit Interface

Each MII supports 3 key sublayers:

PCS – Physical Coding Sublayer

PMA – Physical Medium Attachment

PMD – Physical Media Dependent

Copper implementations also support AN and FEC discussed separately


HSE PHY


IEEE 802.3ba – Nomenclature

40GBASE-KR4

40GBASE-CR4

100GBASE-LR4

100GBASE-SR10

Speed Medium

Coding Lanes

Copper Fiber Copper Optical

40G = 40 Gb/s

100G = 100 Gb/s

K = Backplane PAM2

P = Backplane PAM4

C = Copper (up to 7m)

S = Short Reach (100m)

L = Long Reach (10km)

E = Extended Reach (40km)

R = scRambled

64/66B encoding N = 4 or 10 lanes

N = Number of lanes

or wavelengths


PCS (Physical Coding Sublayer)


HSE PCS Lane Functions

Transfers/receives and

encodes/decodes data to/from the

FEC or PMA

Encodes/decodes 8 frame data

octets (64 bits) to/from 66-bit

blocks (64B/66B); also generates

control blocks for alignment, error

reporting etc.

Compensates for any rate

differences caused by the

insertion or deletion of alignment

markers or due to any rate

difference between the

XLMII/CGMII and PMA through

the insertion or deletion of idle

control characters

Determines when a functional link has

been established and informs the

management entity via the MDIO when

the PHY is ready for use

The 40G PCS is configured for 4 lanes

The 100G PCS is configured for 20 lanes


PCS Encoding Review

64 bits are encoded into a 66-bit code

The Sync Header ensures sufficient 0/1 transitions in the bit stream to make

clock recovery possible; only valid header values are: 01 or 10

Allows for the transmission of control characters (lane mgmt) and data

characters

Preserves the likelihood of detecting single or multiple bit errors

Synchronization headers used by receiver to achieve block alignment


Data Blocks

8 data characters

Control Blocks

8-bit block type field indicating type of block and format for

rest of block

Key Control Codes (Sync = 01)

Idle 0x07

Start of frame 0xFB

Terminate (EoF) 0xFD

Error 0xFE

Sequence Ordered Set 0x9C

Signal Ordered Set 0x5C

Sync header

01 = data block

10 = control block


Scrambler – Purpose of scrambling is to

ensure a relatively even distribution of 1s and

0s are normally found in the transmitted data.

The sync header makes sure there is at least

one 1/0 transition every 66 bits which is

sufficient to recover the clock.

The data is distributed to the PCS lanes one

66-bit block at a time in round robin fashion

starting with lane 0.


PCS Lane Alignment Markers

Special 66-bit control block

Inserted into the data stream after every 16,383 blocks

Inserted into all lanes at the same time

Inter-frame gap bytes are used from the data stream to make room

Not encoded (added after encoding)

Not scrambled (added after scrambling of data)

Not necessary to scramble – markers themselves are balanced 1s and 0s

Not scrambling allows the receiver to find the markers as well

This way the PCS lanes can be deskewed (next slide)

Assemble aggregate stream (transmit block; sync + data blocks)

Then alignment lock, reorder DBs per lane #, marker removal, descramble, last decode on Rx side


Format of Alignment Markers

Bits 0-1: Sync header (10 = control block)

The values M0, M1, M2, M3, M4, M5, M6 are different for each lane but the same for each

alignment marker in that lane.

Because each PCS lane is specifically identified lower layers can transmit the lanes in any

order – the receiver can use the alignment markers to figure out which lanes are received

where and reorder

BIP = Bit-Interleaved Parity. Used to detect bit errors in each PCS lane individually. This is

an 8 bit value.

BIP7 is bit-wise inversion of BiP3 to keep the alignment marker balanced

Each bit is an even parity calculation over all the previous specified bits of a PCS lane from and including the

previous alignment marker but not including the current one

Each bit is assigned 8 or 9 bit positions in the 66-bit block over which the even parity is calculated. For example,

BIP3 bit 0 is assigned bits 2, 10, 18, 26, 34, 42, 50 and 58 of the 66-bit block


PCS Receive Process

Get synchronized on

sync headers for each

66-bit block in each

lane

Get locked on

alignment markers for

each lane

Deskew PCS lanes

Count BIP-8 errors

Re-order the lanes

(can be received in

any order)


Skew

Ideally the sync headers for each PCS lane arrive at the same time

But in reality, the sync header for lane 0 might arrive at a different time than the

header for the adjacent lane 1 for example

This time difference is called “lane skew”

Skew is caused by variations in electrical, thermal or environmental differences

between the lanes

A certain amount of skew is within specification

100GBaseR – maximum = 180ns or ~1,856 bits / Variation 4ns or ~21 bits

40GBaseR – maximum skew is 180ns or ~928 bits / Variation 4ns or ~41 bits

If there is too much skew, the transmitted information on the lanes cannot be

reassembled by the receive PCS

If above the specified tolerance – the PCS layer must remove





spirent.com


HSE Physical Layer Technical Training Part 2


HSE L1 Technical Training Part 2 Discussion Topics

HSE PMA Sublayer

HSE PMD Sublayer

IEEE 802.3bj introduction

25 Gbe single lane introduction

HSE MDI

HSE Auto Negotiation Sublayer

HSE Forward Error Correction Sublayer


PMA (Physical Medium Attachment Sublayer)


HSE PMA

The 40/100G PMA primarily provides:

Transmission

Reception

Clock recovery

For copper implementations it provides collision detection

LR4 PMA w/ CAUI ex:

Upper PMA multiplexes 20 PCS lanes into 10 physical lanes of the CAUI (electric interface)

Lower PMA retimes signals, converts back into 20 lanes, then multiplexes into the 4 lanes needed

for the xR4 PMD (4 lanes x 25G)

The PMA implementation for SR10 is done on a chip at same layer as PCS, the 10 electric signals

are sent on a PPI (Parallel Physical Interface) that is not retimed; CPPI vs. CAUI

The PPI is implemented for short

distances, is self clocked, and

supports 10 x 10G lanes only


802.3ba – 40GE Examples


802.3ba – 100GE Examples


PMD (Physical Medium Dependent Sublayer)


Ethernet PMD Sublayer

IEEE Defines PHYs; examples:

100GBase-CR10: 100Gbps over 10 copper wire lanes

100GBase-SR4: 100Gbps over 4 OM3/OM4 optical fiber lanes

40GBase-LR4: 40Gbps over 4 single-mode optical fiber lanes

Each PHY has a part that is dependent on the physical media

PMD isolates this to a single layer in the protocol stack

The PMD generally defines what is in the transceiver

PMD details Tx and Rx of individual bits on a physical medium

Bit timing

Encoding (associated PCS encoding)

Interacting with the physical medium

Properties of the medium (wire, optical cable, air)


802.3ba – 40G PMDs

Reach <=1m Up to 7m 100m-125m 2km 10km

Name (PMD) 40GBASE-KR4 40GBASE-CR4 40GBASE-SR4 40GBASE-FR 40GBASE-LR4

Standard Status 2010 IEEE

802.3ba

2010 IEEE

802.3ba

2010 IEEE

802.3ba 2011 IEEE 802.3bg 2010 IEEE 802.3ba

Electrical Signalling 4 x 10 4 x10 4 x 10 4 x 10 4 x 10

Media Signalling 4 x 10Gbs copper

backplane

4 x 10Gbs copper

wires 4 x 10Gbs fibers 1 x 40Gbs 4 x 10Gbs DWDM

Media Type Copper backplane

Shielded,

balanced copper

(Twinax)

850nm MMF Tx 1550nm SMF

Rx 1550nm / 1310nm 1310nm SMF

Form Factor Backplane QSFP+ QSFP+, CFP CFP QSFP+, CFP

Connector Backplane N/A MPO12 SC LC/SC

Availability 2010 2010 2010 2012 2012, 2010


802.3ba – 100G PMDs

Courtesy of Brocade


IEEE PMDs Mapping to Media Dependent Interface

IEEE PMD Section Module Reference

40GBase-CR4 85.11.1.1 QSFP+ SFF-8436 (Small Form Factor Committee)

40GBase-CR4 85.11.1.2 IEC 61076 (Int’l Electrotechnical Commission)

100GBase-CR10 85.11.2 CXP SFF-8642

40GBase-SR4

100GBase-SR10

86.10.3.3 MPO/MTP IEC 61754-7

40GBase-LR4

100GBase-LR4

100GBase-ER4

40GBase-FR

87.11.3

88.11.3

88.11.3

89.10.3

QSFP+

CFP/CFP2/CFP4

CFP

IEC 61753-1-1 (general)

IEC 61753-021-2 (single mode perf)

100GBase-CR4

100GBase-CR4

92.12.1.1

92.12.1.2

QSFP+ 28

CFP4

SFF-8665

CFP4 HW Specification

100GBase-SR4 95.11.3.2 MPO/MTP IEC 61754-7


Introduction to 802.3bj

100G over Backplane/copper specification is focused on PMDs for:

100GBase-CR4

100GBase-KR4 (PAM2/NRZ)

100GBase-KP4 (PAM4)

Requires RS-FEC sublayer; RS(528,514) for CR/KR and RS(544,514) for PR

FEC support is a cornerstone and requirement of PMDs that support 25G lanes; otherwise Bit Error Rates and Frame

Loss Rates would be excessive

Reed-Solomon codes used to do error correction on Rx for improved BER/FLR: RS(n,k)

The goal is to reduce Signal to Noise Ratio (SNR) while maintaining required error probability for 100G

copper/backplane

For Ethernet FEC, code selection is based on RS(n, k, t, m);

• Where n is the block code or # symbols in a block

• k is the message length

• t is error correction capability of the RS code

• m is # of bits per symbol or symbol size



Introduces Channel Operational Margin (COM)

Provides an overall budget guidance for designs based on PAM2 vs. PAM4 and the varying

limitations like loss and interference between the two signaling methods

Computes a total noise profile end to end including: noise, jitter, inter symbol interference, and

crosstalk that the system is allowed to have

802.3bj also requires 256B/257B transcoding; transcoding compresses data

before FEC is applied reducing overhead of the RS code itself

Lastly 802.3bj will be based on SR4/LR4 PMD (4 x 25G lanes)



802.3bj adds EEE (Energy Efficient Ethernet)

EEE was part of 802.3az spec from 2010 and is now called “Deep Sleep Mode”

or Normal LPI (low power idle)

Uses AN to announce capability

Uses LLDP for wake sense

802.3bj redefines EEE and calls it Fast Wake LPI:

Main improvement is state transition time by keeping PMA & PMD active eliminating the need for

these two sublayers + FEC sublayer to resynchronize during transition that would normally exceed

the time budget (expected time savings = from us to ns)

PCS will encode/decode LPI signaling on MII

A surrogate signal will be sent during idle periods while PMD/PMA are still up


802.3 25G Proposed PCS/PMD

What PMDs are being proposed:

25GBase-KR

25GBase-CR

25GBase-SR

Basically two options for majority of sublayers;

pull from 10G or pull from 40/100G spec:

Reconciliation Sublayer: use 10G or 40/100G

Physical Coding Sublayer: “

Physical Medium Attachment: “

Forward Error Correction: use from 802.3bj



3 options are under consideration for 25G

Option A:

Use 64b encoding w/out alignment markers based on

40/100G Base-xR, alignment on 64b

Run lower at 25.78125G

Use 256B/257B transcoding from 802.3bj spec

Always encode data w/ RS-FEC w/ potential option for

non-FEC configuration as well

Option B:

Use 64b encoding w/out alignment markers based on

10G Base-xR, alignment on 32b

Run higher at 25.78125G






Option C:

Use 64b encoding with single alignment marker based on

40G Base-xR, alignment on 64b

Run higher at 25.78125G





802.3 25G Cabling/Optics

Optical

MMF & SMF

CAUI-4 single lane

LC & MPO connectors

AOC support proposed

25GBase-CR (two optics modules considered):

QSFP28 for DAC breakout from 100G

SFP28 (from 32G FC)


MDI (Medium Dependent Interface Sublayer)


HSE Medium Dependent Interface

IEEE defines an interface between compliant PHY and physical media

Defines both male and female sides

For fiber typically IEC standard connectors

For copper other connectors used


Medium Dependent – 40GBase-CR4 and 100GBase-CR10

Each signal is carried

on 2 wires in a pair.

So there are either 4

pairs or 10 pairs

carrying the signal –

depending on how

many lanes.


Medium Dependent – 40GBase-CR4

QSFP+ Direct Attach Copper (DAC) SFF-8436

Signaling rate per lane =

10.3125 GBd +/- 100 ppm

Note: Signaling rate for

10GBase-R is also 10.3125

GBd +/- 100 ppm

This is why this connector

supports 40G or 4x10G

Note – only 26 of the 38

contacts are required


100GBase-CR10 MDI SFF-8642 (CXP)

Note here all 84 pins are

used to create 12 different

lanes – each would be

running at 10.1325 GBd

+/- 100 ppm

So with this you could

have 12x10GBase-R with

direct attach copper

Or you could have

3x40GBase-CR4 with

direct attach copper

Or 1x100GBase-CR10

This is what Ixia did with

their “multis” test module


AN (Auto Negotiation Sublayer)


What is Auto Negotiation for 40G and 100G PMDs?

Required for Copper (CR) and Backplane (KR) 40/100G Ethernet

What does AN do?

Port has single media-dependent interface but supports multiple PHY types (such as 40GBase-CR4)

Allows link partners to negotiate PHY type and optional parameters

Defined in Clause 85.3 (refers to 73.1)

40GBase-CR4 and 100GBase-CR10 must support AN_LINK.indication

AN takes place on lane 0 – during AN other lanes are disabled

Uses Differential Manchester Encoding (DME)

A DME page is 48-bits long

The first page is the base page

A “next page” capability allows additional information to be exchanged


Auto Negotiation Base Page (48 bits)

5-bit Selector Field. 32 possible messages outlined in Annex 28A. 00001 = IEEE Std 802.3

5-bit Echoed Nonce Field. Contains the nonce word received from the link partner. When

acknowledge is set to 1, this field contains the nonce word from link partner. Otherwise it is all 0s

5-bit Transmitted Nonce Field. A random or pseudo-random number generated each time AN is

attempted

25-bit Technology Ability Field. If bit 4 is set the device is capable of 40GBase-CR4. If bit 5 is set the

device is capable of 100GBase-CR10. Multiple bits can be set although some combinations are not

allowed. Only bits 0-5 are defined

2-bit FEC Capability Field. Bit 1 advertises the devices FEC capability. Bit 2 requests FEC from the link

partner

2-bit Pause Capability Field. Bit 1 is same as Pause in Annex 28B. Bit 2 is same as ASM_DIR in Annex

28B

1-bit Remote Fault

1-bit Acknowledge

1-bit Next Page


What is Link Training?

Link training facilitates timing recovery and equalization

The mechanism is the exchange of fixed-length training frames

Occurs after AN is complete

Uses same Differential Manchester Encoding at a rate equal to ¼ of the

10GBase-KR rate

Frame structure (548 octets)

Frame marker (for start of frame)

Coefficient update (exchange of current settings)

Status report (signals state information (including “done”) from local PMD to link partner)

Training pattern – PRBS11 pattern (512 octets)


FEC (Forward Error Correction Sublayer)


FEC’s Location in Stack


Forward Error Correction

Redundant information is added to the signal to allow the receiver to detect and

correct errors that may have occurred in transmission

Technique used to improve signal quality on error-prone links

Normal Ethernet link must have BER of better than 1x10-12

100GBase-SR Ethernet link must have BER of better than 5x10-5

Use of FEC improves quality of 100GBase-SR link

A (n,k) code means

k information symbols you want to transmit are

encoded into blocks of n symbols where n > k

Any k symbols can be used to decode the data

Can tolerate n-k losses


FEC Code

n <= 2m-1 when its not equal the code is called “shortened”

there are n-k parity symbols

t symbol errors can be corrected t=(n-k)/2 when n-k is even

original message (k symbols) parity (n-k – 2t symbols)

symbol

(m bits)

code word (n symbols)


Simple FEC Example

Very simple illustration of (3,2) code

Two data bits a and b

Send three symbols a, b and a XOR b

Any 2 out of 3 symbols can be used to decode the data

A B A XOR B What SINGLE Bit Error can be corrected?

0 0 0 Receive 001 – Assume A XOR B must be wrong

Receive 010 – B must be wrong

Receive 101 – A must be wrong

0 1 1 Receive A and A XOR B – B must be 1

Receive B and A XOR B – A must be 0






FEC Block Diagram

20 lanes of 66-bit blocks received from

PCS

Sync on PCS lanes using the sync

headers

Get alignment marker lock

Put lanes in correct order

Take out alignment markers – count BIP-

8 errors

Transcode (re-encode)

Put the alignment markers back in

Run through RS encoder

Distribute on 4 outgoing lanes to PMA

sublayer


References

IEEE 802.3: http://www.ieee802.org/3/

IEEE 802.3ba: http://www.ieee802.org/3/ba/index.html

IEEE 802.3bj: http://www.ieee802.org/3/bj/index.html

IEEE 802.3bm: http://www.ieee802.org/3/bm/index.html

CFP MSA: http://www.cfp-msa.org/

Intel HSE Architecture preso:

http://www.ewh.ieee.org/r6/scv/comsoc/Workshop_101310_StdArch.pdf

Broadcom HSE overview preso: http://www.uknof.org.uk/uknof16/Hankins-

100G.pdf

http://www.ieee802.org/3/

http://www.ieee802.org/3/ba/index.html

http://www.ieee802.org/3/bj/index.html

http://www.ieee802.org/3/bm/index.html

http://www.cfp-msa.org/



http://www.ewh.ieee.org/r6/scv/comsoc/Workshop_101310_StdArch.pdf

http://www.uknof.org.uk/uknof16/Hankins-100G.pdf







spirent.com

Documents

Spirent High Speed Ethernet Training · Spirent Communications PROPRIETARY AND CONFIDENTIAL 10 Key Factors driving the future of HSE in Data Center 3-tier 20:1 and 4:1 oversubscription