Fabric Arch Compet

DATA CENTER FABRIC ARCHITECTURE COMPETITIVE DIFFERENTIATORS

SUTAPA BANSALProduct Marketing Manager, FSG

Juniper Confidential

Dunkleosteus Fish

Ruled the WorldMarlin

In the struggle for survival, the fittest win out b ecause they succeed in adapting themselves best to their enviro nment .

Charles Darwin

INCUMBENT

2 Copyright © 2011 Juniper Networks, Inc. www.juniper.net

Ruled the World

SLOW, Armored FISH – Did Not

Adapt

NOW EXTINCT!

Fast, Agile

Replaced Prehistoric

NEW RULER!

FAST MARLIN


AGENDA

Challenges With Data Centers1

Juniper’s Fabric Architecture2

Competitive Cisco Fabric Path3


Competitive Brocade VCS Architecture4

Summary5

� Fabric Path Architecture� TRILL Protocol


Application

TRENDS IN THE DATA CENTER


Application Architecture

Evolution

VirtualizationData Center Consolidation


Latency Sensitive

CHANGING ROLES OF THE NETWORK

Latency Tolerant

Traditional role – connecting users• North-South traffic

New role – connecting devices• East-West traffic• Ideally one hop away


Application running

Newest role – foundation of the cloud• Any-to-any connectivity


CHALLENGES IN DATA CENTERS

Scale to support DC consolidation

High Performance Networking

Better user experience:� Faster NW with low latency / lower oversubscription / large bandwidth

1

2

3


� Faster NW with low latency / lower oversubscription / large bandwidth

Virtualization support

High reliability

Manage complexity, and reduce cost of networking

4

5

6


Network FabricData Plane

� Flat

DEFINING THE IDEAL NETWORK – A FABRIC

Flat, any-to-any connectivity

Single deviceN=1

Attributes

• Low latency and jitter


� Flat� Any-to-any

Control Plane

� Single device� Shared state

A Network Fabric has the….

Performance and Simplicity of a single switch

Scalability and Resiliency of a network

Andthe

• Low latency and jitter• Lossless• Non L2 and L3 support


AGENDA






Summary5



DATA PLANE IN A SINGLE SWITCH

1. All ports are directly connectedto every other port

Data Plane


2. A single “full lookup” processes packets


CONTROL PLANE IN A SINGLE SWITCH

Single consciousness

Control Plane


All the ports are managed from a single point

Centralized shared table(s) have information about all ports

Management Plane


THE QFABRIC DATA PLANE

Data Plane So, we separate the fabric from the i/o ports

And replace the copper traces with fiber links

For redundancy add multiple devices



THE QFABRIC DATA PLANE

Interconnect

1. All ports are directly connectedto every other port

2. A single “full lookup” at the ingress edge device

Data Plane


Edge

The distributed data plane is implemented on the interconnect and edge devices


Director

SCALING THE CONTROL PLANE

The intelligence and state is federated,

Control Plane


state is federated, distributed across the fabric

New HostAddress

A federated control plane provides both scalability and resiliency


Director

SCALING THE MANAGEMENT PLANE

Management Plane

Single point of management


management

Extensive use of automation

Managed as a single switch - N=1


QFABRIC HARDWARE

Interconnect

Connects all the edge devicesCannot function as a standalone device

Edge node


Media independent I/O ToR device.Can be run in independent or fabric mode

Director

2 RU high fixed configurationX86 based system architecture


QFABRIC VALUE PROPOSITION

� Low latency– <1Us in rack– <5Us across fabric – max cable length

� Low jitter

High Performance Flat Fabric

� Seamless layer 2 and layer 3 with scaleLargeScale


� Non-blocking and lossless

� FCoE gateway and transit switch

� Lower power, HVAC, space

FullWirespeed

Storage Convergence

GreenSolution


COMPETITIVE LANDSCAPE1H2011 Timeframe Estimate

QFabric Cisco (N7K+N5K)FabricPath

BrocadeVDX

Configuration 4 Chassis125 ToRs

24 Chassis171 ToRs

24 Chassis 1

136 ToRs

Layer 3 Support Yes No No 2

Networking Racks (#) 2 24 24

17 Copyright © 2011 Juniper Networks, Inc. www.juniper.net 1Needs L3 core to scale for 6K ports

2L3 supported only in spine/core devices

Networking Racks (#) 2 24 24

Power/Port 14W 37W 54W

Device to Manage 1 191 160

Latency (Inter-rack) <5u >35u >93u

Oversubscription 3:1 3:1 3:1


AGENDA






Summary5



JUNIPER QFABRIC VS. CISCO FABRICPATH3 COMPETITIVE AREAS

Hardware


Architecture ProtocolsHardware

Components


Cisco’s architecture is based on L2MP, Cisco called FabricPath, and the IETF standard is TRILL. Cisco FabricPath works at Layer 2, and need to extend the IS-IS protocol as its control protocol.

IP+MPLS WAN Agg Router

CISCO FABRICPATH ARCHITECTURE – FUTURE

FabricPath

WANDC Core

Nexus 700010GbE Core


DC Access

Nexus 5548 and Nexus 2000

FabricPath

Gigabit Ethernet

10 Gigabit Ethernet

10 Gigabit DCE

4Gb Fibre Channel

10 Gigabit FCoE/DCE


CISCO FABRICPATH ARCHITECTUREREALITY FOR 6000 PORT CONFIG

FabricPath:

1 62 3 4 5

21 Copyright © 2011 Juniper Networks, Inc. www.juniper.net Note: - OS* Over Subscription 3:1- Ports: 4000 server ports

…. ….….…. ….1 112

12

19 20 38 39 57 58 76 77 95 96

1

Chassis: 4 vs 20Access devs: 84 vs 112Links: 336 vs 4288

Managed Devices:N = 1 vs 132


Director

QFABRIC RESILIENCYCONTROL PLANE ISOLATED FROM DATA PLANE

Redundant out-of-band connections

Control PlaneJuniper Interconnect


QFX 3500

X

Dedicated control plane

MAC spoofing

Out of band control plane – data plane flooded but control plane is not blocked

Server

Faster Re-convergence

Unknown traffic


Same path used to propagate control plane changes as

data plane; prone to attacks, less resilient

Control Plane

FABRICPATH CONTROL PLANE PRONE TO ATTACKS:SHARED WITH DATA PATH


Nexus 5548

Control Plane Propagation Over Same

Data Plane

X

Unknown Destination Traffic Flood

Control plane impacted due to data plane flood


QFABRIC: HIGH PERFORMANCE, LOW -LATENCY

Data Plane


231.1.1.1

Stock update

<1Us<5Us

<5Us

QFABRIC: Predictable, consistent performance; every \path < 5Us

Server Server ServerServer Storage Storage ServerBack-End Application Servers

<5Us

<1Us


Different latency depending upon

path; latency across Fabric

<35Us

FABRICPATH: INCONSISTENT PERFORMANCE, HIGH LATENCY

Data Plane


Nexus 5548

FabricPath: Unpredictable, inconsistent performance; QFabric is half the latency of a single Nexus7000

Back-End Application Servers

ServerServerServer231.1.1.1

Stock update

Back-End Application Servers

Storage

14Us 35Us


QFABRIC: SCALED MANAGEMENT PLANE

Director

Management Plane

Single point of management

Extensive use of automation

Admin


Managed as a single switch – N=1


FABRICPATH COMPLEX AND COSTLY MANAGEMENT: 1000 PORT VIEW

Management Plane

Each device managed separately

Multiple touch complex management

OPEX: During image upgrade, provisioning,

M M

M M M M

Admin


6*10G links

Nexus 5548

upgrade, provisioning,

Maintenance

Each Device Managed as an Individual Switch

MMMMMM M



Hardware



Components


No Multipathing With STP

Link Blocked

FABRICPATH TRILL: A BRIEF OVERVIEW

TRILL is a new IETF protocol to perform Layer2 bridgingbased on Layer3 IS-IS link state routing technology

Eliminates STP and increase bandwidth utilization (active-N links) and Fabric efficiency by

B3 B2

B1

B4

X


links) and Fabric efficiency by allowing equal bisectional traffic

Minimal configuration burden on user

Layer2 only technology and does not address Layer3

TRILL supposed to help inter-op but questionable

Trill Allows Multipath

RB2

RB4

RB3

RB1


ARE TRILL IMPLEMENTATIONS INTER -OPERABLE?

ODA OSA

Outer Destination Address (ODA)

Outer Source Address (OSA)

Ethertype = DTAG

Inner Destination Address (IDA)

IDA

Inner Source Address (ISA)

Rest of Original Frame

TTL (6b)FTAG (10b)

ISA

NHDA NHSA

Next Hop Destination Address (NHDA)

Next Hop Source Address (NHSA)

Ethertype = CTAG

Inner Destination Address (IDA)

IDA

Inner Source Address (ISA)

ISA

Next_Hop.VLAN

Egress Nickname Ingress Nickname

Ethertype = TRILL V,R,M,OpL,Hop_Count

Different frameformats,

proprietary extensions – Nointeroperability


FCS

Rest of Original Frame

FCSFabric Path TRILL

TRILL implementation based on FSPF

TRILL implementation based on IS-IS


DATA CENTER DESIGN L3 OR L2?

Capacity planning, traffic engineering, VLANs anywhere

L3 Needed for:

Address Space

L3 Hierarchicalvs. L2 Flat

Domain Size

Need Both Need Both L3 & L2L3 & L2

L3 Fault Containment vs.

L2 Flexibility


VLANs anywhere and G/W routing

TRILL is L2 onlyNeeds external (or internal) router to connect L2 domains

Learning

L3 Control Plane vs. L2 Data Plane

Complexity

L3 Convergence vs. L2 Plug and Play


QFABRIC VS FABRICPATH

QFabric is Layer 2 and Layer 3

QFabric4

1

1

84. . . . . . . . .

FabricPath is only Layer 2

32 Copyright © 2011 Juniper Networks, Inc. www.juniper.net Note: - OS* Over Subscription 3:1- Ports: 4000 server ports

…. ….….…. ….

FabricPath:

1 112

12

1 6

19 20 38 39 57 58 76 77 95 96

2 3 4 5

1


Cisco Strategy

ECONOMICS OF TRILL BASED DATACENTERS

Initial Pitch

• L2-only Traffic

• Position Fcard

• 32x10GE

• $35K ($1K/10GE)

Real Deployment

• L3 or Multicast or FCoE or QCN

• Mcard Required

• 8x10GE

• $70K ($9K/10GE)


Rich Core Rich Core (FP & L3 (FP & L3

Cards)Cards)

Simple Edge

SimpleSimpleCoreCore

Simple EdgeLock-in

Up-Sell

Cost

Scale

Cost

Scale

• $70K ($9K/10GE)


QFABRIC EFFICIENT MULTI-PATHING BY LOAD BALANCING

Efficient Bandwidth

utilization with spraying across

unequal links

Director

14%

25%25%25%25% 28%28%28%28%


Multi-Pathing Dynamic

14%

X


In-proffeciant bandwidth utilization

FABRICPATH TRILL: NOT EFFICIENT LOAD BALANCING FOR MULTI-PATHING

NX7K


Nexus 55484 168 12 28

FP uses TRILL based on ECMP: Only equal cost paths are chosen

X0%25%25% 25% 25%

33%33% 33% 33%


KEY ISSUES WITH FABRICPATH IMPLEMENTATION

Low Reliability and Control over Network

Poor Performance (high latency, high jitter)

Complex Network Management


Limited or No L3(Layer 3 as a Service Limits FabricPath Scale Costly Solution)

Limited Virtualization Support: Small 16K MAC Table, Small Port Density at Full L2/L3

Low on Uplink Bandwidth: Oversubscription



Hardware



Components


Nexus 5548 at Access (ToR)

Cisco Nexus Fabric building blocks..

F: L2, FSS

M: L3, no FSS


Nexus 5548 at Access (ToR)

Nexus 7K in Core

M: L3, no FSS

Line cards in this architecture are either capable of L2 (with FSS) or L3 (without FSS) processing

+ Eliminates STP, allows multi-pathing

+ Scales to many core chassis at L2

- No layer 3 : Does matter whether traffic is within or across VLANs


For Parity: Lossless, 3:1, Oversubscription and Line-rate L3

• Backplane bandwidth of F card is 230G

– Only 23*10 GE ports are line rate per F card

• L3 traffic needs to go via M card which is limited to 80Gbps per card

Closer look at Nexus7k with F & M cards

F (230GE)From N5k


80Gbps per card

F(230G)

M (80GE)

To N5k

Every F Card will need 3xMcards to support line rate L3 traffic

So, a N7K 16 slot chassis will have:• 4 * F cards• 12 * M cards• i.e., 23 * 4 = 92 line-rate ToR

facing ports


QFabric

L2 & L3Non-Blocking

1 125

QFABRIC VS. FABRICPATH – 6000 PORTS

41

TRILL like - “Big Pile” Architecture

QFabric

� 1/3 fewer devices

� 2/3 less power

$300k/year

� 90% less floor space

40 Copyright © 2011 Juniper Networks, Inc. www.juniper.net Note: • OS* Over Subscription 3:1• Ports: 6000 server ports

.. .. .... .. .. ..

L3

L2

105 1671 21 42 63 84 126 147

1 16

1 62 3 4 5 87

space

� 7-10x faster

� 90% fewer links

� Mgd. Devices

1 vs 1931 vs 25 admins


QFABRIC VS. FABRICPATH NUM DEVICES REQUIRED:CAPEX AND COMPLEXITY

100

150

200

250

Servers

$$Juniper QFabric


Servers

$$

0

50

500p 1000p 3000p 6000p

# of Devices JNPR # of Devices CSCO

Servers

Cisco FabricPath


QFABRIC VS. CISCO NEXUS ENVIRONMENTAL FRIENDLINESS

China RoHS

QFabric Environmental Labels Cisco Nexus Environmental Labels

5/6

SR-3580 NEBS Level 3


China RoHS

Recycled Material

SR-3580 NEBS Level 3

• Verizon NEBS compliance

6/6


QFABRIC VS. CISCO NEXUS, ENERGY UTILIZATION

300000

400000

500000

600000

700000

800000


Source for Cisco Power comparisonhttp://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/data_sheet_c78-618603.htmlhttp://www.cisco.com/en/US/docs/switches/datacenter/hw/nexus7000/installation/guide/n7k_sys_specs.html

0

100000

200000

300000

Max Power Nominal Power

Qfabric Cisco 5548, NX7018 L2 Cisco 5548, NX7018 L3


Performance

QFABRIC VS CISCO NEXUS TAKE AWAYS

� 7x lower Latency (Juniper Qfabric <5Us vs. Nexus 35Usec )

� Highest scale (6000 ports )with complete Layer 2 and layer 3

� Invisible IT


� Management Complexity: Minimal. QFabric has operational simplicity of single switch

� High resiliency: no single point of failure

� Operational savings : � 1/20th operators/administrators vs Fabricpath

� Most eco-friendly fabric � 1/3 power/port at 1/10 footprint of FabricPath


AGENDA






Summary5



BROCADE VIRTUAL CLUSTER SWITCH SOLUTION

VDX 6720-24 VDX 6720 -60

No Scale : VCS fabric has only upto 10 switches

Low port utilization: VDX 6720-60 has 60 ports but Ethernet Fabric connections take 12 ports (out of 50) on each switch.


Minimal Virtualization support : 32k total MAC table insufficient for large scale Data center solution.

Lack of basic layer3,QOS features Multicast group supporting is very limited, only 256.

Manageability complexity VCS solution relies on each individual switches managed independently


BROCADE VCS SOLUTION : BRCD REFERENCE ARCHITECTURE

BROCADE CLAIM:

VCS fabric has up to 10 switches

Scale: 600 ports6:1 subscription

ratio in VCS fabric10-switch VCS fabric;

312 usable ports

vLAG

6 links per trunk (24 total)

Core layer: MLX with MCT


Scale: 600 ports

Latency: 600ns *

Up to 36 servers per rack

4 racks per VCS

Servers With 1/10G and DCB Connectivity

10G DCB FCoE/iSCSIStorage

vLAG


QFABRIC VS. BROCADE VDX VCS SCALEBASED ON: BROCADE VDX IN ACCESS, AGGREATION (VCS ) WI TH 450 PORTS

BigIronRx-16

BigIronRx-16

BROCADE REALITY:1. Max 450 ports or only 315 ports server ports

2. Latency >1200ns within switch

Based on: Max # TOR: 7, Aggregation devices: 3, Max MCT MLX: 2


VDX 6720-60

MAX VCS SCALE: 450 Server Ports Only`

VDX 6720-607


QFABRIC VS BROCADE VCSSCALE, SIMPLICITY AND MANAGEABILITY

290

150

200

250

300

350 Manageabilty at 450 ports

4000

5000

6000

7000

Max scale


2 1

40

-

50

100

150

# of Admins # of Links,Interactions

BRCD JNPR

0

1000

2000

3000

JNPR BRCD


QFABRIC VS BROCADE VCS PERFORAMNCE

8

10

12

14

16

18

50 Copyright © 2011 Juniper Networks, Inc. www.juniper.net Based on : BROCADE vdx IN ACCESS,AGGGREATION( vcs ) WITH 450 PORTS

-

2

4

6

Latency (usec)

BRCD JNPR


1

2

3

4

Brocade VCS implementation KNOCKOFSS

Brocade solution does not scale – limited to 600 ports

No Storage Convergence : No FCoE-FC Gateway

No Layer 3 – VCS does not achieve a single-layer DC

POOR PERFORMANCE: HIGH LATENCY ACROSS FABRIC 15Us*


4

5

6

FABRIC 15Us*

Brocade implementation is based on FSPF:non-standard TRILL; it cannot interoperate with other vendors TRILL-based implementation

Brocade VCS is a NOT a single flat “Fabric” technology; similar to Juniper’s Virtual Chassis technology, only 3 years later

*Assuming BigIron as Core switch because of lack of VCS core switch


Performance

QFABRIC VS BROCADE VCS TAKE AWAYS

� Highest scale (6000 ports vs 450 ports )with complete Layer 2 and layer 3

� 3x lower Latency (Juniper Qfabric<5Us vs. BROCADE VCS )


� Invisible IT � Management Complexity: Minimal.

QFabric has operational simplicity of single switch resiliency: no single point of failure

� Operational savings vs Brocade: ½ administrators ,1/7th cables needeusing QFabric vs VCS


AGENDA






Summary5



JNPR VS CSCO DETAIL COMPARISON …FOR MULTIPLE PORT COUNTS

Category #Ports >> 500p 1000p 3000p 6000pPerformance

Latency (usec) CSCO 34 34 34 34JNPR 5 5 5 5

Simplicity# of Devices CSCO 18 32 98 195

JNPR 15 25 67 129# of Admins CSCO 2 2 5 9

JNPR 1 1 1 1# of Cables CSCO 240 482 1,812 3,612

JNPR 44 84 252 500Operational Expenses


Operational Expenses Power & Cooling Cost ($/yr) CSCO $ 61,834 $ 73,409 $ 245,355 $ 489,884

JNPR $ 33,717 $ 39,971 $ 66,237 $ 105,010 Adminstrative Costs ($/yr) CSCO $ 340,000 $ 460,000 $ 1,000,000 $ 1,800,000

JNPR $ 200,000 $ 200,000 $ 200,000 $ 200,000 Maintenance Cost ($/yr) CSCO $ 41,798 $ 73,396 $ 225,288 $ 448,319

JNPRCapital Expenditure

Equipment Costs CSCO $ 2,888,792 $ 3,953,584 $ 12,992,752 $ 25,950,876 JNPR

GreenCO2 Emissions (lbs) CSCO 359,760 427,105 1,427,522 2,850,233

JNPR 196,174 232,559 385,378 610,969

Need PLM Input

Need PLM Input

Documents

Fabric Arch Compet