TF-Storage – FCoE: An Overvie · 2011-06-16 · TF-Storage – FCoE: An Overview Andy Vallely [email protected] June 9th 2011 ... • Preserve SAN architecture for high availability

Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 1

TF-Storage – FCoE: An Overview Andy Vallely

[email protected]

June 9th 2011

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

LAN SAN B SAN A

1 Server 8 * 1GbE 2 * FC

LAN

16 Servers 8 * 1GbE 4 * FC


LAN SAN B SAN A LAN SAN B SAN A


Cable Transceiver Latency (link)

Power (each side) Distance

Connector (Media)

Twinax ~ 0.1µs ~ 0.1W <10m SFP+ CU* copper

MM OM2 MM OM3 ~ 0 1W 82m

300m SFP+ SR MMF,short reach

MM OM2 MM OM3 ~ 0 1W 10m

100m SFP+ USR MMF, ultra short reach

Cat6 Cat6a/7 Cat6a/7

2.5µs 2.5µs 1.5µs

~ 6W*** ~ 6W*** ~ 4W***

55m 100m 30m

RJ45 10GBASE-T copper

100Mb 1Gb 10Gb

UTP Cat 5 UTP Cat 5 MMF, SMF

10Mb

UTP Cat 3

Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s

UTP Cat6a MMF, SMF TwinAx, CX4

Standard

SFF 8431**

IEEE 802.3ae

none

IEEE 802.3an

*** As of 2008; expected to decrease over time * Terminated cable ** Draft 3.0, not final

Twinax 15m X2 CX4 copper IEEE 802.3ak 4W ~ 0.1µs



•  iSCSI No guarantee of No-Packet-Drop No native FC Support No native FC tools or services Heave TCP/IP process – requires TOE

•  Infiniband Skills gap Cost for new network 10Gb les efficient than 10GbE due to encoding

•  Ethernet Pervasive Ethernet Requires additional development (Loss-Less Ethernet)


•  From a Fibre Channel standpoint it’s FC connectivity over a new type of cable called… an Ethernet cloud

•  From an Ethernet standpoints it’s Yet another ULP (Upper Layer Protocol) to be transported, but… a challenging one!

•  And technically…

FCoE is an extension of Fibre Channel onto a Lossless Ethernet fabric


Ethe

rnet

H

eade

r

FCoE

H

eade

r

FC

Hea

der

FC Payload

CR

C

EOF

FCS

Same as a physical FC frame

Control information: version, ordered sets (SOF, EOF)

Normal ethernet frame, ethertype = FCoE

•  10Gbps Ethernet

•  Lossless Ethernet Matches the lossless behavior guaranteed in FC by B2B credits

•  Ethernet jumbo frames Max FC frame payload = 2112 bytes


Ethernet Header

FCoE Header

FC Header

FC Payload

CRC EOF

FCS

12 bytes (MAC addresses) + 4 bytes (802.1Q tag)

16 bytes

24 bytes

Up to 2112 bytes

4 bytes

1 byte (EOF) + 3 bytes (padding)

4 bytes

Total: 2180 bytes


Destination MAC Address

Source MAC Address IEEE 802.1Q Tag

ET = FCoE Ver Reserved

Reserved

Reserved SOF

Encapsulated FC Frame (Including FC-CRC)

EOF Reserved FCS

Reserved


FCoE is Fibre Channel at the host and switch level

Same Operational Model

Same Techniques of Traffic Management

Same Management and Security Models

Easy to Understand

Completely based on the FC model

Same host-to-switch and switch-to-switch behavior of FC

E.g., in order delivery or FSPF load balancing

WWNs, FC-IDs, hard/soft zoning, DNS, RSCN


•  ENode (FCoE Node): a Fibre Channel HBA implemented within an Ethernet NIC aka CNA (Converged Network Adapter)

Enet port

Enet port

FC Node

FCoE FCoE


•  FCF (Fibre Channel Forwarder), the forwarding entity inside an FCoE switch

Eth port

Eth port

Eth port

Eth port

Eth port

Eth port

Eth port

Eth port

Ethernet Bridge

FC port

FC port

FC port

FC port

FCF FCoE

FCoE Switch


PCIe

Ethernet 10G

bE

10GbE

Link

PCIe

Fibre Channel

HB

A

HB

A

Link

Fibre Channel Drivers Ethernet Drivers

Operating System

Fibre Channel Drivers Ethernet Drivers

Operating System

PCIe

Ethernet Fibre C

hannel 10G

bEE

10GbEE

Link

LAN CNA HBA


•  Standard drivers

•  Same management

•  Operating System sees: Dual port 10 Gigabit Ethernet adapter Dual Port 4 Gbps Fibre Channel HBAs


SAN A SAN B 10GE

Backbone

10GE

4/8 Gbps FC

VF_Ports

VN_Ports


Enhanced Ethernet Fabric FCoE

Storage

FCoE

•  Direct attached FCoE storage

•  Unified FCoE network environment

•  Preserve SAN architecture for high availability

SAN A and SAN B

Enhanced Ethernet and FCoE


•  VE_Ports and VF_Ports always use MAC addresses derived from the switch pool

•  VN_Ports may use two types of MAC addresses: SPMA (Server Provided MAC Addresses) FPMA (Fabric Provided MAC Addresses)

•  MAC Addresses are negotiated in FIP

•  Initial deployment will use FPMA only


•  A dedicated MAC address for each FC-ID Assigned by the FC fabrics

Consistent with the FC model OUIs with U/L = 1 (Local addressing), called FC-MAPs Multiple FC-MAPs may be supported (one per FC fabric)

48 bits

24 bits 24 bits

FC-MAP (ex 02-12-34)

FC-ID 7.8.9

MAC address

FC-MAP (ex 02-12-34)

FC-ID 7.8.9


ENode FCoE Switch

VLAN Discovery

FLOGI/FDISC FLOGI/FDISC Accept

FC Command FC Command responses

FIP: FCoE Initialization Protocol

FCOE Protocol

VLAN Discovery

FCF Discovery

Solicitation FCF Discovery Advertisement


•  FCoE frames have: MAC addresses (hop-by-hop) FC addresses (end-to-end)

Ethernet Fabric FC Fabric

FC Domain 7 FC Domain 3

MAC A FCID 7.1.1

FCID 1.1.1 MAC C

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

Ethernet Fabric

FC Domain 1 MAC B

FC Storage

FCoE Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

Dest. = MAC B Srce. = MAC A

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

Dest. = MAC C Srce. = MAC B

FC Fabric


Feature/ Standard Benefit

Lossless Ethernet IEEE 802.1Qbb (PFC)

Enable multiple traffic types to share a common Ethernet link without interfering with each other

Class of Service Based Bandwidth Management IEEE 802.1Qaz

Enable consistent management of Quality of Service at the network level by providing consistent scheduling

Congestion Management IEEE 802.1Qau

End-to-end congestion management for L2 network

Data Center Bridging Exchange, DCBX Management protocol for Enhanced Ethernet capabilities

L2 Multipath for Unicast and Multicast

Increase bandwidth, multiple active paths. No spanning tree


•  PAUSE functionality per Ethernet priority IEEE 802.1Q defines 8 priorities

Traffic classes are mapped to different priorities: no traffic interference

Storage traffic may be paused while IP traffic is being forwarded

Or, vice versa Requires independent resources per priority (buffers)

•  High level of industry support Cisco distributed proposal Standard Track in IEEE 802.1Qbb

Ethertype = IEEE 802.1Q Priority CFI VLAN ID

IEEE 802.1Q tag

16 3 1 12 bits


Priority based Flow Control

•  Enables lossless behavior for each class of service •  PAUSE sent per priority when buffers limit exceeded

Transmit Queues Ethernet Link

Receive Buffers

Zero Zero

One One

Two Two

Five Five

Four Four

Six Six

Seven Seven

Three Three STOP PAUSE

Priority based Bandwidth Management

•  Enables intelligent sharing of bandwidth between traffic classes control of bandwidth •  802.1Qaz Enhanced Transmission

Offered Load

t1 t2 t3

Realized Load

3G/s HPC Traffic 3G/s

2G/s

3G/s Storage Traffic 3G/s

3G/s

LAN Traffic 4G/s

5G/s 3G/s

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

20%

30%

50%


•  IEEE 802.1Qau provides end-to-end congestion management for L2 Ethernet networks


•  Port types VF_port / VN_port VE_port F_port trunking / channeling

•  Device types Fibre Channel Forwarder (FCF) FIP snooping bridge

•  Switching modes FC switching NPV (NPIV gateway) FIP snooping


•  FCoE frames have: MAC addresses (hop-by-hop) FC addresses (end-to-end)

Ethernet link FC Fabric

FC Domain 7 FC Domain 3 MAC A FCID 7.1.1 FCID 1.1.1

MAC C

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

Ethernet link

FC Storage

FCoE Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)


D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)


FC link

FC Domain 1 MAC B

VE_port VE_port VF_port VN_port




MAC C

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

Ethernet link

FC Storage

FCoE Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)


D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)


FC link

No domain MAC B

VF_port VNP_port VF_port VN_port




MAC C

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

Ethernet link

FC Storage

FCoE Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

Dest. = MAC C Srce. = MAC A

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

Dest. = MAC C Srce. = MAC A

FC link

No domain MAC B

VF_port VN_port


•  Intermediate switches in the ethernet cloud All are FIP snooping bridges

Ethernet Fabric FC Fabric

FC Domain 7 FC Domain 3 MAC A

FCID 7.1.1

FCID 1.1.1 MAC C

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

Ethernet Fabric

FC Domain 1 MAC B

FC Storage

FCoE Frame

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)


D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)


FC link

VE_port VE_port VF_port VN_port


•  All links are forwarding

•  IS-IS computes the forwarding tables

L2MP Backbone


10 GbE FCoE

Consolidated I/O Servers to Access Layer

Data Center Ethernet / FCoE Links

Cisco Nexus 5000

Distinct Management and Control Planes Single Data Plane

10 GbE

Fibre Channel

Virtual Port-channels Preserves Existing Dual SAN Environments

MDS

SAN B SAN A vPC

LAN

Nexus


Consolidated I/O Servers to Access Layer

Data Centre Ethernet / FCoE Links

Cisco Nexus 5000

Distinct Management and Control Planes

Single Data Plan

MDS

SAN B SAN A

Preserves Existing Dual SAN Environments

10 GbE FCoE

10 GbE

Fibre Channel

VPC

Virtual Port-channels

LAN

Nexus

Active/Active

VPC

Thank you.


L2MP and FCoE


•  All links are forwarding

•  IS-IS computes the forwarding tables

L2MP Backbone

This will be represented in the following slides as a cloud for simplicity


SAN A SAN B

L2MP Backbone

SAN-A

SAN-B


•  IS-IS provides L2MP for non-FCoE •  Only access switches may be an exit point toward FC

Exit points use 4/8 GFC

•  Access switches may be FCFs Domain Number = Number of access switches

Eight in the example (four per each fabric) FSPF provides L2MP for FCoE

•  In alternative, access switches may be NPIV proxy No additional domains are required

•  Easy to implement, but not really a unified fabric •  SAN-A and SAN-B have no common Data Plane or Control Plane •  SAN Admin and Network Admin roles are clearly separated •  An FCoE host may need to cross the legacy Ethernet Cloud to reach an FCoE Storage

Array


SAN A SAN B

L2MP Backbone

These links are now 10GE FCoE


•  IS-IS provides L2MP for non-FCoE

•  Only access switches may be an exit point toward FC Exit points use 10 GE FCoE

•  Access switches may be FCFs Domain Number = Number of access switches

Eight in the example (four per each fabric) FSPF provides L2MP for FCoE

•  In alternative, Access switches may be NPIV proxy No additional domains are required

•  SAN-A and SAN-B have no common Data Plane or Control Plane

•  SAN Admin and Network Admin roles are clearly separated

•  An FCoE host may reach a native FCoE Storage Array without going through legacy FC

Differences with #1 in orange


SAN A SAN B

All the L2MP switches are also FCFs



•  Any switch may be an exit point toward FC They may use 4/8 GFC or 10 GE FCoE

•  All switches are FCFs Domain Number = Number of access and core switches

Twelve in the example (six per each fabric) FSPF provides L2MP for FCoE




Differences with #1a in orange


•  Let’s focus on an access switch of SAN-B

These 4 links can be used for L2MP of non-FCoE traffic

These 2 links can be used for L2MP of FCoE traffic

These 2 links are more prone to congestion


•  Share the switches, dedicate the links

These 4 links can be used for L2MP of non-FCoE traffic

No link is more prone to congestion

These 2 red links are 10GE links dedicated to L2MP of FCoE traffic

This also provides better traffic isolation than VSANs/VLANs: different ports, different protocols!


SAN A SAN B

A mix of FCFs and NPIV Proxy

FCFs

NPIV Proxies



•  Core switches may be an exit point toward FC They may use 4/8 GFC or 10 GE FCoE

•  Core switches are FCFs and Access switches are NPIV proxy One additional FC domain per core switch is required

4 in the example (2 per each fabric)




Differences with #2 in orange

Thank you.

Documents

TF-Storage – FCoE: An Overvie · 2011-06-16 · TF-Storage – FCoE: An Overview Andy Vallely [email protected] June 9th 2011 ... • Preserve SAN architecture for high availability