Design and Implementations of FCoE for the DC - Cisco

8/13/2019 Design and Implementations of FCoE for the DC - Cisco

1/31

Design and Implementations ofFCoE for the DataCenter

Mike Frase, Cisco Systems


2/31

Design and Implementatio ns of FCoE for the DataCenter 2011 Storage Networking Industry Association. All Rights Reserved. 22

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA unless otherwisenoted.Member companies and individual members may use this material in presentationsand literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modificationThe SNIA must be acknowledged as the source of any material used in the body of anydocument containing material from these presentations.

This presentation is a project of the SNIA Education Committee.Neither the author nor the presenter is an attorney and nothing in thispresentation is intended to be, or should be construed as legal advice or an opinionof counsel. If you need legal advice or a legal opinion please contact your attorney.The information presented herein represents the author's personal opinion andcurrent understanding of the relevant issues involved. The author, the presenter,and the SNIA do not assume any responsibility or liability for damages arising out ofany reliance on or use of this information.NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.


3/31


Abstract

Design and Implementations of FCoE for the DataCenter

This session will discuss the technologies and solutions for implementing FCoE inan end to end design for the Unified Data Center. The terms and technicalrequirements of the 10 GigE Network along with Multi-hop FCoE and Unified I/OCompute will be explained, allowing anyone new or old to the technology the

ability to engage in the discussion. We will focus on what the Data CenterEngineer can deploy in today's designs , intergrading FCoE with current StorageArea Networks (Native Fibre Channel) or complete new FCoEdeployment. Attendees shall learn what new skills are required for theimplementation and management of an FCoE virtualized network.

Learning Objectives:

- FCoE technologies- Skills requirements for Unified I/O Networks- 10 GigE designs for FCoE


4/31

Design and Implementatio ns of FCoE for the DataCenter 2011 Storage Networking Industry Association. All Rights Reserved.

Agenda

Network, Storage, and Server Admin -Positioning your skillsFCoE TechnologyFCoE Deployment Designs TodayFCoE Deployments Designs - Tomorrow

4


5/31


Data Center Engineer Evolution

Network Data Engineer 1, 10 GigESpanning-Tree ProtocolsHighly Available NetworksLow Latency Switching

Layer 2 & 3Network adaptors and driversIP addressing IPV4 & IPV6IP Routing ProtocolsStandard Network services ARP,DHCP, PXE, LLDPQOSSecurity

5

SAN Engineer 2, 4, 8 Gig FCFibre ChannelFSPFHighly Available Fabrics

Low Latency SwitchingFabric Login ServicesFCIDDomain ServicesZone ServiceStorageHBAs and drivers


6/31


DC Engineer

Data Center Engineer LAN EngineerSAN EngineerDCB (Lossless) GigE

CNA adaptors & drivers

Add these extras for Advanced DC Engineer VirtualizationServer Hardware (Memory, CPU, PCIe)Data Center Interconnect Technologies

6

Technologies for LAN & SANiSCSIFCoE


7/31


SAN BSAN ALAN

CNA

CNA

CNA

CNA

Ethernet

Layer 2 & 3protocols

VLANS Fibre Channel

Protocol andFC services

Zoning

SCSI data

Virtual Fabrics

Cabling and Optics

10 GigE

HBA

FCoE

DCB

PerformanceMonitoring

Learning CurveUnified Network Technologies

CNA

O/S & Driversupport

7


8/31


Unified I/O: Network Operations View

LAN

SAN A

SAN B

1 GbE10 GbEFC10GbE FCOE

Network Ops Work Process: Same as today

Configure Ethernet port, VLAN, STP,Port Channels, Trunks etc.

Monitor LAN specific MIBs, Syslog,RMON.

Mgmt :CLI (Role based)XML API

Network Mgmt System:Network/Device Management

SAN Ops Work Process: Same as today

Configure FC Interface mode, speed,Virtual Fabrics, Port Channels, Trunks,NPV etc.

Monitor SAN specific MIBs, Syslog,RMON.

Mgmt :CLI (Role based)XML API

Network Mgmt System:Existing tool SAN Fabric Managers

Server Ops Work Processes: Same as today

Converged Network Adapter (CNA)config equivalent to NIC and HBA.

Mgmt: No impact Network Mgmt System:

Existing tools

Converged Activities Work Processes:

Switch rebootImage upgradeUser rolesInitial creation of virtual interfaces

Mgmt: Creation of Mgmt Interface Day-2 System level support: Memory,CPU error, Call Home features, Span ports.


9/31


FCOE TECHNOLOGY

9


10/31


FCoE Behind the Scenes

Data Center Bridging eXchange (DCBX):DCBX , is a management protocol that extends the Link Layer DiscoveryProtocol (LLDP) defined in IEEE802.1AB. DCBX allows the FibreChannel Forwarder (FCF) to provide Link Layer configurationinformation to the Converged Network Adaptor (CNA) and allows boththe CNA and FCF to exchange status.

FCoE Protocols :FCoE Eternet Frame Type 8906 Hex

FCoE data protocol requires lossless Ethernet and is typicallyimplemented in hardware, and is used to carry most of the FC framesand all the SCSI traffic.

FIP (FCoE Init ialization Protocol) Ethernet Frame Type 8914 Hex A Discovery Protocol whose purpose is to present a multi-accessEthernet network as a set of point-to-point Virtual Links for FCoE to use.

10


11/31


DataCenter Bridging (DCB) for FCoE

Part of the IEEE 802.1 Work GroupLossless Ethernet 802.1Qbb

Priority Grouping 802.1Qaz

Configuration Verification 802.1Qaz

Auto-negotiation of capability andconfiguration

Allows network devices to adver tise theircapabili ties over a link

Enables hosts to get proper configuration from network

Enables switches to verify proper configuration

11


12/31


LLDP & DCB inspection

Organizational TLVs

Basic values of the packet

DCB TLV breakdownsConfiguration Details

FCoE

PFCETSLink up-down


13/31


FIP Protocols Defined

FIP VLAN Discovery - used by the ENode to discover FCOE VLANFIP Discovery Solicitations - used by the ENode to discover a FCFFIP Discovery Advertisement - used by FCF to advertise itself to ENodesFIP FLOGI, FIP LS_ACC and FIP LS_RJT - used by FCF to login and getitself an FCID

FIP LOGO - sent by ENode to logoutFIP NPIV FDISC - sent by VN ports to loginFIP KeepAlives - liveliness mechanism between ENodes and FCFFIP Clear Virtual Link - used by the FCF to signal to the ENode to bring downthe virtual link

13


14/31


FIP: FCoE Initialization Protocol

FCoE VLAN discovery

Automatic discovery of FCoE VLANs

Device discovery

ENodes discover VF_Port capable FCF-MACs for VN_Port to VF_Port Virtual Links

VE_Port capable FCF-MACs discover otherVE_Port capable FCF-MACs for VE_Port toVE_Port Virtual Links

Verifies the Ethernet network supports therequired Max FCoE Size along with manyother settings like FC map value usingDescriptor TLVs

Virtual Link instantiation

Builds on the existing Fibre Channel Login process

Fabric Provided MAC Address (FPMA),

Virtual Links maintenance

Timer based 14


15/31


FCoE, Same Model as FC

A-B SAN separat ion pictureshowing one sideSame host to targetcommunication

Host has 2 CNAs (one per fabric)Target has multiple ports to connect

to fabricConnect to a DCB capable switch

Port Type Negotiation (FC port typewill be handled by FIP)Speed NegotiationDCB Negotiation

Access switch is a Fibre ChannelForwarder (FCF)Dual fabrics are still deployed forredundancy

CNA

ENode

FC FabricTarget

Ethernet Fabric

DCB capable Switchacting as an FCF

Unified Wire

15


16/31


FCOE NETWORKDEPLOYMENTS

16


17/31


18/31


Fibre ChannelDrivers

EthernetDrivers

Operating System

PCIe

E t h er n

e t

F i b r e

C h ann

el

1 0 G b E

1 0 G b E

Link

Ethernet Driverbound toEthernet NIC PCIaddress

FC Driverbound to FCHBA PCIaddress

Unified Wireshared by bothFC and ethernetULPs

Unified Edgesupports both FCand ethernetUPLs

Edge participates in both distinctFC and IP Core topologies

FCF-ASwitchor NPV

FCF-BSwitch

or NPV

Deployments with FCoE

muti-switchport channel

Converged Network Adapter (CNA)presents two PCI address to theOperating System (OS)OS loads two unique sets of driversand manages two uniqueapplication topologiesServer participates in bothtopologies separatelyTwo stacks and thus two differentviews of the same unified wire

SAN Multi-Pathing provides failover

between two fabrics (SAN A andSAN B)NIC Teaming provides failoverwithin the same fabric (VLAN)

18


19/31


Fabric A

DCB

DCB

Generation 2 CNA

Fabric BLAN Fabric

VN

VF Direct attachVN_Port toVF_Port

What is being deployed Single Hop Design

FCF-A FCF-A

Generation 2 CNA

Utilizes Converged EnhancedEthernet & Data Center BridgingExchange protocol (DCB)

Utilizes FCoE Initialization Protocol(FIP) as defined by the T.11 FC-BB-5specification

Supports both direct and multi-hop

attachment (i.e. through a FIPSnooping Bridge or over VE-Ports)

19



20/31


SAN Fabric A

Direct Attach Topologies

SAN Fabric BLAN Fabric

VN

VF Direct attachVN_Port toVF_Port

What is being deployed Initiator connection

FCF-AFCF-B

Physical link is brought online (today

requires 10GE)

DCBX negotiation discovers DCBcapable devices, including but not limitedto lossless Ethernet.

FIP Process discovery and negotiationof FCoE devices and characteristics

FCoE VLAN DiscoveryFCF Discovery on the specific FCoE VLANFabric Login - builds the logical wire fromthe end node to the FCF (VN_port toVF_port)

20



21/31


VLAN 10,30

VLAN 10,20 STP Edge Trunk

SAN Fabric A SAN Fabric BLAN Fabric

FCF-A FCF-B

Single Hop Design FCoE VLAN

A VLAN can be dedicated for everyVirtual Fabric in the SANFIP discovers the FCoE VLAN andsignals it to the hostsTrunking is not required on thehost driver all FCoE frames aretagged by the CNAFCoE VLANs can be pruned fromEthernet links that are notdesignate for FCoEMaintains isolated edge switches

for SAN A and B and separateLAN switches for NIC 1 and NIC 2(standard NIC teaming or Link

Aggregation)

21

VirtualFabric 3

VirtualFabric 2


22/31


VLAN 10,30

VLAN 10,20

VirtualFabric 2

STP Edge Trunk


FCF FCF

Single Hop Design FCoE VLAN Separation

In order to maintain the integrity ofFC forwarding over FCoE, FCoEVLANs should be treated differentlythan LAN VLANs

No flooding, MAC learning,broadcasts, etc.

The FCoE VLAN should beconfigured as a native VLAN

FIP uses native VLAN

Separate FCoE VLANs are used forFCoE in SAN-A and SAN-B

Unified Wires can be configured astrunk ports and STP edge ports

22

VirtualFabric 3

Si l H D i


23/31


Optimal layer 2 LAN design oftenleverages muti-switch port channel

A muti-switch port channel can providenetwork based load sharing andredundancy and some implementationscan do this without introducing layer 2loops in the topology

Muti-switch port channel results indiverging LAN and SAN high availability

topologiesFC maintains separate SAN A and SAN

B topologiesLAN utilizes a single logical topology Direct Host Dual Attached Topology


FCF-A FCF-B

Single Hop DesignUnified Wires and multi-switch port channels

23

muti-switch

port channel

Si l H D i


24/31


TOR Device

Fabric LinksOption 1: SingleHomed PortChannel

Fabric LinksOption 2: StaticPinned

FCoE onindividual links.Ethernet traffic is

Active/Standby

FCoE on a LACPmember PC with asingle link

FIP enabled CNAs (Gen 2)

Single Hop DesignExtending the FCoE Edge to Top of Rack

FCF-A FCF-B

TOR device

Server Ethernet driver connectedto TOR device with NIC Teaming

FCoE can connect over multimember port with a single linkfrom server to TOR

TOR device single homed toupstream core FCF

Top of Rack fabric links can beconnected to FCF with individual

links or a muti-switch port channelCan be oversubscribed

Consistent with separate LAN Access and SAN Edge Topologies

24

SAN Fabric A SAN Fabric B


25/31


SAN BSAN A

Single HomedUnified Wire

MezzanineConverged Network

Adapter

UnifiedDedicated Wire

PCIe

E t h er n

e t

F i b r e

C h

ann

el

1 0 G b E

1 0 G b E

Link

Extending FCoE with FIP Snooping

FCF-A FCF-B

FIP Snooping

Bridge-BFIP SnoopingBridge-A

FIP snooping BridgeDCB enabled

Dual Topology Capable

Still need to maintain the A-BSAN separation

Unified Dedicated Wires from FIP

bridge to FCFSingle Unified Wire Port Channelfrom FIP Bridge to FCF

25


26/31


FCF

Fabric AFabric B

LAN Fabric

Multi - Hop Designs (VE)Considerations for FCoE Multi-hop

FCF

VE Ports

FCoEStorage

Virtual E_Ports (VE)

Design considerations whenextending FCoE beyond the Unif iedEdge?

High Availability for both LAN and SANOversubscription for SAN and LANEthernet Layer 2 and STP design

Where does Unified Wire makesense over Unif ied Dedicated Wire?

Unified Wire provides for sharing ofa single link for both FC andEthernet traffic

26


27/31


UNIFIED I/O IN COMPUTEBLADE CENTER

27

FC E i h C Bl d C


28/31


B l a d e

C h a s s

i s

10GE/FCoE

LANHPC

FCoE in the Compute Blade Center Fabric Consolidation

SAN

B l a d e

B l a d e

B l a d e

B l a d e

28


29/31


Same benefits asnon Blade Center designs

Fewer Cables

Fewer switches

Fewer adapters

Overall less power

Interoperates with existing SANs

FCoE in the Compute Blade Center

UnifiedFCoE

FCoE SolutionMgmt Server Mgmt Server

Fibre Channel

Ethernet

Past Solution Ethernet

FC traffic

29


30/31


Check Out the Hands-On Lab

Check out the Hands-On LAB at SNW:

Unified Storage InfrastructureUnified Storage IP Solutions


31/31

Design and Implementatio ns of FCoE for the DataCenter 2011 St g N t ki g I d t A i ti All Right R d 3131

Q&A / Feedback

Please send any questions or comments on thispresentation to SNIA: [email protected]

Many thanks to the following individualsfor their contributions to this tutorial.

- SNIA Education Committee

Simon Gordon

Joseph White
mailto:[email protected]:[email protected]

Documents

Design and Implementations of FCoE for the DC - Cisco