virtual bridge

BRKDCT-2340

Virtual Bridge Port Extension

Follow us on Twitter for real time updates of the event:

@ciscoliveeurope, #CLEUR

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKDCT-2340 2

Housekeeping

We value your feedback- don't forget to complete your online session evaluations after each session & the Overall Conference Evaluation which will be available online from Thursday

Visit the World of Solutions and Meet the Engineer

Visit the Cisco Store to purchase your recommended readings

Please switch off your mobile phones

After the event don’t forget to visit Cisco Live Virtual: www.ciscolivevirtual.com

Follow us on Twitter for real time updates of the event: @ciscoliveeurope, #CLEUR

http://www.ciscolivevirtual.com/

http://www.ciscolivevirtual.com/


Preface

• The content of this presentation talks about pre-standards, which are not finalized yet !

• Some of the content may be subject of changes till IEEE-802.1 ratifies the standards, however only minor changes expected !

• Cisco is a key contributor to IEEE-802.1 DCB committee in driving a new Datacenter Bridging Standard


[B1] IEEE Std 802.1DTM, IEEE Standard for Local and Metropolitan Area Networks: Media Access Control (MAC) Bridges.

[B2] IEEE Std 802.3TM, IEEE Standard for Information technology—Telecommunications and information exchange between systems—Local and metropolitan area networks—Specific requirements—Part 3: Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications.

[B3] IEEE Std 802.1ACTM, Standard for Local and Metropolitan Area Networks—Media Access Control (MAC) Service.

Bibliography


Agenda

Introduction

Standard Body’s

Edge Virtual Bridging Concepts

Anatomy of IEEE-802.1BR

Cisco FEX Implementation

IEEE-802.1BR and IEEE-802.1Qbg

VNTAG to IEEE-802.1BR Migration

Summary Conclusion

Q&A


IEEE 802.1BR Port Extender

A Port Extender provides the capability to extend bridge Ports to multiple physical servers, to server blades within a blade rack, or to enable logical connection of virtual machines within a server to independent bridge Ports

The Port Extender Control and Status Protocol (PE CSP) is used between a Controlling Bridge and Port Extenders that provides the ability of the Controlling Bridge to assert control over and retrieve status information from its associated Port Extenders.


IEEE 802.1BR Port Extender

A controlling bridge and its port extenders constitute a single entity called an Extended Bridge

An Extended Bridge is a standard 802.1Q bridge

A Port Extender can connect to an OS, VMs, a VEB, a VEPA, a NIC, a bridge


Edge Virtual Bridging (EVB) is an IEEE standard !

It’s a common terminology that involves the interaction between virtual switching environments in a hypervisor and the first layer of the physical switching infrastructure.

IEEE-802.1Qbh Par is dead !

IEEE-802.1Qbh is alive on best way to become to be a standard, has been renamed to IEEE-802.1BR !

VEPA Standard ...

VEPA is a HP proprietary bridging implementation

IEEE-802.1Qbg Standard ?

Qbg like BR, are in Sponor Ballot Phase, very close to be finished and published !

The EVB enhancements are following 2 different paths: 802.1Qbg and 802.1BR.The two proposals are parallel efforts, meaning that both will become standards and both are "optional" for any product being IEEE compliant. The standards are likely to be finished over the next 2-3 Month’s, be published later this Year.

SOME MYTHS …


HSRP NetFlow CDP Tag

Switching Cisco

FabricPath

RSRB

‘92

ISL

‘95

MISTP

‘97 ‘08

Power Over

Ethernet

‘99

VSANs

‘90 ‘94 ‘96 ‘98 ‘09 ‘02 ‘00 ‘01 ‘03 ‘04 ‘05 ‘06 ‘07

‘99

‘95

‘04

‘99

‘09

‘04

‘05

‘00 ‘01 ‘03

‘10

‘09

‘10

VRRP

DLSw

IPFix

802.1q

LLDP

802.1s

MPLS 802.3af

802.3at (PoE

Enhancements)

ANSI T11

802.1BR

ANSI T11

802.1Qbb

VN-Link

FCOE

Lossless 10GbE

IETF TRILL

Priority Flow

control

IEEE 802.1qbb

QCN

IEEE 802.1Qau

FC security

FC-SP at T11 iSCSI

RFC 3270

Cisco Innovations

Resulting Standards

Cisco will support 802.1Qbg and 802.1BR standards

Cisco’s commitment to Standards


T11 Group: Fibre Channel Standards

Who is Who

Part of INCITS

Has defined FC technologies for over 10 years; FCIA markets them

Focuses on all things FC:

- Physical Layer

- Switching

- Framing

- Security

Why it’s important to FCoE:

- Standardized method of transporting Fibre Channel frames over Ethernet

- Standardized method for multi-hop FCoE

10

Fibre Channel


IEEE 802.1 Working Group: LAN Bridging Standards

Who is Who

Part of IEEE 802 (LAN and MAN committee)

Defines LAN Bridging technologies

E.g., all about Ethernet switching

The Data Center Bridging (DCB) Task Group is inside IEEE 802.1

- DCB developed bridging extensions relevant for the Data Center environment

Why it’s important to FCoE:

- Those bridging extensions enable I/O consolidation with FCoE

11

Ethernet

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKDCT-2340 12 12

I/O Consolidation with FCoE

Standards for Unified I/O

FCoE is fully defined in FC-BB-5 standard

FCoE works alongside additional technologies to make I/O Consolidation a reality

T11 IEEE 802.1

FCoE

FC on

other

network

media

FC on Other

Network

Media

FC-BB-5

PFC ETS DCBX

802.1Qbb

DCB

802.1Qaz 802.1Qaz

Lossless

Ethernet

Priority

Grouping

Configuration

Verification


Virtual Bridge Port Extension

Standards for Unified I/O and VBE

PE is fully defined in IEEE-802.1BR standard

PE works alongside additional technologies and is fully 802.1Q standard Bridging compliant to make virtual and cascaded connectivity extensions reality

T11 IEEE 802.1

VBE

FC on

other

network

media

FC on Other

Network

Media

FC-BB-5

PFC ETS DCBX

802.1Qbb

DCB

802.1Qaz 802.1Qaz

Lossless

Ethernet

Priority

Grouping

Configuration

Verification

802.1Qbg 802.1BR

PE EVB

Port-

Extender Edge Virtual

Bridge


IEEE P802.1BR



VEB (Virtual Embedded Bridge)

VEPA (Edge Virtual Bridging) IEEE-802.1Qbg

VBE (Virtual Bridge Port Extension) IEEE-802.1BR

Relevant IEEE Datacenter Standards:

802.1Qau Congestion Notification

802.1Qaz Enhanced Transmission Selection

802.1Qbb Priority based Flow Control

802.1Qbg Edge Virtual Bridging

802.1BR Virtual Bridge Port Extension

802.1aq Shortest Path Bridging

IEEE Bridge Port Extender = Cisco FEX (Fabric Extender)

Normative & Terminologies


IEEE Std 802.1ABTM, IEEE Standard for Local and metropolitan area networks—Station and Media Access Control—Connectivity Discovery.

IEEE Std 802.1QTM-2011, Standards for Local and Metropolitan Area Networks: Virtual Bridged Local Area Networks as modified by the following:

A IEEE Std 802.1QazTM-2011, IEEE Standard for Local and metropolitan area networks—Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks—Amendment 18: Enhanced Transmission Selection for Bandwidth Sharing Between Traffic Classes;

B IEEE Std 802.1QbbTM-2011, IEEE Standard for Local and metropolitan area networks—Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks—Amendment 17: Priority-based Flow Control;

C IEEE Std 802.1QbcTM-2011, IEEE Standard for Local and metropolitan area networks—Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks—Amendment 16: Provider Bridging—Remote Customer Service Interfaces;

D IEEE Std 802.1QbcTM-2011, IEEE Standard for Local and metropolitan area networks—Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks—Amendment 15: Multiple I-SID Registration Protocol; and

E IEEE Std 802.1QbgTM-20XX, Standards for Local and Metropolitan Area Networks—Admendment XX: Edge Virtual Bridging.

IEEE Std 802.3.1TM-2011, Standard for Management Information Base (MIB) definitions for Ethernet.

IETF RFC 1042, A Standard for the Transmission of IP Datagrams over IEEE 802 Networks, Postel, J., and Reynolds, J, February 1988.

IETF RFC 1390, STD 36, Transmission of IP and ARP over FDDI Networks, Katz, D., January 1993.

IETF RFC 2578, STD 58, Structure of Management Information Version 2 (SMIv2), McCloghrie, K., Perkins, D., Schoenwaelder, J., Case, J., Rose, M. and S. Waldbusser, April 1999.

ISO/IEC TR 11802-5:1997, Information technology — Telecommunications and information exchange between systems -- Local and metropolitan area networks — Technical reports and guidelines — Part 5: Media Access Control (MAC) Bridging of Ethernet V2.0 in Local Area Networks


When did you say they would have our resumes printed ?

Different ways in building a Bridge ….


YES, Networking in Today’s DC’s is Key

Many Bridges !!


Networking is powerful …


Server Connectivity Evolution

Servers directly connected to access layer switches

Very little virtualization

Network configuration and policy enforcement for the server done at the switch

All management primarily at the physical element level

Management of Physical ( ) Elements



Shift towards server virtualization

Multiple VMs inside each physical server, connected by virtual switches

Rapid proliferation of logical elements that need to be managed

Feature parity issues between virtual and physical elements

Separate management of physical ( ) and logical ( ) elements

VMs

vNICs

VSwitch

VMs

vNICs

VSwitch

VMs

vNICs

VSwitch

VMs

vNICs

VSwitch

Management Challenges Policy Enforcement Issues



Switch lacks visibility into packets originated by vNICs

Can’t tie packet back to VM, forcing reliance on the software switch for policy enforcement

Leads to policy enforcement and network management issues

Access layer switch lacks visibility into virtual network elements

VMs

vNICs

VSwitch

VMs

vNICs

VSwitch

VMs

vNICs

VSwitch

VMs

vNICs

VSwitch

Management Challenges Policy Enforcement Issues



Virtual Interfaces within VMs are now visible to the switch

Both network configuration and policy enforcement for these interfaces can now be driven from the switch

This allows consolidated management of physical and virtual elements

Consolidated management of physical ( ) and logical elements

VSwitch VSwitch

FEX-Link: Consolidated Management

VMs

vNICs

VSwitch

VMs

vNICs

VSwitch

VMs

vNICs

VMs

vNICs



FEX-Link allows the packets to be tagged

Switch has full visibility into which vNIC originated the packet

Allows switch to forward packets between both physical and virtual elements

FEX-Link capable adapters allow bypassing software based switches

Full visibility into the virtual network elements from switch

VSwitch VSwitch

FEX-Link: Consolidated Policy Enforcement

VMs

vNICs

VSwitch

VMs

vNICs

VSwitch

VMs

vNICs

VMs

vNICs


Traditional Networking The end-station and bridge


Exploiting Switch Adjacency



Modern Networking The end-station and bridge


Identifies and isolates traffic between ports within an Extended Bridge

Specifies a tag format for this identification

Establishes an Extended Bridge consisting of a Controlling Bridge and one or more Bridge Port Extenders

Specifies the functionality and the specific requirements of a Bridge Port Extender

Extends the MAC service of a Bridge Port across the interconnected Bridge Port Extenders, including support of Customer Virtual Local Area Networks (C-VLANs)

Establishes the requirements of bridge components and systems for the attachment of Bridge Port Extenders

Specifies a protocol to provide for the configuration and monitoring of Bridge Port Extenders by a Controlling Bridge

Establishes the requirements for Bridge Management to support Port Extension, identifying the managed objects and defining the management operations.


The purpose of this standard is to extend a bridge, and the management of its objects, beyond its physical enclosure using 802 LAN technologies and interoperable interfaces.

Micro & Macro

Cosmos


Aggregating Port Extender: A Bridge Port Extender that supports the full E-CID space and is capable of aggregating base Port Extenders.

Base Port Extender: A Bridge Port Extender that supports a subset of the E-CID space.

Cascade Port: A Port of a Controlling Bridge or Bridge Port Extender which connects to an Upstream Port. In the case of the connection between two Bridge Port Extenders, the Cascade Port is the Port closest to the Controlling Bridge.

Controlling Bridge: A Bridge that supports one or more Bridge Port Extenders.

Extended Bridge: A Controlling Bridge and at least one Bridge Port Extender under the Controlling Bridge's control.

Extended Port: A Port of a Bridge Port Extender that is not operating as a Cascade Port or Upstream Port. This includes the Ports of a Bridge Port Extender connected via internal LANs to the Port of a C-VLAN component within a Controlling Bridge


E-channel: An instance of the MAC service supported by a set of two E-paths forming a bidirectional service. An E-channel is point-to-point or point-to-multipoint.

E-path: A configured unidirectional connectivity path between an internal Extended Port and one or more external Extended Ports and/or Upstream Ports. E-paths initiating from the Internal Bridge Port Extender can be point-to-point or point-to-multipoint. E-paths can be point-to-point or multipoint-to-point.

E-channel Identifier (E-CID): A value conveyed in a E-TAG that identifies an E-channel.

E-TAG: A tag header with a Tag Protocol Identification value allocated for “802.1BR E-Tag Type.”

External Extended Port: An Extended Port that is part of an External Bridge Port Extender. External Bridge Port Extender: A Bridge Port Extender that is not physically part of a Controlling Bridge but is controlled by the Controlling Bridge.

Internal Extended Port: An Extended Port that is part of an Internal Bridge Port Extender.


Internal Bridge Port Extender: A Bridge Port Extender that is physically part of a Controlling Bridge.

Bridge Port Extender: A device used to extend the MAC service of a C-VLAN component to form a Controlling Bridge and to extend the MAC service of a Controlling Bridge to form an Extended Bridge.

Port Extender Control and Status Agent: The entity within a Bridge Port Extender that implements the Port Extender Control and Status Protocol.

Port Extender Control and Status Protocol (PE CSP): A protocol used between a Controlling Bridge and Bridge Port Extenders that provides the ability of the Controlling Bridge to assert control over and retrieve status information from its associated Bridge Port Extenders.

Replication Group: Within a Controlling Bridge, the set of C-VLAN component Ports connected to a single Bridge Port Extender.

Upstream Port: A Port on a Bridge Port Extender that connects to a Cascade Port. In the case of the connection between two Bridge Port Extenders, the Upstream Port is the Port furthest from the Controlling Bridge.


E-CID E-Channel Identifier

PCID Port E-CID

PE CSP Port Extender Control and Status Protocol

PEISS Port Extender Internal Sublayer Service


A simple two-port Bridge that is capable of acting as a Controlling Bridge


Attachment of a physical Bridge Port Extender to the top port of the two-port Bridge.

At this point, the Bridge and the Bridge Port Extender execute LLDP.

The Bridge learns that a Bridge Port Extender is directly attached

when it receives the Port Extension TLV from the Bridge Port Extender.


Upon detection of the directly attached Bridge Port Extender, the Controlling Bridge

instantiates an Internal Bridge Port Extender between the C-VLAN component and

the External Bridge Port Extender. An E-channel is established for communication

between the Bridge Port Extender and the C-VLAN component. The E-channel used

for communication between the C-VLAN component and the Bridge Port Extender is

identified as E-channel “a” in this example.


Next both the C-VLAN component and the Bridge Port Extender initiate

communication with each other using the Bridge Port Extender Control and Status

Protocol (PE CSP). This is accomplished using the CSP Open message.

Note that prior to completion of the CSP Open message, the Bridge Port Extender

does not know the E-CID of the E-channel to be used for this communication.

It therefore uses a default E-CID of one. Since the E-channel is not tagged, the

communication is established even though the Controlling Bridge and the

Bridge Port Extender are using a different E-CID. After completion of the CSP Open,

the Controlling Bridge informs the Bridge Port Extender of the proper E-CID,

which is “a” in this example, using the E-channel Register message.


The Extended Ports have not been instantiated.

Extended Ports are not necessarily instantiated at the same time the

Bridge Port Extender itself is instantiated. For example, the Extended Ports may be

instantiated coincident with the instantiation of virtual machines.


The instantiation of the virtual machines and the corresponding Extended Ports. When the Extended

Ports are instantiated, the new Bridge Port Extender informs the controlling bridge by issuing an

Extended Port create message for each extended Port. The Controlling Bridge allocates a Port on

the C-VLAN component and an E-channel for each new Extended Port and informs the new Bridge Port

Extender of the E-CID for these E-channels.E-CIDs “d” and “e” are established in this example. In addition,

the Controlling Bridge issues E-channel Register messages to the first Bridge Port Extender to establish the

new E-channels through the first Bridge Port Extender. At this point, the virtual machines have connectivity

to the network.





Representation of Port Extender & Fabric Extender

Server

Hypervisor

VM VM VM VM VM VM

Adapter

Switch

Eth Port Extension

802.1BR

Port

Extender

PE Tag

802.1BR

PE Tag

802.1BR

1 2 3 4 5

Nexus 5K

5

1 2 3 4 5

Port 5

vNIC

3

vNIC

2

vNIC

1

vNIC

5

vNIC

4

Port 0

FEX

(Nexus 2K) 1 2 3

1

6 7 8

NIV Capable

Adapter

IEEE-802.1BR Bridge Port Extender = Cisco FEX (Fabric Extender)


Virtual Networking Standards Components

802.1Q

Virtual Embedded

Bridge

802.1Qbg

Reflective

Relay

802.1Qbg

Multichannel

802.1BR

Port Extension

WITH TAG

OFFLOAD TO UPSTREAM SWITCH

TAGLESS


Virtual Networking Standards Components

802.1Q

Virtual Embedded

Bridge

802.1Qbg

Reflective

Relay

802.1Qbg

Multichannel

802.1BR

Port Extension

NEW BRIDGE

NEW DEVICE NEW BRIDGE

NEW BEHAVIOR

OF EXISTING

BRIDGE

HYPERVISOR-

RESIDENT

BRIDGE


Terminology

For consistency with IEEE and existing IEEE 802.1TM standards terminology, requirements placed upon conformant implementations of this standard are expressed using the following terminology:

shall is used for mandatory requirements;

may is used to describe implementation or administrative choices (“may” means “is permitted to”, and hence, “may” and “may not” mean precisely the same thing); !!

should is used for recommended choices (the behaviors described by “should” and “should not” are both permissible but not equally desirable choices).

The standard avoids needless repetition and apparent duplication of its formal requirements by using is,

is not, are, and are not for definitions and the logical consequences of conformant behavior.

Behavior that is permitted but is neither always required nor directly controlled by an implementer or

administrator, or whose conformance requirement is detailed elsewhere, is described by can.

Behavior that never occurs in a conformant implementation or system of conformant implementations is

described by can not. The word allow is used as a replacement for the phrase “support the ability for”,

and the word capability means “is able to, or can be configured to”.



P802.1BR: Bridge Port Extension

Fully specifies a Port Extender (FEX Equivalent)

Extends ports of a switch to lower entities in a network

Port Extenders are not individually managed

Their ports become ports of the controlling switch

Cascading Port Extenders

Allows one to choose the appropriate controlling switch

Frame replication supported for efficient multicast / flooding

Traffic from each “Extended Port” is reliably segregated to an E-channel and identified by a tag containing an E-channel identifier (ECID)

Does not require prior knowledge of MAC addresses; switch performs standard learning functions

Works with all devices including VEBs, VEPAs, individual VMs, physical services, and devices providing transparent services

Controlling Bridge + PE = Extended Bridge

Single Point of Management

PE

Bridge

PE

PE

PE Port Extender

PE

vFW

Server

VM1

PE

Controlling

Bridge

Extended Bridge

ECID


An Approach - Observations

To the greatest extent possible, all bridging functions are performed in the Controlling Bridge

- Many bridging functions require knowledge of the ingress and/or egress port. The E-TAG provides this information

The ports on the south side of a Port Extender may be physical or virtual ports

Inserting an Port Extender is similar to inserting a line card

- New ports are instantiated in the Controlling Bridge just as if a line card was inserted

- These ports are managed just as if they were part of a new line card

The ports of an embedded Port Extender may be “virtual”

- That is, they are conceptual and connect to a conceptual NIC (commonly referred to as a virtual NIC).

- However, from the point of view of the Controlling Bridge and management of these ports, they are handled just like any other port


VDP

VSI discovery and configuration protocol (VDP)

The VSI discovery and configuration protocol (VDP) associates (registers) a VSI instance with an SBP of an EVB Bridge . VDP simplifies and automates virtual station configuration by enabling the movement of a VSI instance (and its related VSI Type information) from one virtual station to another or from one EVB Bridge to another. VDP supports VSI discovery and configuration across a channel interconnecting an EVBstation and an EVB Bridge. VDP TLVs are exchanged between the station and the Bridge in support of this protocol


What does Virtual Station Interface Discovery Protocol (VDP) Provide?

Provides 3 major functions

1. Identity: provides a MAC address to VM association

Required for VEPA, may be useful for Port Extension

2. Mobility: allows for resources to be reserved in the network

Statically and in support of VM migration

3. Resources: allows the VM to communicate a port profile identifier to the switch

Contents of the port profile are being specified by DMTF.

VM2

Server 1

VM4 VM3

Server 2

veth1 = VM1 = 00:50:56:2E:AE:26

veth2 = VM2 = 00:50:73:10:C9:11

veth3 = VM3 = 00:50:21:46:A6:03

veth4 = VM4 = 00:50:33:AB:29:38

VM1

eMail SAP FTP Web

= Port-Profile

Cisco Position:

Cisco will support VDP since it is fundamental to providing a standards compliant implementation of

FEX-Link technology. Cisco’s pre-standard implementation is widely deployed.


VDP and Port Extension


Port Extension and EVB combined Architecture


VDP Termination in an EVB environment

Architecturally, VDP is a protocol that operates between the EVB Bridge and the Edge Relay (aka VEPA or Bridge)

In reality, the station side of VDP is run by the hypervisor

-802.1 does not specify hypervisors, so we put it in the block we do specify, the ER in this case (which may or may not be part of the hypervisor)

-Either way, it’s the hypervisor that has the knowledge of when a VM is going to migrate, etc.

One instance of ECP per ER

One receive buffer per ER

All VMs may be aggregated

EVB Bridge

Edge Relay

(VEB or VEPA)

VM VM VM VM VM VM

Hypervisor

VDP

Reality

VDP

Architecturally


Observations

VDP is a control protocol between the hypervisor and the EVB-B / Controlling Bridge

- Enables a bridge to configure itself for each VM

- Allows the bridge to identify the traffic to/from a VM

This is the purpose in both an EVB and PE environment

There is no need for the protocol to operate over the data channels

- In many cases, it does not even make sense

- For example, at a pre-associate stage there is no VM, no VSI, and no need for a channel


VDP in an PE environment

Current Model

- Architecturally, VDP is a protocol that executes between the Controlling Bridge and each ER

- One ER per VM

One instance of ECP per VM

One receive buffer per VM

No aggregation

- Reality, VDP runs between the hypervisor and the Controlling Bridge (same as EVB). Hypervisor performs backflips to emulate 1000’s of ERs, ECP sessions, etc.

Proposed model

- Architecturally, VDP is a protocol that operates between the Controlling Bridge and the Port Extender

- In reality, the station side of VDP is run by the hypervisor

One instance of ECP per PE

One receive buffer per PE

All VMs may be aggregated

Controlling Bridge

Port Extender

VM VM VM VM VM VM

ER ER ER ER ER ER

VDP

Reality

VDP

Architecturally

Controlling Bridge

Port Extender

VM VM VM VM VM VM

VDP

Reality

VDP

Architecturally

Hypervisor

Hypervisor


VDP TLV Association

VSI Manager ID:

Identifies the database that should be accessed to get the VSI type.

The value 0 means that the station does not know what VSI Manager ID to use, indicating that the Bridge should select a default value.

Any other value is interpreted as an IPv6 address, as defined in IETF RFC 4291.


802.1BR: Port Extender Control and Status Protocol (PE CSP)

Controlling Bridge configures all of the forwarding tables for each downstream (i.e. cascaded) Port Extender

Occurs at Port Extender initialization

No additional programming required as the result of MAC learning / aging, or MAC migration as the result of VM migration

PE CSP provides this functionality

Transported over ECP

All messages are command / response

Independent instance of PE CSP is executed for each Uplink Port


Why Tag?

What does the controlling switch need?

- An unambiguous indication of the source port

- A mechanism to specify the destination port (or ports)

Why not use the MAC address for the source port?

- MAC addresses may be spoofed

- A single source port may have multiple MAC addresses

- Not possible to always know the MAC address to source port relationship

Why not use the MAC address for specification of the destination port?

- MAC address is not always unambiguous

- e.g. BPDUs, LLDP, etc.

- Complicates Port Extenders

- Requires full implementation of the MAC/VID/FID lookup and forwarding logic

- Prevents Controlling Bridge from performing multicast filtering (ACLs, for example)

- Must be performed by Port Extender instead, further complicating implementation

The tag provides a natural extension to typical switch architecture in support of Port Extension


Tagging and Port Extension - Observations

When a frame enters a bridge, it is internally “tagged” with an indication of the ingress port

The ingress port is used in several frame processing operations, ultimately resulting in determination of the egress port, which is added to the internal tag

The rest of the forwarding through the bridge is performed based on the internal tag

At egress, egress ACL processing is performed based on ingress port, egress port, and Frame Contents (on a per egress port basis for multicast). Frame processing adds or removes a QTag, and potentially other packet rewrite functions

With Port Extension, all of the fundamental bridge functionality remains identical

- Which is a very good thing

- From the outside world, the combination of PEs and the controlling bridge is a single 802.1Q compliant bridge

The PEs are extremely simple

- On ingress, add a tag, then forward north

- Southbound, forward based on ECID as index into forwarding table

- Remove ETAG at the last hop


P802.1BR: Extended Ports & ECIDs

Each Extended Port from a Port Extender to a NIC, VNIC, or bridge is, in effect, a bridge interface

- These are the instantiations of interfaces of the Controlling Bridge

- Each Extended Port is identified by a Extension Channel ID (ECID)Assigned by the Controlling Bridge to each PE Extended Port at initialization

- Scope of uniqueness is the Controlling Bridge Port

- ECIDs are 14 bits:

- Values 1 through 4095 reserved for Extended Ports

- Values 4096 through 16 382 reserved for Multicast use (will be discussed in the next slide)

- Values 0 and 16383 reserved


P802.1BR: Port Extender Forwarding Tables

12 Bits – ECID

(from ETAG)

4 bits – Dport (ECID = 1)




Addre

ss

Dest Port

Unicast forwarding table

One entry per ECID

May support up to 4095 unique ECIDs

Indexed by ECID (part of the ETAG)

Each entry contains a destination port

Multicast table (used for flooding, multicast, SPAN, etc.)

One entry per ECID

May support up to 12k entries

Indexed by ECID

Each entry contains a bit mask indicating which Extended and Cascade Ports are to be used

Width of entry depends on number of ports

14 Bits – ECID

(from ETAG)

n bits – Dportmask (ECID = 4097)




Addre

ss

Dest Port Mask


P802.1BR: Port Extender Basic Functions

From NIC to Controlling Bridge

1 Add ETAG if none present (indicating source ECID)

2 ETAG added only at ingress

3 ETAGs are not “stacked” as the frame passes through successive Port Extenders

4 Forward frame up the Port Extender hierarchy to the Controlling Bridge

1 Forward frame down hierarchy to the NIC

2 Destination port determined by using ECID as index into the forwarding table

3 Replicate multicast frames

4 Filter the frame at the ingress port if it was sourced at the Port Extender

(i.e. if the port’s assigned ECID matches the source ECID in the ETAG)

5 Remove the ETAG if the final downlink has been reached


P802.1BR: Bridge use of Tag

On ingress

- Learn ECID along with MAC address, VID, and port number as part of normal bridge learning function

Forwarding

- Utilize source ECID along with ingress port number as frame source for all normal bridge functions (ACLs, VLAN member set enforcement, etc.)

On egress:

- Populate the ETAG with the ingress ECID and egress ECID


P802.1BR: The E-TAG

DEI and PCP used for traffic class selection

Source ECID contains the identifier of the Port Extender Port that sourced this frame

Port Extender filters the frame from this port

ECID indicates the E-channel on which this frame is being transmitted

ECIDs are 14 bits

First 4k of the range are reserved for E-channels that contain a single Extended Bridge Port

Used for the default ECID of the port

Thus the Ingress ECID field only requires 12 bits

Ethertype

(16 bits)

ECID

(14 bits)

Ingress ECID

(12 bits)

PCP

(3 bits)

DE

I

Resv

(2 bits)

Reserved

(16 bits)


P802.1BR: Port Extender Control and Status Protocol (PE CSP)

Controlling Bridge configures all of the forwarding tables for each downstream (i.e. cascaded) Port Extender

- Occurs at Port Extender initialization

- No additional programming required as the result of MAC learning / aging, or MAC migration as the result of VM migration

PE CSP provides this functionality

- Transported over ECP

- All messages are command / response

- All commands are idempotent enabling repeatability if command or response is lost

- Independent instance of PE CSP is executed for each Uplink Port


Relationship of Port Extension to FEX-Link

With VDP, Provides the exact same capability

- Different format of tags

- Different format of messages in the controlling protocol

E-TAG or VN-TAG may be a port option on the Controlling Switch

- Port Extension and VN-TAG are fully inter-operable


Tagging and Port Extension

Ingress Side of

Line Card

Frame

Processor

Memory

Control

VOQs

Crossbar Egress Side of

Line Card

Frame

Egress

Processing

Port

4

Port

8

Frame enters here,

smac=abc, dmac=xyz,

vlan=123.

Internal tag

added,

sport=4

Frame processor performs several

operations in parallel:

- smac, vlan, sport learned

- Ingress VLAN verified to be part of

member set for sport

-Ingress ACLs processed based on

sport and frame header

- dmac, vlan lookup performed to

determine dport=8

- internal tag updated with dport

Crossbar forwards

frame based on dport

-Egress ACL processed

based on sport, dport,

& frame contents

-Frame rewrite takes place (IP

related, add / delete QTag, etc.)

-Frame transmitted on port 8 based

on dport


Ingress Side of

Line Card

Frame

Processor

Memory

Control

VOQs

Crossbar Egress Side of

Line Card

Frame

Egress

Processing

Ingress

Path PE

Ingress

Path PE

Ingress

Path PE

Egress

Path PE

Egress

Path PE

Egress

Path PE

E-VID

22

Port

4

Port

8

ECID

47

Frame enters here,

smac=abc, dmac=xyz,

vlan=123.

PE adds E-TAG,

ECID=22

PE forwards

frame

unmodified

Internal tag

added,

sport.ECID=4.22

Frame processor performs several

operations in parallel:

- smac,vlan, sport.ECID learned

- Ingress VLAN verified to be part of

member set for sport.ECID

-Ingress ACLs processed based on

sport.ECID and frame header

- dmac, vlan lookup performed to

determine dport.ECID=8.47

- internal tag updated with dport.ECID

Crossbar forwards

frame based on dport

-Egress ACL processed

based on sport.sourceECID,

dport.ECID, & frame contents

-Frame rewrite takes place (IP

related, add / delete QTag, ETAG,

etc.)

-Frame transmitted on port 8 based

on dport

Frame forwarded

to next hop PE

based on ECID=47

Frame forwarded

to egress PE port

based on ECID=47,

ETAG removed

Tagging and Port Extension


VN-TAG frame format (6 bytes)

direction indicates to/from adapter

source virtual interface indicates frame source

- looped indicates frame came back to source adapter

destination virtual interface dictates forwarding

- pointer helps pick specific destination vNIC or vNIC list

Link local scope

- Rooted at Virtual Interface Switch

- 4096 virtual interfaces

- 16,384 Virtual interface lists

Coexists with VLAN (802.1Q) tag

- 802.1Q tag is mandatory to signal data path priority

VNTAG Ethertype

source virtual interface

destination virtual interface d p

l r ver

Frame Payload

CRC[4]

SA[6]

DA[6]

• IEEE 802.1BR and VN-TAG Same architecture and logic

Minor frame format difference

• Future platforms will support both formats Backward compatibility guaranteed

Interoperability even easier


× Management complexity: each VEPA is an independent point of management

× Doesn’t support cascading

Reflective Relay (used in basic VEPA)

× Vulnerable: ACLs based on source MAC (can be spoofed)

× Resource intensive: Hypervisor component consumes CPU cycles

Multichannel (used in advanced VEPA)

× Even more components to manage

× Inefficient bandwidth : separate copy of each Mcast and Bcast packets on the wire

Ease of management: one switch manages all Port Extenders (adapters/switches/virtual interfaces)

Supports cascading of Port Extenders (multi-tier, single point of management)

Virtual Machine aware FEX

Secure: ACLs based on VN-TAG

Scalable: Mcast and Bcast replication performed in HW at line rate

Efficient: no impact to server CPU

Cisco FEX Architecture Advantage

VEPA based on IEEE 802.1Qbg FEX based on IEEE 802.1BR

Switch

FEX

Log

ica

l S

witch

VM- FEX


BR is based on ETAG vs. Cisco’s VNTAG

ETAG offers larger address field for VM’s

VNTAG has single control plane

BR has an individual control plane for each data plane


NIC’s vendor are fine with the dual TAG Implementation

Nexus and UCS-FI TAG translation


Two main implementation categories of IEEE-802.1BR Cisco calls the Fabric Extender “FEX”

Physical

or

Virtual


=

Distributed Modular System

+

Nexus 5000 Parent Switch

Cisco Nexus® 2000 FEX

(Over 3000 production customers!!! Over 3 million Nexus 2000 ports deployed!!!)

Distributed Modular System

Nexus 2000 FEX is a Virtual Line Card to the Nexus 5000

Nexus 5000 maintains all management & configuration

No Spanning Tree between FEX & Nexus 5000

LAN

N7000/

C6500

MDS

SAN

Access

Layer N5000

1 12

N2232 N2232

Physical Example; Nexus + FEX; UCS


Lo

gic

al

Sw

itch

Baseline architecture

Switch

FEX

Hypervisor vSwitch

App

OS

App

OS

App

OS

LAN

Lo

gic

al

Sw

itch

FEX architecture

Switch

FEX

Hypervisor

LAN

App

OS

App

OS

App

OS

VM-FEX

Switch port extended over

cascaded Fabric Extenders to

the Virtual Machine

Lo

gic

al

Sw

itch

Collapse virtual and physical networking tiers!

Virtual Example; Adapter-FEX; Nexus-1000v


VNTAG and BR are not exactly the same - However this is totally transparent for our customers, they won’t feel the difference

from a user perspective

Transition from pre standard to BR is guaranteed

The Standard is finalized - Expected public publication is spring 2012


Most Relevant Datacenter Standards References

TRILL: http://www.ietf.org/rfc/rfc5556.txt http://datatracker.ietf.org/doc/draft-ietf-trill-rbridge-protocol/?include_text=1

MAC-in-MAC: http://standards.ieee.org/getieee802/download/802.1ah-2008.pdf

Bridge Port Extender

http://www.ieee802.org/1/pages/802.1br.html

http://www.ietf.org/rfc/rfc5556.txt

http://datatracker.ietf.org/doc/draft-ietf-trill-rbridge-protocol/?include_text=1









http://standards.ieee.org/getieee802/download/802.1ah-2008.pdf




http://www.ieee802.org/1/pages/802.1br.html

Recommended Reading

Please visit the Cisco Store for suitable reading.


Please complete your Session Survey

Don't forget to complete your online session evaluations after each session.

Complete 4 session evaluations & the Overall Conference Evaluation

(available from Thursday) to receive your Cisco Live T-shirt

Surveys can be found on the Attendee Website at www.ciscolivelondon.com/onsite

which can also be accessed through the screens at the Communication Stations

Or use the Cisco Live Mobile App to complete the

surveys from your phone, download the app at

www.ciscolivelondon.com/connect/mobile/app.html

We value your feedback

http://m.cisco.com/mat/cleu12/

1. Scan the QR code

(Go to http://tinyurl.com/qrmelist for QR code reader

software, alternatively type in the access URL above)

2. Download the app or access the mobile site

3. Log in to complete and submit the evaluations

http://www.ciscolivelondon.com/onsite

http://www.ciscolivelondon.com/connect/mobile/app.html

http://tinyurl.com/qrmelist



Thank you.

Documents

virtual bridge