Scaling The Edge Bridge Address TableIn Datacenter Networks

June-2012

Agenda

Motivation

Protocol properties, concepts and operation

Protocol details

2

A

B

C

Motivation

4

Problem Statement

Address learning methods Control plane learning Data plane learning

Data-plane learning is simpler than control plane learningyet, it leads to bad scaling of forwarding tables

Question: can we have both data-plane learning simplicity and forwarding tables scaling?

A

B

C

B

VM EB/Port

A

B

C

A.1

A.2

C.1

B.1

B.2

C.2

OverlayNetwork

5

Dataplane Learning On Edge Bridges (EB)

VM1BC (e.g. ARP Request)

VM1

VM2

A

VM EB/Port

C

VM EB/Port

BC1

DS

1 A.1

BC1

DS

BCABC1

DS

BCA

1 A

1 A

BC1

DS

BC1

DS

BC1

DS

BC1

DS

Dataplane learning EB table size = # of VMs in the VLAN/Tenant DomainSevere FDB Scaling Problem in EB

Protocol Concepts and Operation

Properties of The Proposed Solution

Bridge address table scaling for data-center networks with support for hot VM migration FDB size = # of EBs in the network + # of locally attached VMs

Layer-2 only No higher layers awareness

End point (Hypervisor) is blind to overlay network protocol Can work with any overlay protocol

7

Protocol Concepts

The protocol defines Data-plane format between the hypervisor and the Edge-Bridge

– Modify 802.1BR or extend 802.1Qbg

Control-plane negotiates the protocol capabilities between the EB and the hypervisor– Extend DCBX 802.1Qaz

Protocol concepts A handshake between the EB and the hypervisor

– Capabilities exchange using control-plane– Dynamic operation uses the data-plane

EB– Learns addresses of local VMs & remote EBs – Uses data-plane signaling to informs the hypervisor of the path in the overlay network– Uses the path signaled by the hypervisor to forward traffic to remote VMs over the overlay network

Hypervisor– Sends data traffic to EB with path indication– Updates its path database (Path$) using the indications received from the EB

8

9

Protocol Databases and Signaling

VM1

VM2

B

VM Port

A

B

C

A.1

A.2

C.1

B.1

B.2

C.2

DS

DS

B

EB

1 A

2 B

3 CA.1$

VM Path

DS S.Path

Generated by VM

DS T.Path

DS

ServerEB

OverlayNetwork

EBServer

Rx byVM

EB

Hypervisor

Path$

Overlay FDB

LocalFDB

A

EB

1 A

2 B

3 C

10

Protocol Operation #1

VM1VM2 flooded Unicast forwarding

VM1

VM2

A

VM Port

C

VM Port

B

VM Port

A

B

C

A.1

A.2

C.1

B.1

B.2

C.2

21

DS

1 A.1

21

DS

BCA21

DS

BCA

Dataplane learning EB table size = # of local VMs + # of EBs in the network

C

EB

1 A

2 B

3 C

B

EB

1 A

2 B

3 C

A.1$

VM Path

B.1$

VM Path

21

DS

1

s.Path

21

DS

1

s.Path

21

DS

1

s.Path

21

DS

1

s.Path

1 1

Learn only in B.1

A

EB

1 A

2 B

3 C

11

Protocol Operation #2

VM2VM1 reply

VM1

VM2

A

VM Port

C

VM Port

B

VM Port

A

B

C

A.1

A.2

C.1

B.1

B.2

C.2

21

SD

1 A.1

BA

D S

21

Dataplane learning EB table size = # of local VMs + # of EBs in the network

C

EB

1 A

2 B

3 C

B

EB

1 A

2 B

3 C

A.1$

VM Path

B.1$

VM Path

11

DT.Path

2

S

1 1

12

DS.Path

2

S

21

SD

2 2

2 B.1

Properties Of Hypervisor Path$

Acts like ARP$ - holds active sessions only Inactive entries are aged-out Not contaminated by ARP-BC received from the network

Path$ entry insert/update ETH DA is UC/MC and conforms to a VM hosted by this hypervisor, OR ETH DA is BC and the Layer-3 DA conforms to a VM hosted by this hypervisor

Path$ entry delete/refresh Using an activity timer

12

A

B

C

Protocol Details

14

Protocol Details

Control protocol Capabilities negotiation between the Hypervisor and the Edge Bridge Modify 802.1Qaz (DCBx)

Data-plane protocol (2 options) Add Path-ID Tag (P-Tag)

– S-channel/E-Tag is outer– P-Tag is inner:

–16b source/target-path-id–Source/target depends on direction

Modify BPE E-Tag– HypervisorEB

–I-ECID – identical use to BPE–E-CID – target-path-id

– EBHypervisor–I-ECID

–I-ECID < 4K local virtual port (identical to BPE)–I-ECID =>4K source-path-id

–E-CID – identical use to BPE

DA (6B)

SA (6B)

S-Channel/E-Tag

(4B)

P-Tag (4B) VLAN (4B)

Payload+

FCC

Summary of Protocol Properties

Bridge address table scaling for data-center networks with support for hot VM migration FDB size = # of EBs in the network + # of locally attached VMs

Layer-2 only No higher layers awareness

Hypervisor is blind to overlay network protocol Can work with any overlay protocol

Easy to implement Local scope: hypervisor to edge-bridge protocol Simple control-plane – only need to negotiate capabilities

– Extend DCBX 802.1Qaz

Simple extension of existing data-plane protocols– Modifies 802.1BR E-Tag or extends 802.1BR/802.1Qbg with a P-Tag

Easy to deploy Co-exists with 802.1Qbg/802.1BR protocols Support for incremental upgrade in per EB granularity

15

Detailed Packet Walkthrough

Identical To The Animation

Walkthrough in a Nutshell (VM1VM2) #1

VM1VM2 (VM2 ETH address is known to VM1) and back

Initial state: all FDBs are empty

Hypervisor hosting VM1 Receive packet from VM1 If VM2 is registered in Path$, forward with the registered T.Path

Else forward with T.Path=BC

EB-A Learn on FDB-A (VM1,A.1) T.Path=BCFlood to Overlay and to local ports

EB-B Replace tunnel-header with S.Path=A Forward to VM1 if VM1 is registered in FDB-B

Else flood to local ports

Hypervisor hosting VM2 Receive the packet and update Path$ (VM1,Path=A) if:

– ETH DA conforms to a VM hosted by this hypervisor, OR– ETH DA is BC and the Layer-3 DA conforms to a VM hosted by this hypervisor

Pass packet to VM2 if any of the above conditions is true17

A

B

C

VM1

VM2

Walkthrough in a Nutshell (VM2VM1) #2

Hypervisor hosting VM2 Receive packet from VM2 VM1 is registered in Path$ send with T.Path=A

EB-B Learn on FDB-B (VM2,B.1) Send over Path A to EB-A

EB-A Replace tunnel-header with S.Path=B VM1 is registered in FDB-A (thanks to VM1VM2 path) Forward to VM1

Hypervisor hosting VM2 Receive the packet and update Path$ (VM2,Path=B) if:

– ETH DA conforms to a VM hosted by this hypervisor, OR– The Layer-3 DA conforms to a VM hosted by this hypervisor

Pass packet to VM1 if any of the above conditions is true

18

A

B

C

VM1

VM2

Thank you