HUAWEI TECHNOLOGIES CO., LTD. Page 1Huawei Confidential

Neutron cascading to address scalability issue

Chaoyi Huang ( joehuang@huawei.com )

Last edited May 18, 2015

HUAWEI TECHNOLOGIES CO., LTD. Page 2Huawei Confidential

Compute node

Compute node

Compute node

Compute node

Compute node

Compute node

Compute node

Neutron - like a “multi-cast”/“burst” domain

• Security-group: project-level, neutron-levelsecurity_groups_rule_updatedsecurity_groups_member_updated…

• DVR: router level:add_arp_entrydel_arp_entryrouter_deletedrouters_updateddvr_mac_address_updatesync_router…

• L2-population: network-leveladd_fdb_entriesremove_fdb_entries

…

Neutron ServerNeutron ServerNeutron Server

DBDBDB

Msg. Bus

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

Compute node

Lots of nodes involved in Neutron API request processing

HUAWEI TECHNOLOGIES CO., LTD. Page 3Huawei Confidential

Because Neutron has to touch regarding nodes where the VMs (in network level, router level…) resides for networking purpose

( It’s quite different from Nova/Cinder which mostly manipulate single VM/volume entity !!! )

Neutron - Why challenge?

HUAWEI TECHNOLOGIES CO., LTD. Page 4Huawei Confidential

Neutron ServerNeutron ServerNeutron Server

DBDBDB

Msg. Bus

VM

Compute node

VM

Compute node

Compute node

Compute node

Compute node

VM

Compute node

Compute node

Compute node

VM

Compute node

VM

Compute node

VM VM VM VM VM

As VMs/nodes increased, “multi-cast”/”burst” heavily impacts the Neutron scalability

HUAWEI TECHNOLOGIES CO., LTD. Page 5Huawei Confidential

Neutron ServerDB

Msg. Bus

VM

Compute node

VM

Compute node

Compute node

Compute node

Compute node

VM VM VM

Then the large “multi-cast”/”burst” domain should be partitioned into smaller workable ones

Neutron ServerDB

Msg. Bus

VM

Compute node

VM

Compute node

Compute node

Compute node

Compute node

VM VM VM

Portioning according to availability zone(AZ) in Nova, one AZ one small “multi-cast”/”burst” domain

AZ1 AZ2

Neutron ServerNeutron ServerNeutron Server

DBDBDB

Msg. Bus

HUAWEI TECHNOLOGIES CO., LTD. Page 6Huawei Confidential

Why partitioning in AZs ?

1. Availability zone is also often meaning fault isolation domain. Crash of one AZ should not affect other AZ’s running

2. Application often independently (or weak relationship) deployed in multiple availability zones for redundancy purpose

3. Virtual machines in one network often located inside one availability zone. (Amazon, for example ).

HUAWEI TECHNOLOGIES CO., LTD. Page 7Huawei Confidential

Neutron ServerDB

Msg. Bus

VM

Compute node

VM

Compute node

Compute node

Compute node

Compute node

VM VM VM

Neutron ServerDB

Msg. Bus

VM

Compute node

VM

Compute node

Compute node

Compute node

Compute node

VM VM VM

AZ1 AZ2

After partition, the Neutron server API should still be reserved for API compatibility.

Neutron ServerDB

Neutron API

HUAWEI TECHNOLOGIES CO., LTD. Page 8Huawei Confidential

ProxyProxy

Neutron ServerDB

Msg. Bus

VM

Compute node

VM

Compute node

Compute node

Compute node

Compute node

VM VM VM

Neutron ServerDB

Msg. Bus

VM

Compute node

VM

Compute node

Compute node

Compute node

Compute node

VM VM VM

AZ1 AZ2

And the API request should be proxy to regarding partitioned Neutron...

Neutron ServerDB

Neutron API

HUAWEI TECHNOLOGIES CO., LTD. Page 9Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Forward the network/subnet/port/SG creation to regarding partitioned Neutron where VM resides

Neutron ServerDB

Create Port ( device_owner: AZ2)

if device_owner belongs to AZ1Create Network/Subnet/Port/SG here

AZ1

VM

HUAWEI TECHNOLOGIES CO., LTD. Page 10Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Forward the network/subnet/port creation to regarding partitioned Neutron where VM resides

Neutron ServerDB

Create Port ( device_owner: AZ2)

if device_owner belongs to AZ2Create Network/Subnet/Port/SG here

VM

AZ2

HUAWEI TECHNOLOGIES CO., LTD. Page 11Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Router will be created where network/VM resides

Neutron ServerDB

Router1 . add_router_interface( Network1 )

AZ1

VM R1

Network1

Create Router1Router1 . add_router_interface( Network1 )

HUAWEI TECHNOLOGIES CO., LTD. Page 12Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Router will be created where network/VM resides

Neutron ServerDB

Router1 . add_router_interface( Network1 )Router1 . add_router_interface( Network2 )

Create Router1Router1 . add_router_interface( Network2 )

AZ2

VM R1

Network1

VMR1

Network2

Create Router1Router1 . add_router_interface( Network1 )

AZ1

HUAWEI TECHNOLOGIES CO., LTD. Page 13Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Connect distributed router in different AZs…. in many ways, and automatically by the proxy

Neutron ServerDB

AZ2

VM R1

Network1

VMR1

Network2

Provider L2 network

HUAWEI TECHNOLOGIES CO., LTD. Page 14Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

connect distributed router in different AZs…. in many ways, and automatically by the proxy

Neutron ServerDB

AZ2

VM R1

Network1

VMR1

Network2

GRE/VPN… tunneling

HUAWEI TECHNOLOGIES CO., LTD. Page 15Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Different Neutron backend ( SDN controller ) co-existence

Neutron ServerDB

AZ2

VM R1

Network1

VMR1

Network2

AZ1

OpenDayLight OVN

GRE /VPN..tunneling

HUAWEI TECHNOLOGIES CO., LTD. Page 16Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Even in different data centers (geographically)

Neutron ServerDB

AZ2

VM R1

Network1

VMR1

Network2

AZ1

OpenDayLight OVN

DC1 DC2GRE /VPN…tunneling

HUAWEI TECHNOLOGIES CO., LTD. Page 17Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Even inter-connection network between AZs can be dynamically created by DCI controller

Neutron ServerDB

AZ2

VM R1

Network1

VMR1

Network2

AZ1

OpenDayLight OVN

Data Center Interconnection

Controller

creation dynamically

Provider network

HUAWEI TECHNOLOGIES CO., LTD. Page 18Huawei Confidential

ProxyProxy

Neutron ServerDB

Neutron ServerDB

Neutron ServerDB

AZ2AZ1

Proxy

Neutron ServerDB

AZ3

Router_updateRouter_update

Router_update

No challenge for “multi-cast/burst” in the cascading Neutron

For one tenant’s resource will be allocated in limited AZs ( 3 AZs by default in Amazon ), the “multi-cast/burst” will only happen in limited proxy nodes, which reduce load greatly in cascading Neutron layer

HUAWEI TECHNOLOGIES CO., LTD. Page 19Huawei Confidential

1 million ports can be supported by Neutron cascading,

verified by semi-simulation test

http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

HUAWEI TECHNOLOGIES CO., LTD. Page 20Huawei Confidential

More information:

Wiki: https://wiki.openstack.org/wiki/OpenStack_cascading_solutionPoC Source Code: https://github.com/stackforge/tricircle