Topic Database Consistency Solution for large-scale

Topic：Database Consistency Solution for large-scale OpenStack SDN Architecture

Speaker：马力, 海云捷迅（AWcloud）

Omer Anson, 华为（Huawei）

Agenda

• Introduction• Dragonflow Overview• What's the Problem• How We Solve It• To the Next Stage

Introduction

• Li Ma– Principle Architect in AWcloud– Core in OpenStack Dragonflow– Concentrated on large-scale cloud infrastructure

• Omer Anson– Software Engineer in Huawei– Core in OpenStack Dragonflow

Dragonflow Overview• Integral “Big Tent” project in OpenStack• Designed for High Scale, Performance and Low Latency• Lightweight and Simple• Easily Extendable• Distributed SDN Control Plane• Focus on advanced networking services• Distributes Policy Level Abstraction to the Compute Nodes

Neutron-Server

Dragonflow Plugin

DBOVSDragonflow

DBDriver

Compute Node

OVSDragonflow

DBDriver

Compute Node

OVSDragonflow

DBDriver

Compute Node

OVSDragonflow

DBDriver

Compute Node

DB

VM VM..VM VM..

VM VM.. VM VM..

Distributed SDN

“Under The Hood”

Compute Node Compute Node Compute Node

Dragonflow

Network DB

OVS

NeutronServer

Redis

OVSDB-Server

ETCD RethinkDBRMC

Kernel Datapath Module

NIC

User Space

Kernel Space

DB Drivers

Redis ETCD RethinkDBRMC

Future

Dragonflow Plugin

Route CoreAPI SG

vswitchd

Container

VM Dragonflow ControllerAbstraction Layer

L2 App L3 App DHCP

FWaaS LBaaS …FIP/DNAT

Pluggable DB Layer

NB DB Drivers

SB DB Drivers

SmartNIC OVSDB

ZK

ETCD

RMC

Redis

OpenFlow

SG

ZK

ZK

Pub/Sub Drivers

Redis ØMQ

Current Release Features (Mitaka)L2 core API, IPv4, IPv6 § GRE/VxLAN/STT/Geneve tunneling protocols

Distributed L3 Virtual RouterDistributed DHCP Pluggable Distributed Database§ ETCD, RethinkDB, RAMCloud, Redis, ZooKeeper

Pluggable Publish-Subscribe§ ØMQ, Redis

Security Groups§ OVS Flows leveraging connection tracking integration

Distributed DNATSelective Proactive Distribution§ Tenant Based

Pluggable Database

Requirements§ HA + Scalability§ Different Environments have different requirements

§ Performance, Latency, Scalability, etc.

Why Pluggable?§ Long time to productize§ Mature Open Source alternatives§ Allow us to focus on the networking services only

9

DistributedDatabase

DB Data 3

DB Data 2

DB Data 1

Selective Proactive Distribution

Compute Node 1

DragonflowLocal Cache

OVS

DB Data 1

Compute Node N

Dragonflow

OVS

Local Cache

DB Data 3

DB Data 2

Dragonflow DB Drivers

Redis ETCD ZKRMC

1 VM Send DHCP_DISCOVER

2 Classify Flow as DHCP, Forward to Controller

3 DHCP App sends DHCP_OFFER back to VM

4 VM Send DHCP_REQUEST

5 Classify Flow as DHCP, Forward to Controller

6 DHCP App populates DHCP_OPTIONS from DB/CFG and send DHCP_ACK

Distributed DHCP

DHCP DISCOVER

VM DHCP SERVER

DHCP OFFER

DHCPREQUEST

DHCPACK

1

34

67

Compute Node

Dragonflow

VM

OVS

VM

1 2

br-intqvoXXX qvoXXX

OpenFlow

14

25

7

Dragonflow Controller

Abstraction Layer

L2App

L3App

DHCPApp SG

36

Pluggable DB

Distributed

DB

This haseverythingin it

11

Compute Node

DragonflowLocal Controller

SubscriberRedis ØMQ

Compute Node



Compute Node



Compute Node



Neutron ServerDragonflow

PluginPublisher

Redis ØMQ


PluginPublisher

Redis ØMQ


PluginPublisher

Redis ØMQ...

DB

...

Pluggable Pub/Sub

12

Is Dragonflow Ready?

AWcloud Point of View

Dispatch Network Policy to Compute Nodes

Requirements:

ScalabilityReliability

Currently, we use Neutron OVS plugin…but as workloads increase…

Limitations in Large-scale deployments

- Messaging- Distributed Messaging System for OpenStack at Scale- Presented in Vancouver Summit 2015

- Persistent HA DB- Dragonflow DONE the SDN way- Presented in Austin Summit 2016

Scalability in Persistent Storage

We prefer BASE systems for data backends

• Basically Available• Soft-state• Eventual consistent

Is there any open source solution that can meet our requirements?

Scalable Persistent Storage in Dragonflow

• A pluggable Key-Value Interface Layer• Supported Solutions

• ETCD• RAMCloud• ZooKeeper• Redis• RethinkDB

Is it enough?Scalable and reliable?

DB Consistency: Common Problem to all SDN Solutions

SDN ControllerNorth-bound Interface (REST?)

South-bound Interface (Openflow)

SDN Apps

SDN DB

NeutronDB

Neutron-serverML2-Core-Plugin

ML2.Drivers.Mechanism.XXX

Services-PluginService

Network

Neutron API Nova API

CLI / Dashboard (Horizon) / Orchestration Tool (Heat)

Switch

Nova

Nova Compute

VM VM

Nova Compute

VM VM

Virtual Switch Virtual SwitchNeutron

Plugin AgentNeutron

Plugin Agent

Message Queue (AMQP)

Neutron-L3-Agent

Neutron-DHCP-AgentLoad BalancerFirewallVPNL3 Services

Topology Mgr.Overlay Mgr.Security

Vendor-specific API

DB Consistency: Common Problem to all SDN Solutions

Neutron DB

Relational Database

ACID system

Stores the whole virtualized network topology for OpenStack

Dragonflow DB

Key-value Store

BASE system

Stores a ‘partial’ virtualized network topology used in Dragonflow

DB Consistency: Common Problem to all SDN Solution Problem 1: Dragonflow DB operation has failed

• Neutron DB operation is committed• But the related Dragonflow DB operations have failed

DB Consistency: Common Problem to all SDN Solution Problem 2: Multiple Parallel Transactions

• Neutron DB can deal with multiple parallel transactions.• How about Dragonflow DB?

DB Consistency: Common Problem to all SDN Solution Problem 3: Nested Transactions

• Neutron DB can deal with nested transactions.• How about Dragonflow DB?

DB Consistency: Common Problem to all SDN Solution Additional Problems

• There may be other issues.

Some thoughts on DB Consistency

• Database in Multi-node/Multi-core System• Multi-Version Concurrency Control• Transaction Isolation

• REPEATABLE READ• READ COMMITTED• READ UNCOMMITTED• SERIALIZABLE



• Remove Neutron DB• Complicated Solution when involving ML2• Cannot be done in a short period of time



SDN Apps

SDN DB

NeutronDB




Network



Switch

NovaNova Compute

VM VM

Nova Compute

VM VM


Plugin AgentNeutron

Plugin Agent


Neutron-L3-Agent



Vendor-specific API


• Introduce the pluggable key-value store into Neutron• How to work with SQLAlchemy?

• ROME: https://github.com/BeyondTheClouds/rome• Need much more time on evaluation and deep discussion.



SDN Apps

SDN DB

NeutronDB




Network



Switch

NovaNova Compute

VM VM

Nova ComputeVM VM


Plugin AgentNeutron

Plugin Agent


Neutron-L3-Agent



Vendor-specific API


• Are there any other solutions?• That are simple?• That are straightforward?

DB Consistency in Dragonflow—— Distributed Lock

• Introduce a distributed lock for coordination– Guarantee the atomicity of a given API– Implemented in the Neutron core plugin layer– Project-based lock allows concurrency

DragonflowNorth-bound Interface


SDN Apps

DF DB

NeutronDB

Neutron-serverCore Plugin

Dragonflow Neutron Plugin

Neutron API



Obtain distributed lock

Dragonflow NB API

• Initial Solution: Introduce a distributed lock for coordination– Initially it was implemented by SELECT-FOR-UPDATE statement– Not compatible with Galera clustering– Performance penalty when involving retry_for_deadlock operation


• Improved Solution: SQL-based compare-and-swap operation– Compatible with Galera clustering– No performance penalty


• Introduce an object synchronization mechanism– All the objects stored in both databases are versioned.– Sync the object when something unexpected happens.

SDN DBNeutronDB

Network_ID Name Status MTU VLAN Availability Zone Subnets

Object_ID = Network_ID Version = 5

Read

Notify

compare & swap <- Version Compute Node Compute Node Compute Node

Dragonflow Local Controller

Subscriber

vSwitch Flush Flows

DB Consistency in Dragonflow—— Object Synchronization

• Introduce auto-recovery mechanism– Periodically detect inconsistency by version comparison.– Recover the object data from Neutron DB to Dragonflow DB.– Compatible for multi-node deployment.

• Introduce Master Election• Introduce Load Balancing in the later phase

DB Consistency in Dragonflow—— Auto-Recovery

DB Consistency in Dragonflow—— Auto-Recovery

Neutron-Server

Dragonflow Plugin

DB

Neutron-Server

Dragonflow Plugin

DB

Neutron-Server

Dragonflow Plugin

DB

VOTE

MASTER

SLAVE

SLAVE

1 Obtain the distributed lock

2 Get Object data from Neutron DB

3 Get the corresponding data from Dragonflow DB

4 Verify the consistency of that data

5 Call update_object for that data if it is not consistent

6 Check the next object.

7 Release the distributed lock

To the Next Stage

OpenStack Challenges

• Scalability– Networking does not scale (< 500 compute nodes)

• Performance– Networking performance is low (namespace overhead, huge

control plane overhead, …)

• Operability– Reference implementation has lots of maintenance problems

(e.g. thousands of concurrent DHCP servers, namespaces, etc.)

Scalability

Scale(# Compute

Nodes)

today

10,000

2,500

Time to Market

n+2

1,000

Dragonflow& Redis

Dragonflow &RAMCloud &

ØMQOptimized Hybrid

(Reactive/Proactive)Dragonflow4,000

n+1soon

Roadmap

Additional DB Drivers ZooKeeper, Redis…Selective Proactive DB Pluggable Pub/Sub Mechanism DB Consistency Distributed DNATSecurity Group

Ø Hierarchical Port Binding (SDN ToR) move to ML2Ø Containers (Kuryr plugin and nested VM support)Ø Topology Service Injection / Service ChainingØ Inter Cloud Connectivity (Border Gateway / L2GW)Ø Optimize Scale and Performance

38

Newton Release New Applications

l IGMP Applicationl Distributed Load Balancing (East/West)l Brute Force preventionl DNS servicel Distributed Metadata proxyl Port Fault Detection

Ride the Dragon!• Documentation

– https://wiki.openstack.org/wiki/Dragonflow• Bugs & blueprints

– https://launchpad.net/dragonflow• DF IRC channel

– #openstack-dragonflow– Weekly on Monday at 0900 UTC in

#openstack-meeting-4 (IRC)

Placeholder Footer Copy / BU Logo or Name Goes Here

Thanks

Documents

Topic Database Consistency Solution for large-scale