NFVInfrastructureManagerwith HighPerformanceSoftwareSwitch ... · • KVM,QEMU,Xen,VMware,Hyper V,LXCandothers • Integrated many network appliance and storageappliance Ref:

Copyright©2015 NTT corp. All Rights Reserved.

NFV Infrastructure Manager with High Performance Software Switch Lagopus

Hirofumi IchiharaNTT Software Innovation Center

Collaborator: Tomoya Hibi(NTT), Hiroki Kumazaki(NTT)

2Copyright©2015 NTT corp. All Rights Reserved.

• NFV• NFV Infrastructure Manager• OpenStack• Performance requirement for NFV• OpenvSwitch and Lagopus• Lagopus vs OVS• Lagopus advantage• Demo(video)

Agenda


Hirofumi Ichihara

• Engineer at NTT Software Innovation Center

• OpenStack developer•  Neutron and Devstack contributor in OpenStack projects

Who


• NFV(Network Function Virtualization)

What is NFV?

ref: https://portal.etsi.org/nfv/nfv_̲white_̲paper.pdf

•  Rapid development

•  Reducing CAPEX

•  Reducing OPEX

•  Avoiding vendor lock in


ETSI NFV

Today main topic

ref: http://www.etsi.org/deliver/etsi_̲gs/nfv/001_̲099/002/01.01.01_̲60/gs_̲nfv002v010101p.pdf


Storage

• Manage infrastructure for VNF•  Compute•  Network•  Storage

• IaaS Softwares•  OpenStack•  CloudStack•  Eucalyptus•  VMware

NFV Infrastructure Manager

Hypervisor Network

Physical Switch

Virtual Network

VM VM

LRFW

VM

VM

LR

LVFW LR Logical Router

Logical VolumeFirewall

VM Virtual Machine

LV

LV

LV


OpenStack

• Cloud Operating System•  Provides API to users

• Multi Hypervisor support•  KVM, QEMU, Xen, VMware, Hyper-‐‑‒V, LXC and others

• Integrated many network appliance and storage appliance

Ref: http://www.openstack.org/software/


OpenStack Conceptual Architecture

Ref: http://docs.openstack.org/admin-‐‑‒guide-‐‑‒cloud/content/conceptual-‐‑‒architecture.html


OpenStack Release Schedule

2010 2011 2012 2013 2014

Majorrelease

Providedfunctions

▲Oct 21Austin

▲Feb 3Bexar

▲Apr 15Cactus

▲Sep 22Diablo

▲Apr 5Essex

▲Sep 27Folsom

▲Apr 4Grizzly

▲Oct 17Havana

▲Apr 1７Icehouse

NovaSwiftGlanceKeystoneHorizonCinderQuantum

NovaSwift

NovaSwiftGlance Virtual machine image

Object storage

Virtual machine NovaSwiftGlanceKeystoneHorizonCinderNeutronCeilometer

HeatVirtual environment provisioning

Metering/Monitoring

NovaSwiftGlanceKeystoneHorizonCinderNeutronCeilometer

HeatTrove

▲Oct 16Juno

NovaSwiftGlanceKeystoneHorizonCinderQuantum

Block storage

Database

NovaSwiftGlanceKeystoneHorizonWeb user interface

Authentication service

Virtual network

NovaSwiftGlanceKeystoneHorizonCinderNeutronCeilometer

HeatTroveSaharaHadoop

▲Apr 30Kilo

NovaSwiftGlanceKeystoneHorizonCinderNeutron

・・・・・・

• 1 release / half a year• Apr 30, 2015 Kilo release


• Provides API enables to define Virtual Machine

• Supporting a wide variety of virtualization technologies, including KVM, Xen, Docker, etc

OpenStack Nova

KVM Host Docker Host

VM VM

VM VM

Container

Container

ContainerVM VM


• Provides API enables to define virtual network• Neutron core resources

•  Network: Virtual network （L2 Switch）•  Subnet: Manage IP address assign to network•  Port: Port connected with network

• Service plugin resources•  Router(include Floating IP)•  Load balancer•  VPN•  Firewall

OpenStack Neutron

L2 Switch

Virtual Machine

Router

VPN Firewall

Load balancer

Tenant A Tenant B


• OpenStack has been designed for Data Center use cases

NFV History in OpenStack

Hong Kong Summit 2014

Past

Present

Telco and vendor expect to use OpenStack to build NFV infrastructure

They realizedOpenStack NOT NFV orchestrator for CARRIER

They didnʼ’t realized

Cloud Provider Telco Carrier

What is NFV requirement?


• Canʼ’t help failure• User must HA• A few network down• User gets angry• Guaranteed by service contract

• Sometimes Fail

Gaps between Cloud and Telco

• Must help failure• Provider must HA• Donʼ’t down network• Government gets angry too

• Guaranteed by law• 24/365 on Ready!

Cloud Provider Telco Carrier

What is NFV requirement?


• Performance•  Packet processing speed with short packet•  Low latency

• High Availability• Interface for management• Stable• Monitoring

•  Fault detection•  Security

• Fault tolerance

NFV Telco Requirements


• OpenStack has been designed for Data Center use cases

NFV History in OpenStack

Hong Kong Summit 2014

Past

Present

Telco and switch vendor expects to use OpenStack to build NFV infrastructure

They realizedOpenStack NOT NFV orchestrator for CARRIER

They didnʼ’t realized

After Summit Telco Working Group and NFV subteam was organized

Oct. 2014 OPNFV was organized

Juno, Kilorelease Some NFV requirements was merged in OpenStack


• Proposed in OpenStack•  VLAN aware VM•  VM Scheduler•  High Availability method•  Service chaining API•  Driver for network high performance

• Liberty summit sessions related to NFV•  41 sessions (my grep)•  OPNFV Days

OpenStack Activity for NFV


# of packet to be proceededfor 10Gbps with 1 CPU core

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

14,000,000

16,000,000

0 256 512 768 1024 1280

# of packets per seconds

Packet size (Byte)

Short packet 64Byte14.88 MPPS, 67.2 ns•  2Ghz: 134 clocks•  3Ghz: 201 clocks

Computer packet 1KByte1.2MPPS, 835 ns

•  2Ghz: 1670 clocks•  3Ghz: 2505 clocks


• OpenvSwitch• Multilayer software switch•  Supports OpenFlow 1.4 protocol•  Including DPDK implement•  Source code: https://github.com/openvswitch/ovs

• Lagopus•  High performance software switch with DPDK•  Supports OpenFlow 1.3 protocol•  Source code: https://github.com/lagopus/lagopus

OpenvSwitch/Lagopus


OpenFlow

OpenFlow controller

OpenFlow switch

Data-‐‑‒planeFlow Table

Control plane(Routing / switching)

Data-‐‑‒plane(ASIC, FPGA)

Control plane(routing/ switching)

Flow match action

Flow match action counter

OpenFlow Protocol

Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #)

Action (frame processing)(output port, drop,

modify/pop/push header)

Flow statistics(# of packet, byte size, flow duration,… )

counter

Conventional NW node OpenFlow

Flow Table#2

Flow Table#3

OpenFlow switch agent


high-‐‑‒performance user-‐‑‒space packet processing with Intel DPDK

NIC

sk_̲buf

Ethernet Driver API

Socket API

vswitch

packet buffer

Dataplane

1. Interrupt& DMA

2. system call (read)

Userspace

Kernel space

Driver

4. DMA

3. system call (write)

NIC

Ethernet Driver API

Socket API

vswitch

packet buffer

agentagent

1. DMA Write

DPDKdataplane

Userspace packet processing (Event-‐‑‒based)

DPDK apps (polling-‐‑‒based)

2. DMA READ


• Many rx interrupt causes overhead•  Polling-‐‑‒based packet receiving

• Lock with multiple thread causes bottleneck•  Thread local storage and lockless-‐‑‒queue

• High TLB miss rates causes slowing performance

•  Huge DTLB for miss of memory controller• Context switch and memory copy causes overhead

•  Direct data copy from NIC buffer to CPU•  Kernel stack bypass of network

Issues about high performancepacket processing on x86


• OpenFlow software switch •  Run on x86 server

• High performance packet processing• > 10Gbps

• Multiple protocol•  Conform OpenFlow1.3•  Protocol(MPLS, PBB) for WAN

• OSS

About Lagopus

OpenFlow Controller

Control Plane

OpenFlow Protocol

OpenFlow SwitchData Plane

Flow TableFlow Pattern Action

Flow Pattern Action Counter

CounterFlow Table#2

Flow Table#3

Flow Table#4


• OVS-‐‑‒DPDK master•  Back for DPDK 1.8.0•  commit 66cabc46ecc09eeae536277a0fc7d5e44836f845

• Lagopus v0.1.2• DPDK 1.8.0• System

•  CPU: Xeon E5-‐‑‒2697v2@ 2.70GHz(12 cores to switch)• Memory: 64GB(hugepages: 1GBx8)•  NIC: Intel X520 10GNIC

• OpenFlow Controller•  Ryu (https://github.com/osrg/ryu)

• Traffic Generator•  IXIA

Switch Performance Conditionscommit 543342a41cbceffaac30ace2c66b6e489eb359c8Author: Mark Kavanagh <[email protected]>Date: Mon Apr 20 12:37:14 2015 -‐‑‒0700 DPDK: add support for v2.0.0 Update relevant artifacts to add support for DPDK v2.0.0 -‐‑‒ INSTALL.DPDK.md -‐‑‒ travis build script -‐‑‒ acinclude.m4: add 'mssse3' flag to OVS_̲CFLAGS -‐‑‒ netdev-‐‑‒dpdk: fix build with unified offload types in DPDK v2.0.0 Note that this breaks compatibility with DPDK v1.8.0


diff -‐‑‒-‐‑‒git a /config /common_̲linuxapp b /config /common_̲linuxappindex 2f9643b..78738dc 100644-‐‑‒-‐‑‒-‐‑‒ a /config /common_̲linuxapp+++ b /config /common_̲linuxapp@@ -‐‑‒81,7 +81,7 @@ CONFIG_̲RTE_̲BUILD_̲SHARED_̲LIB=n # # Combine to one single l ibrary #-‐‑‒CONFIG_̲RTE_̲BUILD_̲COMBINE_̲LIBS=n+CONFIG_̲RTE_̲BUILD_̲COMBINE_̲LIBS=y CONFIG_̲RTE_̲LIBNAME="intel_̲dpdk"

#@@ -‐‑‒160,7 +160,7 @@ CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲DEBUG_̲TX_̲FREE=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲DEBUG_̲DRIVER=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲PF_̲DISABLE_̲STRIP_̲CRC=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲RX_̲ALLOW_̲BULK_̲ALLOC=y-‐‑‒CONFIG_̲RTE_̲IXGBE_̲INC_̲VECTOR=y+CONFIG_̲RTE_̲IXGBE_̲INC_̲VECTOR=n CONFIG_̲RTE_̲IXGBE_̲RX_̲OLFLAGS_̲ENABLE=y

#@@ -‐‑‒372,7 +372,7 @@ CONFIG_̲RTE_̲KNI_̲VHOST_̲DEBUG_̲TX=n # fuse-‐‑‒devel is needed to run vhost. # fuse-‐‑‒devel enables user space char driver development #-‐‑‒CONFIG_̲RTE_̲LIBRTE_̲VHOST=n+CONFIG_̲RTE_̲LIBRTE_̲VHOST=y CONFIG_̲RTE_̲LIBRTE_̲VHOST_̲DEBUG=n

#

DPDK config


• Full route•  Set about 51,000 flow rule (IPv4)•  packets: 1flow, random ipv4_̲dst 100000flow

• Port VLAN•  Set port VLAN to two port (2 flow rule for round trip)•  packets: 1flow, random ipv4_̲dst 10000flow

Test scenario


Evaluation -‐‑‒ full route

64

512 1024 1500

012345678910

0 500 1000 1500

Gbps

byte/packet

Wire-‐‑‒rate

OVS-‐‑‒DPDK

Lagopus

• Lagopus wins with 1flow packets• OVS-‐‑‒DPDK wins with 100000flow packets

64

512 1024 1500

012345678910

0 500 1000 1500

Gbps

byte/packet

Wire-‐‑‒rate

OVS-‐‑‒DPDK

Lagopus

Result 1flow packets Result 100000flow packets


Evaluation – port VLAN

64

512 1024 1500

012345678910

0 500 1000 1500

Gbps

byte/packet

Wire-‐‑‒rate

OVS-‐‑‒DPDK

Lagopus

64

512 1024 1500

012345678910

0 500 1000 1500

Gbps

byte/packet

Wire-‐‑‒rate

OVS-‐‑‒DPDK

Lagopus

• Lagopus wins with 1flow packets• Lagopus wins with 10000flow packets

Result 1flow packets Result 10000flow packets


• Lagopus has some advantages

• Telco friendly network protocol•  QinQ, PBB, MPLS

• Comfortable more OpenFlow protocols than OVS

•  Important protocol is Meter for NFV• OVS is used by general services but lagopus is designed for network services

Lagopus vs OVS-‐‑‒DPDK


OpenFlow 1.3 Conformance StatusType Action Set field Match Group Meter Total

# of test scenario (mandatory,

optional)

56 (3 , 53)

161 (0 , 161)

714 (108 , 606)

15 (3 , 12)

30 (0 , 30)

991 (114 , 877)

Lagopus 2015.3.19

56 (3, 53)

161 (0, 161)

714 (108, 606)

15 (3, 12)

30 (0, 30)

976 (114, 862)

OVS (kernel) 2014.08.08

34 (3, 31)

96 (0, 96)

534 (108, 426)

6 (3, 3)

0 (0, 0)

670 (114, 556)

OVS (netdev) 2014.11.05

34 (3, 31)

102 (0, 102)

467 (93, 374)

8 (3, 5)

0 (0, 0)

611 (99, 556)

IVS 2015.02.11

17 (3, 14)

46 (0, 46)

323 (108, 229)

3 (0, 2)

0 (0, 0)

402 (111, 291)

ofswitch 2015.01.08

50 (3, 47)

100 (0, 100)

708 (108, 600)

15 (3, 12)

30 (0, 30)

962 (114, 848)

LINC 2015.01.29

24 (3, 21)

68 (0, 68)

428 (108, 320)

3 (3, 0)

4 (0, 4)

523 (114, 409)

Trema 2014.11.28

50 (3, 47)

159 (0 , 159)

708 (108, 600)

15 (3, 12)

34 (0, 34)

966 (114, 854)

http://osrg.github.io/ryu/certification.html


• System•  Intel(R) Core(TM) i7-‐‑‒4790 CPU @ 3.60GHz• Memory 16GB(hugepages: 1GBx8)•  NIC Intel X520 10GNIC

• Traffic Generator•  Pktgen DPDK(master 2015.4.28)

Meter Demo

LagopusData PlaneFlow Table

Flow Pattern Action

in:port1 Meter 100000kbps out:port2

Pktgen DPDK

port1 port2


• How do we setup lagopus for NFV?

• Lagopus is designed as network switch not switch for VM on hypervisor

• Big issue•  Canʼ’t add/delete port•  OpenFlow controller essential•  Canʼ’t try easily

Using Lagopus for NFV


• Made by me + other developers•  Tomoya Hibi: Lagopus developer•  Hiroki Kumazaki: Lock free master

• What is Gondola?• Manage KVM and Docker with Hybrid tenants and support for various tunneling protocols(VLAN, MPLS)

•  Try to run Lagopus as Hypervisor switch easily•  Exhibited in SDN/Cloud Program Contest 2014 Okinawa

• Not OSS yet

Gondola


• Using etcd for DB and notification

• Master node•  REST/GUI•  Scheduler•  Alive monitoring • Manage agent nodes

• Agent node• Manage VMs and containers

Gondola Architecture

HV（KVM+Docker） HV（KVM+Docker）

gondola gondola

HV（KVM+Docker） HV（KVM+Docker）

gondola master

HV（KVM+Docker） lagopus

Container VM

Ryu Libvirt API NW IF API

REST API

VM

management db

etcd

Docker API gondola agent

DC Network

Virtual/Phisical port

Physical machine

GUI User App

heart beat

change notification

write

Network

Gondola SW App.


• Demo1•  Create VM/Container and ping/ssh between them•  Isolation between multi-‐‑‒tenant networks

• Demo2•  OpenStack Integration

Demo video

Host (ubuntu02)

kvm-‐‑‒test192.168.0.10

Host (ubuntu03)

docker-‐‑‒test192.168.0.5

lagopus lagopus

docker-‐‑‒test2192.168.0.5

Host (ubuntu02)

docker-‐‑‒kvm

Host (ubuntu03)

docker-‐‑‒openstacklagopus lagopus

kvm-‐‑‒openstack2OpenStack API/GUI

mistake -> kvm-openstack �


• OpenStack as NFV Infrastructure Manager

• NFV requirements discussion

• DPDK for high performance packet processing

• Advantage of Lagopus

Summary

Documents

NFVInfrastructureManagerwith HighPerformanceSoftwareSwitch ... · • KVM,QEMU,Xen,VMware,Hyper V,LXCandothers • Integrated many network appliance and storageappliance Ref: