Copyright©2015 NTT corp. All Rights Reserved.
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hiorufmi IchiharaNTT Software Innovation Center
Collaborator: Tomoya Hibi(NTT), Hiroki Kumazaki(NTT)
2Copyright©2015 NTT corp. All Rights Reserved.
• NFV• NFV Infrastructure Manager• OpenStack• Performance requirement for NFV• OpenVSwtich and Lagopus• Lagopus vs OVS• Lagopus advantage• Demo(video)
Agenda
3Copyright©2015 NTT corp. All Rights Reserved.
Hirofumi Ichihara
• Engineer at NTT Software Innovation Center
• OpenStack developer• Neutron and Devstack contributor in OpenStack projects
Who
4Copyright©2015 NTT corp. All Rights Reserved.
• NFV(Network Function Virtialization)
What is NFV?
ref: https://portal.etsi.org/nfv/nfv_̲white_̲paper.pdf
• Rapid development
• Reducing CAPEX
• Reducing OPEX
• Avoiding vendor lock in
5Copyright©2015 NTT corp. All Rights Reserved.
ETSI NFV
Today main topic
ref: http://www.etsi.org/deliver/etsi_̲gs/nfv/001_̲099/002/01.01.01_̲60/gs_̲nfv002v010101p.pdf
6Copyright©2015 NTT corp. All Rights Reserved.
Storage
• Manage infrastructure for VNF• Compute• Network• Storage
• IaaS Softwares• OpenStack• CloudStack• Eucalyptus• VMware
NFV Infrastructure Manager
Hypervisor Network
Physical Switch
Virtual Network
VM VM
LRFW
VM
VM
LR
LVFW LR Logical Router
Logical VolumeFirewall
VM Virtual Machine
LV
LV
LV
7Copyright©2015 NTT corp. All Rights Reserved.
OpenStack
• Cloud Operating System• Provides API to users
• Multi Hypervisor support• KVM, QEMU, Xen, VMware, Hyper-‐‑‒V, LXC and others
• Integrated many network appliance and storage appliance
Ref: http://www.openstack.org/software/
8Copyright©2015 NTT corp. All Rights Reserved.
OpenStack Conceptual Architecture
Ref: http://docs.openstack.org/admin-‐‑‒guide-‐‑‒cloud/content/conceptual-‐‑‒architecture.html
9Copyright©2015 NTT corp. All Rights Reserved.
OpenStack Release Schedule
2010 2011 2012 2013 2014
Majorrelease
Providedfunctions
▲Oct 21Austin
▲Feb 3Bexar
▲Apr 15Cactus
▲Sep 22Diablo
▲Apr 5Essex
▲Sep 27Folsom
▲Apr 4Grizzly
▲Oct 17Havana
▲Apr 17Icehouse
NovaSwiftGlanceKeystoneHorizonCinderQuantum
NovaSwift
NovaSwiftGlance Virtual machine image
Object storage
Virtual machine NovaSwiftGlanceKeystoneHorizonCinderNeutronCeilometer
HeatVirtual environment provisioning
Metering/Monitoring
NovaSwiftGlanceKeystoneHorizonCinderNeutronCeilometer
HeatTrove
▲Oct 16Juno
NovaSwiftGlanceKeystoneHorizonCinderQuantum
Block storage
Database
NovaSwiftGlanceKeystoneHorizonWeb user interface
Authentication service
Virtual network
NovaSwiftGlanceKeystoneHorizonCinderNeutronCeilometer
HeatTroveSaharaHadoop
▲Apr 30Kilo
NovaSwiftGlanceKeystoneHorizonCinderNeutron
・・・・・・
• 1 release / half a year• Apr 30, 2015 Kilo release
10Copyright©2015 NTT corp. All Rights Reserved.
• Provides API enables to define Virtual Machine
• Supporting a wide variety of virtualization technologies, including KVM, Xen, Docker, etc
OpenStack Nova
KVM Host Docker Host
VM VM
VM VM
Container
Container
ContainerVM VM
11Copyright©2015 NTT corp. All Rights Reserved.
• Provides API enables to define virtual network• Neutron core resources
• Network: Virtual netowork (L2 Switch)• Subnet: Manage IP address assign to network• Port: Port connected with network
• Service plugin resources• Router(include Floating IP)• Load balancer• VPN• Firewall
OpenStack Neutron
L2 Switch
Virtual Machine
Router
VPN Firewall
Load balancer
Tenant A Tenant B
12Copyright©2015 NTT corp. All Rights Reserved.
• OpenStack has been designed for Data Center use cases
NFV History in OpenStack
Hong Kong Summit 2014
Past
Present
Telco and vendor expect to use OpenStack to build NFV infrastructure
They realizedOpenStack NOT NFV orchestrator for CARRIER
They didnʼ’t realized
Cloud Provider Telco Carrier
What is NFV requirement?
13Copyright©2015 NTT corp. All Rights Reserved.
• Canʼ’t help failure• User must HA• A few network down• User gets angry• Guaranteed by service contract
• Sometimes Fail
Gaps between Cloud and Telco
• Must help failure• Provider must HA• Donʼ’t down network• Government gets angry too
• Guaranteed by law• 24/365 on Ready!
Cloud Provider Telco Carrier
What is NFV requirement?
14Copyright©2015 NTT corp. All Rights Reserved.
• Performance• Packet processing speed with short packet• Low latency
• High Availability• Interface for management• Stable• Monitoring
• Fault detection• Security
• Fault tolerance
NFV Telco Requirements
15Copyright©2015 NTT corp. All Rights Reserved.
• OpenStack has been designed for Data Center use cases
NFV History in OpenStack
Hong Kong Summit 2014
Past
Present
Telco and switch vendor expects to use OpenStack to build NFV infrastructure
They realizedOpenStack NOT NFV orchestrator for CARRIER
They didnʼ’t realized
After Summit Telco Working Group and NFV subteam was organized
Oct. 2014 OPNFV was organized
Juno, Kilorelease Some NFV requirements was merged in OpenStack
16Copyright©2015 NTT corp. All Rights Reserved.
• Proposed in OpenStack• VLAN aware VM• VM Scheduler• High Availability method• Service chaining API• Driver for network high performance
• Liberty summit sessions related to NFV• 41 sessions (my grep)• OPNFV Days
OpenStack Activity for NFV
17Copyright©2015 NTT corp. All Rights Reserved.
# of packet to be proceededfor 10Gbps with 1 CPU core
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
14,000,000
16,000,000
0 256 512 768 1024 1280
# of packets per seconds
Packet size (Byte)
Short packet 64Byte14.88 MPPS, 67.2 ns• 2Ghz: 134 clocks• 3Ghz: 201 clocks
Computer packet 1KByte1.2MPPS, 835 ns
• 2Ghz: 1670 clocks• 3Ghz: 2505 clocks
18Copyright©2015 NTT corp. All Rights Reserved.
• OpenvSwitch• Multilayer software switch• Supports OpenFlow 1.4 protocol• Including DPDK implement• Source code: https://github.com/openvswitch/ovs
• Lagopus• High performance software switch with DPDK• Supports OpenFlow 1.3 protocol• Source code: https://github.com/lagopus/lagopus
OpenvSwitch/Lagopus
19Copyright©2015 NTT corp. All Rights Reserved.
OpenFlow
OpenFlow controller
OpenFlow switch
Data-‐‑‒planeFlow Table
Control plane(Routing / swithching)
Data-‐‑‒plane(ASIC, FPGA)
Control plane(routing/ switching)
Flow match action
Flow match action counter
OpenFlow Protocol
Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #)
Action (frame processing)(output port, drop,
modify/pop/push header)
Flow statistics(# of packet, byte size, flow duration,… )
counter
Conventional NW node OpenFlow
Flow Table#2
Flow Table#3
OpenFlow switch agent
20Copyright©2015 NTT corp. All Rights Reserved.
high-‐‑‒performance user-‐‑‒space packet processing with Intel DPDK
NIC
sk_̲buf
Ethernet Driver API
Socket API
vswitch
packet buffer
Dataplane
1. Interrupt& DMA
2. system call (read)
Userspace
Kernel space
Driver
4. DMA
3. system call (write)
NIC
Ethernet Driver API
Socket API
vswitch
packet buffer
agentagent
1. DMA Write
DPDKdataplane
Userspace packet processing (Event-‐‑‒based)
DPDK apps (polling-‐‑‒based)
2. DMA READ
21Copyright©2015 NTT corp. All Rights Reserved.
• Many rx interrupt causes overhead• Polling-‐‑‒based packet receiving
• Lock with multiple thread causes bottleneck• Thread local storage and lockless-‐‑‒queue
• High TLB miss rates causes slowing performance
• Huge DTLB for miss of memory controller• Context switch and memory copy causes overhead
• Direct data copy from NIC buffer to CPU• Kernel stack bypass of network
Issues about high performancepacket processing on x86
22Copyright©2015 NTT corp. All Rights Reserved.
• OpenFlow software switch • Run on x86 server
• High performance packet processing• > 10Gbps
• Multiple protocol• Conform OpenFlow1.3• Protocol(MPLS, PBB) for WAN
• OSS
About Lagopus
OpenFlow Controller
Control Plane
OpenFlow Protocol
OpenFlow SwitchData Plane
Flow TableFlow Pattern Action
Flow Pattern Action Counter
CounterFlow Table#2
Flow Table#3
Flow Table#4
23Copyright©2015 NTT corp. All Rights Reserved.
• OVS-‐‑‒DPDK master• Back for DPDK 1.8.0• commit 66cabc46ecc09eeae536277a0fc7d5e44836f845
• Lagopus v0.1.2• DPDK 1.8.0• System
• CPU Intel(R) Xeon(R) CPU E5-‐‑‒2697 v2 @ 2.70GHz• Memory 64GB(fugapages: 1GBx8)• NIC Intel X520 10GNIC
• OpenFlow Controller• Ryu (https://github.com/osrg/ryu)
• Traffic Generator• IXIA
Switch Performance Conditionscommit 543342a41cbceffaac30ace2c66b6e489eb359c8Author: Mark Kavanagh <[email protected]>Date: Mon Apr 20 12:37:14 2015 -‐‑‒0700 DPDK: add support for v2.0.0 Update relevant artifacts to add support for DPDK v2.0.0 -‐‑‒ INSTALL.DPDK.md -‐‑‒ travis build script -‐‑‒ acinclude.m4: add 'mssse3' flag to OVS_̲CFLAGS -‐‑‒ netdev-‐‑‒dpdk: fix build with unified offload types in DPDK v2.0.0 Note that this breaks compatibility with DPDK v1.8.0
24Copyright©2015 NTT corp. All Rights Reserved.
diff -‐‑‒-‐‑‒git a /config /common_̲linuxapp b /config /common_̲linuxappindex 2f9643b..78738dc 100644-‐‑‒-‐‑‒-‐‑‒ a /config /common_̲linuxapp+++ b /config /common_̲linuxapp@@ -‐‑‒81,7 +81,7 @@ CONFIG_̲RTE_̲BUILD_̲SHARED_̲LIB=n # # Combine to one single l ibrary #-‐‑‒CONFIG_̲RTE_̲BUILD_̲COMBINE_̲LIBS=n+CONFIG_̲RTE_̲BUILD_̲COMBINE_̲LIBS=y CONFIG_̲RTE_̲LIBNAME="intel_̲dpdk"
#@@ -‐‑‒160,7 +160,7 @@ CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲DEBUG_̲TX_̲FREE=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲DEBUG_̲DRIVER=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲PF_̲DISABLE_̲STRIP_̲CRC=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲RX_̲ALLOW_̲BULK_̲ALLOC=y-‐‑‒CONFIG_̲RTE_̲IXGBE_̲INC_̲VECTOR=y+CONFIG_̲RTE_̲IXGBE_̲INC_̲VECTOR=n CONFIG_̲RTE_̲IXGBE_̲RX_̲OLFLAGS_̲ENABLE=y
#@@ -‐‑‒372,7 +372,7 @@ CONFIG_̲RTE_̲KNI_̲VHOST_̲DEBUG_̲TX=n # fuse-‐‑‒devel is needed to run vhost. # fuse-‐‑‒devel enables user space char driver development #-‐‑‒CONFIG_̲RTE_̲LIBRTE_̲VHOST=n+CONFIG_̲RTE_̲LIBRTE_̲VHOST=y CONFIG_̲RTE_̲LIBRTE_̲VHOST_̲DEBUG=n
#
DPDK config
25Copyright©2015 NTT corp. All Rights Reserved.
• Full route• Set about 51,000 flow rule (IPv4)• packets: 1flow, random ipv4_̲dst 100000flow
• Port VLAN• Set port VLAN to two port (2 flow rule for round trip)• packets: 1flow, random ipv4_̲dst 10000flow
Test scenario
26Copyright©2015 NTT corp. All Rights Reserved.
Evaluation -‐‑‒ full route
64
512 1024 1500
012345678910
0 500 1000 1500
Gbps
byte/packet
Wire-‐‑‒rate
OVS-‐‑‒DPDK
Lagopus
• Lagopus wins with 1flow packets• OVS-‐‑‒DPDK wins with 100000flow packets
64
512 1024 1500
012345678910
0 500 1000 1500
Gbps
byte/packet
Wire-‐‑‒rate
OVS-‐‑‒DPDK
Lagopus
Result 1flow packets Result 100000flow packets
27Copyright©2015 NTT corp. All Rights Reserved.
Evaluation – port VLAN
64
512 1024 1500
012345678910
0 500 1000 1500
Gbps
byte/packet
Wire-‐‑‒rate
OVS-‐‑‒DPDK
Lagopus
64
512 1024 1500
012345678910
0 500 1000 1500
Gbps
byte/packet
Wire-‐‑‒rate
OVS-‐‑‒DPDK
Lagopus
• Lagopus wins with 1flow packets• Lagopus wins with 10000flow packets
Result 1flow packets Result 10000flow packets
28Copyright©2015 NTT corp. All Rights Reserved.
• Lagopus has some advantages
• Telco friendly network protocol• QinQ, PBB, MPLS
• Comfortable more OpenFlow protocols than OVS
• Important protocol is Meter for NFV• OVS is used by general services but lagopus is designed for network services
Lagopus vs OVS-‐‑‒DPDK
29Copyright©2015 NTT corp. All Rights Reserved.
OpenFlow 1.3 Conformance StatusType Action Set field Match Group Meter Total
# of test scenario (mandatory,
optional)
56 (3 , 53)
161 (0 , 161)
714 (108 , 606)
15 (3 , 12)
30 (0 , 30)
991 (114 , 877)
Lagopus 2015.3.19
56 (3, 53)
161 (0, 161)
714 (108, 606)
15 (3, 12)
30 (0, 30)
976 (114, 862)
OVS (kernel) 2014.08.08
34 (3, 31)
96 (0, 96)
534 (108, 426)
6 (3, 3)
0 (0, 0)
670 (114, 556)
OVS (netdev) 2014.11.05
34 (3, 31)
102 (0, 102)
467 (93, 374)
8 (3, 5)
0 (0, 0)
611 (99, 556)
IVS 2015.02.11
17 (3, 14)
46 (0, 46)
323 (108, 229)
3 (0, 2)
0 (0, 0)
402 (111, 291)
ofswitch 2015.01.08
50 (3, 47)
100 (0, 100)
708 (108, 600)
15 (3, 12)
30 (0, 30)
962 (114, 848)
LINC 2015.01.29
24 (3, 21)
68 (0, 68)
428 (108, 320)
3 (3, 0)
4 (0, 4)
523 (114, 409)
Trema 2014.11.28
50 (3, 47)
159 (0 , 159)
708 (108, 600)
15 (3, 12)
34 (0, 34)
966 (114, 854)
http://osrg.github.io/ryu/certification.html
30Copyright©2015 NTT corp. All Rights Reserved.
• System• Intel(R) Core(TM) i7-‐‑‒4790 CPU @ 3.60GHz• Memory 16GB(fugapages: 1GBx8)• NIC Intel X520 10GNIC
• Traffic Generator• Pktgen DPDK(master 2015.4.28)
Meter Demo
LagopusData PlaneFlow Table
Flow Pattern Action
in:port1 Meter 100000kbps out:port2
Pktgen DPDK
port1 port2
31Copyright©2015 NTT corp. All Rights Reserved.
• How do we setup lagopus for NFV?
• Lagopus is designed as network switch not switch for VM on hypervisour
• Big issue• Canʼ’t add/delete port• OpenFlow controller essential• Canʼ’t try easily
Using Lagopus for NFV
32Copyright©2015 NTT corp. All Rights Reserved.
• Made by me + other developers• Tomoya Hibi: Lagopus developer• Hiroki Kumazaki: Lock free master
• What is Gondola?• Manage KVM and Docker with Hybrid tenants and support for various tunneling protocols(VLAN, MPLS)
• Try to run Lagopus as Hypervisor switch easily• Exhibited in SDN/Cloud Program Contest 2014 Okinawa
• Not OSS yet
Gondola
33Copyright©2015 NTT corp. All Rights Reserved.
• Using etcd for DB and notification
• Master node• REST/GUI• Scheduler• Alive monitoring • Manage agent nodes
• Agent node• Manage VMs and containers
Gondola Architecture
HV(KVM+Docker) HV(KVM+Docker)
gondola gondola
HV(KVM+Docker) HV(KVM+Docker)
gondola master
HV(KVM+Docker) lagopus
Container VM
Ryu Libvirt API NW IF API
REST API
VM
management db
etcd
Docker API gondola agent
DC Network
Virtual/Phisical port
Physical machine
GUI User App
heart beat
change notification
write
Network
Gondola SW App.
34Copyright©2015 NTT corp. All Rights Reserved.
• Demo1• Create VM/Container and ping/ssh between them• Isolation between multi-‐‑‒tenant networks
• Demo2• OpenStack Integration
Demo video
Host (ubuntu02)
kvm-‐‑‒test192.168.0.10
Host (ubuntu03)
docker-‐‑‒test192.168.0.5
lagopus lagopus
docker-‐‑‒test2192.168.0.5
Host (ubuntu02)
docker-‐‑‒kvm
Host (ubuntu03)
docker-‐‑‒openstacklagopus lagopus
kvm-‐‑‒openstack2OpenStack API/GUI
mistake -> kvm-openstack �
35Copyright©2015 NTT corp. All Rights Reserved.
• OpenStack as NFV Infrastructure Manager
• NFV requirements discussion
• DPDK for high performance packet processing
• Advantage of Lagopus
Summary