Software Stacks to enable SDN and NFV

Software Stacks to enable Software-Defined Networking and Network Functions VirtualizationYoshihiro Nakajima <nakajima.yoshihiro@lab.ntt.co.jp, ynaka@lagopus.org>NTT Network Innovation Laboratories

Yoshihiro NakajimaWork for Nippon Telegraph and Telephone Corporation,

R&D DivisionNetwork Innovation Laboratories

Project lead of Lagopus SDN software switchBackground

• High performance computing• High performance networking and data processsing

About me

TrendsSoftware-Defined Networking (SDN)Network Functions Virtualization (NFV)

DataplaneLagopus: SDN/OpenFlow software switch

• Overview and NFV• Trials

Controller/C-planeO3 projectRyu and gobgpZebra2

Future plan

Agenda

Network is so stuck….

Many technologies has emerged in system development areaLanguage, Debugger, Testing framework, Continuous Integration,

Continuous Deployment….

Network area …CLI with vender format CLI with serial or telnet Expect script

Good for Human,But not for software

Trend shift in networking

Closed (Vender lock-in)

Yearly dev cycle

Waterfall dev

Standardization

Protocol

Special purpose HW / appliance

Distributed cntrl

Custom ASIC / FPGA

Open (lock-in free)

Monthly dev cycle

Agile dev

DE fact standard

Commodity HW/ Server

Logically centralized cntrl

Merchant Chip

What is Software-Defined Networking?6

Innovate services and applications in software development speed!

Reference: http://opennetsummit.org/talks/ONS2012/pitt-mon-ons.pdf

Decouple control plane and data plane→Free control plane out of the box（ APIs: OpenFlow, P4, …）

Logically centralized view→Hide and abstract complexity of networks, provide entire view of the network

Programmability via abstraction layer→Enables flexible and rapid service/application development

SDN conceptual model

Why SDN?7

Differentiate services (Innovation)• Increase user experience• Provide unique service which is not in the market

Time-to-Market（ Velocity）•Not to depend on vendors’ feature roadmap•Develop necessary feature when you need

Cost-efficiency•Reduce OPEX by automated workflows•Reduce CAPEX by COTS hardware

Open-Source Cloud Computing

Open Northbound APIs

Open-Source Controller

Open-Source Hardware

Open Southbound Protocol

Open-Source Switch Software

Infrastructure Layer

Application Layer

Business Applications

Control LayerNetwork Services

Network Services

API API API

Open source network stacks for SDN/NFV

SDN/NFV history

201020092008 2011 2012 2013　　　　　　　　　　　　　　　　OpenFlow deveopment

Open Networking Foundation

ETSI NFV

OpenDaylight

OpenFlow Switch Consortium

Stanford UniversityClean Slate Program

SDNactivity

Standarlization OpenFlow

trema RyuOpenDaylight

OF1.1 OF1.2OpenFlowprotocols OF1.4OF 0.8 OF 0.9 OF 1.0

NOXLagopus

History of software vswitch/router and I/O library1995 2000 2005 2010 2015

DPDKR1.0 OSS LF join

2012 BT vBRAS demo

Research phaseInternal product phaseOSS phase

netmap

ClickBESS

OVSNicira OVN

Lagopus

Network Functions Virtualization

Replace dedicated network nodes to virtual applianceRuns on general servers

Leverage cloud provisioning andmonitoring system for managementVirtual network function (VNF) may be run on

virtual machine or baremetal server

Evaluate the benefits of SDN by implementing our control plane and

switch

High performance network I/O for all packet sizesEspecially in smaller packet size (< 256 bytes)

Low-latency and less-jitterNetwork I/O & Packet processing

IsolationPerformance isolation between NFV VMs Security-related VM-to-VM isolation from untrusted apps

Reliability, availability and serviceability (RAS) function for long-term operation

NFV requirements from 30,000 feet

Still poor performance of NFV apps Lower network I/O performanceBig processing latency and big jitter

Limited deployment flexibility SR-IOV has limitation in performance and configurationCombination of DPDK apps on guest VM and DPDK-enabled vSwitch

is configuration Limited operational support

DPDK is good for performance, but has limited dynamic reconfiguration

Maintenance features are not realized

What’s matter in NFV

Not enough performance Packet processing speed < 1Gbps 10K flow entry add > two hours Flow management processing is too heavy

Develop & extension difficulties Many abstraction layer cause confuse for me

• Interface abstraction, switch abstraction, protocol abstraction Packet abstraction…

Many packet processing codes exists for the same processing • Userspace, Kernelspace,…

Invisible flow entries cause chaos debugging

Existing vswitch is @2013

vSwitch requirement from user side

Run on the commodity PC server and NIC Provide a gateway function to allow connect different

various network domainsSupport of packet frame type in DC, IP-VPN, MPLS and access NW

Achieve 10Gbps-wire rate with >= 1M flow ruleslow-latency packet processingflexible flow lookup using multiple-tablesHigh performance flow rule setup/delete

Run in userland and decrease tight-dependency to OS kerneleasy software upgrade and deployment

Support various management and configuration protocols.

Lagopus: High-performance SDN/OpenFlow Software Switch

Goal of Lagopus project

Provide NFV/SDN-aware switch software stackProvide dataplane API with OpenFlow protocol and gRPC100Gbps-capable high-performance software dataplaneDPDK extension for carrier requirementsCloud middleware adaptation

Expand software-based packet processing to carrier networks

Lagopus is a small genus of birds in the grouse subfamily, commonly known as ptarmigans. All living in tundra or cold upland areas.Reference: http://en.wikipedia.org/wiki/Lagopus

What is Lagopus (雷鳥属 )?

Provide High performance software switch on Intel CPUOver-100Gbps wire-rate packet processing / portHigh-scalable flows handling

Expands SDN idea to many network domain Datacenter, NFV environment, mobile networkVarious management /configuration interfaces

Target of Lagopus switch

Virtual Switch

Hypervisor

Virtual Switch

Hypervisor

NFV NFV

Virtual Switch

Hypervisor

Gateway CPE

Data Center Wide-area Network Access Network Intranet

Open Source High performance SDN software switchMulticore-CPU-aware packet processing with DPDKSupports NFV environmentRuns on Linux and FreeBSD

Best OpenFlow 1.3 compliant software switch by Ryu certificationMany protocol frame matches and actions support

• Ethernet, VLAN, MPLS, PBB, IPv4, IPv6, TCP, UDP, VxLAN, GRE, GTPMultiple-Flow table, Group table, meter table1M flow entries handling (4K flow mod/sec)Over-40Gbps-class packet processing (20MPPS)

Open source under Apache v2 licensehttp://lagopus.github.io/

What is Lagopus SDN software switch

0 256 512 768 1024 12800

2,000,0004,000,0006,000,0008,000,000

10,000,00012,000,00014,000,00016,000,000

Packet size (Byte)

How many packets to be proceeded for 10Gbps

Short packet 64Byte14.88 MPPS, 67.2 ns• 2Ghz: 134 clocks• 3Ghz: 201 clocks

Computer packet 1KByte1.2MPPS, 835 ns• 2Ghz: 1670 clocks• 3Ghz: 2505 clocks

L1 cache access: 4 clocksL2 cache access: 12 clocksL3 cache access: 44 clocksMain memory: 100 clocksMax PPS between cores: 20MPPS

PC architecture and limitation

CPU CPUMemory Memory

NICNIC

PCI-Exp PCI-Exp

Reference: supermicro X9DAi

L2 forwarding

IP routing

L2-L4 classification

TCP termination

Simple OF processing

0 1000 2000 3000 4000 5000 6000# of required CPU cycles

# of CPU cycle for typical packet processing

10Gbps 1Gbps

Simple is better for everythingPacket processingProtocol handling

Straight forward approachFull scratch (No use of existing vSwitch code)

User land packet processing as much as possible, keep kernel code smallKernel module update is hard for operation

Every component & algorithm can be replaced

Approach for vSwitch development

What is Lagopus vSwitch

switch configuration datastore(config/stats API, SW DSL)

None-DPDK NIC DPDK NIC/vNIC

DPDK libs/PMD driver

Lagopus soft dataplane

flow lookup flow cache

OpenFlow pipeline

queue/policer

Flow tableFlow table

flow tableFlow table

Flow tableGrouptable

Flow tableFlow tablemetertable

switch HAL

OpenFlow 1.3 agent

JSON IF

stackAgent

SDN switch Agent• Full OpenFlow 1.3 support• Controller-less basic L2 and

L3 support with action_normal

SDN-aware management API• JSON-based control• Ansible support

DPDK-enabled OpenFlow-aware software dataplane•Over-10-Gbps performance•Low latency packet processing•high performance multi-layer flow lookup•Cuckoo hash for flow cache

Switch configuration datastore• Pub/sub mechanism• Switch config DSL• JSON-based control

OS NIFVarious I/O support• DPDK-enabled NIC• Standard NIC with raw socket• tap

Virtualization support• QEMU/KVM• Vhost-user• DPDK-enabled VNF

General packet processing on UNIX

skb_buf

Ethernet Driver API

Socket API

vswitch

packet buffer

Data plane

User-space implementation(Event-triggered)

1. Interrupt& DMA

2. system call (read)

Userspace

Kernel space

Driver

4. DMA

3. system call (write)

Kernel-space implementation(Event-triggered)

skb_buf

Ethernet Driver API

Socket API

vswitch

packet buffer

1. Interrupt& DMA

vswitchData plane

agentagent

2. DMA

Contexts switch

MassiveInterrupt

Many memory copy / read

x86 architecture-optimized data-plane library and NIC driversMemory structure-aware queue, buffer

managementpacket flow classificationpolling mode-based NIC driver

Low-overhead & high-speed runtime optimized with data-plane processing

Abstraction layer for hetero server environments

BSD-license

Data Plane Development Kit (DPDK)

Ethernet Driver API

Socket API

DPDK apps

packet buffer

1. DMA Write

2. DMAREAD

DPDKdataplane

DPDK apps

What DPDK helps

Processing bypass for speed

skb_buf

Ethernet Driver API

Socket API

vswitch

packet buffer

packet buffermemory

Standard linux application

1. Interrupt & DMA

2. system call (read)

User space

Kernel space

Driver

4. DMA

3. system call (write)

Ethernet Driver API

User-mode I/O & HAL

vswitch

packet buffer

Application with intel DPDK

1. DMA Write 2. DMA READ

DPDK Library

Polling-basepacket handling

Event-basepacket handling

Implementation strategy for vSwitch

Massive RX interrupts handling for NIC device=> Polling-based packet receiving Heavy overhead of task switch=> Thread assignment (one thread/one physical CPU) Lower performance of PCI-Express I/O and memory

bandwidth compared with CPU=> Reduction of # of access in I/O and memory Shared data access is bottleneck between threads=> Lockless-queue, RCU, batch processing

Basic packet processing

Network I/O RX

packet

Frame processing

Flow lookup &Action

QoS ・ Queue

Network I/OTX

Packet classification &packet distribution to buffers

Packet parsing

lookup, Header rewriteEncap/decap

Policer, ShaperMarking

packet

What we did for performance

Network I/O RX

packet

Frame processing

Flow lookup &Action

QoS ・ Queue

Network I/OTX

packet

• Delayed packet frame evaluation

• Delayed action (processing) evaluation

• Packet batching to improve CPU $ efficiency

• Delayed flow stats evaluation

• Smart flow classification• Thread assignment optimization

• Parallel flow lookup• Lookup tree compaction• High-performance lookup

algorithm for OpenFlow(multi-layer, mask, priority-aware flow lookup)

• Flow $ mechanism

• Batch size tuning

Exploit many core CPUsReduce data copy & move (reference access)Simple packet classifier for parallel processing in I/O RX

Decouple I/O processing and flow processingImprove D-cache efficiency

Explicit thread assign to CPU core

Packet processing using multi core CPUs

NIC 1RX

NIC 2RX

I/O RXCPU0

I/O RXCPU1

NIC 1TX

NIC 2TX

I/O TXCPU6

I/O TXCPU7

Flow lookuppacket processing

NIC 3RX

NIC 4RX

NIC 3TX

NIC 4TX

NIC RX buffer

Ring buffer

Ring buffer NIC TX buffer

OpenFlow semantics includesMatch

• Protocol headers– Port #, Ethernet, VLAN, BPP, MAC-in-MAC, MPLS, IPv4, IPv6, ARP, ICMP, TCP,

UDP…– Mask-enabled

• PriorityAction

• Output port• Packet-in/Packet-out• Header rewrite, Header push, Header push

How OpenFlow match is so hard

Many header match

And GRE, L2TP, OSPF… Same as TCP, SCTP L3 header continues Includes IP and

other protocols

Linear search (first implementation)

eth_dsteth_srceth_type(ip_v, ip_hl)

ip_tos(ip_len)(ip_id, ip_off)

ip_ttlip_p(ip_sum)ip_srcip_dsttcp_srctcp_dst

Is eth_type0x0800?Is ip_p17?Is udp_dst53?

0x0800?

packet entry 0 entry 1

Is eth_type

Is ip_p

Is tcp_dst

0x0800?

entry 2

Is eth_type

Is ip_p

Is tcp_dst

0: Linear search (first implementation)

Is eth_type0x0800?Is ip_p17?Is udp_dst53?

0x0800?

packet entry 0 entry 1

Is eth_type

Is ip_p

Is tcp_dst

0x0800?

entry 2

Is eth_type

Is ip_p

Is tcp_dst

Simplify comparison routineFlow entry is composed by mask and valueStill linear search

1: bitmap comparison with bitmask

eth_dsteth_src0xffff(ip_v, ip_hl)

ip_ttl0xff(ip_sum)ip_srcip_dsttcp_src0xffff

eth_dsteth_src0x0800(ip_v, ip_hl)

ip_ttl6(ip_sum)ip_srcip_dsttcp_src80

packet mask flow entry

Compose search tree for dedicate fields to narrow lookup space

Then compare each fileds

2: Fixed-type search tree

eth_dsteth_srceth_type(ip_v, ip_hl)ip_tos(ip_len)(ip_id, ip_off)

packet

hash table

Linear sarch

Scan all flow entries and compose search tree with arrangement in order from the most frequently searched field

3: Search tree with most searched field first

packet

(ip_v, ip_hl)ip_tos(ip_len)(ip_id, ip_off)

ip_ttlip_p(ip_sum)ip_src

tcp_srctcp_dst

hash table

eth_dsteth_srceth_type

ip_dst

Linear search

Reduce # of lock in flow lookup tableFrequent locks are required

• Switch OpenFlow agent and counter retrieve for SNMP• Packet processing

Packet batching

Input packet

Lookup tableInput packet

Lookup table

Lock Unlock

ロック Unlock

●Naïve implementation

●Packet batching implementation

Lock is required for each packet

Lock can be reduced due to packet batching

Reduce # of flow lookup in multiple tableGenerate composed flow hash function that contains flow tablesIntroduce flow cache for each CPU core

Bypass pipeline with flow $

●Naïve implementation table1 table2 table3Input packet

table1 table2 table3

Input packet

Flow cache

1. New flow

2. Success flow

Output packet

Multiple flow $generation & mngmtWrite flow $

●Lagopus implementationOutput packet

Packet

Best OpenFlow 1.3 compliant switch

Type Action Set field Match Group Meter Total# of test scenario

(mandatory, optional)

56(3 , 53)

161(0 , 161)

714(108 , 606)

15(3 , 12)

36(0 , 36)

991(114 , 877)

Lagopus2014.11.09

59(3, 56)

161(0, 161)

714(108, 606)

15(3, 12)

34(0, 34)

980(114, 866)

OVS (kernel)2014.08.08

34(3, 31)

96(0, 96)

534(108, 426)

6(3, 3)

0(0, 0)

670(114, 556)

OVS (netdev)

2014.11.05

34(3, 31)

102(0, 102)

467(93, 374)

8(3, 5)

0(0, 0)

611(99, 556)

IVS2015.02.11

17(3, 14)

46(0, 46)

323(108, 229)

3(0, 2)

0(0, 0)

402(111, 291)

ofswitch2015.01.08

50(3, 47)

100(0, 100)

708(108, 600)

15(3, 12)

30(0, 30)

962(114, 848)

LINC2015.01.29

24(3, 21)

68(0, 68)

428(108, 320)

3(3, 0)

4(0, 4)

523(114, 409)

Trema2014.11.28

50(3, 47)

159(0 , 159)

708(108, 600)

15(3, 12)

34(0, 34)

966(114, 854)

SummaryThroughput: 10Gbps wire-rateFlow rules: 1M flow rules

Evaluation modelsWAN-DC gateway

• MPLS-VLAN mappingL2 switch

• Mac address switching

Performance Evaluation

Evaluation setup

Server spec. CPU: Dual Intel Xeon E5-2660

• 8 core(16 thread), 20M Cache, 2.2 GHz, 8.00GT/s QPI, Sandy bridge Memory: DDR3-1600 ECC 64GB

• Quad-channel 8x8GB Chipset: Intel C602 NIC: Intel Ethernet Converged Network Adapter X520-DA2

• Intel 82599ES, PCIe v2.0

Performance Evaluation

ServerLagopus

Flow table

tester Flows

Throughput (bps/pps/%)

Flowrule

Packet size

Flowcache

(on/off)

WAN-DC Gateway

Throughput vs packet size, 1 flow, flow-cache

Throughput vs flows, 1518 bytes packet

10000 IP subnet entries test

L3 forwarding performance with 40G NIC and

CPU E5-2667v3 3.20GHz x2Memory DDR4 64GBNIC Intel X710 x2OS Ubuntu 14.04LTSDPDK 2.2Lagopus 0.2.4 with default options

Full OpenFlow 1.3 support Limited OpenFlow 1.5 support (Flowmod-related instructions) General tunnel encap/decap extension (EXT-382 and EXT-566) support

• GRE, VxLAN, GTP, Ethernet, IPv4, IPv6• Updated draft will be implemented soon

Flexible & high-performance dataplane Hybrid-mode support

• ACTION_NORMAL (L2, L3) Various network I/O support

• DPDK NIC, vNIC (vhost-user with virtio-net), none-DPDK NIC (raw-socket) Leverage network stack in OS kernel for OpenFlow switch

• ARP, ICMP, Routing control packet are escalated to network stack (with tap IF) Queue, Meter table

Linux, FreeBSD, NetBSD support Virtualization support for NFV

DPDK-enabled VNF on DPDK-enabled vSwitch QEMU/KVM support through virsh

Lagopus version 0.2.10

Hands-on and seminar

Collaboration with Lagopus

Bussiness Research institutes and networks

Software switch collaboration

White box switch collaboration

High-performance vNIC framework for hypervisor-based NFV with userspace vSwitch

• To provide novel components to enable high-performance NFV with general propose hardware

• To provide high-performance vNIC with operation-friendly features

Issues on NFV middleware

Performance bottleneck in NFV with HV domain

Pkt recv / send cause

VM transition

HW emulation needs CPU cycle & VM transition

Privileged register accesses for vNIC

cause VM transition

System call cause context

switch on guest VM

VM transition: 800 CPU cycles

Use para-virtualization NIC frameworkNo full-virtualization (emulation-based)

Global-shared memory-based packet exchangeReduce memory copy

User-space-based packet data exchangeNo kernel-userspace packet data exchange

vNIC strategy for performance & RAS

DPDK apps or legacy apps on guest VM + userspace DPDK vSwitchConnected by shared memory-based vNICReduce OS kernel implementation

Target NFV architecture with hypervisor

Run in userspace to

avoid VM transition and context switch Memory based

packet transfer

Existing vNIC for u-vSW and guest VM (1/2)DPDK e1000 PMD with QEMU's e1000 FV andvSwitch connected by tap

DPDK virtio-net PV PMD with QEMU virtio-net framework andvSwitch connected by tap

DPDK virtio-net PV PMD with vhost-net framework and vSwitch connected by tap

Pros: legacy and DPDK support, opposite status detectionCons: bad performance, many VM transitions, context switch

Pros: legacy and DPDK support, opposite status detectionCons: Cons: bad performance, many VM transitions, context switch

Existing vNIC for u-vSW and guest VM (2/2)DPDK ring by QEMU IVSHMEM extension and vSwitch connected by shared memory

DPDK virtio-net PV PMD with QEMU virtio-net framework and vSwitch with DPDK vhost-user API to connect to virtio-net PMD.

Pros: Best performanceCons: only DPDK support, static configuration, no RAS

Pros: good performance, both support of legacy and DPDKCons: no status tracking of opposite device

High performance vNIC framework for NFV

This patch has been already merged to DPDK

High-Performance10-Gbps network I/O throughputNo virtualization transition between a guest VM and u-vSWSimultaneous support DPDK apps and DPDK u-vSW

Functionality for operationIsolation between NFV VM and u-vSWFlexible service maintenance support Link status notification on the both sides

Virtualization middleware supportSupport open source hypervisor (KVM) DPDK app and legacy app support No OS (kernel) modification on a guest VM

vNIC requirements for NFV with u-vSW

vNIC as an extension of virtio-net frameworkPara-virtualization network interfacePacket communication by global shared memoryOne packet copy to ensure VM-to-VM isolationControl msg by inter-process-communication between pseudo

devices

vNIC deisgn

User space on host

Kernel space on host

DPDK-enabled vSwitch

Software dataplane

Guest VM

Kernel space

User space

virtio-net-compatible device

DPDK NW apps

DPDK ETHDEV/virtio-net PMD

Global shared memory

DPDK ETHDEV / PMD

pseudo PMD-enabled device

IPC-basedcontrol

communication

Virtq-PMD driver: 4K LOC modificationVirtio-net device with DPDK extensionDPDK API and PV-based NIC (virtio-net) APIGlobal shared memory-based packet transmission on hugeTLBUNIX domain socket based control message

• Event notification (link-status, finalization)• Pooling-based the opposite device check mechanism

QEMU: 1K LOC modificationvirtio-net-ipc device on shared memory spaceShared memory-based device mapping

vNIC implementation

Performance

vhost application on host

Measurement point

virtio-net PMD

virtq PMD

testpmd on Guest VM

Null PMD

Vhost app

testpmd on host

Null PMD

Bare-metal configuration

virtq PMDMeasureme

nt point

Testpmd on host

Measurement pointpcap PMD

testpmd on Guest VM

Null PMD

virtq PMD

virtqueue

TAP driverVirtio-net

driver

Kernel-drivertestpmd on host

virtq PMD

Measurement point

virtio-net PMD

virtqueue

testpmd on Guest VM

Null PMD

virtq-pmd

micro benchmarking tool: Testpmd appsPolling-based DPDK bridge app that reads data from a NIC and

writes data to another NIC in both directions. null-PMD: a DPDK-enabled dummy PMD to allow packet generation

from memory buffer and packet discard to memory buffer

Performance benchmark

Performance evaluationMPPSGBPS

Virtq PMD achieved great performance 62.45 Gbps (7.36 MPPS) unidirectional throughput 122.90 Gbps (14.72 MPPS) bidirectional throughput 5.7 times faster than Linux driver in 64B, 2.8 times faster than Linux drvier in 1500B

Virtq PMD achieved better performance in large packet to vhost app

Container adaptation

Vhost-user for container

Vhost-user compatiblePMD for containerVirtio-net-based backendShared-memory-based

packet data exchangeEvent-trigger by shared file27.27 Gbps throughtput

Packet flowpktgen -> physical-> vswitch -> Container(L2Fwd) -> vswitch -> physical ->

pktgen

Performance

Lagopus or docker0

Server

ContainerL2Fwd or Linux

Bridge Container

pktgen-dpdk

OS: Ubuntu 16.04.1CPU: Xeon E5-2697 v2 @ 2.70GHzMem: 64GB

Performance

SDN IX @ Interop Tokyo 2015 ShowNet

Interop Tokyo is the biggest Internet-related technology show in Japan.This trial was collaboration with NECOMA project (NAIST & University of Tokyo)

IX (Internet eXchange)Packet exchange point between ISP

and DC-SPBoarder router of ISP exchanges

route information Issue

Enhance automation in provisioning and configuration

DDoS attack is one of the most critical issues

• ISP wants to reduce DDoS-related traffic in origin

• DDoS traffic occupies link bandwidth

Motivation of SDN-IX

ISP-CISP A ISP-DISP B

ISP-EISP F

What is SDN IX?

Next generation IX with SDN technology Web portal-based path provisioning between ISPs

• Inter-AS L2 connectivity– VLAN-based path provisioning– Private peer provisioning

Protect network from DDoS attack• On-demand 5-tuple-baesd packet filtering

SDN IX controller and distributed SDN/OpenFlow IX core switch

Developed by NECOMA project (NAIST and University of Tokyo)

ISP-EISP F

Two Lagopus (soft switch) are deployed forSDN-IX core switchMultiple 10Gbps linksDual Xeon E5 8core CPUs

Lagopus @ ShowNet 2015

Lagopus @ ShowNet rack

Connectivity between AS

qfx10k ne5kx8

AS290AS131154

DIX-IEJPIXKDDI

10G-LR

lagopus-1(DPID:2)

pf5240-1(DPID:1)

ax.noteJGNX

lagopus-2(DPID:4)

pf5240-2(DPID:3)

xg-89:0.1(port 4)

xg-83:0.0(port 1)

xg-89:00.0(port 3)xg-83:00.1

(port 2)

xg-83:0.0(port 1)

xg-83:0.1(port 2)

xg-89:0.0(port 3)

xg-1-0-49(port 49)

xg-1-0-51(port 51)

xg-1-0-52(port 52) xg-1-0-50

(port 50)

xg-1-0-49(port 49)

xg-1-0-50(port 50)

xg-1-0-51(port 51)

799, 1600, 1060, 810, 910, 920 (tmporally)2, 3000

OtemachiMakuhari (Veneue)

Average 2Gbps throughputNo packet dropNo reboot & no trouble for 1 week during Interop TokyoSometimes 10Gbps burst traffic

Traffic on Lagopus @Makuhari

Big change happened

Before After

vSwitch has

lots of issues on performance,

scalability, stability, …..

vSwitch works well

without any trouble!

Good performance,Good stability.

The SDI special prize of Show Award in Interop Tokyo 2015http://www.interop.jp/2015/english/exhibition/bsa.html

FinalistThe SDIShowNet demonstration

DPDK-enabled SDN/NFV middleware with Lagopus & VNF with Vhost @Interop Tokyo 2016

This trial was collaboration with University of Tokyo and IPIfusion

NFV middleware for scale-out VNFs

Flexible load balance for VNFs with smart hash calculation and flow direction Hash calc: NetFPGA-SUME

• Hash calculation using IP address pairs• Hash value are injected to MAC src for flow direction for VNF

Classification and flow direction: Lagopus • Flow direction with MAC src lookup

HV VNF VNF VNF

lagopus

uplink

downlink

hash calc & mac rewrite

MAC-based classification for

hash dl_srctype1 52:54:00:00:00:01type2 52:54:00:00:00:02

… …Type 256

52:54:00:00:00:FF

Bird in ShowNet

Two Lagopus deploymentsNFV domain, SDN-IX

https://www.facebook.com/interop.shownet

Challenges in Lagopus

vNIC between DPDK-enabled Lagopus and DPDK-enabled VNF (Virnos)

Many vNICs and flow director (loadbalancing)8 VNFs and total 18 vNICs

HV VirNOS VirNOS VirNOS VirNOS

lagopus

port4 port6 port8 port10

port9port7port5port3

Explicit resource assignment for performance

Packet processing workload aware assignment is required for Lagopus and VNF

MemoryMemory

CPU0 CPU1

Traffic

Resource assign impacts in packet processing performance

Memory Memory

CPU0 CPU1

Traffic

Lagopus 8 VNFs

Memory Memory

CPU0 CPU1

Traffic

Lagopus8 VNFs

10Gbps

4.4Gbps

lagopus

DPDK-based system needs CPUs for I/O because polling-based network I/O in DPDK

Physical I/O are intensive compared to vNICs

CPU resource assignment for I/O (1/2)

10/4 Gbps 10Gbps

lagopus

Traffic-path-aware CPU assign 4 CPU core were assigned to I/O thread of Lagopus

CPU resource assignment for I/O (2/2)

10Gbps

5Gbps 5Gbps

Performance evaluation

Good performance and scalability But long packet journey

Packet-in -> Physical NIC -> Lagopus -> vNIC -> VNF -> vNIC -> Lagopus -> Physical NIC -> Packet-out

[byte]

[Mbps]

0 200 400 600 800 1000 1200 14000

100020003000400050006000700080009000

wire ratelagopus

Packet size

Traffi

Other trials

Location-aware packet forwarding + Service Chain (NFV integration)Location-aware transparent

security check by NFV Virtual Network

Intra network• Web service and clients• Malware site blocking

Lab network• Ixia tester for demo• Policy management (Explicit routing for

#1: Segment routing with Lagopus for campus network

lago0-0

lago0-1 lago1-1

lago1-0

lago2-1

lago2-0

Serv01

Serv00

win01 p1

vFW vFW

Untrusted server block service

lago0-0

lago0-1

lago1-1

lago1-0

lago2-1

lago2-0

Flexible video stream transmission for multiple-sites and devicesLagopus switch as stream duplicatorSimultaneous 4K video – 50 sites streaming

#2: transparent video stream duplication

EncoderDecoder

Cinema or public space

#2: Blackboard streaming without Teacher’s shadow

Human shadow transparent module

Pattern APattern B

Realtime image processing with flow direction switchUser can select modes with teacher or without teacherNo configuration change without video options

Students can not see due to shadow Transparent processing help students

lagopus lago

Realtime imageprocessing

O3 project: providing flexible wide area networks with SDN

This research is executed under a part of a Research and Development of Network Virtualization Technology” program commissioned by the Ministry of Internal Affairs and Communications. (Y2013-2016)

Open Innovation over Network Platform

• 3 kinds of Contributions for User-oriented SDN(1) Open development with OSS(2) Standardization of architecture and interface(3) Commercialization of new technologies

Toward open User-oriented SDN

©O3 Project 93

(1) Open (2) Standardization (3) Commercialization

• Open, Organic, Optima– Anyone, Anything, Anywhere– Neutrality & Efficiency for Resource, Performance, Reliability, ….– Multi-Layer, Multi-Provider, Multi-Service

• User-oriented SDN for WAN– Softwarization: Unified Tools and Libraries– On-demand, Dynamic, Scalable, High-performance

• Features– Object-defined Network Framework– SDN WAN Open Source Software– SDN Design & Operations Guideline

• Accelerates– Service Innovation, Re-engineering, Business Eco-System

The O3 Project Concept, Approach, & Goal

©O3 Project 94

The O3 User-oriented SDN Architecture

©O3 Project 95

Path Nodes（ Opt ・ Pkt Transport） Switch Nodes（ Lagopus, OF ）D-planeC-plane

D-plane consists of Switch and Path Nodes; Switching Nodes provide programmability, and Path Nodes provide various type of network resources.

Orchestrator & Controllers can create and configure virtual networks according to SDN Users, and enable to customized control on individual D-Plane.

Virtual NW Virtual NW

OTTsCarriers

OTT-ACnt. Appl.

OTT-BCnt. Appl.

Controls on Virtual NW

View from Virtual NW

Network Orchestrator

Switch Nodes（ Lagopus, OF）

Controller( スイッチ部 )

Controller（ Switch Nodes ）

Controller( パス部 )

Controller（ Path Nodes ）

Controller( スイッチ部 )

Controller（ Switch Nodes ）

Common ControlFramework

SDN Nodes

Multi-Layer,Multi-DomainControl

• WAN experiments with Multi-vendor Equipment

Proof-of-Concept: Physical Configuration

©O3 Project 96

PoC on Multi-Layer & Domain Control

©O3 Project 97

PoC on Network Visualization

Ryu SDN Framework

http://osrg.github.io/ryu/

OSS SDN Framework founded by NTTSoftware for building SDN control plane agilelyFully implemented in PythonApache v2 licenseMore than 350 mailing list subscribers

Supporting the latest southbound protocolsOpenFlow 1.0, 1.2, 1.3, 1.4 (and Nicira extensions)BGPOfconfig 1.2OVSDB JSON

What’s RYU?

Many users

and more…

Developed mainly for network operatorsNot for one who sells the specific hardware switch

Integration with the existing networksGradual SDN’ing’ the existing networks

Ryu development principles

Your application are free from OF wire format (and some details like handshaking)

What ‘supporting OpenFlow’ means?

PythonObject

OF wireprotocol

DataPlane

Ryu converts it

PythonObject

OF wireProtocol

Ryu generates

Your applicationdoes something here

Ryu development is automated

github

Push the new code

Unit tests are executed

Docker hub image is updated

Ryu certification is executed on test lab

Ryu certification site is updated

You can update your Ryu environmentwith one command

Lessons leaned

What’s OpenStack?OSS for building IaaSYou can run lots of VMsMany SDN solutions are supported

What SDN means for OpenStack?The network for your VMs are separated from othersVirtual L2 network on the top of L3 network

SDN in OpenStack

Virtual L2 on tunnels (VXLAN, GRE, etc)

Typical virtual L2 implementation

Compute node

Tunnel

People advocated something like this

DataPlane

OpenFlow Controller

OpenFlow Protocol

Application Logic

Same as other OpenFlow controllersThe controller are connected with all the OVSes

Our first version OpenStack integration

Plugin

NeutronServer

RYUAgent

Compute node

Custom REST API

OpenFlow

Compute node

OpenStack REST API

SDNOperationalIntelligence

Scalability Availability

What’s the problems?

How many a single controller can handle?Can handle hundreds or thousands?

Controller does more than setting up flowsReplying to ARP packet requests rather than sending ARP packets to

all compute nodesMaking OVS work as L3 router rather than sending packets to a

central routerYou could add more here

Scalability

The death of a controller leads to the dead of the whole cloudNo more network configuration

Availability

OFC on every compute node One controller handles only one OVS

Our second verion (OFAgent driver)

NeutronServer

RYUAgent

Compute node

RYUAgent

Compute node

RYUAgent(OFC)

Compute node

OpenStack standard RPCOver queue system

Released in Icehouse

OpenStack REST API

Openflow is used only inside a compute node• Scalable with the number of compute nodes• No single point of failure in OFAgent

SDNOperationalIntelligence

Push more features to edgesDistribute featuresPlace only a feature (e.g. TE) on central node you can’t distribute

Couple loosely a central node and edgesTight coupling doesn’t scale (e.g. OpenFlow connections between a

controller and switches)The existing technology like queue works

SDN deployment for scale

NSA (National Security Agency)

More users: Tracking network activities

“The NSA is using NTT’s Ryu SDN controller. Larish says it’s a few thousand

lines of Python code that’s easy to learn, understand, deploy and troubleshoot”

http://www.networkworld.com/article/2937787/sdn/nsa-uses-openflow-for-tracking-its-network.html

TouIX (IX in France) : Replacing expensive legacy switch with whitebox switch and Ryu

More users: IX (Internet Exchange)

“The deployment is leveraging Ryu, the NTT Labs open-source controller”

http://finance.yahoo.com/news/pica8-powers-sdn-driven-internet-120000932.html

Zebra 2.0Open Source Routing Software

Open Source Revisited• Apache License• Written From Scratch in Go• Go routine & Go channel is used for

multiplexing• Task Completion Model + Thread Model• Single SPF Engine for OSPFv2/OSPFv3/IS-

IS• Forwarding Engine Abstraction for

DPDK/OF-DPA• Configuration with Commit/Rollback• gRPC for Zebra control

Architecture

• Single Process/Multithread Architecture

BGP OPSF RSVP-TE LDP

OpenConfigd• Commit & Rollback support configuration system• Configuration is defined by YANG• CLI, NetConf, REST API is automatically

generated• `confsh` - bash based CLI command• OpenConfig is fully supported

OpenConfigd Architecture

• gRPC is used for transport• completion/show/config APIs between shell

OpenConfigd

Zebra 2.0 Lagopus

confsh

completion show config

API API

Forwarding Engine Abstraction• Various Forwarding Engine Exists Today

• OS Forwarder Layer• DPDK • OF-DPA

• FEA provides Common Layer for Forwarding Engine• FEA provides

• Interface/Port Management• Bridge Management• Routing Table• ARP Table

Current limitation of software switch # of OF apps is limited

Leverage Network OS for whtitebox switch • Opensnaproute, Linux kernel, OpenSwitch

Hard to integrate with other management systemOpenStack, libvirt, nFlow, sFlow, BGP-extension

OF pipeline does not cover all our requirementsTunnel termination

• IPsec, VxLAN,Control packet escalation/injectionUser-defined OAM functionality

Heavy packet processing for OpenFlow flow entryLookup, action, long pipeline

Balance of programmability and existing network protocol supportL2, L3 (IPv4, IPv6), GRE, VxLAN, MPLSHybrid traffic control

Provide router-aware programmable dataplane for network OSProtocol-aware pipeline and APIs, OpenFlowIntegration with network OSExisting forwarding &routing protocols

support (BGP, OSPF) VPN framework over IP networks

IP as a transport protocolVxlan, GRE, IPsec tunnel support

Decouple OpenFlow semantics and Wireprotocol from OpenFlow protocolProvide gRPC switch control API

Next major upgrade: Lagopus SDN switch router

Forwarding Engine Integration

dataplane

Lagopus

BGP OPSF RSVP-TE

Zebra 2.0

OpenConfigd DB

Dataplane Manager

Configuration datastore

RIB/FIB control Interface/Port

bridge mngmt

Interface/Portbridge mngmt

C-plane related packet

User-traffic User-traffic

C-plane related packet

C-plane packet escalation via tap IF

New version will available this summer

Availability?

Hirokazu Takahashi, Tomoya Hibi, Ichikawa, Masaru Oki,

Motonori Hirano, Kiyoshi Imai, Takaya Hasegawa, Tomohiro Nakagawa, Koichi Shigihara, Keisuke Kosuga, Takanari Hayama, Tetsuya Mukawa, Saori Usami, Kunihiro Ishiguro

Thanks our development team

Comments and collaboration are very welcome!

Webhttps://lagopus.github.io

GithubLagopus vswitch

• https://github.com/lagopus/lagopusLagopus book

• http://www.lagopus.org/lagopus-book/en/html/Ryu with general tunnel ext

• https://github.com/lagopus/ryu-lagopus-ext

Conclusion

Questions?

Software Stacks to enable SDN and NFV

Software

Begin Monetising SDN & NFV Today - Telecoms.comtelecoms.com/wp-content/blogs.dir/1/files/2015/12/... · Begin Monetising SDN & NFV Today ... As SDN and NFV become more prevalent for

SDN and NFV - events.static.linuxfound.org · NFV phase 1 Virtualize The technology journey: convergence of the SDN & NFV stages 3 NFV phase 0 Decouple NFV & SDN phase 3 We are starting

SDN & NFV Orchestration Platform Development & QAsdn.calsoftlabs.com/.../SDN-NFV-Orchestration-Platform-Developmen… · SDN & NFV ORCHESTRATION PLATFORM DEVELOPMENT & QA THE CLIENT

SDN/NFV: Service Chaining

SDN, NFV and Cloud

INTRACOM TELECOM: SDN/NFV Lab: OpenDaylight … · INTRACOM TELECOM: SDN/NFV Lab: OpenDaylight Performance Stress Tests Report v1.0: Lithium vs Helium Author: sdn-nfv@intracom-telecom.com

Dynamic Service Chaining for NFV/SDN · PDF fileDynamic Service Chaining for NFV/SDN Kishore Inampudi . 2 ! Introduction – NFV Reference Architecture ... (SFC, Metadata) – Design

Devtest Orchestration for SDN & NFV

MPLS SDN NFV WORLD'17 - SDN NFV deployment update

LISP for SDN and NFV

Orchestrating SDN/NFV Solutions

Know about SDN and NFV

SDN and NFV white paper

NFV & SDN Customer Deployments

NFV SDN Summit March 2014 D3 03 bruno_rijsman NFV with OpenContrail

SDN and NFV: Driving Network Transformation

SDN and NFV - DSPCSP Pages · Fundamentals of Communications Networks 2 Why SDN and NFV ? Before explaining what SDN and NFV are we need to explain why SDN and NFV are Its all started

NFV TUTORIAL SESSION - SDN Usage in an NFV Architectural … · 2015-11-23 · NFV TUTORIAL SESSION - SDN Usage in an NFV Architectural Framework Marie-Paule Odini, Rapporteur, HP

SDN-NFV: Network Transformation - · PDF fileSDN-NFV: Network Transformation Seamless transition towards network virtualization with Prodapt’s SDN/NFV Offerings

Delivering Security Virtually Everywhere with SDN … ROLE OF SECURITY IN SDN AND NFV ... Delivering Security Virtually Everywhere with SDN and NFV NFV AND SECURITY: DELIVERING SERVICES