Advanced Networking Technologies€¦ · Forwarding Hardware Advanced Networking (SS 19): 05 -Software Defined Networking From [2] 27 Views on SDN: Redefining abstractions (“Shenker

1

Advanced NetworkingTechnologies

Chapter 5Software Defined Networking

This chapter is heavily based on material by Holger Karl (Universität Paderborn), Michael Pfeiffer (TU

Ilmenau) and Nick McKeown (Standford University)

Advanced Networking (SS 19): 05 - Software Defined Networking

2

Content

q SDN – High-level motivationq Views on the SDN architectureq Background and contextq OpenFlowq An alternative/extension: P4q White Box Switching


3

Computer

Application

Computer

Application Application

OS

OS abstracts hardware substrate➡ Innovation in applications

Innovation in computers


4

Vertically integratedClosed, proprietary

Slow innovationSmall industry

SpecializedOperatingSystem

SpecializedHardware

AppAppAppAppAppAppAppAppAppAppApp

SpecializedApplications

HorizontalOpen interfacesRapid innovation

Huge industry

Microprocessor

Open Interface

Linux MacOS

Windows(OS) or or

Open Interface

Computer systems development


5

Million of lines

of source code

8,000 RFCs

Billions of gates

• Vertically integrated, complex, closed, proprietary• Networking industry with “mainframe” mind-set

Custom Hardware

OS

Routing, management, mobility management, access control, VPNs, …

Feature Feature

We Have Lost Our Way


Bloated Power Hungry

6

Reality is Even Worse


App

AppApp

Specialized Packet Forwarding Hardware


OperatingSystem

App App App

• Lack of competition means glacial innovation• Closed architecture means blurry, closed interfaces• Innovation all carried out by hardware vendor

Operating System

7

Vertically integratedClosed, proprietary

Slow innovation

AppAppAppAppAppAppAppAppAppAppApp

HorizontalOpen interfacesRapid innovation

ControlPlane

ControlPlane

ControlPlane or or

Open Interface

SpecializedControlPlane

SpecializedHardware

SpecializedFeatures

MerchantSwitching Chips

Open Interface

Router development


8

A simple stable common substrate

q Allows applications to flourish q Internet: Stable IPv4 lead to the web

q Allows the infrastructure on top to be defined in softwareq Internet: Routing protocols, management, …

q Rapid innovation of the infrastructure itselfq Internet: er...? What’s missing? What is the substrate…?


9

(Statement of the obvious)

q In networking, despite several attempts…

q We have never agreed upon a clean separation between:q A simple common hardware substrateq And an open programming environment on top


But things are changing fast in data centers and service provider networks.

10

We need…

1. A clean separation between the substrate and an open programming environment

2. A simple hardware substrate that generalizes, subsumes and simplifies the current substrate

3. Very few preconceived ideas about how the substrate will be programmed

4. Strong isolation


11

New function!

Operators, users, 3rd party developers, researchers, …

Step 1: Separate intelligence from datapath


12

From Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud by William Stallings (0134175395) Copyright © 2016 Pearson Education, Inc. All rights reserved.

FIGURE 3.2 Control and Data Planes

One Possibility: Traditional vs. SDN networking


13

Step 2: Cache decisions in minimal flow-based datapath


“If header = x, send to port 4”

FlowTable

“If header = ?, send to me”

“If header = y, overwrite header with z, send to ports 5,6”

14

Types of action

§ Allow/deny flow§ Route & re-route flow§ Isolate flow§ Make flow private§ Remove flow

What is a flow?

§ Application flow§ All http§ Jim’s traffic§ All packets to Canada§ …

What do we need to cache?


15

Packet-switching substrate


PayloadEthernetDA, SA, etc

IPDA, SA, etc

TCPDP, SP, etc

Collection of bits to plumb flows (of different granularities)

between end points

16

Properties of a flow-based substrate

q We need flexible definitions of a flowq Unicast, multicast, waypoints, load-balancingq Different aggregations

q We need direct control over flowsq Flow as an entity we program: To route, to make private, to move, …

q Exploit the benefits of packet switchingq It works and is universally deployedq It‘s efficient (when kept simple)


17

Substrate: “Flowspace”


PayloadEthernetDA, SA, etc

IPDA, SA, etc

TCPDP, SP, etc

Collection of bits to plumb flows (of different granularities)

between end points

PayloadHeaderUser-defined flowspace

18

Flowspace: Simple example


IP Source Address

IP D

estin

atio

nAd

dres

s

Single flow All flows from A

A

All flows between two

subnets

19

Flowspace: Generalization


Field 2

Field 1

Single flowSet of flows

Field n

20

A substrate

q Flow-basedq Small number of actions for each flow

q Plumbing: Forward to port(s)q Control: Forward to controllerq Routing between flow-spaces: Rewrite headerq Bandwidth isolation: Min/max rate

q External open API to flow-table


21


FIGURE 3.3 Software-Defined Architecture

And (the/a) result: The SDN architecture


22

The SDN Stack


ControllerNOX

SlicingSoftwareFlowVisor

FlowVisorConsole

22

ApplicationsLAVIENVI (GUI) …n-Casting

NetFPGASoftware Ref. Switch

Broadcom Ref. Switch

OpenWRT PCEngineWiFi AP

Commercial Switches

OpenFlowSwitches

…

Monitoring/debugging toolsoflopsoftrace openseer

Open vSwitch

HP, NEC, Pronto, Juniper.. and many

more

Open Daylight ONOS Ryu

From [2]

23

Content



24

Views on SDN: Refactoring & Functionality (“McKeown View”)

q SDN is defined by where it places which functionality q From today’s fully distributed protocols in closed boxes…q … to a network where node exposes control APIs, managed by a

network operating system, which runs Apps


Specialized*Packet*Forwarding*Hardware*

App# App# App#


App# App# App#


App#

App#

App#


App# App# App#


Opera(ng#System#

Opera(ng#System#

Opera(ng#System#

Opera(ng#System#

Opera(ng#System#

App# App# App#

Network#Opera(ng##System#

App# App# App#

2# From [2]

25

App

Simple Packet Forwarding Hardware



App App

Simple Packet Forwarding Hardware Simple Packet

Forwarding Hardware

Network Operating System

1. Open interface to hardware

3. Well-defined open API2. At least one good operating system

Extensible, possibly open-source

The “Software-defined Network”


From [2]

26


Network Operating System 1

Open interface to hardware

Virtualization or “Slicing” Layer




App App App App App App App App

Many operating systems, orMany versions

Open interface to hardware

Isolated “slices”






From [2]

27

Views on SDN: Redefining abstractions (“Shenker view”)

q SDN is defined by the abstractions it provides to software and to a software developerq Following summary [2]

q Network control planes need abstractionsq Abstractions solve architectural problems and enable evolvabilityq Today’s layers (L2, L3, ..) are good abstractions but terrible for control

interfacesq Networks work because we can master complexity

q but what we should be doing is extracting simplicity, with the right abstractions


http://www.slideshare.net/martin_casado/sdn-abstractions

From [2]

http://www.slideshare.net/martin_casado/sdn-abstractions

28


q Abstraction level in CS rises continuously q Machine Language to operating system to modern OO/multi-threaded

languages q Abstractions simplify programming: they make it easier to write,

maintain, and reason about programs

q Could networking follow this same path?

q Important abstraction: Specify forwarding behavior via a control program


From [2]

29


q Control program should not have to handle distributed-state detailsq Proposed abstraction: global network viewq Control program operates on network view

q Input: global network view (graph)q Output: configuration of each network device

q Network OS provides network view

q Put briefly: Control programs operate on graphs, not networks (or even individual network elements)q It needs abstract (!) view of network q View needs to be detailed enough such that goals can be specified, not

needed to implement them


From [2]

30

Views on SDN: Opening up design Axes (“Heller view”)

q SDN is defined by the flexibility it provides q E.g., centralized vs. distributed control


Centralized+Control+

OpenFlow Switch

OpenFlow Switch

OpenFlow Switch

Controller+

Distributed+Control+

OpenFlow Switch

OpenFlow Switch

OpenFlow Switch

Controller+

Controller+

Controller+

5+ From [2]

31

Microflow vs. Aggregated

Microflow

• Every flow is individually set up by controller• Exact-match flow entries• Flow table contains one

entry per flow

• Good for fine grain control, policy, and monitoring, e.g. campus

Aggregated

• One flow entry covers large groups of flows• Wildcard flow entries• Flow table contains one

entry per category of flows

• Good for large number of flows, e.g. backbone


From [2]

32

Reactive vs. Proactive (pre-populated)

Reactive

• First packet of flow triggers controller to insert flow entries• Efficient use of flow table• Every flow incurs small

additional flow setup time• If control connection lost,

switch has limited utility• Extremely simple fault

recovery

Proactive

• Controller pre-populates flow table in switch• Zero additional flow setup

time• Loss of control connection

does not disrupt traffic• Essentially requires

aggregated (wildcard) rules

Advanced Networking (SS 19): 05 - Software Defined NetworkingFrom [2]

33

Virtual vs Physical

Virtual

• Assumes configurable switching within a host: in the OS or hypervisor

• Software! Memory, processing, arbitrary modifications

• Massive flow rates• Limited to the hardware

below

Physical

• No assumption of software changes; unmodified end hosts• Greater control over

expensive forwarding resources

Advanced Networking (SS 19): 05 - Software Defined Networking From [2]

34

Fully Consistent vs Eventually Consistent

Fully Consistent

• Certainty about state• Consistent state is harder to

scale• Easier to reason about state

and its transitions• May eliminate route flaps

Eventually Consistent

• Uncertainty about state now, but eventually converges• Probabilistic state is easier to

scale• Introduces the possibility of

long-lived route flaps and unstable control systems

Advanced Networking (SS 19): 05 - Software Defined Networking From [2]

35

Examples for design choice combinations


Distributed

Microflow Aggregated

Reactive Proactive

Virtual Physical

Fully Consistent Eventually Consistent

Centralized

BGP todayDistributed


Reactive Proactive

Virtual Physical


Centralized

Ethane

Distributed


Reactive Proactive

Virtual Physical


Centralized

Your own SDN solution/network

From [2]

36

Leading to new challenges

q Availability of controllerq Communication path of controllerq Consensus on routing decisions

q Graceful degradationq Impact of software bugs in controllerq Honest but curious or hostile (compromised) controller

q Performanceq #Flows in switching hardwareq CPUsq Latency

q Consistency in rule sets in switchesq Interoperability

q “Legacy” networksq “Apps” controlling the network


37

Content

q SDN – High-level motivationq Views on the SDN architectureq Background and context

q SDN predecessorsq CAP theorem

q OpenFlowq An alternative/extension: P4q White Box Switching


38

IETF ForCES

q IETF working group: Forwarding and Control Element Separation (ForCES) q Main RFC 3746 published in 2004, older than Ethane/OpenFlow but

overshadowed by it q Closed in 2014

q Similar core ideas, slightly different terminology q Forwarding Element (FE) q Control Element (CE) – instructs one or more FEs how to process

packets, uses the ForCES protocol to do so q Network Element (NE) – an abstraction of a network, hiding internal

organization, providing a single point of managementq Pre-association phase: when FE and CE find out whether FE can join NE


https://datatracker.ietf.org/wg/forces/

39

ForCES elements (as of RFC 3746)


CE Manager

FE Manager

ForCES Network Element

CE 1 CE 2

FE 1 FE 1

CE-FE interface

(ForCES!)

FE-FE interface

CE-CE interface

CE-MgrCE

interface

FE-MgrFE

interface

CE-MgrFE Mgrinterface

CE examples: OSPF, RIP, BGP, RSVP, LDPFE examples: LPM Fwd, Meter, Shaping, NAT, Classifier

40

Early work: Ethane

q Conceptual precursor of SDN in its OpenFlow interpretationq Designed (mostly) by Martin Casado, PhD thesis, 2007

q Main principles: q Network is governed by policies, declared over high-level names

■ Names for users, hosts, … rather than (dynamically allocated) addresses

q Network routing should be policy-aware ■ Policies mandate connectivity, routing should respect that ■ Rather than building filtering as an on-top mechanism■ E.g.: “all packets of guest users go via this middlebox”

q Network enforces strong binding between packet and its origin■ Rather than: not feasible today


M. Casado, M. J. Freedman, J. Pettit, N. Gude, N. McKeown, and S. Shenker,

“Rethinking Enterprise Network Control,” IEEE/ACM Transactions on Networking,

vol. 17, no. 4, pp. 1270–1283, Aug. 2009.

41

So what does this mean architecturally?

q Compare our three “standard architectures”: IP, MPLS, SDNq From the perspective of the main three interfaces that exist in any

network architecture: q Host-to-Network, Operator-to-Network, Packet-to-Switch


Idea from[4], 4WARD papers

Host-to-Network

Packet-to-Switch

Operator-to-Network

IP Header fields in IP packet

Same None

MPLS Header fields in IP packet

Label somewhat (e.g., PCE)

SDN Depends Depends Main focus

42

Content

q SDN – High-level motivationq Views on the SDN architectureq Background and context

q SDN predecessorsq CAP theorem: the bad, the good & the ugly

q OpenFlowq An alternative/extension: P4q White Box Switching


43

The bad: The CAP theorem’s proposition & proof

q Suggested by Eric Brewer in 1999/2000, proof by Gilbert and Lynch in 2002 [12]

q In many networks, the absence of partitions cannot be guaranteed (firmware bugs, administrative errors, . . . )

➡ choice between CP and AP


Central propositionIn a distributed system, it is impossible to provide• Consistency,• Availability, and• Partition toleranceall at once, i.e. at least one of them has to be sacrificed.

44

CAP theorem: Formal model

q Network partitionq All messages between nodes in different parts of the network/components

are lost.q Availability: Available data objects

q Every request received by a non-failing node must result in a response.q No time boundary, but network partition can last ‘forever’, thus a strong

availability requirement.q Consistency: Atomic data objects

q ∃ total order on all operations such that each operation looks as if it were completed at a single instant.

q Equivalent: Requests must act as if they were processed on a single node, one at a time.


45

CAP theorem: Sketch of proof

q By contradiction:

q Q: Why does this apply to centrally controlled routing protocols?


S1 S2

C1 C2

1. X := 42 2. OK 3. X? 4. ???

46

CAP theorem: Classic strategies for CP and AP

q CP systemsq Delay the acknowledgement of a write operation until new value has been

propagated to all nodes q Examples:

■ Relational database with synchronous replication■ Two-phase commit protocols

q Twist: transaction is completed if write operation has propagated to majority or some specific node

q AP systems q Answer with the (possibly stale) last known value q Examples:

■ Slave DNS servers■ NoSQL databases


47

The good: A more graduated perspective on CAP [13]

q This means:q No partition → No problemq But during a partition, all systems must decide eventuallyq Permanently retrying is in fact a choice for C over A


If a partition occurs during the processing of an operation, each node can decide to • cancel the operation (favor C over A), or • proceed, but risk inconsistencies (favor A over C).

But: It is possible to decide differently every time, based on the circumstances.

48

CAP: Mitigation strategies

q Generally to keep consistency:q Some operations must be forbidden during a partition q Others are okay (e.g. read queries)

q Often: Guarantee to consistency to a certain degree q Example: Read-your-own-writes consistency

q Facebook: A user’s timeline is stored at master copy and cached at slaves q Usually users see (potentially stale) copies at slaves q But when they post something, their reads are redirected to the respective

master for a certain time q Different strategies on different levels possible, e.g. inside a single site

and between sites (latency!)

q Often: In one component changes may be allowed, multiple consensus algorithms available (e.g. dynamic voting)


49

CAP: Partition recovery (I)

q What if we still want to continue service during partition? 1. Detect partition 2. Enter a special partition mode 3. Continue service 4. After partition: Recovery

q The small problem: Partition detection q Nodes might disagree whether a partition existsq Consensus about partition state not possibleq Nodes may enter the partition mode at different timesq A distributed commit protocol is required (2PCP, Paxos, ...)


50

CAP: Partition recovery (II)

q The bigger problem: Partition recovery

q A (very) simple example: q Users register on a web site q Every user is assigned an unique ID (SQL: serial, auto_increment) q During partition: Same ID might be assigned twice q Recovery: Recreate uniqueness of IDs

q Root cause:q In a consistent system, invariants are guaranteed q The system’s designer may or may not know them q In an available system, invariants must be explicitly restored after a

partition q System’s designer must know the invariants and (correctly) restore them!


51

CAP: Partition recovery – Example algorithm

q Commutative/Conflict-free Replicated Data Types (CRDTs) are data types that provably/eventually converge

q Example: Google Docs serializes edits into a list of insert and delete operations

➡ Application-specific invariants are not ensured automatically


On Monday, the ANT lecture is at 13:00.

On Thursday, the ANT lecture is at 13:00.

On Monday, the ANT lecture is at 17:00.

On Thursday, the ANT lecture is at 17:00.

52

CAP: Partition recovery (III)

q Recovery is tedious and error-prone, and seldom testedq Brewer: Similar to going from single-threaded to multi-threaded

programming q Sometimes only possibility: Ask the user (e.g. git merge)

q Balance between availability and consistency: q ATMs: When partitioned, limit withdrawal to amount Xq Invariant: Not more withdrawals than allowedq Manual correction afterwards

q Usual tools: q Version vectors (vector clocks)q Logging, replay and rollback


53

The ugly: CAP in SDN infrastructures (I)

q So far focus on distributed systems at application layer (databases, web services, ...)

q SDN is much more basic (layer 2/3)

q Network functionality is essential q Must be able to recover itselfq pure CP is not an option

q AP means partition recovery is required!


54

The ugly: CAP in SDN infrastructures (II)

q Possible without the network up and running?q What about dependency loops?q Fallback to non-SDN networking possible?

q Even if VLANs (and similar features) are replaced by SDN?q How can we handle conflicting policy decisions?

q Relying on user input rather unrealistic …q Possible to figure out all the invariants?

q Most SDN publications ignore the issue …q Even BGP is know to not stabilize in all cases [14] …


55

SDN – Next steps?

q One concrete example for an SDN control protocol: OpenFlowq Design choices

q E.g., scalability, distributed implementations for a centralized architecture q Controllers – design and concrete examples

■ NOX, Onix, Floodlight, OpenDaylightq Switches – what changes in a switch suited for SDN?

■ Example: NetFPGAsq Programming abstractions for SDN

q FlowVisor – virtualize and isolate networks q Ideas, concrete example (Frenetic, Procera) q Abstractions, in particular, composition of networks

q Practical environments, tools, proofs q E.g., debuggers, verification of network invariants

q Network management in SDN q Application areas

q SDN & data center, Cloud, Big data (e.g. Hedera)q SDN & optics

Advanced Networking (SS 19): 05 - Software Defined NetworkingSe

e ht

tp://

www.

nec-

labs

.com

/~lu

me/

sdn-

read

ing-

list.h

tml

for v

ery

good

list o

f pap

ers!

56

Content

q SDN – High-level motivationq Views on the SDN architectureq Background and context q OpenFlow

q Switch assumptions and protocol q OpenFlow switches, some examples

■ Hardware■ Software

q OpenFlow controllers – a survey ■ Case study: ONOS

q An alternative/extension: P4q White Box Switching


57

Where are we?


FIGURE 4.2 Data Plane Network Device


58

OpenFlow

q OpenFlow is just a forwarding table management protocol! (Heller, OpenFlow tutorial)

q It exists in several versions, all concurrently coexisting q 1.0 – 1.5

q The following slides followq OpenFlow Switch Specification, Version 1.5.1., Open Networking

Foundation, 2015 q Stallings, Modern Networking q B. Heller et al., Tutorial: SDN for Engineers, Open Networking Summit,

Santa Clara, April 2012. www.opennetsummit.org/tutorials.html


59


FIGURE 4.3 OpenFlow Switch Context


60


FIGURE 4.4 OpenFlow Switch


61

Components of an OpenFlow switch

q Tables: Programmable forwarding tables q One or more flow tables q One group table

q Channels to OpenFlow controllers (one or more)q Ports (see next slide)

q May be changed by configuration messages

q Optionally: an “underlying”, traditional switching logicq A hybrid OF switch


62

Ports in an OpenFlow switch

q OpenFlow ports: Network interfaces as seen by the OF tables q Physical ports: actual, physical network interfaces q Logical ports: Abstraction on top of physical ports; e.g., for link

aggregation or tunnels q Packets from logical ports may have Tunnel-ID field q Provided by switch, transparent to OF processing

q Standard ports: logical or physical portq Reserved ports: Abstractions for certain actions on packets

q ALL: output port to represent all egress ports q CONTROLLER: Abstracts channel to/from controller q TABLE: Start of OF pipeline q IN_PORT: represents port at which packet arrived q NORMAL: hand packet over to standard switch processing q Plus: ANY, UNSET, LOCAL, FLOOD


63

OpenFlow pipeline processing

q Packets arriving at switch travers through a sequence of ingress and egress flow tables q Simplest case: only a single ingress table

q Processing always starts at table 0q Later pipeline stages can be skipped depending on packet

OF 1.5.1 specification

Optional


64

OpenFlow pipeline processing: FlowTable

q When packet is processed by FlowTable, match it against all entries in priority order q First match: execute instruction set (do not proceed in this table)

■ Instruction set might forward packet to a higher-numbered table (no cycles allowed!)

q No match: Action specified by table miss configuration ■ E.g., drop, forward to controller, forward to other table


65


FIGURE 4.5 OpenFlow Table Entry Formats

Flow table entries


66

Flow table entries

q match fields: to match against packets. These consist of the ingress port and packet headers, and optionally other pipeline fields such as metadata specified by a previous table.

q priority: matching precedence of the flow entry. q counters: updated when packets are matched. q instructions: to modify the action set or pipeline processing. q timeouts: maximum amount of time or idle time before flow is expired

by the switch. q cookie: opaque data value chosen by the controller. May be used by

the controller to filter flow entries affected by flow statistics, flow modification and flow deletion requests. Not used when processing packets.

q flags: flags alter the way flow entries are managed, for example the flag OFPFF_SEND_FLOW_REM triggers flow removed messages for that flow entry.


67

Examples

Switching

*

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport Action

* 00:1f:.. * * * * * * * port6

Flow Switching

port3

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport Action

00:2e.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6

Firewall

*

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport Forward

* * * * * * * * 22 drop


68

Examples

Routing

*

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport Action

* * * * * 5.6.7.8 * * * port6

VLAN

*

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport Action

* * * vlan1 * * * * *port6, port7,port9


69

Matching in OF flow tables

q Matches fixed and wildcard fields in flow table entries againstq Header fields q Ingress port q Meta data

■ Between flow tables, packet maybe be enhanced by metadata generated by actions in previous tables

q Several matches with same (highest) priority: behaviour is undefined!


70

Instructions in flow entries; action set

q Instructions how to process a packet, specified as actions q Can be applied immediately q Or collected in action set for later application (empty by default)

■ Carried between flow tables ■ Executed when pipeline processing stops (i.e., when no action to go to

another table) q Instructions:

q Apply actions: Execute any given actions immediately q Clear actions: Clear action set, do not execute q Write actions: Merge new actions into action set q Write metadata: add values to metadata of a packetq Stat-trigger: trigger message to controller if some counter thresholds are

executed q Goto Table: continue processing of this packet with the given table


71

Actions in action set

q Output to port (potentially starting egress processing there)q Group: Process by group (see below) q Set queue: enqueue packet into a given queue at the output port q Meter (see below) q Push/pop tags: Add or remove tags for MPLS, Provider Backbone

Bridging (PBB), VLAN headersq Set field: write values into packet or metadata q Copy field: Especially between header and metadata fieldsq Change TTL

q Property: When packet arrives at output port, its action set must only have a single output action


72

Actions: Summary


73


FIGURE 4.9 Example of Nested Flows

Nested flows treated by multiple flow tables


74

From OPenFlow specification 1.5.1

FIGURE 4.6 Simplified Flowchart Detailing Packet Flow Through an OpenFlow Switch


75Advanced Networking (SS 19): 05 - Software Defined Networking


77


FIGURE 4.7 Packet Flow Through an OpenFlow Switch: Ingress Processing


78


FIGURE 4.8 Packet Flow Through OpenFlow Switch: Egress Processing


79

Counters

q Counters for: for each flow table, flow entry, port, queue, group, group bucket, meter and meter band

q Counters can be queried from controller q But beware of significant overhead!


80

Group table

q Group entries q group identifier: a 32 bit unsigned integer uniquely identifying the group on

the OpenFlow switch. q group type: see next slide q counters: updated when packets are processed by a groupq action buckets (zero or more): an ordered list of action buckets, where

each action bucket contains a set of actions to execute and associated parameters


81

Group types

q All: q For each bucket in group, create a clone of the packet q Process each clone by actions in the bucket q Typical use case: multi-/broadcast

q Select: q Pick one bucket, process packet via its actions q How to pick determined by switch; e.g., round-robin or hash-basedq Typical use case: load balancing

q Fast failover: q Pick first alive bucket q Needs a liveness monitoring mechanism to check alive or not


82


FIGURE 4.10 Group Types


83


FIGURE 10.9 OpenFlow QoS-Related Formats

Metering in OF


84

Controller and control channel

q So far, we looked into processing inside an OF switch q Relationship between switch an OF controller?

q Control channel: TLS (usually) or plain TCP q Message types

q Controller-to-switch q Asynchronous (switch to controller, without solicitation by controller) q Symmetric (sent by either)


85

Controller to switch messages

q Feature request: Controller enquiries switch about identity or capabilities

q Configuration: enquire or set configuration parameters q Read stateq Modify state: mostly, add/delete/change flow table entries q Packet out: Instruct switch to send a packet out of a given port

q Packet either included as payload in Packet-Out command q Or specified as a buffer ID, pointing to a buffered packet in the switch

q Barrier messages to ensure synchronizationq Note: OF protocol does NOT guarantee in-order delivery; need barriers if

desired by controllerq Implies committing all changes to data path, flow tables reconfigured, …

q Role request: to deal with multi-controller setups q Configure asynchronous messages: set filter on messages sent from

switch to controller (again, for multi-controller setups)


86

Asynchronous messages

q Sent from switch to controller to indicate event, state change in switch

q Packet-In: Any packet forwarded to the CONTROLLER reserved port gets sent to controller using this message type q Switch can include actual packet or buffer the packet locally; then only

sent a buffer ID to controller (configuration dependent) q For buffered packets, only packet header is forwarded to controller

q Flow removed: When a flow entry is removed from a table, typically upon timeout

q Port status: Inform controller about port up/down events, etc.q Role status: Multi-controller cases; e.g., new master controller, inform

old master q Flow status: A monitor has locally detected a change in flow status;

inform controller


87

Symmetric messages

q Usually bookkeeping messages

q Helloq Echo q Errorq Experimenter: Announce additional features, for future extensions


88

Content







89

OpenFlow switches

q Commodity hardwareq Internal structure of many switches was “SDN-like” for a long time q Today, a lot of vendors offer OpenFlow compatibility

■ Sometimes, as an (expensive) add-on q Issues to consider: How fast is OF processing?

■ Common bottleneck: link between flow table and OF processor ■ Can severely curtail flow installation rates

q Experimental hardwareq NetFPGA-based hardware switches

q Software q Used in both experimental as well as production q Important case: software switch for virtual machines!q Main example: Open vSwitch


90

Open vSwitch

q Starting point: How to connect virtual machines to a network? q VMs have virtual NICs with their own MAC addresses

VM 1

vNIC

vNIC

VM 2vN

IC

vNIC

Real machine

NIC

VM 1

vNIC

vNIC

VM 2

vNIC

vNIC

Real machine

NIC

NIC

?? ??


91

Open vSwitch: Basic job

q Build functions of a standard L2 switch in software

q Conceptually simple, but main challenge: speedq I.e., a user-level implementation would be

trivial to put into Linuxq But far too slow, if kernel is involved at any

time q Approach:

q Put core forwarding functionality into an operating system kernel module

q Keep control functionality outside of the kernel

q First packets of a flow leave kernel, treated by user level part, which programs kernel module ■ Similar to OF

VM 1

vNIC

vNIC

VM 2

vNIC

vNIC

Real machine

NIC


92

Open vSwitch main components

Database holding switch configuration


93

Content







94

SDN controllers


FIGURE 5.2 SDN Control Plane Functions and Interfaces


95

SDN controller interfaces

From Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud by William

Stallings (0134175395) Copyright © 2016 Pearson Education, Inc. All

rights reserved.

FIGURE 5.3 SDN Controller Interfaces


96

Open Controllers


Name Lang Platform License OriginalAuthor

Notes

OpenFlowReference

C Linux OpenFlowLicense

Stanford/Nicira

not designed for extensibility

NOX/POX Python, C++

Linux GPL Nicira Also no longer actively developed

Beacon Java Win, Mac, Linux, Android

GPL (core), FOSS Licenses for your code

DavidErickson (Stanford)

runtime modular, web UI framework, regression test framework

Trema Ruby,C

Linux GPL NEC includes emulator, regression test framework

RouteFlow ? Linux Apache CPqD(Brazil)

virtual IP routing as a service

97

Open Controllers (2)Name Lang Platform(

s)License Original

AuthorNotes

OpenDaylight (ODL)

Java Eclipse Public License

Main focus: Cloud

ONOS Java Apache Main focus: Operators

Floodlight Java Any Apache BigSwitch, based on Beacon

Maestro Java Win, Mac, Linux

LGPL Zheng Cai(Rice)

Ryu Python Apache Nice to play with

OpenMUL C GPLv2 Abandoned, relatively fast

Too many to easily list or keep track of…Advanced Networking (SS 19): 05 - Software Defined Networking

98


FIGURE 5.7 OpenDaylight Architecture

OpenDaylight Architecture


99


FIGURE 5.8 Service Abstraction Layer Model

ODL Service Abstraction


100

OpenDaylight Beryllium


101

ONOS

q Concerted effort to build a modular, fast-moving, potentially distributed SDN controller

q Actually, broader vision:

Open Network Operating System (ONOS) is an open source SDN network operating system. Our mission is to enable Service Providers to build real SDN/NFV Solutions.

q Smaller than ODL, seemingly more stringent organization q Focus on working code, less working philosophy

q Following slides: ONOS architecture by ON.LAB


https://docs.google.com/presentation/d/1Y4S82YZyskqnKAzW4kKm-6IIO04h_nnrRQyfrsCdt-I/edit?usp=sharing

102

Content

q SDN – High-level motivationq Views on the SDN architectureq Background and contextq OpenFlow

q Scalability & Resilience■ Scalability options ■ ONIX■ Kandoo■ DevoFlow■ Trade-offs



103

Scalability & resilience concerns in SDN

q Possible conception: SDN has a single, centralized controllerq Main point: Controller of SDN is considered logically centralized; need not

to be physically centralized q Central view is important, not implementation

q Line of argument 1: Even if it were centralized, it would not be a problem (or not a bigger problem than in non-SDN networks) q … for flow setup times q … for controller performance (# flow establishments / second) q ... for resilience (need to reach controller)

q Line of argument 2: Let’s build distributed SDN controllers


104


FIGURE 5.10 SDN Domain Structure

Option: Multiple independent or cooperative domains


105


FIGURE 5.11 Federation of SDN Controllers [GUPT14]

Example: Federated access to cloud


106


FIGURE 5.12 Heterogeneous Autonomous Systems with OpenFlow and Non-OpenFlow Domains

Interaction


107


FIGURE 5.13 East-West Connection Establishment, Route, and Flow Setup

Connection Establishment


108

Controller scalability

q Flow establishment in early implementations (NOX): about 30.000 requests/s, to keep establishment time reasonable

q Improved implementations achieve factor 10 higher rate q Provide defaults for short-lived flows; only talk to controller for longer-

lived flows (DevoFlow) q Physically distribute controller (Onix)


109

Flow establishment overhead

q Push state proactively, do not wait for actual requests

q Keep controllers close to switches (to keep propagation time small)

q Improve update rates on flow tables q Bottleneck: slow switch CPUs, limited

bandwidth inside a switch between fabric and control

IEEE Communications Magazine • February 2013138

scalability issues associated with a centralizedcontroller, albeit for a more restricted set ofcontrol applications satisfying certain properties.

Kandoo [9] takes a different approach to dis-tributing the control plane. It defines a scope ofoperations to enable applications with differentrequirements to coexist: locally scoped applica-tions (i.e., applications that can operate usingthe local state of a switch) are deployed close tothe data path in order to process frequentrequests and shield other parts of the controlplane from the load. A root controller, on theother hand, takes charge of applications thatrequire network-wide state, and also acts as amediator for any coordination required betweenlocal controllers.

An interesting observation is that controlplane scalability challenges in SDN (e.g., conver-gence and consistency requirements) are notinherently different than those faced in tradi-tional network design. SDN, by itself, is neitherlikely to eliminate the control plane design com-plexity or make it more or less scalable.1 SDN,however:

• Allows us to rethink the constraints tradi-tionally imposed on control protocol designs(e.g., a fixed distribution model) and decideon our own trade-offs in the design space

• Encourages us to apply common softwareand distributed systems development prac-tices to simplify development, verification,and debugging

Unlike traditional networks, in SDN, we do notneed to address basic but challenging issues liketopology discovery, state distribution, andresiliency over and over again. As demonstratedin Onix, control applications can rely on the con-trol platform to provide these common func-tions; functions such as maintaining a cohesiveview of the network in a distributed and scalablefashion. In fact, it is significantly easier to devel-op applications for such cohesive distributedcontrol platforms than a swarm of autonomousapplications running on heterogeneous forward-ing elements.

OTHER SDN SCALABILITY CONCERNSIncreased load on the controller is only one ofthe voiced concerns about SDN scalability. Here,we briefly explain other causes of concern, alongwith potential solutions.

Flow Initiation Overhead — Ethane [10], anearly SDN security system, puts a controller incharge of installing forwarding state on switcheson a per-flow basis. Even though this reactiveform of flow handling introduces a great degreeof flexibility (e.g., easy fine-grained high-levelnetwork-wide policy enforcement in the case ofEthane), it introduces a flow setup delay and,depending on implementation, may limit scala-bility. Early designs, such as Ethane and NOX,lead to the widespread assumption that all SDNsystems are reactive. In reality, however, proac-tive designs — in which forwarding entries areset up before the initiation of actual flows — areperfectly acceptable in SDN, and can avoid theflow setup delay penalty altogether.

Let us review the flow setup process toexplain the bottlenecks and show how a gooddesign can avoid them. As illustrated in Fig. 2,the flow setup process has four steps:• A packet arrives at the switch that does not

match any existing entry in the flow table.• The switch generates a new flow request to

the controller.• The controller responds with a new for-

warding rule.• The switch updates its flow table.The performance in the first three steps and par-tially the last depends on the switch capabilitiesand resources (management CPU, memory, etc.)and the performance of its software stack. Thedelay in the third step is determined by the con-troller’s resources along with the control pro-gram’s performance. Finally, the switch’s FIBupdate time contributes to the delay in complet-ing the flow setup process.

Assuming controllers are placed in closeproximity of switches, the controller-switch com-munication delay is negligible. On the controllerside, even on a commodity machine with a singleCPU core, state-of-the-art controllers are wellcapable of responding to flow setup requests

Figure 2. The four steps in the flow setup process.

Rule

Rule

Packet Miss inflow table

Switch updates

the flow table.

Switch

4

1

Flow table

New rule

32

Controller sends a newforw

arding rule.

Controller

Flow request is sent to

the controller.

1 After all, one can repli-cate a traditional networkdesign with SDN by collo-cating equal numbers offorwarding and controlelements. Even thoughthis obviates the benefitsof SDN, it is technicallypossible.

YEGANEH LAYOUT_Layout 1 1/28/13 3:43 PM Page 138

From [5]


110

Link failure recovery

q Controllers need to be informed about link failure

q But only the controller! No need to flood network

q Possible problem: control network itself can be damaged q In particular, by

misconfiguration q Use separate, slow, reliable

out-of-band signalling network

q At least similar convergence times in this case

within a millisecond when the flow initiationrequests are on the order of several hundredthousand per second.

While Open vSwitch — an OpenFlow-enabledsoftware switch — is capable of installing tens ofthousands of flows per second with sub-millisec-ond latency, hardware switches only support a fewthousand installations per second with a sub-10 mslatency at best. This poor performance is typicallyattributed to lack of resources on switches (weakmanagement CPUs), poor support for high-fre-quency communication between the switchingchipset and the management CPU, and non-opti-mal software implementations. We expect theseissues to be resolved in a few years as more spe-cialized hardware is built. It is foreseeable that theFIB update time will become the main factor inthe switch-side flow setup latency.

While we argue that controllers and, in thenear future, switches would be able to sustainsufficient throughput with negligible latency forreactive flow setup, in the end the control logicdetermines the scalability of a reactive design. Acontrol program installing an end-to-end path ona per-flow basis does not scale, because the per-switch memory is fixed but the number of for-warding entries in the data path grows with thenumber of active flows in the network. However,the control program may install aggregate rulesmatching a large number of micro-flows (therebyfacing the same scalability challenges as a proac-tive design), or proactively install rules in thenetwork core to provide end-to-end connectivityand identify quality of service (QoS) classes,while classifying and reactively labeling flows atthe edge. A viable solution to the scalabilitychallenges of the proactive designs in the formerclass due to data path memory scarcity is pro-posed in DIFANE [5]; while the scalability ofthe latter class follows from the observation thatthe fanout of an edge switch and thus the num-ber of flows initiated there is bounded (just addedge controllers as the network grows in size).

Resiliency to Failures — Resiliency to failuresand convergence time after a failure have alwaysbeen a key concern in network performance.SDN is no exception, and, with the early systemssetting an example of designs with a single cen-tral control, resiliency to failures has been amajor concern. A state-synchronized slave con-troller would be sufficient to recover from con-troller failures, but a network partition wouldleave half of the network brainless. In a multi-controller network, with an appropriate con-troller discovery mechanisms, switches canalways discover a controller if one exists withintheir partition. Therefore, given a scalable dis-covery mechanism, controller failures do notpose a challenge to SDN scalability.

Let us decompose the process of repairing abroken link or switch to see how it is differentfrom the traditional networks. As shown in Fig. 3,convergence in response to a link failure has fivesteps. The switch detects a change. Then theswitch notifies the controller. Upon notification,the control program computes the repair actionsand pushes updates to the affected data path ele-ments, which, in turn, update their forwardingtables.2 In traditional networks, link failure noti-

fications are flooded across the network, where-as with SDN, this information is sent directly toa controller. Therefore, the information propa-gation delay in SDN is no worse than in tradi-tional networks. Also, as an advantage for SDN,the computation is carried out on more capablecontroller machines as opposed to weak man-agement CPUs of all switches, regardless ofwhether they are affected by the failure or not.

Note that the above argument was built on theimplicit assumption that the failed switch or linkdoes not affect the switch-controller communica-tion channel. The control network itself needs tobe repaired first if a failed link or switch was partof it. In that case, if the control network — builtwith traditional network gear — is running anIGP, the IGP needs to converge first beforeswitches can communicate with the controller torepair the data network. In this corner case,therefore, convergence may be slower than in tra-ditional networks. If this proves to be a problem,the network operator should deploy an out-of-band control network to alleviate this issue.

Overall, the failure recovery process in SDNis no worse than in traditional networks. Conse-quently, similar scalability concerns exist, andthe same techniques used to minimize downtimein traditional networks are applicable to SDN.For instance, SDN design can and should alsoleverage local fast failover mechanisms availablein switches to transiently forward traffic towardpreprogrammed backup paths while a failure isbeing addressed. We stress that, as demonstrat-ed in Onix [2], the control platform provides theessential failover and recovery mechanisms thatcontrol applications can reuse and rely upon.

IEEE Communications Magazine • February 2013 139

Figure 3. The five steps when converging on a link failure.

Switch

A switch detectsa link failure.

Controller pushesupdate to sw

itches.

Switch

4

Then notifies thecontroller aboutthe failure.

2

1

Controllercomputes therequired updates.

3

Switches updatetheir forwardingtable.

5

Switch

Controller

2 For switch failures, theprocess is very similar withthe exception that the con-troller itself detects thefailure.

YEGANEH LAYOUT_Layout 1 1/28/13 3:43 PM Page 139

From [5]


111

Scaling controllers: Case study Onix [7]

q Goal: Design a flexible and reliable control platform for SDN q Platform should take care of distribution details; shield protocol developer

from them

q Main contributions of Onixq Provide an API general enough to be applicable to scenarios ranging from

WAN to clouds to data centres, … q Provide flexible distribution primitives for network state/rules

q Requirements q Scale to million of ports, high-performance networks


112

Onix components

q NIB = Network Information Base, synchronized between devicesq Management network run in-band or out-of-band (e.g. IS-IS or OSPF)


From [7]

113

Onix network model and API

1

Node

ForwardingEngineHost Forwarding

Tablen

nPort

1Network

Link

21

Figure 2: The default network entity classes provided byOnix’s API. Solid lines represent inheritance, while dashed linescorrespond to referential relation between entity instances. Thenumbers on the dashed lines show the quantitative mappingrelationship (e.g., one Link maps to two Ports, and twoPorts can map to the same Link). Nodes, ports and linksconstitute the network topology. All entity classes inherit thesame base class providing generic key-value pair access.

For example, there is a Port entity class that canbelong to a list of ports in a Node entity. Figure 2illustrates the default set of typed entities Onix provides –all typed entities have a common base class limited togeneric key-value pair access. The type-set within Onix isnot fixed and applications can subclass these basic classesto extend Onix’s data model as needed.6

The NIB provides multiple methods for the controllogic to gain access to network entities. It maintains anindex of all of its entities based on the entity identifier,allowing for direct querying of a specific entity. It alsosupports registration for notifications on state changesor the addition/deletion of an entity. Applications canfurther extend the querying capabilities by listening fornotifications of entity arrivals and maintaining their ownindices.

The control logic for a typical application is thereforefairly straightforward. It will register to be notified onsome state change (e.g., the addition of new switches andports), and once the notification fires, it will manipulatethe network state by modifying the key-value pairs of theaffected entities.

The NIB provides neither fine-grained nor distributedlocking mechanisms, but rather a mechanism to requestand release exclusive access to the NIB data structureof the local instance. While the application is given theguarantee that no other thread is updating the NIB withinthe same controller instance, it is not guaranteed thestate (or related state) remains untouched by other Onixinstances or network elements. For such coordination,it must use mechanisms implemented externally to theNIB. We describe this in more detail in Section 4; for now,we assume this coordination is mostly static and requirescontrol logic involvement during failure conditions.

All NIB operations are asynchronous, meaning thatupdating a network entity only guarantees that the updatemessage will eventually be sent to the corresponding

6Subclassing also enables control over how the key-value pairs arestored within the entity. Control logics may prefer different trade-offsbetween memory and CPU usage.

Category PurposeQuery Find entities.Create, destroy Create and remove entities.Access attributes Inspect and modify entities.Notifications Receive updates about changes.Synchronize Wait for updates being exported to

network elements and controllers.Configuration Configure how state is imported

to and exported from the NIB.Pull Ask for entities to be imported

on-demand.

Table 1: Functions provided by the Onix NIB API.

network element and/or other Onix instances – noordering or latency guarantees are given. While thishas the potential to simplify the control logic and makemultiple modifications more efficient, often it is useful toknow when an update has successfully completed. Forinstance, to minimize disruption to network traffic, theapplication may require the updating of forwarding stateon multiple switches to happen in a particular order (tominimize, for example, packet drops). For this purpose,the API provides a synchronization primitive: if calledfor an entity, the control logic will receive a callback oncethe state has been pushed. After receiving the callback,the control logic may then inspect the contents of the NIBand verify that the state is as expected before proceeding.We note that if the control logic implements distributedcoordination, race-conditions in state updates will eithernot exist or will be transient in nature.

An application may also only rely on NIB notificationsto react to failures in modifications as they would anyother network state changes. Table 1 lists available NIB-manipulation methods.

3 Scaling and ReliabilityTo be a viable alternative to the traditional networkarchitecture, Onix must meet the scalability and reliabilityrequirements of today’s (and tomorrow’s) production net-works. Because the NIB is the focal point for the systemstate and events, its use largely dictates the scalability andreliability properties of the system. For example, as thenumber of elements in the network increases, a NIB thatis not distributed could exhaust system memory. Or, thenumber of network events (generated by the NIB) or workrequired to manage them could grow to saturate the CPUof a single Onix instance.7

This and the following section describe the NIBdistribution framework that enables Onix to scale to very

7In one of our upcoming deployments, if a single-instanceapplication took one second to analyze the statistics of a single Portand compute a result (e.g., for billing purposes), that application wouldtake two months to process all Ports in the NIB.

From [7]


114

Onix itself

q Onix: distributed system, consisting of multiple instances, running on (typically) multiple physical servers q Facilitates reading/writing network state towards control logic q Coordinates with peer instances, exchanges state q Does not provide any particular network behaviour as such (has to be

realized by control logic)

q Onix network information base (NIB) q Collection of network entities q One network entity: set of key/value pairs, identified by flat, 128bit key

■ Optionally with a type (class): adds methods to key/value pairs q Read, write, synchronous updates, register for notification when value

changes


115

Distributing the NIB

q Partitioning: State can be partitioned across replicated controllers q Controllers only connected to some subset of network elements

q Aggregation: Controller can group together network elements, expose them as a single element to peer / higher-layer controllers q E.g., make a campus network appear as a single switch

q Consistency: Replicated state needs to be consistent under changes q Control logic application can choose (= is responsible) q Option 1: strong consistency via a database with distributed transactions

(e.g., forwarding state) q Option 2: One-hop DHT (e.g., link utilization), eventually consistent


116

Case study: Kandoo – Scaling by hierarchy

q Source of scaling problems for controllers: Too frequent events in network-global control plane q We could process many events in the data plane, but that is exactly what

we do not want

q Kandoo proposal: Build two layers of controllersq Bottom controllers only have local view of their network entities

■ Only run local control applications■ Do not even talk to their peer bottom controllers ■ Reduce rate of events to be processed on global level

q Top controllers have global view


117

Kandoo design

Switch

Local Controller

Switch Switch Switch Switch

Local Controller

LocalController

Root Controller

RareEvents

FrequentEvents

Figure 1: Kandoo’s Two Levels of Controllers. Localcontrollers handle frequent events, while a logicallycentralized root controller handles rare events.

control applications. As illustrated in Figure 1, severallocal controllers are deployed throughout the network; eachof these controllers controls one or a handful of switches.The root controller, on the other hand, controls all localcontrollers.It is easy to realize local controllers since they are

merely switch proxies for the root controller, and theydo not need the network-wide state. They can even beimplemented directly in OpenFlow switches. Interestingly,local controllers can linearly scale with the number ofswitches in a network. Thus, the control plane scales as longas we process frequent events in local applications and shieldthe root controller from these frequent events. Needlessto say, Kandoo cannot help any control applications thatrequire network-wide state (even though it does not hurtthem, either). We believe such applications are intrinsicallyhard to scale, and solutions like Onix [8] and HyperFlow [18]provide the right frameworks for running such applications.Our implementation of Kandoo is completely compliant

with the OpenFlow specifications. Data and controlplanes are decoupled in Kandoo. Switches can operatewithout having a local controller; control applicationsfunction regardless of their physical location. The mainadvantage of Kandoo is that it gives network operatorsthe freedom to configure the deployment model of controlplane functionalities based on the characteristics of controlapplications.The design and implementation of Kandoo are presented

in Sections 2. Our experiments confirm that Kandoo scalesan order of magnitude better than a normal OpenFlownetwork and would lead to more than 90% of eventsbeing processed locally under reasonable assumptions, asdescribed in Section 3. Applications of Kandoo are notlimited to the evaluation scenarios presented in this paper.In Section 4, we briefly discuss other potential applicationsof Kandoo and compare it to existing solutions. We concludeour discussion in Section 5.

2. DESIGN AND IMPLEMENTATIONDesign objectives. Kandoo is designed with the followinggoals in mind. First, Kandoo must be compatible withOpenFlow: we do not introduce any new data planefunctionality in switches, and, as long as they supportOpenFlow, Kandoo supports them, as well. Second, Kandooautomatically distributes control applications without anymanual intervention. In other words, Kandoo control

Root Controller

Local ControllerLocal Controller

SwitchSwitch

Appdetect

End-host Switch

Appdetect

End-host

Flow-Entry

Eelephant

LegendLogical Control ChannelDatapath Connection

Local Controller

Appdetect

Appreroute

Flow-Entry

Figure 2: Toy example for Kandoo’s design: Inthis example, two hosts are connected using a simpleline topology. Each switch is controlled by one localKandoo controller. The root controller controls thelocal controllers. In this example, we have two controlapplications: Appdetect is a local control application, butAppreroute is non-local.

applications are not aware of how they are deployed inthe network, and application developers can assume theirapplications would be run on a centralized OpenFlowcontroller. The only extra information Kandoo needs is aflag showing whether a control application is local or not.In what follows, we explain Kandoo’s design using a

toy example. We show how Kandoo can be used toreroute elephant flows in a simple network of three switches(Figure 2). Our example has two applications: (i) Appdetect,and (ii) Appreroute. Appdetect constantly queries each switchto detect elephant flows. Once an elephant flow is detected,Appdetect notifies Appreroute, which in turn may install orupdate flow-entries on network switches.It is extremely challenging, if not impossible, to implement

this application in current OpenFlow networks withoutmodifying switches [5]. If switches are not modified, a(logically) centralized control needs to frequently query allswitches, which would place a considerable load on controlchannels.

Kandoo Controller. As shown in Figure 3, Kandoo hasa controller component at its core. This component hasthe same role as a general OpenFlow controller, but ithas Kandoo-specific extensions for identifying applicationrequirements, hiding the complexity of the underlyingdistributed application model, and propagating events in thenetwork.A network controlled by Kandoo has multiple local

controllers and a logically centralized root controller.1 Thesecontrollers collectively form Kandoo’s distributed controlplane. Each switch is controlled by only one Kandoocontroller, and each Kandoo controller can control multipleswitches. If the root controller needs to install flow-entrieson switches of a local controller, it delegates the requeststo the respective local controller. Note that for highavailability, the root controller can register itself as the slavecontroller for a specific switch (this behavior is supported inOpenFlow 1.2 [1]).

1We note that the root controller in Kandoo can itself belogically/physically distributed. In fact, it is straightforwardto implement Kandoo’s root controller using Onix [8] orHyperflow [18].

20

Switch

Local Controller

Switch Switch Switch Switch

Local Controller

LocalController

Root Controller

RareEvents

FrequentEvents

Figure 1: Kandoo’s Two Levels of Controllers. Localcontrollers handle frequent events, while a logicallycentralized root controller handles rare events.

control applications. As illustrated in Figure 1, severallocal controllers are deployed throughout the network; eachof these controllers controls one or a handful of switches.The root controller, on the other hand, controls all localcontrollers.It is easy to realize local controllers since they are

merely switch proxies for the root controller, and theydo not need the network-wide state. They can even beimplemented directly in OpenFlow switches. Interestingly,local controllers can linearly scale with the number ofswitches in a network. Thus, the control plane scales as longas we process frequent events in local applications and shieldthe root controller from these frequent events. Needlessto say, Kandoo cannot help any control applications thatrequire network-wide state (even though it does not hurtthem, either). We believe such applications are intrinsicallyhard to scale, and solutions like Onix [8] and HyperFlow [18]provide the right frameworks for running such applications.Our implementation of Kandoo is completely compliant

with the OpenFlow specifications. Data and controlplanes are decoupled in Kandoo. Switches can operatewithout having a local controller; control applicationsfunction regardless of their physical location. The mainadvantage of Kandoo is that it gives network operatorsthe freedom to configure the deployment model of controlplane functionalities based on the characteristics of controlapplications.The design and implementation of Kandoo are presented

in Sections 2. Our experiments confirm that Kandoo scalesan order of magnitude better than a normal OpenFlownetwork and would lead to more than 90% of eventsbeing processed locally under reasonable assumptions, asdescribed in Section 3. Applications of Kandoo are notlimited to the evaluation scenarios presented in this paper.In Section 4, we briefly discuss other potential applicationsof Kandoo and compare it to existing solutions. We concludeour discussion in Section 5.

2. DESIGN AND IMPLEMENTATIONDesign objectives. Kandoo is designed with the followinggoals in mind. First, Kandoo must be compatible withOpenFlow: we do not introduce any new data planefunctionality in switches, and, as long as they supportOpenFlow, Kandoo supports them, as well. Second, Kandooautomatically distributes control applications without anymanual intervention. In other words, Kandoo control

Root Controller

Local ControllerLocal Controller

SwitchSwitch

Appdetect

End-host Switch

Appdetect

End-host

Flow-Entry

Eelephant

LegendLogical Control ChannelDatapath Connection

Local Controller

Appdetect

Appreroute

Flow-Entry

Figure 2: Toy example for Kandoo’s design: Inthis example, two hosts are connected using a simpleline topology. Each switch is controlled by one localKandoo controller. The root controller controls thelocal controllers. In this example, we have two controlapplications: Appdetect is a local control application, butAppreroute is non-local.

applications are not aware of how they are deployed inthe network, and application developers can assume theirapplications would be run on a centralized OpenFlowcontroller. The only extra information Kandoo needs is aflag showing whether a control application is local or not.In what follows, we explain Kandoo’s design using a

toy example. We show how Kandoo can be used toreroute elephant flows in a simple network of three switches(Figure 2). Our example has two applications: (i) Appdetect,and (ii) Appreroute. Appdetect constantly queries each switchto detect elephant flows. Once an elephant flow is detected,Appdetect notifies Appreroute, which in turn may install orupdate flow-entries on network switches.It is extremely challenging, if not impossible, to implement

this application in current OpenFlow networks withoutmodifying switches [5]. If switches are not modified, a(logically) centralized control needs to frequently query allswitches, which would place a considerable load on controlchannels.

Kandoo Controller. As shown in Figure 3, Kandoo hasa controller component at its core. This component hasthe same role as a general OpenFlow controller, but ithas Kandoo-specific extensions for identifying applicationrequirements, hiding the complexity of the underlyingdistributed application model, and propagating events in thenetwork.A network controlled by Kandoo has multiple local

controllers and a logically centralized root controller.1 Thesecontrollers collectively form Kandoo’s distributed controlplane. Each switch is controlled by only one Kandoocontroller, and each Kandoo controller can control multipleswitches. If the root controller needs to install flow-entrieson switches of a local controller, it delegates the requeststo the respective local controller. Note that for highavailability, the root controller can register itself as the slavecontroller for a specific switch (this behavior is supported inOpenFlow 1.2 [1]).

1We note that the root controller in Kandoo can itself belogically/physically distributed. In fact, it is straightforwardto implement Kandoo’s root controller using Onix [8] orHyperflow [18].

20

From [8]

Non-local ApplicationNon-local

Application

Local ApplicationLocal

Application

Data-path Element

Local Controller

Local Applications

RegisterApplication

ActionsEvents

Events

Root Controller

RelayEvents

Subscribe to Events

OpenFlowPackets

Data-path ElementSwitches

API

API Non-local

ApplicationsActionsEvents

Events

RegisterApplication

Kandoo Controllers

Figure 3: Kandoo’s high level architecture.

Deployment Model. The deployment model of Kandoocontrollers depends on the characteristics of a network.For software switches, local controllers can be directlydeployed on the same end-host. Similarly, if we can changethe software of a physical switch, we can deploy Kandoodirectly on the switch. Otherwise, we deploy Kandoolocal controllers on the processing resources closest to theswitches. In such a setting, one should provision thenumber of local controllers based on the workload andavailable processing resources. Note that we can use ahybrid model in real settings. For instance, consider avirtualized deployment environment depicted in Figure 4,where virtual machines are connected to the network usingsoftware switches. In this environment, we can place localcontrollers in end-hosts next to software switches and inseparate nodes for other switches.In our toy example (Figure 2), we have four Kandoo

controllers: three local controllers controlling the switchesand a root controller. The local controllers can be physicallypositioned using any deployment model explained above.Note that, in this example, we have the maximum numberof local controllers required.

Control Applications. Control applications functionusing the abstraction provided by the controller and are notaware of Kandoo internals. They are generally OpenFlowapplications and can therefore send OpenFlow messagesand listen on events. Moreover, they can emit Kandooevents (i.e., internal events), which can be consumed byother applications, and they can reply to the applicationthat emitted an event. Control applications are loaded inlocal name spaces and can communicate using only Kandooevents. This is to ensure that Kandoo does not introducefaults by o�oading applications.In our example, Eelephant is a Kandoo event that carries

matching information about the detected elephant flow (e.g.,its OpenFlow match structure) and is emitted by Appdetect.A local controller can run an application only if the

application is local. In our example, Appreroute is not local,i.e., it may install flow-entires on any switch in the network.Thus, the root controller is the only controller able to runAppreroute. In contrast, Appdetect is local; therefore, allcontrollers can run it.

Event Propagation. The root controller can subscribe

Processing Resource

End-Host

Physical SwitchVM

Root Controller

Local Controller

Local Controller

Software Switch

VM

LegendLogical Control ChannelData Path Links

Physical SwitchVM

Figure 4: Kandoo in a virtualized environment. Forsoftware switches, we can leverage the same end-host for local controllers, and, for physical switches,we use separate processing resources.

to specific events in the local controllers using a simplemessaging channel plus a filtering component. Once thelocal controller receives and locally processes an event, itrelays the event to the root controller for further processing.Note that all communications between Kandoo controllersare event-based and asynchronous.In our example, the root controller subscribes to events

of type Eelephant in the local controllers since it is runningAppreroute listening on Eelephant. Eelephant is fired by anAppdetect instance deployed on one of the local controllersand is relayed to root controller. Note that if theroot controller does not subscribe to Eelephant, the localcontrollers will not relay Eelephant events.

It is important to note that the data flow in Kandoo is notalways bottom-up. A local application can explicitly requestdata from an application deployed on the root controller byemitting an event, and applications on the root controllerscan send data by replying to that event. For instance, we canhave a topology service running on the root controller thatsends topology information to local applications by replyingto events of a specific type.

Reactive vs. Proactive. Although Kandoo provides ascalable method for event handling, we strongly recommendpushing network state proactively. We envision Kandoo tobe used as a scalable, adaptive control plane, where thedefault configuration is pushed proactively and is adaptivelyrefined afterwards. In our toy example, default paths canbe pushed proactively, while elephant flows will be reroutedadaptively.

Implementation Details. We implemented Kandoo in amixture of C, C++, and Python. Our implementation hasa low memory footprint and supports dynamically loadableplug-ins, which can be implemented in C, Python, andJava. It also provides an RPC API for more generalintegration scenarios. Our implementation of Kandoois extremely modular; any component or back-end canbe easily replaced, which simplifies porting Kandoo tophysical switches. Currently, Kandoo supports OpenFlow1.0 (OpenFlow 1.1 and 1.2 support is under development).For the applications, we created a “central application

repository” and developed a simple package management

21


118

Credible suggestion?

q Questions to ask: 1. Is there a significant amount of control problems that can be solved

based on local view? ■ If not, what’s the point?

2. Is processing power for such local controllers available locally? ■ If not, we would have to ship all the “local” control traffic to a remote

place, and then again, what’s the point?

q Tentative answers: 1. Yes, for some problems: Link discovery, policy enforcement, detect big

flows 2. Yes, in some environments: data centre networks, possibly also

enterprise networks

q Main difference to ONIX: No attempt is made to keep state even only eventually consistent


119

Kandoo performance

0 500

1000 1500 2000 2500 3000

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Mes

sage

s Per

Sec

ond

Ratio of Elephant to Mouse Flows

Normal OpenFlow Controller Kandoo's Root Controller

(a) Average number of messages received by thecontrollers

0 50

100 150 200 250 300

10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Band

widt

h Co

nsum

ptio

ns

(kB/

s)

Ratio of Elephant to Mouse Flows

Normal OpenFlow Controller Kandoo's Root Controller

(b) Average number of bytes received by the con-trollers

Figure 6: Control Plane Load for the ElephantFlow Detection Scenario. The load is based on thenumber of elephant flows in the network.

4. RELATED WORKDatapath Extensions. The problem that we tackle inthis paper is a generalization of several previous attemptsat scaling SDNs. A class of solutions, such as DIFANE [21]and DevoFlow [5], address this problem by extending dataplane mechanisms of switches with the objective of reducingthe load towards the controller. DIFANE tries to partlyo⇥oad forwarding decisions from the controller to specialswitches, called authority switches. Using this approach,network operators can reduce the load on the controller andthe latencies of rule installation. DevoFlow, on the otherhand, introduces new mechanisms in switches to dispatch farfewer “important” events to the control plane. Kandoo hasthe same goal, but, in contrast to DIFANE and DevoFlow,it does not extend switches; instead, it moves control planefunctions closer to switches. Kandoo’s approach is moregeneral and works well in data centers, but it might havea lower throughput than specific extensions implemented inhardware.

Interestingly, we can use Kandoo to prototype and testDIFANE, DevoFlow, or other potential hardware extensions.For instance, an authority switch in DIFANE can beemulated by a local Kandoo controller that manages a subsetof switches in the network. As another example, DevoFlow’sextensions can also be emulated using Kandoo controllersdirectly installed on switches. These controllers not onlyreplace the functionality of DIFANE or DevoFlow, but they

0 1000 2000 3000 4000 5000 6000

2 3 4 5 6 7

Mes

sage

s Per

Sec

ond

Fanout

Normal Openflow Root Controller

(a) Average number of packets received by the controllers

0 100 200 300 400 500 600

2 3 4 5 6 7 Ba

ndwi

dth

Cons

umpt

ion

(k

B/s)

Fanout

Normal Openflow Root Controller

(b) Average number of bytes received by the controllers

Figure 7: Control Plane Load for the ElephantFlow Detection Scenario. The load is based on thenumber of nodes in the network.

also provide a platform to run any local control applicationin their context.

Distributed Controllers. HyperFlow [18], Onix [8],SiBF [10], and Devolved Controllers [17] try to distributethe control plane while maintaining logically centralized,eventually consistent network state. Although these ap-proaches have their own merits, they impose limitations onapplications they can run. This is because they assumethat all applications require the network-wide state; hence,they cannot be of much help when it comes to local controlapplications. That said, the distributed controllers can beused to realize a scalable root controller, the controller thatruns non-local applications in Kandoo.

Middleboxes. Middlebox architectures, such as Flow-Stream [7], SideCar [15] and CoMb [13], provide scalableprogrammability in data plane by intercepting flows usingprocessing nodes in which network applications are de-ployed. Kandoo is orthogonal to these approaches in thesense that it operates in the control plane, but it providesa similar distribution for control applications. In a networkequipped with FlowStream, SideCar or CoMb, Kandoo canshare the processing resources with middleboxes (given thatcontrol and data plane applications are isolated) in orderto increase resource utilization and decrease the number ofnodes used by Kandoo.

Active Networks. Active Networks (AN) and SDNs repre-sent di�erent schools of thought on programmable networks.SDNs provide programmable control planes, whereas ANsallow programmability in networking elements at packet

23

From [8]


120

Case study: DevoFlow [6]

q Claims: OpenFlow suffers from overhead and restrictions inq … the amount of work the controller has to do (“distributed system costs”) q … the amount of effort necessary for the switch to invoke its control plane

(“switch-implementation cost”) ■ For their environment, data-plane to control-plane is 4x slower than

forwarding q Relevant both for flow setup and for gathering statistics

q This overhead is caused by the desire to have all flows globally visible (at the controller) and globally controlled q Maybe that was an exaggerated goal?? q Maybe only bother the controller with “important” flows, let the switches

deal with the ordinary ones?


121

DevoFlow design goals

q Keep flows in the data plane as much as possible q Maintain enough visibility over network flows for effective centralized

flow management q Simplify design and implementation of fast switches

q Goals motivated by measurements on prototype OpenFlow hardware q E.g.: HP ProCurve 5406zl switch: 300 Gbps data plane in linecard, but

only 80 Mbps between linecard and switch management CPU q Question: How representative are these measurements?


122

DevoFlow: Mechanisms for devolving control

q Rule cloning: The “action” of a wildcard rule is extended by a CLONE flag q If set: upon packet arrival, copy the wildcard rule into a specific rule,

replacing the wildcard fields by values from actual packetq If unset: follow usual OF rules q Purpose:

■ Packets from such a flow will then update counters for the specific rule, not the wildcard rule

q Local actions: Allow more powerful actions q E.g., switch between alternative ports (realizing multi-path protocols) q Intermediate between “trivial forwarding” and “talk to controller”


123

DevoFlow: Extended statistics collection

q Sampling: randomly pick 1/n packet of flow, send (header) to controllerq Triggers: Once a counter crosses a threshold, send a report packet to

a controller q Approximate counters: Maintain counters for k largest flows more in

detail


124

DevoFlow example usage: Load balance big flows

q Claims: q Load-balancing on all flows is infeasible (controller overhead,…)q Only the big flows (“elephants”) matter; small flows can take random route

■ Define: a flow is an elephant flow if it has transferred at least X bytes (e.g., 1-10 Mbytes)

q Approachq Treat all flows with random multi-path wildcards (ECMP) q Once counter crosses threshold, report to controller q Controller computes routes for this newly-so-classified elephant flow and

distributes according forwarding table entries

q Discuss: q Relationship to sFlow/Netflow/IPFIX?q What if flows are Markovian?q Is bandwidth orientated routing a good thing in general?


125

Controller placement

q Given a large SDN: q How many controllers do we need? q Where should they be placed?

q Important for convergence time, flow establishment, … q E.g.: Internet2 about to run SDN in a 34 node production network

q Metrics: Minimize average-case or worst-case latency


126

Controller placement – formalization

q All variations of facility location, all are NP-hard q (see http://www.nada.kth.se/~viggo/wwwcompendium)q Minimum k-median: Not in APX q Minimum k-center:

■ Under triangle inequality, approximable within 2, but not 2-𝜖■ Else: Not in APX; capacitated version approximable within 5 (!)

q Maximum cover: greedy algorithm has best possible 1-1/e approximation

2. CONTROL PLANE DESIGNBoth decoupled and more traditional distributed archi-

tectures have a control plane: a network for propagatingevents, such as routing updates, new tra⇥c engineering poli-cies, or topology changes, to each packet-forwarding devicein the network. The key di�erence between these designsis the structure of their control-plane network. The controltopology of traditional distributed architectures like BGP,OSPF, and IS-IS is peer-to-peer: each forwarding devicehears events from its peers and makes autonomous decisionsbased on a local (and likely inconsistent) view of global state.

In contrast, the control networks in decoupled architec-tures are closer to client-server networks. Packet-forwardingdevices (the “clients”) have limited decision-making capabil-ities and must implement control decisions made by con-trollers (the “servers”). A common example is a BGP routereflector that presents each edge router with a subset of ad-vertised prefixes, rather than propagating the full mesh oflearned routes. A more recent example is the relationshipbetween an OpenFlow switch and controller; the switch hasno local control-plane logic and relies entirely on the con-troller to populate its forwarding table.1 Control networksfor SDNs may take any form, including a star (a single con-troller), a hierarchy (a set of controllers connected in a fullmesh, which connect to forwarding nodes below), or evena dynamic ring (a set of controllers in a distributed hashtable [16]).

Regardless of the exact form, the layout of controllers willa�ect the network’s ability to respond to network events.Understanding where to place controllers2 and how manyto use is a prerequisite to answering performance and faulttolerance questions for SDNs, and hence also a prerequisitefor quantitatively comparing them to traditional architec-tures. We call this design choice the controller placementproblem. In this paper, we consider only wide-area networkswhere the “best” controller placement minimizes propaga-tion delays; in a data center or in the enterprise, one mightinstead maximize fault tolerance or actively balance con-trollers among administrative domains.

For WANs, the best placement depends on propagationlatency, a quantity fixed by physics and physical topology.Propagation latency bounds the control reactions with a re-mote controller that can be executed at reasonable speedand stability. With enough delay, real-time tasks (like link-layer fault recovery) become infeasible, while others mayslow down unacceptably (BGP convergence). Note that re-gardless of the state consistency mechanisms in the controlplane implementation, these lower bounds apply.

In this paper, we compare placements using node-to-controller latency, for the fundamental limits imposed onreaction delay, fault discovery, and event propagation e⇥-ciency. Other metrics matter, such as availability and fair-ness of state, processing, and bandwidth — but our focus isthe WAN, where latency dominates. One can always reducethe e�ective delay by adding autonomous intelligence intoa switch or pushing failover plans, but these may add com-plexity and make network evolution harder. One goal of thispaper is to understand if, and for which networks, extensionsto the “dumb, simple switch” model are warranted.

1 We ignore “bootstrap state” for the control connection.2 We use“controllers” to refer to geographically distinct con-troller locations, as opposed to individual servers.

3. MOTIVATING EXAMPLESHaving defined the core problem, we show three types of

SDN users and motivate their use for controller placementdesign guidelines.

Network Operators. Rob Vietzke is the VP of Net-work Services at Internet2, and his team has committed toa SDN deployment of 34 nodes and about 41 edges, shownin Figure 1. This network, the Open Science, Scholarshipand Services Exchange (OS3E) [4], needs to peer with theoutside world through BGP. Placement matters to Rob be-cause his network should minimize downtime and multiplecontrollers are a requirement for high availability.

Controller Application Writers. Nikhil Handigol is agrad student who created Aster*x [13], a distributed loadbalancer that reactively dispatches requests to servers aswell as managing the network path taken by those requests.Nikhil would like to demonstrate the advantages of his algo-rithm on a service with real users, and ideally on a range oftopologies, like GENI. Placement matters to Nikhil becausehe can’t get users if his service goes down or does not per-form, but at the same time he would prefer to keep thingssimple with one controller. Ideally, we could provide Nikhilwith guidelines to evaluate the response-time potential ofdi�erent approaches, from centralized to distributed, beforehe implements extra code or does a deployment.

Network Management Software Writers. Rob Sher-wood built FlowVisor [19], a centralized network slicing toolthat enables network access and control to be split amongmultiple controllers or versions of controllers, given a controlpolicy. Since FlowVisor’s only consistent state is its configu-ration, multiple instances might be used to scale FlowVisor.Placement matters to Rob because FlowVisor sits betweencontrollers and switches, where its presence adds a delay topotentially every network command; this delay should beactively minimized, especially with multiple instances.

In each case, the SDN user must ask the question: “Howmany controllers should I use, and where should they go?”and benefits from practical methods for analyzing tradeo�s.

4. PLACEMENT METRICSWe now introduce and compare definitions of whole-

network latency, along with their corresponding optimiza-tion problems. Each is called a facility location problemand appears in many contexts, such as minimizing firefight-ing response times, locating warehouses near factories, andoptimizing the locations of content distribution nodes andproxy servers. All are NP-hard problems with an input fork, the number of controllers to place, and all have weightedvariations where nodes have varying importance.

Average-case Latency. For a network graph G(V,E)where edge weights represent propagation latencies, whered(v, s) is the shortest path from node v � V to s � V ,and the number of nodes n = |V |, the average propagationlatency for a placement of controllers S0 is:

Lavg(S0) =

1n

�

v2V

min(s2S0)

d(v, s) (1)

In the corresponding optimization problem, minimum k-median [6], the goal is to find the placement S0 from the setof all possible controller placements S, such that |S0| = kand Lavg(S

0) is minimum. For an overview of the approachesto solving this problem, along with extensions, see [20].

8

location in average-latency-optimized placement!

k = 1 k = 5

location in worst-case-latency-optimized placement!

Figure 1: Optimal placements for 1 and 5 controllersin the Internet2 OS3E deployment.

Worst-case latency. An alternative metric is worst-caselatency, defined as the maximum node-to-controller propa-gation delay:

Lwc(S0) = max

(v2V )min

(s2S0)d(v, s) (2)

where again we seek the minimum S0 � S. The relatedoptimization problem is minimum k-center [21].

Nodes within a latency bound. Rather than mini-mizing the average or worst case, we might place controllersto maximize the number of nodes within a latency bound;the general version of this problem on arbitrary overlap-ping sets is called maximum cover [14]. An instance ofthis problem includes a number k and a collection of setsS = S1, S2, ..., Sm, where Si � v1, v2, ..., vn. The objectiveis to find a subset S0 � S of sets, such that |

�Si2S0 Si| is

maximized and |S0| = k. Each set Si comprises all nodeswithin a latency bound from a single node.

In the following sections, we compute only average andworst-case latency, because these metrics consider the dis-tance to every node, unlike nodes within a latency bound.Each optimal placement shown in this paper comes fromdirectly measuring the metrics on all possible combinationsof controllers. This method ensures accurate results, but atthe cost of weeks of CPU time; the complexity is exponentialfor k, since brute force must enumerate every combinationof controllers. To scale the analysis to larger networks orhigher k, the facility location problem literature providesoptions that trade o� solution time and quality, from simplegreedy strategies (pick the next vertex that best minimizeslatency, or pick the vertex farthest away from the current se-lections) to ones that transform an instance of k-center intoother NP-complete problems like independent set, or evenones that use branch-and-bound solvers with Integer LinearProgramming. We leave their application to future work.

5. ANALYSIS OF INTERNET2 OS3EHaving defined our metrics, we now ask a series of ques-

tions to understand the benefits of multiple controllers forthe Internet2 OS3E topology [4]. To provide some intuitionfor placement considerations, Figure 1 shows optimal place-ments for k = 1 and k = 5; the higher density of nodes in thenortheast relative to the west leads to a di�erent optimal setof locations for each metric. For example, to minimize av-erage latency for k = 1, the controller should go in Chicago,which balances the high density of east coast cities with thelower density of cities in the west. To minimize worst-caselatency for k = 1, the controller should go in Kansas Cityinstead, which is closest to the geographic center of the US.

k = 5!4! 3! 2! 1! k = 5!4! 3!2! 1!

Figure 2: Latency CDFs for all possible controllercombinations for k = [1, 5]: average latency (left),worst-case latency (right).

Figure 3: Ratio of random choice to optimal.

5.1 How does placement affect latency?In this topology, placement quality varies widely. A few

placements are pathologically bad, most are mediocre, andonly a small percent approach optimal. Figure 2 shows thisdata as cumulative distributions, covering all possible place-ments for k = 1 to k = 5, with optimal placements at thebottom. All graphs in this paper show one-way network dis-tances, with average-optimized values on the left and worst-case-optimized values on the right. If we simply choose aplacement at random for a small value of k, the averagelatency is between 1.4x and 1.7x larger than that of the op-timal placement, as seen in Figure 3. This ratio is largerfor worst-case latencies; it starts at 1.4x and increases up to2.5x at k = 12. Spending the cycles to optimize a placementis worthwhile.

5.2 How many controllers should we use?It depends. Reducing the average latency to half that at

k = 1 requires three controllers, while the same reductionfor worst-case latency requires four controllers. Assumingwe optimize for one metric, potentially at the expense of theother, where is the point of diminishing returns? Figure 4shows the benefit-to-cost ratios for a range of controllers, de-fined as (lat1/latk)/k. A ratio of 1.0 implies a proportionalreduction; that is, for k controllers, the latency is 1/k of

Figure 4: Cost-benefit ratios: a value of 1.0 indicatesproportional reduction, where k controllers reducelatency to 1

k of the original one-controller latency.Higher is better.

9

From [9]


127

Controller placement in Internet2 topology

q Lessons: q Random choice is bad q Sometimes, even one controller is

enough q More data from Internet Topology

ZooFrom [9]


k = 1 k = 5


Figure 1: Optimal placements for 1 and 5 controllersin the Internet2 OS3E deployment.

Worst-case latency. An alternative metric is worst-caselatency, defined as the maximum node-to-controller propa-gation delay:

Lwc(S0) = max

(v2V )min

(s2S0)d(v, s) (2)

where again we seek the minimum S0 � S. The relatedoptimization problem is minimum k-center [21].

Nodes within a latency bound. Rather than mini-mizing the average or worst case, we might place controllersto maximize the number of nodes within a latency bound;the general version of this problem on arbitrary overlap-ping sets is called maximum cover [14]. An instance ofthis problem includes a number k and a collection of setsS = S1, S2, ..., Sm, where Si � v1, v2, ..., vn. The objectiveis to find a subset S0 � S of sets, such that |

�Si2S0 Si| is

maximized and |S0| = k. Each set Si comprises all nodeswithin a latency bound from a single node.

In the following sections, we compute only average andworst-case latency, because these metrics consider the dis-tance to every node, unlike nodes within a latency bound.Each optimal placement shown in this paper comes fromdirectly measuring the metrics on all possible combinationsof controllers. This method ensures accurate results, but atthe cost of weeks of CPU time; the complexity is exponentialfor k, since brute force must enumerate every combinationof controllers. To scale the analysis to larger networks orhigher k, the facility location problem literature providesoptions that trade o� solution time and quality, from simplegreedy strategies (pick the next vertex that best minimizeslatency, or pick the vertex farthest away from the current se-lections) to ones that transform an instance of k-center intoother NP-complete problems like independent set, or evenones that use branch-and-bound solvers with Integer LinearProgramming. We leave their application to future work.

5. ANALYSIS OF INTERNET2 OS3EHaving defined our metrics, we now ask a series of ques-

tions to understand the benefits of multiple controllers forthe Internet2 OS3E topology [4]. To provide some intuitionfor placement considerations, Figure 1 shows optimal place-ments for k = 1 and k = 5; the higher density of nodes in thenortheast relative to the west leads to a di�erent optimal setof locations for each metric. For example, to minimize av-erage latency for k = 1, the controller should go in Chicago,which balances the high density of east coast cities with thelower density of cities in the west. To minimize worst-caselatency for k = 1, the controller should go in Kansas Cityinstead, which is closest to the geographic center of the US.

k = 5!4! 3! 2! 1! k = 5!4! 3!2! 1!

Figure 2: Latency CDFs for all possible controllercombinations for k = [1, 5]: average latency (left),worst-case latency (right).

Figure 3: Ratio of random choice to optimal.

5.1 How does placement affect latency?In this topology, placement quality varies widely. A few

placements are pathologically bad, most are mediocre, andonly a small percent approach optimal. Figure 2 shows thisdata as cumulative distributions, covering all possible place-ments for k = 1 to k = 5, with optimal placements at thebottom. All graphs in this paper show one-way network dis-tances, with average-optimized values on the left and worst-case-optimized values on the right. If we simply choose aplacement at random for a small value of k, the averagelatency is between 1.4x and 1.7x larger than that of the op-timal placement, as seen in Figure 3. This ratio is largerfor worst-case latencies; it starts at 1.4x and increases up to2.5x at k = 12. Spending the cycles to optimize a placementis worthwhile.

5.2 How many controllers should we use?It depends. Reducing the average latency to half that at

k = 1 requires three controllers, while the same reductionfor worst-case latency requires four controllers. Assumingwe optimize for one metric, potentially at the expense of theother, where is the point of diminishing returns? Figure 4shows the benefit-to-cost ratios for a range of controllers, de-fined as (lat1/latk)/k. A ratio of 1.0 implies a proportionalreduction; that is, for k controllers, the latency is 1/k of






Figure�3:� Ratio�of� random� choice� to�optimal.


k = 1 k = 5


Figure�1:�Optimal�placements�for�1�and�5�controllers�in� the� Internet2�OS3E�deployment.

128

Content



129

Deficits of OpenFlow

q Development of OpenFlow calmed downq OpenFlow 1.5 not deployed in HWq OpenFlow 1.6 standard is not being adopted

q Technical reasonsq Heavily burdens TCAMs,

e.g. OF 1.4 has 41 header fields to be matched uponq Not adjustable to new protocolsq Communications with controller introduces delayq Many optional parts Very few features developers may depend on

q Other reasonsq Cisco favors a declarative approach: ACI


130

“P4”

q Domain-specific language to program switches/routersq Named after “Programming Protocol-Independent Packet Processors”

SIGCOMM paper [10] (McKeown among others)

q Approach: q Program functionality for protocol parsing (entirely) in a DSL

■ C-style, but more declarative and no loops…■ May be compiled to different platforms ASICs, FPGAs, CPUs■ Compiler envisioned to be provided by switch manufacturer

q Export run-time configuration interface, i.e., tables, to OpenFlow-like mechanisms

q No direct functionality to store state (e.g. variables), but manipulation of tables possible


131

P4 – Programming model


From [15]

132

P4 – Program structure

q Each program consists of three parts

q Parser:q Runs through a state machine for each packetq Determines all relevant header structuresq May either accept or reject a packet

q Match-Action Pipeline: Looks up output port and other “actions” based on extracted headers & input port, e.g. in a TCAM

q Deparser: Emits a packet to an output port based on parsed packet

q Pipelines and parsers may be run in a loop to process more complex packets…


133

P4 – Example: A simple IPv4 router (I)


From [11]

134

P4 – Example: A simple IPv4 router (II)


parser TopParser(packet_in b, out Parsed_packet p) {Checksum16() ck; // instantiate checksum unitstate start {

b.extract(p.ethernet);transition select(p.ethernet.etherType) {

// no default rule: all other packets rejected0x0800: parse_ipv4; }

}

state parse_ipv4 {

b.extract(p.ip);

verify(p.ip.version == 4w4, error.IPv4IncorrectVersion);verify(p.ip.ihl == 4w5, error.IPv4OptionsNotSupported);

ck.clear();

ck.update(p.ip);

// Verify that packet checksum is zeroverify(ck.get() == 16w0, error.IPv4ChecksumError);

transition accept; }

}

135

P4 – Example: A simple IPv4 router (III)


control TopPipe(inout Parsed_packet headers,in error parseError, // parser errorin InControl inCtrl, // input port

out OutControl outCtrl) {

IPv4Address nextHop; // local variable

....

apply {if (parseError != error.NoError) {

Drop_action(); // invoke drop directly

return; }

ipv4_match.apply(); // Match result will go into nextHopif (outCtrl.outputPort == DROP_PORT) return;

dmac.apply(); // Set destination MAC based on nextHopif (outCtrl.outputPort == DROP_PORT) return;

smac.apply(); // Set source MAC }

}

136

P4 – Example: A simple IPv4 router (IV)


action Drop_action() {outCtrl.outputPort = DROP_PORT;

}

action Set_nhop(IPv4Address ipv4_dest, PortId port) {

nextHop = ipv4_dest;headers.ip.ttl = headers.ip.ttl - 1;outCtrl.outputPort = port;

}

table ipv4_match {key = { headers.ip.dstAddr: lpm; } // longest-prefix match

actions = {

Drop_action;

Set_nhop;

}

size = 1024;

default_action = Drop_action;

}

137

P4 – Example: A simple IPv4 router (V)


// deparser sectioncontrol TopDeparser(inout Parsed_packet p, packet_out b) {

Checksum16() ck;

apply {

b.emit(p.ethernet);

if (p.ip.isValid()) {

ck.clear(); // prepare checksum unitp.ip.hdrChecksum = 16w0; // clear checksum

ck.update(p.ip); // compute new checksum.

p.ip.hdrChecksum = ck.get();

}

b.emit(p.ip);

}

}

138

P4 - Discussion

q SRAM/TCAM resources may be used much more fine grainedq Unknown protocols may be parsed after software update

q Adoption by hardware vendors?q Currently there are NICs by Netronome, FPGA software by Xilinxq Most research uses software implementations

q How often do protocol specs change really?q What does line speed mean for P4 switch?q Table definition varies: Handling in controllers?q Externs will vary for different vendors/products – How can we write

portable code?q What about network slicing?q …


139

Content



140

Whitebox Switching/Bare Metal Switching

q Taking the idea of customer influence a step further: q OS on switches interchangeableq Standardized interface to switching hardware



OperatingSystem

App App App

yesterday


YourOperatingSystem

App App App

in a few hours


OperatingSystem

App App App

today

OEM Chip

OEM

141

Vision of a white box switch


Switching Chassis etc.

OperatingSystem

App App App

Silicon

• Using OpenFlow, traditional protocols or something else

• Cumulus, PicaOS, OpenNetworkLinux, SONiC …

• Quanta, Cisco, Dell...

• Mellanox, Broadcom, Marvel

142

Open Network Install Environment (ONIE)

q Open source component to bootstrap custom OS on switchesq De-facto standardq Supported by many big players: Big Switch, Quanta, Broadcom, Mellanox

etc.q Based on bootloader & ~4MB Linux imageq Provides install environment

q Over network (loaded over some URL)q Over USB stickq Includes automation support (think of data centers)

q Located in extra partition on x86

q Note: only a prerequisite to boot your own OS, not a guarantee!


143

Switch Abstraction Interface (SAI)

q Standardized interface for different hardware platforms from network OS

q Introduced by DELL & pushed by Microsoft (think of Azure!), wide support from industry

q Basically bunch of C header filesq “Objectified” calling convention (pain to debug)q >10 modules in API in 37 filesq >5000 lines of pure C code (22731 lines total)q Still changing rapidly


144

Whitebox Switching/Bare Metal Switching

q What does it mean to SDN?q What does it mean to OpenFlow?


https://xkcd.com/927/

145

References

1. Various talks by Nick McKeown on SDN, http://yuba.stanford.edu/~nickm/talks.html

2. B. Heller et al., Tutorial: SDN for Engineers, Open Networking Summit, Santa Clara, April 2012. http://www.opennetsummit.org/tutorials.html

3. M. Casado, M. J. Freedman, J. Pettit, N. Gude, N. McKeown, and S. Shenker, “Rethinking Enterprise Network Control,” IEEE/ACM Transactions on Networking, vol. 17, no. 4, pp. 1270–1283, Aug. 2009.

4. M. Casado, T. Koponen, S. Shenker, and A. Tootoonchian, “Fabric: a retrospective on evolving SDN,” in Proceedings of the first workshop on Hot topics in software defined networks - HotSDN ’12, 2012, pp. 85–89.


http://yuba.stanford.edu/~nickm/talks.html

http://www.opennetsummit.org/tutorials.html

146

References

5. S. Yeganeh, A. Tootoonchian, and Y. Ganjali, “On scalability of software-defined networking,” IEEE Communications Magazine, no. February, pp. 136–141, 2013.

6. A. R. Curtis, J. C. Mogul, J. Tourrilhes, P. Yalagandula, P. Sharma, and S. Banerjee, “DevoFlow : Scaling Flow Management for High-Performance Networks,” in ACM SIGCOMM, 2011, pp. 254–265.

7. T. Koponen, M. Casado, N. Gude, J. Stribling, L. Poutievski, M. Zhu, R. Ramanathan, Y. Iwata, H. Inoue, T. Hama, and S. Shenker, “Onix: a distributed control platform for large-scale production networks,” in Proceedings of the 9th USENIX conference on Operating systems design and implementation, 2010.

8. S. H. Yeganeh and Y. Ganjali, “Kandoo: a framework for efficient and scalable offloading of control applications,” in Proceedings of the first workshop on HotSDN, 2012, pp. 19–24.

9. B. Heller, R. Sherwood, and N. McKeown, “The controller placement problem,” in ACM SIGCOMM Computer Communication Review, 2012, vol. 42, no. 4, p. 473.


147

References

10. Bosshart, Pat, et al. "P4: Programming protocol-independent packet processors." ACM SIGCOMM Computer Communication Review 44.3 (2014): 87-95.

11. The P4 Language Consortium, “P416 Language Specification,” May 2017.

12. Gilbert, Seth, and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." Acm Sigact News 33.2 (2002): 51-59.

13. Brewer, Eric. "CAP twelve years later: How the" rules" have changed." Computer 45.2 (2012): 23-29.

14. Griffin, Timothy G., and Gordon Wilfong. "An analysis of BGP convergence properties." ACM SIGCOMM Computer Communication Review 29.4 (1999): 277-288.

15. The P4 Language Consortium, “P4 Language Tutorial,” 2017.


Documents

Advanced Networking Technologies€¦ · Forwarding Hardware Advanced Networking (SS 19): 05 -Software Defined Networking From [2] 27 Views on SDN: Redefining abstractions (“Shenker