OpenNF: Enabling Innovation in Network Function Control

Aditya Akella

With: Aaron Gember, Raajay Vishwanathan, Chaithan Prakash, Sourav Das, Robert Grandl, and Junaid Khalid

University of Wisconsin—Madison

Network functions, or middleboxes

CachingProxy

IntrusionPrevention

Firewall

WAN optimizer

Traffic scrubber

SSL Gateway

Introduce custom packet processing functions into the network

Load balancer

… [Sherry et al., SIGCOMM 2012]

Common in enterprise, cellular, ISP networks

Stateful: detailed book-keeping for

network flows

3

Network functions virtualization (NFV)– Lower cost

– Easy prov., upgrades

Software-defined networking (SDN)– Decouple from physical

– Better performance, chaining

Xen/KVM

State-of-the-art

Extract maximal performance at a given $$

SDN Controller

MBoxMBox

Load balancing

Dynamic reallocation to coordinate processing across instances

NFV + SDN: distributed processing

SDN Controller

MBox

Key abstractions

1. Elastic2. Always updated3. Dynamic enhancement

MBox MBoxMBox

Hand-off processing fora traffic subset

MBox

Dynamic reallocation to coordinate processing across instances

SDN Controller

MBoxMBox

Load balancing

Extract maximal performance at a given $$

NFV + SDN: distributed processing

What’s missing?The ability to simultaneously…

Meet SLAs– E.g., ensure deployment throughput > 1Gpbs

Ensure accuracy/efficacy– E.g., IDS raises alerts for all HTTP flows

containing known malware packages

Keep costs low, efficiency high– E.g., shut down idle resources when not needed

… needs more control than NFV + SDN

7

Example

Not moving flows bottleneck persists SLAs!

Naively move flows no associated state Accuracy!

Scale down: wait for flow drain out Efficiency!

Transfer live state while updating n/w forwarding

Firewall CachingProxy

IntrusionPrevention

Web ServerHomeUsers

Scaling in/outsustain thr’put at low $

OpenNF

Quick and safe dynamic reallocation of processing across NF instances

Quick: Any reallocation decision invoked any time finishes predictably soon

Safe: Key semantics for live state transfers No state updates missed, order preserved, etc.

Rich distributed processing based applications toflexibly meet cost, performance, security objectives

Outline

Overview and challenges

Design– Requirements– Key ideas– Applications

Evaluation

OpenNFOpenNF Controller

SDN Controller

Elas

tic sc

alin

g

Hot s

tand

by

Dyna

mic

en

hanc

emen

t

MBox MBox

NF APIs and control plane for joint control over internal NF

state and network forwarding state

Overview and challenges

Reallocationoperations

State import/export

Coordination w/ network

11

1: Many NFs, minimal changes– Avoid forcing NFs to use special state structures/

allocation/access strategies Simple NF-facing API; relegates actions to NFs

2: Reigning in race conditions– Packets may arrive while state is being moved; Updates

lost or re-ordered; state inconsistency Lock-step NF state/forwarding update

3: Bounding overhead– State transfers impose CPU, memory, n/w overhead Applications control granularity, guarantees

ChallengesOverview and challenges

12

State created or updated by an NF applies to either a single flow or a collection of flows

Classify state based on scope

Flow provides a natural way for reasoning about which state to move, copy, or share

NF state taxonomy

Connection

Connection

TcpAnalyzer

HttpAnalyzer

TcpAnalyzer

HttpAnalyzer

Per-flow state

ConnCount

Multi-flow state

All-flows stateStatistics

C1: Minimal NF Changes

13

API to export/import stateThree simple functions: get, put, delete(f)– Version for each scope (per-, multi-, all-flows)– Filter f defined over packet header fields

NFs responsible for– Identifying and providing all state matching a filter– Combining provided state with existing state

No need to expose internal state organization No changes to conform to a specific allocation strategy

C1: Minimal NF Changes

14

Operations

move flow-specific NF state at various granularities

copy and combine, or share, NF state pertaining to multiple flows

Semantics for move (loss-free, order-preserving), copy/share (various notions of consistency)

“Reallocate port 80 to NF2”

15

Move

OpenNF Controller

Control Application

move (port=80,Inst1,Inst2,LF&OP)

getPerflow(port=80)[ID1,Chunk1] putPerflow(ID1,Chunk1)

delPerflow(port=80)[ID2,Chunk2] putPerflow(ID2,Chunk2)

forward(port=80,Inst2)

SDN Controller

Inst2Inst1

Load-balanced network monitoring

vulnerable.bro reconstruct MD5’s for HTTP responses– Not robust to losses

weird.bro SYN and data packets seen in unexpected order – Not robust to reordering

HTTP req HTTP req

vulnerable.broweird.bro

move

C2: Race conditions

Moving live state: some updates (packets) may be lost, or arrive out of order

17

• Packets may arrive during a move operation

• Fix: suspend traffic flow and buffer packets– May last 100s of ms– Packets in-transit when buffering starts are dropped

Lost updates during move

R1B1

B2

B2

Inst2 is missing updates

Inst2Inst1

move(blue,Inst1 ,Inst2 )Loss-free: All state updates due to packet processing should be reflected in the transferred state, and all

packets the switch receives should be processed

Key idea: Event abstraction to prevent, observe and sequence state updates

C2: Race conditions

1. enableEvents(blue,drop) on Inst1;2. get/delete on Inst1 3. Buffer events at controller4. put on Inst2

5. Flush packets in events to Inst2

6. Update forwarding

Controller

Loss-free move using events

18

R1 Inst2Inst1

B3B1

dropB1 B1,B2B2

B2

B1,B2,B3

C2: Race conditions

Stop processing; buffer at controller

19

Order-preserving: All packets should be processed in the order they were forwarded

to the NF instances by the switch

Controller Switch Inst2

5. Flush buffer

6. Issue fwd update

Inst1

B2

B2B3

B4B3

B3

B3

B2

B4

B3

C2: Race conditions

Re-ordering of updates

Two-stage update to track last packet at NF1

Flush packets in events to Inst2 w/ “do not buffer”

enableEvents(blue,buffer) on Inst2

Forwarding update: send to Inst1 & controller

Wait for packet from switch (remember last)

Forwarding update: send to Inst2

Wait for event from Inst2 for last Inst1 packet

Release buffer of packets on Inst2

Order-preserving move

20

R1drop B1B1,B2

B2

B1,B2,B3buf

B3

B3B3B4

B1,B2,B3, B4

C2: Race conditions

Track last packet; sequence updates

Bounding overheadApps decide, based on NF type, objective:

granularity of reallocation operationsmove, copy or sharefilter, scope

guarantees desiredmove: no-guarantee, loss-free, loss-free + order-preservingcopy: no or eventual consistencyshare: strong or strict consistency

C3: Applications

HTTP req HTTP req

Load-balanced network monitoring

movePrefix(prefix,oldInst,newInst): copy(oldInst,newInst,{nw_src:prefix},multi) move(oldInst,newInst,{nw_src:prefix},per,LF+OP) while (true): sleep(60) copy(oldInst,newInst,{nw_src:prefix},multi) copy(newInst,oldInst,{nw_src:prefix},multi)

scan.brovulnerable.broweird.bro

C3: Applications

scan.bro

scan.bro

vuln.bro

weird.bro

23

Implementation

OpenNF Controller (≈4.7K lines of Java)– Written atop Floodlight

Shared NF library (≈2.6K lines of C)

Modified NFs (4-10% increase in code)– Bro (intrusion detection)– PRADS (service/asset detection)– iptables (firewall and NAT)– Squid (caching proxy)

Testbed: HP ProCurve connected to 4 servers

Impl & Eval

24

Microbenchmarks: NFs

Serialization/deserialization costs dominate

Cost grows with state complexity

Impl & Eval

Average Maximum0

50

100

150

200

250

Per-

pack

et La

tenc

y In

crea

se (m

s)

NG NG PL LF PL LF PL+ER

OP PL+ER

050

100150200250300350400450

Mov

e Ti

me

(ms)

• State: 500 flows in PRADS; Traffic: 5000 pkts/sec• Move per-flow state for all flows

Microbenchmarks: operations

25

Packetsdropped!

686 462

881 packetsin events

Guarantees come at a cost!Copy (MF state) – 111msShare (strong) – 13ms per pkt

D = f(load,state,speed)

Impl & Eval

1120 packets buffered at Inst2

838 packetsin events

+

Macrobenchmarks: end-to-end benefits

Load balanced monitoring with Bro IDS– Load: replay cloud trace at 10K pkts/sec– At 180 sec: move HTTP flows (489) to new Bro– At 360 sec: move HTTP flows back to old Bro

OpenNF scaleup: 260ms to move (optimized, loss-free)– Log entries equivalent to using a single instance

VM replication: 3889 incorrect log entries– Cannot support scale-down

Forwarding control only: scale down delayed by more than 1500 seconds

Impl & Eval

27

Wrap up!

• OpenNF enables rich control of the packet processing happening across instances of an NF

• Quick, key safety guarantees,

• Low overhead, minimal NF modifications

http://opennf.cs.wisc.edu

Backup

29

Copy and shareUsed when multiple instances need to access a

particular piece of state

Copy – eventual consistency– Issue once, periodically, based on events, etc.

Share – strong– All packets reaching NF instances trigger an event– Packets in events are released one at a time– State is copied between packets

C2: Race conditions

Example app: Selectively invoking advanced remote processing

enhanceProcessing(flowid,locInst): move(locInst,cloudInst,flowid,per,LF)

scan.brovulnerable.broweird.bro

scan.brovulnerable.broweird.brodetect-MHR.bro!

C3: Applications

HTTP req HTTP req

Enterprise n/w

Internet

checks md5sum of HTTP reply

No need for: (1) order-preservation (2) copying multi-flow state

31

Existing approaches• Control over routing (PLayer, SIMPLE, Stratos)• Virtual machine replication– Unneeded state => incorrect actions– Cannot combine => limited rebalancing

• Split/Merge and Pico/Replication– Address specific problems => limited suitability– Require NFs to create/access state in specific

ways => significant NF changes

32

Controller performance

Improve scalability with P2P state transfers

Macrobenchmarks: Benefits of Granular Control

Two clients make HTTP requests– 40 unique URLs

Initially, both go to Squid120s later reassign client 1 to Squid2

Metric Ignore Copy-client Copy-all

Hits @ S1 117 117 117

Hits @ S2 crashed 39 50

State transferred

0 4MB 54MB

Granularitiesof copy

Impl & Eval