“Hashing Out” the Future of Enterprise and Data-Center Networks

Jennifer RexfordPrinceton University

http://www.cs.princeton.edu/~jrex

Joint with Changhoon Kim, Minlan Yu, Matthew Caesar, and Alex Fabrikant

2

Challenges in Edge Networks

• Large number of hosts–Tens or even hundreds of thousands of hosts

• Dynamic hosts–Host mobility–Virtual machine migration

• Cost conscious–Equipment costs–Network-management costs

Need a scalable and efficient self-configuring network

3

An All-Ethernet Solution?

• Self-configuration–Hosts: MAC addresses–Switches: self-learning

• Simple host mobility–Location-independent flat addresses

• But, poor scalability and performance–Flooding frames to unknown destinations–Large forwarding tables (one entry per address)–Broadcast for discovery (e.g., ARP, DHCP)–Inefficient delivery of frames over spanning tree

4

An All-IP Solution?

• Scalability–Hierarchical prefixes

(smaller tables)–No flooding

• Performance–Forwarding traffic over shortest paths

• But, several disadvantages–Complex joint configuration of routing and DHCP–Clumsy mobility (location-dependent addresses)–Expensive longest-prefix match in data plane

5

Compromise: Hybrid Architecture

Ethernet-based IP subnets interconnected by routers

R

R

R

R

Ethernet Bridging - Flat addressing - Self-learning - Flooding - Forwarding along a tree

IP Routing - Hierarchical addressing - Subnet configuration - Host configuration - Forwarding along shortest paths

R

Sacrifices Ethernet’s simplicity and IP’s efficiency for scalability

6

Can We Do Better?

• Shortest-path routing on flat addresses–Shortest paths: scalability and performance–MAC addresses: self-configuration and mobility

• Scalability without hierarchical addressing–Limit dissemination and storage of host info–Sending packets on slightly longer paths

SH

S

S

S

S

S S

S S

S S S

S

H

H

H

H

HH

H

H

7

Outline

• SEATTLE–Distributed directory service

• BUFFALO–Compact forwarding tables

• SeaBuff–Combining Seattle and Buffalo

• Deployment scenarios–Enterprises vs. data centers

8

SEATTLE

Scalable Ethernet Architecture for Large Enterprises

9

SEATTLE Design Decisions

Objective Approach Solution

1. Avoiding flooding

Never broadcast unicast traffic Network-layer

one-hop DHT2. Restraining

broadcastingBootstrap hosts

via unicast

3. Reducing routing state

Populate host infoonly when and where

it is needed

Traffic-driven resolution with caching

4. Shortest-path forwarding

Allow switches to learn topology

L2 link-state routingmaintaining only

switch-level topology

* Meanwhile, avoid modifying end hosts

10

Network-Layer One-hop DHT

• Maintains <key, value> pairs with function F – Consistent hash mapping a key to a switch–F is defined over the set of live switches

• One-hop DHT– Link-state routing ensures

switches know each other

• Benefits– Fast and efficient

reaction to changes– Reliability and capacity

naturally grow with size of the network

0 12128-1

Location Resolution

11

SwitchesEnd hosts

Control messageData traffic

<key, val> = <MAC addr, location><key, val> = <MAC addr, location>

Host discovery

B

x

HashF(MACx) = B

Store<MACx, A>

Traffic to x

HashF(MACx ) = BTunnel

to A

Notify<MACx, A>

E

Forward directly from D to A

A

Tunnel to B

C

D

yOwner

User

Resolver

Publish<MACx, A>

Address Resolution

12

<key, val> = <IP addr, MAC addr><key, val> = <IP addr, MAC addr>

Traffic following ARP takes a shortest pathwithout separate location resolution

B

DHash

F(IPx) = B

Store<IPx, MACx, A>

BroadcastARP requestfor IPx

HashF(IPx ) = B

Unicast reply<IPx, MACx, A>

E

A

Unicastlook-up to B

C

<IPx ,MACx>

x y

13

Handling Network and Host Dynamics

• Network events–Switch failure/recovery

Change in <key, value> for DHT neighbor Fortunately, switch failures are not common

–Link failure/recovery Link-state routing finds new shortest paths

• Host events–Host location, MAC address, or IP address –Must update stale host-information entries

14

Handling Host Information Changes

Resolver

y

Host talkingwith x

< x, A >

< x, A >

< x, A >

D< x, D >

Oldlocation

New location

< x, D >

< x, D >

< x, D >

Dealing with host mobilityDealing with host mobility

MAC- or IP-address change can be handled similarly

B

xA

C

E

F

15

Packet-Level Simulations

• Large-scale packet-level simulation–Event-driven simulation of control plane–Synthetic traffic based on LBNL traces –Campus, data center, and ISP topologies

• Main results–Much less routing state than Ethernet–Only slightly more stretch than IP routing–Low overhead for handling host mobility

16

Amount of Routing State

SEATTLEw/o caching

SEATTLEw/ caching

Ethernet

SEATTLE reduces the amount of routing state by more than an order of magnitudeSEATTLE reduces the amount of routing

state by more than an order of magnitude

17

Cache Size vs. StretchStretch = actual path length / shortest path length (in latency)

SEATTLE offers near-optimal stretch with very small amount of routing state

SEATTLE offers near-optimal stretch with very small amount of routing state

SEATTLE

ROFL

18

Sensitivity to Mobility

SEATTLE rapidly updates routing statewith very low overhead

SEATTLE rapidly updates routing statewith very low overhead

SEATTLEw/o caching

SEATTLEw/ caching

Ethernet

19

Prototype Implementation

Host-info registrationand notification msgs

User/Kernel Click

XORP

OSPFDaemon

OSPFDaemon

RingManager

RingManager

Host InfoManagerHost InfoManager

SeattleSwitchSeattleSwitch

Link-stateadvertisements

Data Frames

Data Frames

RoutingTable

RoutingTable

NetworkMap

NetworkMap

ClickInterface

ClickInterface

Throughput: 800 Mbps for 512B packets, or 1400 Mbps for 896B packets

20

Conclusions on SEATTLE

• SEATTLE –Self-configuring–Scalable–Efficient

• Enabling design decisions–One-hop DHT with link-state routing–Reactive location resolution and caching–Shortest-path forwarding

21

BUFFALO

Bloom Filter Forwarding Architecture for Large Organizations

Data Plane Scaling Challenge

• Large layer-two networks–Many end-host MAC addresses–Many switches

• Forwarding-table size becomes a problem–Requires a large, fast memory–Expensive and power hungry–Over-provisioning to avoid running out

• Buffalo’s goal–Given a small, fast memory–… make the most of it!

23

Bloom Filters

• Bloom filters in fast memory – A compact data structure for a set of elements– Calculate s hash functions to store element x– Easy to check set membership – Reduce memory at expense of false positives

h1(x) h2(x) hs(x)01000 10100 00010

x

V0Vm-1

h3(x)

24

Bloom Filter Forwarding

• One Bloom filter (BF) per next hop– Store all addresses forwarded to that next hop

Nexthop 1

Nexthop 2

Nexthop T

……packet

query

Bloom Filters

hit

25

BUFFALO Challenges

26

1. Optimize Memory Usage

• Goal: Minimize overall false-positive rate– Probability that one BF has a false positive

• Input:– Fast memory size M– Number of destinations per next hop– The maximum number of hash functions

• Output: the size of each Bloom filter– Larger BF for next hop with more destinations

27

1. Optimize Memory Usage (cont.)

• Constraints– Memory constraint

Sum of all BF sizes <= fast memory size M

– Bound on number of hash functions To bound CPU calculation time Bloom filters share the same hash functions

• Convex optimization problem– An optimal solution exists– Solved by IPOPT optimization tool– Runs in about 50 msec

28

1. Minimize False Positives

• Forwarding table with 200K entries, 10 next hops

• 8 hash functions

29

1. Comparing with Hash Table

• Save 65% memory with 0.1% false positive

65%

30

2. Handle False Positives

• Design goals– Should not modify the packet– Never go to slow memory– Ensure timely packet delivery

• BUFFALO solution – Exclude incoming interface

Avoid loops in one false positive case

– Random selection among the rest Guarantee reachability with multiple FPs

31

2. One False Positive

• Most common case: one false positive–When there are multiple matching next hops–Avoid sending to incoming interface

• We prove that there is at most a 2-hop loop with a stretch <= l(AB)+l(BA)

False positive

AA

Shortest path

BB

dst

32

2. Multiple False Positives

• Handle multiple false positives– Random selection from matching next hops– Random walk on shortest path tree plus a few

false positive links– To eventually find out a way to the destination

dstdst

Shortest path tree for destination

False positive link

33

2. Multiple False Positives (cont.)

• Provable stretch bound–With k false positives–… expected stretch is at most O(k2*3k/3)–Proved by random walk theories

• Stretch bound is actually not that bad–False positives are independent

Different switches use different hash functions

–k false positives for one packet are rare Probability of k false positives drops exponentially in k

• Tighter bounds in special cases: e.g., tree

34

2. Stretch in Campus Network

When fp=0.001%99.9% of the packets

have no stretch

When fp=0.001%99.9% of the packets

have no stretch0.0003% packets have a stretch of

shortest path length

0.0003% packets have a stretch of

shortest path length

When fp=0.5%, 0.0002% packets

have a stretch 6 times of shortest path length

When fp=0.5%, 0.0002% packets

have a stretch 6 times of shortest path length

3. Update on Routing Change

• Use CBF in slow memory– Assist BF to handle forwarding-table updates– Easy to add/delete a forwarding-table entry

CBF in slow memory

BF in fast memory

Delete a route

3. Occasionally Resize BF

• Under significant routing changes–# of addresses in BFs changes significantly–Re-optimize BF sizes

• Use CBF to assist resizing BF–Large CBF and small BF–Easy to expand BF size by contracting CBF

1 0

Hard to expand to size 4

CBF BF

Easy to contract CBF to size 4

BUFFALO Switch Architecture

37

Prototype implemented in kernel-level Click

38

Prototype Evaluation

• Environment– 3.0 GHz 64-bit Intel Xeon– 2 MB L2 data cache (fast-memory size M)– 200K forwarding-table entries and 10 next hops

• Peak forwarding rate– 365 Kpps, 1.9 μs per packet– 10% faster than hash-based EtherSwitch

• Performance with forwarding-table updates– 10.7 μs to update a route– 0.47 s to reconstruct BFs based on CBFs

39

Conclusion on BUFFALO

• BUFFALO–Small, bounded memory requirement–Small stretch–Fast reaction to routing updates

• Enabling design decisions–Bloom filter per next hop–Optimizing of Bloom filter sizes–Preventing forwarding loops–Dynamic updates using counting Bloom filters

40

SeaBuff

Seattle + Buffalo

Seattle + Buffalo

• Seattle–Shortest-path routing and scalable control plane–Fewer host MAC addresses stored at switches

• Buffalo–Less memory for given # of routable addresses–Graceful handling of increase in # of addresses

• Combined data plane

cachedestination

egressswitch

outgoinglink

Seattle

Bloomfilter

Buffalo

Two Small Sources of Stretch

• Seattle: diverting some packets through relay B

• Buffalo: extra stretch from D to B, or B to A

B

E

C

D

A

x y

Traffic to x

Relay for x

Choosing the Right Solution

• Spatial distribution of traffic–Sparse: caching in Seattle is effective–Dense: caching is not as effective

• Temporal distribution of traffic–Stable: shortest path routing is effective–Volatile: forwarding through relay spreads traffic

• Topology–Arbitrary: false positives have more impact–Tree-like: false positives have less impact

Conclusions

• Growing important of edge networks–Enterprise networks–Data center networks

• Shortest-path routing and flat addressing–Self-configuring and efficient–Scalability in exchange for stretch

• Ongoing work–SeaBuff = Seattle + Buffalo–Theoretical analysis of stretch in Buffalo–Efficient access control in OpenFlow