Prototyping, Implementation & Large-Scale-Experimentation of

VIRO in GENI

Zhi-Li ZhangQwest Chair Professor

Department of Computer Science and Engineering

University of Minnesota Email: zhzhang@cs.umn.edu

Scalability: capability to connect tens of thousands, millions or more devices (& users) routing table size, constrained by router memory, lookup

speed Availability & Reliability: must be resilient to failures

need to be “proactive” instead of reactive need to localize effect of failures

Mobility: hosts (both clients & servers) are more mobile need to separate location (“addressing”) and identity

(“naming”) Manageability: ease of deployment, “plug-&-play”

need to minimize manual configuration self-configure, self-organize, while ensuring security and

trust Agility: dynamically adapt to demand; net expands/shrinks,

… ......

VIRO: Scalable, Robust Virtual-Id Routing for Future Large, Dynamic Networks

Designed to meet the following challenges & requirements

VIRO: A Scalable and Robust “Plug-&-Play” Routing Architecture

Decoupling routing from naming/”addressing” “native” naming/address-independent

“future-proof” & capable of supporting multiple namespaces Introduce a “self-organizing” virtual id (vid) layer

a layer 2 (LLC)/layer-3 convergence layer -- replacing IP subsume layer-2/layer-3 routing/forwarding functionality

except for first/last hop: host to switch or switch to host layer-3 addresses (or higher layer names): global addressing

or naming for inter-networking and “persistent” identifiers DHT-style routing using a topology-aware, structured vid

space highly scalable and robust

– O(log N) routing table size, localize failures, enable fast rerouting

support multiple topologies or virtualized network services

Virtual ID layer and VID space Topology-aware, structured virtual id (vid) space

Kademlia-like “virtual” binary tree vid: embed topological “location” information

structurally self-configurable and self-organizing

Layer 2 Physical Network Topology

IPv4/IPv6

Virtual ID Layer

Other NamespaceDNS Names

M N HGJ LK

C

FEBDA

1

1

1

1 1

0

0

0 0 011

0

0

1

10

0

0

0

1

1

1

1

1

0 0

- - - -- - - - - - - - -- - - - - -FE HGBA DC NM J LK

1

0

0 1

0 01

1

0 0 0 0 01 1 1 11 1

VIRO: Three Core Components Virtual id space construction and vid assignment

performed most at the bootstrap process (i.e., network set up):

a vid space “skeleton” is created once network is set up/vid space is constructed:

a new node (a “VIRO switch”) joins: assigned based on neighbors’ vid’s

end-host/device: inherits a vid (prefix) from “host switch” (to which it is attached), plus a randomly assigned host id; host may be agnostic of its vid

VIRO routing algorithm/protocol: DHT-style, but needs to build end-to-end

connectivity/routes a bottom-up, round-by-round process, no network-wide control

flooding O(log N) routing entries per node, N: # of VIRO switches

(Persistent) layer-2/3 address/name resolution and vid look-up DHT directory services built on top of the same vid space

“persistent” identifier (e.g., MAC/IP address) hashed to a “vid” key, which is then used for (pid, vid) mapping registration, look-up, etc.

Data forwarding among VIRO switches using vid only

VIRO Routing & Forwarding Routing: Bottom-up, round-by-round process

highly scalable: O(L) routing entries per node no single point of failure, and localize effect of failures completely eliminate “network-wide” control flooding

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

Bucket Distan

ce

Gateway

Nexthop

1 - -2 A B3 A C,D4 C, D C,D5 B,D,M,N B,C,D

Forwarding: a packet to a destination node, say, L compute the logical distance (“level”) to the dest. node look up gateway & next-hop pair at the distance level

(bucket) if no routing entry, drop the packet

Robustness: Localized Failures

Routing table for node A does not change despite the failure!

Link F-K failsInitial Topology

After link F-K fails

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

Bucket Distan

ce

Gateway

Nexthop

1 - -2 A B3 A C,D4 C, D C,D5 B,

D,N,MB,C,D

Built-in Multi-Path Routing, Load-Balancing and Resilient Fast Re-

Routing Learn multiple gateways at each level

Default gateway is the one that is logically closest

Use additional gateways for multi-path routing, load balancing & fast failure re-routing Requires consistent gateway selection

strategiesotherwise forwarding loops may occur

Use appropriate “forwarding directive” carried in packets to select appropriate gateways to ensure consistent gateway selection

Forwarding directives may be modified to dynamically adapt to network conditions

Key (Other) Features & Advantages

Load balancing & fast rerouting can be naturally incorporated no additional complex mechanisms needed

Can support multiple namespaces, and inter-operability among such namespaces (e.g., IPv4<->IPv6, IP<->Phone#, NDN, etc.) VIRO: “native” naming/address-independent simply incorporate more <namespace, vid> directory

services Support multiple topologies or virtualized network

services e.g., support for multiple virtual networks multiple vid spaces may be constructed

e.g., by defining different types of “physical” neighbors Also facilitate security support

host and access switches can perform access control “persistent” id is not used for data forwarding

eliminate flooding attacks

GENI-VIRO Project: GoalsVIRO as a “non-IP” layer 3 protocol & service Implement & prototype VIRO in GENI

what capabilities does GENI, & how flexible GENI is to support prototyping VIRO?

Large-scale Experimentation of VIRO in GENI test & evaluate VIRO functionality & performance what GENI capabilities/tools facilitate such large-

scale “shake-up” experiments, e.g., dynamic link/node failures?

Longer-Term Goal: VIRO as “long-lived” non-IP layer 3 prototype service Potentially incorporated in GENI as a non-IP

alternative to support research, experiments & educational

activities by other GENI researchers

GENI-VIRO: Prototype & Implementation

Challenges Plan to implement using the GENI OVS platform, as a GENI slice OVS (open virtual switch) platform derived from

the openflow switch spec & SDN paradigm centralized control/management planes more flexible forwarding behavior, but still tied to

existing Ethernet/IP/TCP protocol suite Can we implement VIRO using OVS platform?

has its own “topology-aware” addressing (vid’s) scheme, centrally managed and/or “self-configured”

perform distributed routing, using novel “pub-sub” paradigm

forwarding behavior: using both destination vid (via prefix matching) and a forwarding directive to select gateway & next-hop, with built-in multipath & fast failure (re)routing

GENI-VIRO: Data Plane Prototype Challenges

• Re-use MAC address (40 bits) as vid’s• Virtual id space construction and virtual id

assignment Centralized vid construction & management plane

via a centralized OVS/SDN controller distributed topology discover, but centralized topology

management: e.g., VIRO switch join or leave operations end host: dynamically assigned vid; report to controller

VIRO packet header: similar to Ethernet frame format, but need an additional “forwarding directive” field as an extension to Ethernet, similar to VLAN extension

VIRO forwarding behavior: dest. vid + forwarding directive difficult to implement directly using openflow OVS need to define new flow table format & implement our

own flow classifier, as an extension to standard OVS

GENI-VIRO: Control Plane Prototype Challenges

Can implement a centralized version using the openflow OVS/SDN paradigm, but lose some of the key advantages of VIRO, e.g., its

ability to adapt to failures dynamically Implement distributed VIRO routing via OVS:

equip each OVS node with a local OVS controller configure local flow table to forward VIRO routing

messages to local OVS controller local OVS controller implements & emulates VIRO

routing protocols additional OVS controllers as rendezvous points

(RPs) Additional functions & modules for neighbor

discovery, monitoring, etc.

GENI-VIRO “Shakedown” Experiments Test & evaluate functionality & performance of

VIRO VIRO GENI experiment designs, topology configuration

and workload generation tools Large-scale experimental evaluation of VIRO scalability

as # of nodes and links, or richness of topology grows metrics: routing entities, memory requirements, routing stretches,

routing overheads, etc.; Large-scale experimental evaluation of VIRO reliability

how well does it adapt to dynamic topology changes, failures metrics: convergence speed, response time, packet loss, control

load, … Experimental evaluation of VIRO multi-path routing at

scale- What GENI facilities or tools can we leverage to

facilitate these experiments at scale? workload generation? dynamic (experiment) topology

control? failure generation? monitoring and measurement tools?

what scale (# nodes, links, topology richness) can we attain?

SummaryVIRO as a non-IP (layer 3) service Test, evaluate & shake-down GENI in terms of its

flexibility & limitations for realizing non-IP services in particular, the OVS platform used by GENI hope to report our results on these by end of Year 1

Test, evaluate & shake-down GENI in terms of its support for large-scale evaluation of a non-IP layer 3 routing protocol & service What control do experimenters have in experiment

design? What scale can we attain? How realistic can we get? hope to report our results on these by end of Year 2

If successful, incorporate into GENI as long-running service? To support other research & experiments?

Will also enable our own research on SDN, CDN/CCN and future network architecture designs

QUESTIONS?

THANK YOU!

Pros & Cons of Existing Technologies (Layer-2)

Ethernet/Wireless LANs Pluses:

plug-&-play, minimal configuration, better mobility

Minuses: (occasional) data

plane flooding, sub-optimal routing (using spanning tree), not robust to failures

Not scalable to large (& wide-area) networks

(Layer-3) IPv4/IPv6 Pluses: • better data plane scalability,

more “optimal” routing, … Minuses:• control plane flooding, global

effect of network failures• poor support for mobility• difficulty/complexity in

“network renaming” • Esp., changing addressing

schemes (IPv4 -> IPv6 transition) requires modifications in routing and other network protocols

Vid Assignment : Key Properties

00010 10010 11100

MN H

GJ L

K

C

FEB

DA00000

00100

00110

0100001001

10000 10100

10110

1111011000

Key invariant properties

• closeness: if two nodes are close in the vid space, then they are also close in the physical topology esp., any two logical neighbors must be directly connected.• connectivity: any logical sub-trees must be physically connected.

1

1

1

1 1

0

0

0 0 011

0

0

1

10

0

0

0

1

1

1

1

1

0 0

- - - -- - - - - - - - -- - - - - -FE HGBA DC NM J LK

1

0

0 1

0 01

1

0 0 0 0 01 1 1 11 1

Vid based distance: Logical distance

Logical distance defined on the vid space d(vidx, vidy) = L – lcp (vidx,vidy)

L: max. tree height; lcp: longest common prefix

e.g. d(00001, 00111) = 5 – lcp(00001, 00111) = 5 – 2 = 3

d(01001, 01011) = 5 – lcp(01001, 01011) = 5 – 3 = 2

Vid Assignment: Bootstrap Process Initial vid assignment and vid space construction

performed during the network bootstrap process Performed using either a centralized or distributed

vid assignment algorithm.

0

0M

N HG

JL

K

C

FE

B

DA

00

000

00

00 0

00 10

00

10

MN H

G

JL

K

\

FE

B

DA

MN H

G

JL

K

C

FE

B

DA00

00

10

10 1000

10

0000

00

010

000

000M

N HG

JL

K

C

FE

B

DA

100110

010

010

000 100

110

110

100

000

0010

0010

1100

MN H

G

JL

K

C

FE

B

DA0000

0100

0110

1000

1001

0000

0100

0110

1110

1000

VIRO Routing: Routing Table Construction

Bottom-up, “round-by-round” process: round 1: neighbor discovery

discover and find directly/locally connected neighbors round k ( 2 ≤ k ≤L):

build routing entry to reach level-k bucket Bk(x)-- a list of one or more (gateway, next-hops)

use “publish-query” (rendezvous) mechanismsAlgorithm for building Bk(x) routing entry at node x:

if a node(x) is directly connected to a node in Bk(x), then it is a gateway for Bk(x), and also publishes it within Sk-1(x).

nexthop to reach Bk(x) = direct physical neighbor in Bk(x) else node x queries within Sk-1(x) to discover gateway(s) to reach

Bk(x), choose the logically closest if multiple gateways. nexthop to reach Bk(x) = nexthop(gateway)

Correctness of the algorithm can be formally established!

VIRO Routing: Key Features Inspired by Kademlia DHT

but need to build end-to-end connectivity/routes! Bottom-up, round-by-round process

first: neighbor discovery then: build routing entries to reach nodes within each level of the

vid space (virtual binary tree) use “publish-query” mechanisms

Highly scalable: O(L) routing entries per node L = O(log N), N: number of nodes (VIRO switches) more importantly, path diversity (e.g., multi-path routing) can be

naturally exploited for load-balancing, etc. routing is no longer “shortest” path based !

No single point of failure, and localize effect of failures unlike link state routing, node/link failure/recovery is not flooded

network-wide; impact scope is limited also enable localized fast rerouting

Completely eliminate “network-wide” control flooding

VIRO Routing: Some DefinitionsFor k =1, …, L, and any node x:• (level-k) sub-tree, denoted by Sk(x): • set of nodes within a logical distance of k from x

• (level-k) bucket, denoted by Bk(x): • set of nodes exactly k logical distance from node x

• (level-k) gateway, denoted Gk(x): • a node in Sk-1(x) which is connected to a node in Bk(x) is a gateway

to reach Bk(x) for node x; a direct neighbor of x that can reach this gateway node is a next-hop for this node

Example: S1(A)= {A}, S2(A) ={A,B}, B2(A)={B}G2(A)={A},S3(A) = {A,B,C,D}B3(A) = {C,d}G3(A) = {A,B}

1

1

1

1 1

0

0

0 0 011

0

0

1

10

0

0

0

1

1

1

1

1

0 0- - - -- - - - - - - - -- - - - - -FE HGBA DC NM J LK

1

0

0 1

0 01

1

0 0 0 0 01 1 1 11 1

VIRO Routing: Routing Table

1

1

1

1 1

0

0

0 0 011

0

0

1

10

0

0

0

1

1

1

1

1

0 0- - - -- - - - - - - - -- - - - - -FE HGBA DC NM J LK

1

0

0 1

01

1

0 0 0 0 01 1 1 11 1

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000Level Gateway

Nexthop

1 - -2 - -3 - -.. .. ..L - -

VIRO Routing: Example Round 1:

each node x discovers and learns about its directly/locally connected neighbors

build the level-1 routing entry to reach nodes in B1(x)

Bucket Distan

ce

Gateway

Nexthop

1 - -23

Routing Table for node A

E.g. Node A: discover three direct neighbors, B,C,D; build the level-1 routing entry to reach B1(A)={}

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

VIRO Routing: Example … Round 2:

if directly connected to a node in B2(x), enter self as gateway in level-2 routing entry, and publish in S1(x)

otherwise, query “rendezvous point” in S1(x) and build the level-2 routing entry to reach nodes in B2(x)

Routing Table for node A

E.g. Node A: B2(A)={B}; node A directly connected to node B; publish itself as gateway to B2(A) Bucket Distan

ce

Gateway

Nexthop

1 - -2 A B3 00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

VIRO Routing: Example … Round 3:

if directly connected to a node in B3(x), enter self as gateway in level-3 routing entry, and publish in S2(x)

otherwise, query “rendezvous point” in S2(x) and build the level-2 routing entry to reach nodes in B3(x)

E.g. Node A: B3(A)={C,D}; A publishes edges A->C, A->D to “rendezvous point” in S2(A), say, B; Bucket

Distance

Gateway

Nexthop

1 - -2 A B3 A C,D

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

VIRO Routing: Example … Round 4:

if directly connected to a node in B4(x), enter self as gateway in level-4 routing entry, and publish in S3(x)

otherwise, query “rendezvous point” in S3(x) and build the level-4 routing entry to reach nodes in B4(x)

E.g. Node A: B4(A)={M,N}; A queries “rendezvous point” in S3(A), say, C; learns C as

gateway Bucket Distan

ce

Gateway

Nexthop

1 - -2 A B3 A C,D4 C C

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

VIRO Routing: Example … Round 5:

if directly connected to a node in B5(x), enter self as gateway in level-5 routing entry, and publish in S4(x)

otherwise, query “rendezvous point” in S4(x) and build the level-4 routing entry to reach nodes in B5(x)

E.g. Node A: B5(A)={E,F,G,H,J,K,L}; A queries “rendezvous point” in S4(A), say, D; learns B as gateway

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

Bucket Distan

ce

Gateway

Nexthop

1 - -2 A B3 A C,D4 C C5 B B

VIRO Routing: Packet Forwarding To forward a packet to a destination node,

say, L compute the logical distance to that node Use the nexthop corresponding to the logical distance

for forwarding the packet If no routing entry:

drop the packet

00010 10010 11100

MN H

G

JL

K

C

FE

B

DA00000

00100

00110

01000

01001

10000 10100

10110

11110

11000

Bucket Distan

ce

Gateway

Nexthop

1 - -2 A B3 A C,D4 C C5 B B

<pid, vid> Mapping and vid Lookup

pid: persistent identifier of end host/device, or switch e.g., MAC/IP address, or any other layer 2/3 address, “flat”

host id, or higher layer names can simultaneously support multiple namespaces

<pid, vid> mapping registration and look-up using Kademlia DHT on top of the same vid space Hash(pid) -> vidkey: used for registration & look-up mapping stored at “access switch” whose vid is “closest” to

vidkey Look-up speed, scalability & mobility support trade-off

can use one-hop or multi-hop DHT or use hierarchical (or “geographically scoped”) hash tables

vid look-up and data forwarding may be combined use hierarchical (or “geographically scoped”) rendezvous

points provide better support for mobility

Re-use 48-bit MAC addresses for vid vid structure: divided into two fields

VEIL: a VIRO Realization over Ethernet (and 802.11, etc)

switch vid host id

L

switch vid (32 bits) assigned to switches using the

vid assignment process L: default 24 bits

host id (16 bits) assigned by “host-switches” uniquely identify hosts directly

connected to a switch. End hosts agnostic of their vid’s Host switch performs vid/MAC address translation Backward compatible w/ Ethernet, 802.11, etc.

VEIL: <IP/MAC, vid> Mapping Host-switch:

a switch directly connected to the host discover host MAC/IP through (gratuitous) ARP, and assign vid to

host host-switch publishes pid vid mappings at an “access-switch”

Access-switch: a switch whose vid is closest to hash (pid of the host)

VIRO Switch

Sx

VIRO Switch

Sy

VIROSwitch

Sz

x yHost-switch for y

IPy MACy VIDy

IPy VIDyAccess-switch for y

register mappingIPy VIDy

An example using IP address as pid

Address/vid Lookup & Data Forwarding Use DHT look-up for address/vid

resolution with local cache

vid to MAC address translation at last-hop

VIROSwitchSx

VIRO Switch

Sy

VIRO SwitchSz

x y

1. ARP Query (IPy MAC?)

2. ARP query forwarded as

unicast request(IPy MAC?)

3. ARP Reply (IPy VIDy)

4. Ethernet packet (MACx VIDy)

5. Sx changes source MAC address

Ethernet Packet (VIDx VIDy)

6. Sy changes destination

MAC addressEthernet packet (VIDx MACy)

Mapping Table at Sz

VID IP Address MAC Address

VIDy IPy MACy

… … …

More Than One Gateway Exist and Can be Used !

1

1

1

1 1

0

0

0 0 011

0

0

1

10

0

0

0

1

1

1

1

1

00

- - - -- - - - - - - - -- - - - - -FE HGBA DC NM J LK

1

0

0 1

0 01

1

0 0 0 0 01 1 1 11 1

VIRO: Scalable, Robust & Name-

Independent Virtual Id Routing for (future) Large-scale, Dynamic

Networks

main student contributors: Sourabh Jain Yingying Chen Saurabh Jain Guor-Huar Lu