Sponsored by the National Science Foundation

Systematic Experimentation Case Study:

Virtual Router Failure Restoration

Xuan Liu

UMKC/GENI Project Office

Sponsored by the National Science Foundation 2Systematic Experimentation – GEC 20 – June 23, 2014

Motivation

Design Failure

Restoration Scheme

DesignOptimization

Model

Heuristic Algorithm

DeployExperiments

On GENI

Goal: Dynamically recover multiple virtual router failures using standby virtual routers in a virtual network environment.

Collect Data(bw, delay,

etc)

Virtual network

Find the optimal Standby Virtual

Router

Complex Topology requires systematic

experimentation setup

Sponsored by the National Science Foundation 3Systematic Experimentation – GEC 20 – June 23, 2014

Prior Work

• Experimental Study on GpENI Testbed– Test the conceptual dynamic reconfiguration scheme

for virtual router failure restoration– Lessons learned

• The proposed scheme is practical in a distributed network • Substrate network topology has impact to the delay between

virtual nodes

GpENI Network: US Sites GpENI Network: European Sites

Sponsored by the National Science Foundation 4Systematic Experimentation – GEC 20 – June 23, 2014

Why Choosing GENI Testbed

• A distributed virtual network testbed is desired to deploy the heuristic multi-criteria S-VR selection algorithm, so that the impact to the virtual networks can be better understood – GENI Testbed provides such capabilities.

• Geographically distributed • Customized software installation on VMs is allowed• Arbitrary virtual topology creation• Backbone topology is more general

GpENI Network GENI Network

Sponsored by the National Science Foundation 5Systematic Experimentation – GEC 20 – June 23, 2014

Target Topology on GENI

Stitched Links between aggregates

LANs within single aggregates

End Hosts Virtual Routers

(Colors shows different geographical locations)

Sponsored by the National Science Foundation 6Systematic Experimentation – GEC 20 – June 23, 2014

Systematic Experimentation Design

Install Software on One Node

Automate Experimental Configuration

End-to-End Validation

Scaling Up

Automate Measurement

Use Smallest Topology

Increases nodes/sites

• Important Tips:• Make One Change at A Time• Version Control

Sponsored by the National Science Foundation 7Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#1 Install Software on One Node

MANUAL Install XORP (took > 1 hour)

• Download XORP software• Install all required

dependences• $apt-get install libssl-dev• $apt-get install ncurses-dev• $apt-get install libpcap-dev• $apt-get install g++• $apt-get install traceroute• $apt-get install scons

• Compile XORP• Build XORP• $sudo groupadd xorp

AUTOMATE Create custom image

VM router

Load the URL of the Custom image

How to Create Custom Image:http://groups.geni.net/geni/wiki/HowTo/ManageCustomImagesInstaGENI

Install XORP on one Node

Create Custom image

Save the URL to the Custom Image

Sponsored by the National Science Foundation 8Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#2 Automate Configuration

MANUAL Configure OSPF and starts XORP

AUTOMATE: Configures OSPF and starts XORP

router

router router

router

Starts with smallest reasonable topology

Get Familiar with the Syntax of the

Configuration File

Validate the OSPF Routing in

the Topology

Write Shell Script to Automate OSPF

Configuration Procedure

Load the Automation Script as a Postboot Script in Flack

to each VM

• One pair of nodes are not adjacent• Each node has at least two interfaces

router

Sponsored by the National Science Foundation 9Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#3 End-to-end Validation

router

router router

router

client

server

router

router router

router

Manually Install Iperf to client/server

Validate End-to-End

Communication

Add Client and

Server

Take similar procedure in step #2 to automate iperf installation using a postboot script.

Sponsored by the National Science Foundation 10Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#4 Measurement

• Track routing table updates– MANUAL Script to track routing table updates– AUTOMATE Periodically Capture Routing table

• Tracking routing table updates by killing XORP process (mimic virtual router failure)

– Tracking the routing table changes using terminal– Display graphs of routing table changes using Labwiki

• Followed step #1 to create a new custom image with both LabWiki and XORP installed.

router

router router

router

client

server

Sponsored by the National Science Foundation 11Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#3 End-to-End Validation

router-1

router router-2

router-3

client

server

Outputs from the server:

Sponsored by the National Science Foundation 12Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#5 Scaling up

• Scale to more aggregatesTOPOLOGY Stitch between 2/3 aggregates

TEST Repeat earlier tests

router

router router

router

client

server

router

router router

router

client

server

Sponsored by the National Science Foundation 13Systematic Experimentation – GEC 20 – June 23, 2014

Single Aggregate:

Multiple Aggregates with Stitched Links:

Case Study#5 Scaling up

Sponsored by the National Science Foundation 14Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#5 Scaling Up

•Scale nodes at aggregateTOPOLOGY Add more routers and links

TEST Repeat earlier tests

router

router router

router

client

server

router

router

router

router

routerrouter

router

client

server

Sponsored by the National Science Foundation 15Systematic Experimentation – GEC 20 – June 23, 2014

router-3

router-1

router-2

router-4

router-7router-6

router-5

client

server

Traceroute from server to client

Iperf TCP test outputs from server

Sponsored by the National Science Foundation 16Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#5 Scaling Up

Display the routing table update from three routers over time in Labwiki

Sponsored by the National Science Foundation 17Systematic Experimentation – GEC 20 – June 23, 2014

Case Study#5 Scaling Up

• Scaling up at both dimensions (nodes and aggregates)

router

router router

router

client

server

router

router

router

router

routerrouter

router

client

server

router

router router

router

client

server

To be done

Sponsored by the National Science Foundation 18Systematic Experimentation – GEC 20 – June 23, 2014

Lessons Learned / Best Practices

• Lessons Learned– Learn tool first. Then use in experiment.– Underlying physical topology– Experience with GENI rack

• Challenges with debugging postboot script execution– Always start with a small postboot script execution

• Easy to get help from geni-user group mailing list

• Best Practices– Version control

• Software• Scripts• RSpec

– Keep the RSpec (both request and manifest RSpec) at each step, so we can track the updates/errors

Sponsored by the National Science Foundation 19Systematic Experimentation – GEC 20 – June 23, 2014

• Demo