Sponsored by the National Science Foundation

Systematic Experimentation in GENI

Sarah Edwards

GENI Project Office

Sponsored by the National Science Foundation 2Systematic Experimentation – GEC 20 – June 23, 2014

General Principle #1: One thing at a time

Experimentation 101 and Debugging 101

Only change one thing at a time

Examples:• Software/image• Configuration• Number of nodes/links• Geographic distribution of nodes/links

Sponsored by the National Science Foundation 3Systematic Experimentation – GEC 20 – June 23, 2014

General Principle #2: Start Small

Start by building smallest possible topology by hand.

Automate as needed.

Test and measure as you go.

client server

host

OVSswitch

host

host

router

router router

node

Sponsored by the National Science Foundation 4Systematic Experimentation – GEC 20 – June 23, 2014

General Principle #3: Save what you do

• Log all of your experimental artifacts for every experiment that works– RSpec – image– install script– custom software– measurements– etc

• Use version control to store your artifacts• Always know the last configuration that worked

Sponsored by the National Science Foundation 5Systematic Experimentation – GEC 20 – June 23, 2014

Process Recommendation

A. Build (smallest possible) topology by hand at a single aggregate

B. Automate topology creation at a single aggregate

C. Orchestrate and Instrument

D. Increase scale

E. More nodes

F. More aggregates

Automate

Sponsored by the National Science Foundation 6Systematic Experimentation – GEC 20 – June 23, 2014

Design Choices

• Custom images vs install scripts vs both• Single aggregate vs multi-aggregate• Which aggregates?• Bare metal vs virtual machine• What tools are you comfortable with?

There are no right answers. Pick what’s needed for your experiment.

Sponsored by the National Science Foundation 8Systematic Experimentation – GEC 20 – June 23, 2014

“How To” pages

• Listed under the “Experimenters” section

• Each “How To” is a short descriptions of how to do various tasks

• New entries being added all the time

Sponsored by the National Science Foundation 9Systematic Experimentation – GEC 20 – June 23, 2014

Finding other resources

• GENI wiki– Pages for Instructors and Experimenters

http://groups.geni.net/geni/wiki

Sponsored by the National Science Foundation 10Systematic Experimentation – GEC 20 – June 23, 2014

Sponsored by the National Science Foundation 11Systematic Experimentation – GEC 20 – June 23, 2014

Systematic Experimentation

Running an experiment on a testbed requires best

practices and methodology from a combination of:– Science (Scientific method)

– Programming

– System Administration

Today, talking about strategies and techniques for

bringing up experiments in a systematic way.

Sponsored by the National Science Foundation 12Systematic Experimentation – GEC 20 – June 23, 2014

Why experiment systematically?

Debugging Easier to debug experiments. Easier to get help debugging. Especially for complex experiments.

Scalability and Repeatability A well designed experiment will be repeatable and systematically scalable

Valid Develop a solid understanding of and validate your experimental setup

Rigorous Perform a scientifically rigorous (and publishable) study

Sponsored by the National Science Foundation 13Systematic Experimentation – GEC 20 – June 23, 2014

GENI-FIRE 150 node topology

Courtesy of Thierry Rakotoarivelo, NICTA, GEC 20 Demo

Clear plan AutomationScale and

Visualization

Sponsored by the National Science Foundation 14Systematic Experimentation – GEC 20 – June 23, 2014

Virtual Computer Networking Lab

Courtesy of Cong Wang and Divya Bhat, UMass Amherst, GEC 20 Demo

Start small

Sponsored by the National Science Foundation 15Systematic Experimentation – GEC 20 – June 23, 2014

Intelligent Data Movement System

Courtesy of Ezra Kissel, Indiana University, GEC 20 Demo

Long-lived slicefor stitched, shared VLAN

Dynamically add/remove nodes

as needed

Sponsored by the National Science Foundation 16Systematic Experimentation – GEC 20 – June 23, 2014

Why doesn’t this work?(Anonymous) Past Bug Reports

repeatable

incremental

test controller

in one AM?

GRE works

Stitch

test controllerin one AM

Multi-site Hadoop

Used control plane to send high volume data between

racks

Interfere with rack !

OpenFlow multi-site

OpenFlow multi-site

Sponsored by the National Science Foundation 17Systematic Experimentation – GEC 20 – June 23, 2014

Process: Start Small

A. Build smallest possible topology by hand at a single aggregate

B. Automate topology creation at a single aggregate – Store RSpecs, software, etc in version control – TOOLS Use install scripts and custom images– Determine how to update software on your nodes

Repeat A/B as needed.

C. Orchestrate and Instrument your experiment procedureTOOLS GENI Desktop and LabWiki

Automate as needed.

worker

master

worker

node

Sponsored by the National Science Foundation 18Systematic Experimentation – GEC 20 – June 23, 2014

Process: Scale

D. Increase scale (number of nodes, volume of traffic, etc) TOOLS Flack for small scale. geni-lib for larger scale

client

server

client

client clientclient

client server

Sponsored by the National Science Foundation 19Systematic Experimentation – GEC 20 – June 23, 2014

Process: Split across aggregates

E. Split an existing topology across second aggregate – How connect across aggregates matters (stitching vs OF etc)

TOOLS stitcher for VLAN stitching and GRE tunnels. ExoSM for Intra-ExoGENI stitching. Flack for GRE tunnels.

F. Split across multiple aggregates

client client

server

client clientclient

Colors represent different aggregates

Inter-aggregate link