Winter 2008 Evaluation Tools1 Brief Overview of Networking Evaluation Methods and Tools

winter 2008 Evaluation Tools 1

Brief Overview ofNetworking Evaluation

Methods and Tools


Outline

• Tools– Simulation

• ns2

– Cluster based network emulation• Emulab

– Live Distributed Testbed • PlanetLab

• Questions– What is the right tool(s) for my research?– What does it take to get up and running?


Outline

• For each tool/testbed– How does it work?– A “hello world” in ___– What kind of research is it good for?

• Tradeoffs between tools


Simulation(ns2)


What is ns?• Network simulator• a discrete event simulator• focused on modeling network protocols

– wired, wireless, satellite– TCP, UDP, multicast, unicast– Web, telnet, ftp– Ad hoc routing; sensor networks– Infrastructure: stats, tracing, error models etc.


ns --- what is it good for?

• Evaluate performance of existing network protocols.• Prototyping and evaluation of new protocols.• Large-scale simulations not possible in real experiments.

Used to:


ns

• Event-driven simulator– Model world as events– Simulator has list of events– Process: take next one, run it, until done– Each event happens in instant of virtual time, but takes arbitrary real time

• Single thread of control• Packet level

How does it work:


Ns models

• Traffic/applications– CBR, FTP, telnet, web

• Routing/Queuing– Drop-tail, FQ, SFQ, RED, DRR– Wired routing, adhoc routing etc

• Transport– TCP (variants), UDP, multicast (SRM)


ns - software structure• Object oriented (C++, OTcl) – code reuse• Scalability + Extensibility

– Control/”data” separation– Split C++/OTcl object

• C++ for packet-processing (fast to run)• OTcl for control - (fast to write)

– Simulation setup and configuration


otcl and C++: The Duality

OTcl C++

Pure OTclobjects

Pure C++objects

C++/OTcl split objects

ns

Your ns-script


Outline• Overview• Tcl, OTcl basics• ns basicsns basics• Extending ns• ns internals


Basic structure of ns-scripts

• Creating the event scheduler• [Tracing]• Creating network topology• Creating Transport Layer - Agents• Creating Applications - Applications• Events!


Creating Event Scheduler• Create scheduler

– set ns [new Simulator]

• Schedule event– $ns at <time> <event>– <event>: any legitimate ns/tcl commands

• Start scheduler– $ns run


“Hello World” in nssimple.tcl

set ns [new Simulator]$ns at 1 “puts \“Hello World!\””$ns at 1.5 “exit”$ns run

bovik@gs19% ns simple.tclHello World!bovik@gs19%


Creating Network• Nodes

– set n0 [$ns node]– set n1 [$ns node]

• Links & Queuing– $ns duplex-link $n0 $n1 <bandwidth> <delay>

<queue_type>– Queue type: DropTail, RED, CBQ, FQ, SFQ, DRR


Routing + traffic• Unicast

– $ns rtproto <type> – <type>: Static, Session, DV

• Multicast support also.• Traffic

– Simple two layers: transport and application.– Transport: TCP, UDP etc.– Applications: web, ftp, telnet etc.


The transport layer: UDP• UDP

– set udp [new Agent/UDP]– set null [new Agent/NULL]

– $ns attach-agent $n0 $udp– $ns attach-agent $n1 $null

– $ns connect $udp $null


The transport layer: TCP• TCP

– set tcp [new Agent/TCP]– set tcpsink [new Agent/TCPSink]

– $ns attach-agent $n0 $tcp– $ns attach-agent $n1 $tcpsink

– $ns connect $tcp $tcpsink


Creating Traffic: On Top of TCPFTP

– set ftp [new Application/FTP]– $ftp attach-agent $tcp– $ns at <time> “$ftp start”

Telnet– set telnet [new Application/Telnet]– $telnet attach-agent $tcp


Creating Traffic: On Top of UDP

• CBR– set src [new Application/Traffic/CBR]

• Exponential or Pareto on-off– set src [new Application/Traffic/Exponential]– set src [new Application/Traffic/Pareto]

• Trace driven traffic– Inter-packet time and packet-size


Attaching a traffic source

• set cbr [new Application/Traffic/CBR]

• $cbr attach-agent $udp

• $ns at <time> “$cbr start”


TracingTrace packets on all links:

– set f[open out.tr w]– $ns trace-all $f

– $ns flush-trace– close $f

<event><time><from><to><type><size>--<flags>--<flow id><src><dst><seqno> <pckt id>+ 1 0 2 cbr 210 ------- 0 0.0 3.1 0 0- 1 0 2 cbr 210 ------- 0 0.0 3.1 0 0r 1.00234 0 2 cbr 210 ------- 0 0.0 3.1 0 0

Is tracing all links always the best thing to do?


More Tracing

• Tracing specific links– $ns trace-queue $n0 $n1 $f

• Tracing variables – set cwnd_chan_ [open all.cwnd w]– $tcp trace cwnd_– $tcp attach $cwnd_chan_


Controlling object parameters

• Almost all ns objects have parameters– ex. Application/Traffic/Exponential has rate and

packetSize– set parameters in OTcl

• set etraf [new Application/Traffic/Exponential]• $etraf set rate_ 1Mb• $etraf set packetSize_ 1024


Putting it all togetherset ns [new Simulator]

set n0 [$ns node]set n1 [$ns node]$ns duplex-link $n0 $n1 1.5Mb 10ms DropTail

Creating Topology

set tcp [$ns create-connection TCP $n0 TCPSink $n1 0] Creating Transport layer

set ftp [new Application/FTP]$ftp attach-agent $tcp Creating Applications

$ns trace-queue $n0 $n1 $f

Schedule Events$ns at 0.2 "$ftp start“

$ns at 1.2 ”exit“

$ns run

n0 n11.5Mb 10msFTP/TCP


Example: XCP vs CSFQ

• Compare utilization of links

• Compare throughput of flows


nam – the network animator

n0 n1

set nf [open out.nam w]

$ns namtrace-all $nf

exec nam out.nam &…


ns “components”• ns,ns, the simulator itself• namnam, the Network AniMator

– Visualize ns output– GUI input simple ns scenarios

• Pre-processing:– Traffic and topology generators

• Post-processing:– Simple trace analysis, often in Awk, Perl, or Tcl


Network Dynamics: Link failures

• $ns rtmodel-at <time> <up|down> $n0 $n1

• $ns rtmodel Trace <config_file> $n0 $n1

• $ns rtmodel <model> <params> $n0 $n1 <model>: Deterministic, Exponential


Issues in Simulations

• Suppose you want to study the way TCP sources share a bottleneck link…Agent/FTP

Agent/FTP

Agent/FTP

Agent/FTP

Which topology?

When to start sources?

What else affects results?

Which traffic sources? Background Traffic?


Cluster Based Emulation(Emulab)

368 nodes3 big Ciscos

Slides based on SOSP poster by Jay Lepreau et. al. University of Utah www.emulab.net


Why?• “We evaluated our system on five nodes.”

-job talk from university with 300-node cluster• “We evaluated our Web proxy design with 10 clients on 100Mbit

ethernet.”• “Simulation results indicate ...”• “Memory and CPU demands on the individual nodes were not

measured, but we believe will be modest.”• “The authors ignore interrupt handling overhead in their evaluation,

which likely dominates all other costs.”• “You have to know the right people to use the cluster.”• “The cluster is hard to use.”• “<Experimental network X> runs FreeBSD 2.2.x.”• “February’s schedule for <Testbed Y> is…”• “<Network Z> is tunneled through the Internet”


What?• An instrument for experimental CS research• A completely configurable “Internet emulator” in a

room– At its core, it’s bare hardware, with…– … complete remote access and control

• But, also simple to use– Lots of fast tools for common case

• Automatic topology, node and link configuration• Automatic traffic generation

– Universally available• Universities, research labs, companies

– Zero-penalty for remote research


Key Design Aspects

• Allow experimenter complete control– Configurable network properties

• link bandwidth, Latency, and loss rates via transparently interposed “traffic shaping” nodes that provide WAN emulation

– Configurable OS image• Linux, FreeBSD, Windows

• Virtualization– of all experimenter-visible resources– node names, network interface names, network addrs

• e.g., node-0.esmexp1.esm.emulab.net


Emulab Architecture

Programmable “Patch Panel”

PCPC

Web/DB/SNMPSwitch MgmtUsers

Internet

Control Switch/Router

Serial

Sharks Sharks

160168

PowerCntl


Experiment Creation Process


Using Emulab• Submit ns script via web form

– Specify number of nodes– Specify link properties

• Relax while emulab …– Generates config from script & stores in DB– Maps specified virtual topology to physical nodes– Provides user accounts for node access– Assigns IP addresses and host names– Configures VLANs– Loads disks, reboots nodes, configures OSs

• Two ways to run experiments– Interactive

• Works only if there are enough free nodes right now– Batch

• Runs experiment when enough free nodes available


Using Emulab

• Time sharing model for nodes– If you check out 50 nodes, they are

exclusively yours until you release them (or forced to release them!)


Using Emulab

set ns [new Simulator]source tb_compat.tcl

##setup the core nodesfor {set i 0} {$i <= 4} {incr i} { set node($i) [$ns node] tb-set-node-os $node($i) FC4-STD}## setup the core linksfor {set i 0} {$i < 4} {incr i} { set j [expr $i + 1] $ns duplex-link $node($i) $node($j) 100Mb 0ms DropTail}$ns duplex-link $node(4) $node(0) 100Mb 0ms DropTail

## setup the edge nodes and linksfor {set i 5} {$i <= 10} {incr i} { set node($i) [$ns node] tb-set-node-os $node($i) FC4-STD set j [expr $i % 5] set link($i) [$ns duplex-link $node($i) $node($j) 1000Kb 0ms

DropTail] tb-set-link-simplex-params $link($i) $node($i) 0ms 500Kb 0.0}

# Go!$ns rtproto Static$ns run

D

D

D

D

D D


What is Emulab Good For?(creators words)

• Simulation– Fast prototyping, easy to use, but less realistic

• Small static testbeds– Real hardware and software, but hard to configure and

maintain, and lacks scale

• Live networks– Realistic, but hard to control, measure, or reproduce results

Emulab complements and also helps validate these environments


What is Emulab Good For?

• Studying routing protocols– Difficult to create richly connected nodes– Experiments that need access to the network core

• Studying peer-to-peer protocols– But, lacks scale

• Scaling experiments with “raw” machines with no link emulation

• Good controlled environment for testing before going to PlanetLab


Live Distributed Testbed(PlanetLab)

644 nodes at 304 siteswww.planet-lab.org


Resource Sharing

• Statistical sharing of resources– Each application has a “slice” of the overlay

resources• Bandwidth, CPU, memory

– No guarantees on bandwidth, CPU or memory fairness

– Slices guaranteed 1Mbps of bandwidth, rest shared by all slices


Resource SharingEmulab PlanetLab


Slices


Slices


Slices


Per-Node View

Virtual Machine Monitor (VMM)

NodeMgr

LocalAdmin

VM1 VM2 VMn…


PlanetLab Virtualization: VServers

• Kernel patch to mainstream OS (Linux)• Gives appearance of separate kernel for

each virtual machine– Root privileges restricted to activities that do not

affect other vservers

• Some modification: resource control (e.g., File handles, port numbers) and protection facilities added


How to Use PlanetLab

• Get a slice– Request local PI– Setup SSH keys (no passwords on PlanetLab)

• Add nodes to your slice through the web interface

• Write (or borrow) scripts to deploy and run your system on PlanetLab and gather data from PlanetLab– Pssh/pscp (parallel versions of ssh/scp)

• Remember, you’re running on a live network!– Test your code before you deploy


Not Just a Testing Infrastructure

• PlanetLab started by Intel …

“…to support seamless migration of an application from an early prototype,through multiple design iterations,

to a popular service that continues to evolve.”


Testbed Comparison

•ns2 •Emulab •PlanetLab

•Event-driven simulated network•Scale of 100s depending of configuration•Simulated topology•Controlled environment

•Emulated network•Scale of 10s most of the time•Real controllable topology•Controlled environment

•Real overlay network•Scale of 100s (goal is over 1000)•Real fixed topology•Live network, uncontrolled environment

Documents

Winter 2008 Evaluation Tools1 Brief Overview of Networking Evaluation Methods and Tools