UW EE Colloquium Feb. 2009 1
Network Simulation and Emulationin a Testbed Era
Tom HendersonUniversity of Washington
The Boeing Company
UW EE ColloquiumFebruary 24, 2009
Engineering, Operations & Technology | Boeing Research & Technology
Copyright Š 2009 Boeing. All rights reserved.
Networked Systems Technology
Information Assurance
Digital Communications &
Network Technology
Information Management and Transformation
Applied Mathematics
Architecture and Networked Systems
Technology
Contract Research and Development
Programs
Boeing Research & Technology organization
⢠Development of high assurance multi-level security technologies
â EAL7 common criteria evaluation experience
â Formal methods in support of EAL 7 evaluation
⢠High assurance guard solutions for edge, core, & tactical environments
⢠Security modeling & formal method analysis of end to end systems
⢠Policy-based secure computingâ Systems of Systems
IA approach⢠Intrusion detection correlation
technologies⢠Secure workflow and SOA
technologies⢠Information operations⢠System security engineering⢠Advanced trusted system
implementations⢠Text analysis for information
assurance⢠Proprietary program support in IA
technologies⢠Secure RFID infrastructure⢠Anti-tampering technologies
Networked Systems Technology
⢠Phased array antennas⢠Free air laser
communications⢠High capacity RF
communications⢠Bandwidth efficient
modulation⢠On board wireless
communications⢠Network processors⢠AFRL RAPID⢠Integrated communications
systems⢠MANET protocols⢠Reliable routing⢠Network management for
tactical environments⢠Auto-configuration⢠Network gateways⢠End to end QoS⢠Network security⢠UAV network router
⢠Data & Text Mining⢠Human Language technologies⢠Decision support information⢠Information visualization⢠Machine translation⢠Knowledge representation⢠Data integration⢠Simulation visualization⢠3D visualization⢠Model-based collaboration⢠Social networks analysis⢠Intelligent graphics & docs.⢠Collab. human robot systems⢠Autonomous systems
â Architecturesâ World modelingâ Multi-vehicle behaviors
⢠Reasoning⢠Adaptive planning⢠Learning ⢠Distributed multi-agents ⢠Global information sharing⢠RFID info. mgmt. services⢠Adaptive intelligent info. integration⢠Semantic modeling, ontologies,
adaptive workflow modeling⢠Quality of Service info. algorithms⢠Wireless mobile dynamic info. mgmt.⢠Sensor networks⢠Data & information fusion⢠Info. assessment aids⢠Trends analysis/intent inferencing⢠Situation assessment⢠Resource allocation techniques⢠Deconfliction algorithms⢠Course of action analysis/generation
⢠Variation analysis & reduction⢠Reliability & time dependent data
analysis⢠Analysis of large datasets⢠Financial modeling & decisions⢠Survey & test design⢠Statistical training⢠Quality assurance & Lean+⢠Design of Experiments⢠Computer-aided geometric design⢠Numerical optimization⢠Design space exploration⢠Multidisciplinary Design
Optimization ⢠Geometric modeling⢠Multivariate data fitting⢠Optimal control⢠Engineering modeling & analysis⢠Computational mechanics⢠Sensor Fabric design & operation⢠System dynamics & control
modeling⢠Data structures for complex
product families⢠Kalman filter applications⢠Operations research⢠Systems of Systems modeling &
analysis⢠Discrete/network/stochastic/
Systems of Systems optimization⢠Data fusion⢠Discrete-event simulation⢠Transportation, manufacturing,
business, communication & military system modeling
⢠UML architecting⢠Arch. analysis⢠CONOPS development⢠System requirements⢠NR-KPP compliance⢠SoSE⢠Systems engineering⢠NCO demo.⢠LVC technologies⢠LVC tools⢠DIS router⢠Data link translation⢠Gateways⢠Networked manufacturing⢠Location services⢠Planning / visibility tools⢠Functional integration⢠High performance computing⢠GRID computing⢠Resource management⢠Functional integration⢠CINS lab⢠System/network mgmt.⢠QoS⢠Mobile agents⢠Situation awareness⢠Reconfigurable computing⢠S/W architecture⢠S/W processes⢠Arch. evaluation (ATAM)⢠Performance assessment
(PAPM)⢠QoS
⢠Real-time system development⢠Tactical info. mgmt. with an
emphasis in C2/tactical collaboration
⢠Quality of Service network utilization
⢠Software enabled control for UAV automation
⢠Dynamic network management⢠Affordable validation &
verification of flight critical software & systems
⢠Collaborative anticipatory systems
⢠Cyberwarfare / Cybercraft⢠Effects-based operations⢠System & vehicle health
management⢠Diagnostics⢠InfoCenter & information
management⢠Database technology &
implementation⢠Distributed sensor system
technology
Homeorganization
UW EE Colloquium Feb. 2009
What is a network simulator?
⢠A software tool and set of models used to evaluate computer network behavior
⢠A discrete-event network simulator describes networks in terms of a sequence of discrete eventsâ The simulation kernel temporally orders and
dispatches events to modelsâ Models generate output or more eventsâ Time advances in discrete steps
UW EE Colloquium Feb. 2009 4
Application
The basic model
Application
Protocolstack
Node
NetDeviceNetDevice
ApplicationApplication
Protocolstack
Node
NetDevice
Channel
Packet
Simul
UW EE Colloquium Feb. 2009
The basic model
// Define your topologyNode n0 = new Node;Node n1 = new Node;AddInternetStack (n0, n1);Channel c0 = new Channel;.Connect (n0, n1, c0);Application a0 = new TrafficGenerator;
// Configure thingsa0.SetDataRate (1Mb/s);a0.Start (10.0 seconds);
// Define outputsWriteTraceFile (âoutfileâ);
// Run the simulatorSimulator::Run();
UW EE Colloquium Feb. 2009 6
Network simulation remains popular
⢠Many commercial productsâ e.g. OPNET, QualNet, Shunra VE, Matlab Simulink,
OMNet++, NCTUns, SSFNet, Extend
⢠A few free, open source projectsâ ns-2 is the most widely used such tool
â roughly 10,000 downloads/month
⢠Widely used in research
Source: Search of ACM Digital Library papers citing simulation, 2001-04
UW EE Colloquium Feb. 2009 7
Skepticism in the research community
âFor years, the community had to rely on simulators, which now seem a little dated, and itâs not clear who was convinced to adopt anything new based on ns2 simulations;â
Nick McKeown, VINI public review, ACM Sigcomm 2006
UW EE Colloquium Feb. 2009
What has changed?
⢠Simulators have historically been used for experiments in place of real machinesâ A tradeoff of realism for scale
But now...
⢠Cost of test hardware has plummeted
⢠Virtualization technologies proliferate
UW EE Colloquium Feb. 2009
Emulab (University of Utah)
⢠Source: http://www.emulab.net
UW EE Colloquium Feb. 2009
Emulab
⢠Original Emulab allowed experiments of roughly ~100 nodesâ FreeBSD Dummynet and VLANs to interconnect the
nodes
⢠Emulab now offers virtualizationâ Experiments of up to ~2000 (virtual) nodes
â Requires new approaches for experimental configuration
⢠Also, some 802.11 wireless nodesSource: M. Hibler et al., âLarge-scale Virtualization in the Emulab
Network Testbed,â Proc. 2008 Usenix Technical Conference.
UW EE Colloquium Feb. 2009
PlanetLab: Global-scale Internet testbed
⢠Source: http://www.planet-lab.org
PlanetLab âslicesâ
UW EE Colloquium Feb. 2009
PL-VINI and Trellis
⢠PL-VINI adds virtual networking capabilities (via network overlay) to PlanetLabâ A. Bavier et al, âIn VINI Veritas: Realistic and
Controlled Network Experimentation,â Proc. ACM Sigcomm, 2006
⢠Trellis is a kernel-based implementation of the same conceptâ S. Bhatia et al, âTrellis: A Platform for Building
Flexible, Fast Virtual Networks on Commodity Hardware,â ROADS Workshop, Dec. 2008
UW EE Colloquium Feb. 2009
ORBIT (Rutgers WINLAB)
⢠Source: http://www.winlab.rutgers.edu/docs/focus/ORBIT.html
⢠802.11a/b/g radios
⢠Bluetooth
⢠GNU/USRP radios
⢠Noise generators
UW EE Colloquium Feb. 2009 14
Simulation âout of the loopâ
⢠Source: NSF GENI project: http://www.geni.net/docs/GENIOvrvw092908.pdf
UW EE Colloquium Feb. 2009 15
Skepticism in the research community
âFor years, the community had to rely on simulators, which now seem a little dated, and itâs not clear who was convinced to adopt anything new based on ns2 simulations;â
Nick McKeown, VINI public review, ACM Sigcomm 2006
UW EE Colloquium Feb. 2009 16
Overheard* on e2e-interest mailing list
â...Tragedy of the Commons...â
â...around 50% of the papers appeared to be... bogus...â
âWho has ever validated NS2 code?â
âTo be honest, I'm still not sure whether I will use a simulation in a paper.â
â...I will have a hard time accepting or advocating the use of NS-2 or any other simulation toolâ
...
* September 2005 archives of the e2e-interest mailing list
UW EE Colloquium Feb. 2009
Questions
⢠Do we need to simulate networks anymore?
⢠What kind of network simulator is needed?
⢠Why should you believe network simulation results?
UW EE Colloquium Feb. 2009
Why simulate?
⢠Wirelessâ Reproducibilityâ Fidelity (especially, real-time constraints)â Radios may not exist or be availableâ Field tests in realistic conditions cost $$
⢠Scalabilityâ 10,000+ nodes?â For smaller configurations, execution time
Simulation is often an appropriate tool for the job
UW EE Colloquium Feb. 2009 19
Challenges for ns (and simulators)
⢠Align with how research is now conducted
⢠Improve credibility
Can a new simulator help with these problems?
UW EE Colloquium Feb. 2009 20
What is ns-3?
An open source project building a new network simulator to replace ns-2
UW EE Colloquium Feb. 2009 21
ns-3 project goal
Develop a preferred, open simulation environment for networking research1) a tool aligned with the simulation needs of
modern networking research
2) an open-source project that encourages community contribution, peer review, and validation of the software
UW EE Colloquium Feb. 2009 22
Relationship to ns-2
ns-3 is a new simulator, without backward compatibility
Similarities to ns-2:⢠C++ software core⢠GNU GPLv2 licensing⢠ported ns-2 models: random variables, error models,
OLSR, Calendar Queue scheduler, (more planned)
Differences: ⢠Python scripting (or C++ programs) replaces OTcl⢠most of the core rewritten⢠also based on the yans and GTNetS simulators⢠new animators, configuration tools, etc. are in work
UW EE Colloquium Feb. 2009 23
Funding support
ns-3 project is supported by:
⢠NSF CISE CRI awards (2006-10)â PIs: Tom Henderson and Sumit Roy (University of Washington),
George Riley (Georgia Institute of Technology)
â NSF CNS-0551686 (University of Washington)
⢠INRIA Planete and the French government
With additional support from:
⢠2008 Google Summer of Code
⢠University of Washington
⢠Georgia Institute of Technology
â
UW EE Colloquium Feb. 2009 24
Challenges for ns (and simulators)
⢠Align with how research is now conducted
⢠Improve credibility
UW EE Colloquium Feb. 2009
Test and Evaluation Options
⢠When mathematical analysis alone wonât do
Increasing realism
Increasing complexity
Puresimulation
Simulationcradles
Virtual/Physicaltestbeds
Fieldexperiments
Livenetworks
Test and evaluation options
Can we develop tools to span this space?
UW EE Colloquium Feb. 2009
Some ns-3 design goals
1. Realism
2. Software reuse
3. Emulation
4. Composition
UW EE Colloquium Feb. 2009 27
1) Attention to realism
Research should often involve a mix of simulations and testbed or live experiments
⢠If the simulator cannot be made to closely model a real system:â hard to compare results or validate the model
â hard to reuse software between the two domains
When in doubt, do what implementations do
UW EE Colloquium Feb. 2009 28
Attention to realism (example)
An ns-3 Node is a husk of a computer to which applications, stacks, and NICs are added
ApplicationApplication
Application
âDTNâ
UW EE Colloquium Feb. 2009 29
Realism (cont.)
⢠Align ns-3 to be more faithful representation of real computersâ sockets APIâ packets are âpackedâ binary representations
ready for direct serialization to/from network devices
â alignment with Linux architectureâ multiple interfaces handled properly
UW EE Colloquium Feb. 2009 30
ns-3 follows a Linux architecture
classNetDevice
classIpv4Protocol
⢠Linux corollariesâ each NetDevice is modelled
like struct net_device
â each layer-3 protocol has a struct in_device component for each NetDevice
â this interface (transmit side) is aligned with Linux dev_queue_xmit()
â on receive side, demux is modelled after Linux protocol handlers
UW EE Colloquium Feb. 2009 31
2) software integration
Do not reimplement models and tools for which open-source implementations abound
⢠ns-3 conforms to standard input/output formats so that other tools can be reused.â e.g., pcap trace output, ns-2 mobility scripts
⢠ns-3 is adding support for running implementation codeâ Network Simulation Cradle integration has met with
success: Linux TCP code
â ns-3 âprocessâ API
UW EE Colloquium Feb. 2009
ns-3 simulation cradle
⢠Port by Florian Westphal of Sam Jansenâs Ph.D. work
Figure reference: S. Jansen, Performance, validation and testing with the Network Simulation Cradle. MASCOTS 2006.
UW EE Colloquium Feb. 2009
ns-3 simulation cradle
For ns-3:⢠Linux 2.6.18⢠Linux 2.6.26⢠Linux 2.6.28
Others:⢠FreeBSD 5⢠lwip 1.3⢠OpenBSD 3
Other simulators:⢠ns-2⢠OmNET++
Figure reference: S. Jansen, Performance, validation and testing with the Network Simulation Cradle. MASCOTS 2006.
UW EE Colloquium Feb. 2009 34
ns-3 âprocessesâ and Posix API
Support for a synchronous Posix socket API
⢠each Posix type and function is redefined in the simulator
⢠processes get their own private stackâ somewhat like a lightweight virtual machine
⢠Benefits:â makes porting real world application code much easier
â makes writing applications easier because the BSD socket API is faithfully followed
⢠see the âmathieu/ns-3-simuâ code repository
UW EE Colloquium Feb. 2009 35
3) emulation support
Support moving between simulation and testbeds or live systems
⢠A real-time scheduler, and support for two modes of emulation
UW EE Colloquium Feb. 2009 36
ns-3 emulation modes
virtualmachine
ns-3
virtualmachine
1) ns-3 interconnects real or virtual machines
realmachine
ns-3
Testbed Testbed
realmachine
ns-3
2) testbeds interconnect ns-3 stacks
real machine
Various hybrids of the above are possible
UW EE Colloquium Feb. 2009 37
Example: ORBIT and ns-3
⢠Support for use of Rutgers WINLAB ORBIT radio grid
UW EE Colloquium Feb. 2009
4) Composition
ns-3 is a component of an overall workflow
Define ns-3 withinterfaces to supportintegration with othertools
⢠Topology generators⢠Traffic generators⢠Data analysis frameworks⢠Configuration tools⢠Animators
Engineering, Operations & Technology | Boeing Research & Technology
Copyright Š 2009 Boeing. All rights reserved.
Networked Systems Technology
Common Open Research Emulator
Scalable Network Emulator
⢠Network lab âin a boxâ⢠Efficient and scalable⢠Easy-to-use GUI canvas
⢠Kernel-level networking efficiency⢠Reference passing packet sending
⢠Runs real binary code⢠No need to modify applications
⢠Connects with real networks⢠Hardware-in-the-loop⢠Distributed - runs on multiple servers⢠Virtual nodes process real packets
⢠Fork of the IMUNES project⢠University of Zagreb
⢠Open Source⢠http://cs.itd.nrl.navy.mil/work/core
Engineering, Operations & Technology | Boeing Research & Technology
Copyright Š 2009 Boeing. All rights reserved.
Networked Systems Technology
netgraphsystem
Tcl/Tk GUIcore_wlan
core_spanCOREAPI
ng_wlan
NIC
FreeBSDkernel
virtual images(vimages)
tunnels
userspace
Overview of CORE Components
Three key components
1. virtual machines
3. graphical user interface
2. networking subsystem
Engineering, Operations & Technology | Boeing Research & Technology
Copyright Š 2009 Boeing. All rights reserved.
Networked Systems Technology
CORE features
IMUNES base:⢠Flexible GUI-based management of FreeBSD lightweight
virtual containers⢠Virtual topology can be hooked to real devices
CORE extensions:⢠GUI enhancements, kernel enhancements, wireless
networking, Linux support (OpenVz and NetEm), distributed emulation, and packaging.
⢠Software leads: Jeff Ahrenholz and Claudiu Danilov, Boeing⢠http://cs.itd.nrl.navy.mil/work/core
Engineering, Operations & Technology | Boeing Research & Technology
Copyright Š 2009 Boeing. All rights reserved.
Networked Systems Technology
CORE and ns-3 integration
ns-3 tap-wifi-dumbbell.cc program
UW EE Colloquium Feb. 2009 43
ns-3 and research alignment
In summary, make it easier to move from simulation to emulation to experiments
⢠Realism and alignment with popular interfaces⢠Support use of real code⢠Emulation capabilities⢠Combine ns-3 with other tools
UW EE Colloquium Feb. 2009 44
Challenges for ns (and simulators)
⢠Align with how research is now conducted
⢠Improve credibility
UW EE Colloquium Feb. 2009 45
Background on simulation credibility
[1] âWhy We STILL Donât Know How to Simulate Networksââ Mostafa Ammar, Georgia Institute of Technology, Annual
Simulation Symposium 2005
[2] âMaintaining a Critical Attitude Towards Simulation Resultsâ â Sally Floyd, WNS2 Workshop Keynote, October 2006
[3] âMANET Simulation Studies: The Incrediblesâ â Kurkowski, Camp, and Colagrosso, ACM Sigmobile, MC2R,
Volume 9, Issue 4, October 2005
[4] âAn Integrated Approach to Evaluating Simulation Credibilityâ â Muessig, Laack, and Wrobleski, U.S. Naval Air Warfare Center,
August 2001
UW EE Colloquium Feb. 2009 46
Criteria for Credibility
⢠Repeatable
⢠Unbiased
⢠Realistic Scenarios
⢠Statistically Sound
⢠Model Accuracy
⢠Results Accuracy (Validation)
⢠Data Accuracy
⢠Usability
from [3]
from [4]
UW EE Colloquium Feb. 2009 47
Repeatability
A hallmark of the scientific method...
⢠Papers should identify simulator, version, operating system, parameters, etc.
⢠Better yet, make code and configuration scripts available to the communityâ Yet, 0 out of 84 ACM Mobihoc MANET
simulation papers (2000-2004) referenced publicly available code (from [3])
UW EE Colloquium Feb. 2009
ns-3 approach
⢠We are adding support to systematically document how simulations were conductedâ A configuration subsystem that dumps
simulation parameters to a configuration file (input and output)
⢠We will host code, publications, validation results, etc.
UW EE Colloquium Feb. 2009 49
Model accuracy
⢠âerror-free-nessâ of software and models
⢠ns-3 goals here:â Support real code where possibleâ Open source modelsâ Maintainers
UW EE Colloquium Feb. 2009 50
Open source models
⢠âGiven enough eyeballs, all bugs are shallowâ â Eric Raymond, âThe Cathedral and the Bazaarâ
⢠ns-3 needs ways to certify models, tooâ capture level of community acceptanceâ publication lists, cross-referenceâ need to identify maintainers, or state the
absence of a maintainerâ validation techniques and results
UW EE Colloquium Feb. 2009
Example: ns-3 Wifi development
Several research groups are maturing the original INRIA model:
⢠Karlsruhe Institute of Technology: 802.11 PHY, 802.11e
â Equalizing PHY models including capture effects, user-definable coding rates (e.g. 5.9 GHz from 802.11p), EDCA QoS extensions of 802.11e, Nakagami/Rayleigh propagation loss model
⢠University of Florence: 802.11n features
â Frame Aggregation, Block ACK, HCF (EDCA and support for HCCA),TXOP, HT terminal (also with protection modes), MIMO
⢠Russian Academy of Sciences: 802.11s
â a complete model of IEEE802.11s D2.0 Draft Standard
⢠Deutsche Telekom Laboratories in Berlin: 802.11 PHY
⢠Boeing: 802.11b channel models, validation
⢠(and others...)
UW EE Colloquium Feb. 2009
Pledge break
⢠ns-3 needs participation from the research community1) improving simulation credibility2) contributed and supported models3) maintainers
Please support your open source projects!
UW EE Colloquium Feb. 2009 53
Summary of simulation credibility
⢠Learn from good and bad examples of simulation research, to produce credible simulations
⢠Consider open source (or publishing of models and scripts) to be integral part of your research
⢠Please give back to the simulators that you use
UW EE Colloquium Feb. 2009 54
Acknowledgments
Thanks to:
⢠the core development team and research project leadsâ Raj Bhattacharjea, Gustavo Carneiro, Walid Dabbous, Craig Dowell, Joe
Kopena, Mathieu Lacage (software lead), George Riley, Sumit Roy
⢠2008 Google Summer of Code mentors and students
⢠many code authors and testers
⢠the ns-2 PIs and developers for creating ns-2 and for supporting ns-3 activities
⢠USC ISI for hosting project mailing lists
Contact: [email protected] (office M354)