146 Int. J. Simulation and Process Modelling, Vol. 5, No. 2, 2009

Real-time network simulation support for scalable

routing experiments

Yue Li*, Jason Liu and Raju Rangaswami

School of Computing and Information Sciences,Florida International University,Miami, Florida 33199, USAE-mail: yueli@cis.fiu.eduE-mail: liux@cis.fiu.eduE-mail: raju@cis.fiu.edu*Corresponding author

Abstract: This paper describes a new software infrastructure that combines the scalability andflexibility benefits of real-time network simulation with the realism of open-source routingprotocol implementations. The infrastructure seamlessly integrates the open-source XORProuter implementation with a real-time large-scale network simulation engine. The design usesa novel forwarding plane offloading approach that decouples routing from forwarding andconfines the more resource consuming forwarding operations inside the simulation engine toreduce I/O overhead. Experiments demonstrate superior performance of the software routinginfrastructure without impairing accuracy. The infrastructure is shown to be able to supportlarge-scale routing experiments on light-weight virtual machines.

Keywords: network simulation; real-time simulation; network emulation; routing protocols;large-scale routing experiments.

Reference to this paper should be made as follows: Li, Y., Liu, J. and Rangaswami, R. (2009)‘Real-time network simulation support for scalable routing experiments’, Int. J. Simulationand Process Modelling, Vol. 5, No. 2, pp.146–156.

Biographical notes: Yue Li is a Postdoctoral Research Associate in the School of Computingand Information Sciences, Florida International University. He received PhD Degree inComputer Science from Xi’an Jiao Tong University, China in 2005. He joined TsinghuaNational Laboratory for Information Science and Technology as a postdoc from 2005 to 2007.His research interests include network simulation and emulation, high performance simulation,and multimedia networking.

Jason Liu is an Assistant Professor in the School of Computing and Information Sciences,Florida International University. His research focuses on parallel discrete-event simulation,performance modelling and simulation of computer systems and communication networks.He received a BA in Computer Science from Beijing University of Technology in China in1993, an MS in Computer Science from College of William and Mary in 2000, and a PhD inComputer Science from Dartmouth College in 2003.

Raju Rangaswami is an Assistant Professor of Computer Science at the Florida InternationalUniversity inMiami.He received aBTechDegree inComputer Science from the Indian Instituteof Technology, Kharagpur, India. He obtained MS and PhD Degrees in Computer Sciencefrom the University of California at Santa Barbara. His research interests include operatingsystems, storage systems, virtualisation, security, and real-time systems.

1 Introduction

Researchers are continuously searching for techniquesthat can better evaluate the dynamic behaviour of routingprotocols in complex, large-scale network scenarios.Such techniques can help evolve our understanding ofcomplex network problems, such as the stability issueswith BGP and OSPF (Basu and Riecke, 2001; Shaikh

et al., 2000, 2002; Shaikh and Greenberg, 2001), BGPconvergence (Obradovic, 2002), and BGP security andmisconfiguration (Goodell et al., 2003; Chim and Yeung,2004; Mahajan et al., 2002; Nordstrom and Dovrolis,2004). Existing techniques for understanding andevaluating network routing protocols can be categorised asanalytical, simulation-based, and emulation-based. Herewe exclude direct experimentation on physical networks,

Copyright © 2009 Inderscience Enterprises Ltd.

Real-time network simulation support for scalable routing experiments 147

because it is infeasible to conduct custom large-scalerouting experiments directly on routers deployed deep inthe internet.

Analytical techniques, which can often bring valuableinsight to the design of large-scale complex systems, areill-equipped to address the complexities and dynamicsinherent with the actual protocol execution. Simulationis used commonly by researchers to perform large-scalerouting experiments since this technique can capture thenetwork dynamics with a great amount of detail. Baueret al. (2006), Liu and Chien (2004) and Dimitropoulos andRiley (2004). However, a simulation model of a router’sbehaviour can be quite different from a real routingprotocol implementation and thus may not providethe necessary realism of protocol execution behaviour.Further, simulation implementation of routing protocolstypically comes with only a modest selection of protocolsand protocol versions, thus limiting its utility.

Emulation-based techniques allow the execution ofnetwork routing protocols ‘as is’ within an emulatednetwork environment, providing the necessary realism inthe protocol execution.With the emergence of open-sourcerouter software, such as XORP (2008), Zebra (2008)and Quagga (2008), researchers can now prototype andevaluate routing protocols on actual systems as partof the emulation testbed. Emulation-based techniques,however, must address the issues of scalability andflexibility. The scalability of an emulation testbed isconstrained by the available physical resources. Forexample, experimental network scenarios must observethe limits on bandwidth, latency, and node counts ofthe underlying physical infrastructure. Emulation-basedsolutions are thus incapable of creating arbitrary networktopologies and controlling background network trafficconditions.

Our position is that the design of a successfulframework for routing experiments must comprehensivelyaddress the three challenges of realism, scalability, andflexibility. In this paper, we describe an infrastructure thatwe have built to support scalable routing experimentsbased on a real-time network simulation system. A keyadvantage of our approach is that, since it provides areal-time communication interface to real client machines,commodity applications can be executed unmodifiedwithin this framework (Liu, 2008). Consequently, ifengineered correctly, such a simulation framework canprovide a realistic network infrastructure indistinguishablefrom a real network from the standpoint of the realapplications. When this framework is coupled withmachines running open-source router software, it canprovide a realistic, scalable, and flexible infrastructurefor conducting routing experiments. We address the threechallenges comprehensively as we describe below:

• Realism. Our infrastructure uses a previouslydeveloped real-time network simulator, calledPRIME (2008). PRIME provides an emulationinfrastructure which supports a seamless integrationof a large number of prototype network applications

or protocols into the experimental environment.The infrastructure we have developed integratesPRIME with the open-source (XORP, 2008) routersoftware, making it possible to conduct large-scalerouting experiments. This technique also achieves ahigh level of realism since actual network protocolimplementations are directly included into theexperiments.

• Scalability. Traffic on the virtual network must beforwarded based on routing decisions made byXORP instances. If each packet must be exportedfrom the simulator and sent to an XORP routerinstance, and then immediately received andimported back into the simulator as soon asforwarding is achieved at the router, the I/Ooverhead would be substantial and might limit thescale of the routing experiments. We propose aforwarding plane offloading approach, which allowsthe XORP router to designate the packet forwardingfunction to the simulator and communicates with thesimulator its routing decisions through anout-of-band channel. In doing so we can confine thebackground traffic (unrelated to routing) withinsimulation, thereby eliminating the I/O overheadassociated with the bulk of the network traffic.Furthermore, since packet transmissions can beparallelised somewhat easily, we are able conductrouting experiments with a network size far beyondwhat can be supported by emulation testbeds.

• Flexibility. It is straightforward to apply real-timesimulation to explore a wide spectrum of networkscenarios and address what-if questions (e.g., byinjecting network failures). Real-time simulation isbased on simulation. Once a simulation model isdeveloped, reasonably verified and validated, it takeslittle effort to conduct simulation experiments tocover a large parameter space. The proposedinfrastructure also allows incorporating analyticalmodels in the real-time network simulation.For example, a user of the infrastructure can employlow-resolution models to describe aggregate internettraffic (e.g., Liu, 2006) and thus considerablyincrease the size of the network under test. Further,it is relatively easy to manipulate and controldynamic network behaviour with simulation. Forinstance, the user can bring down a link in the virtualnetwork and observe the reaction mechanism of therouting protocols.

2 Background

Our infrastructure for scalable routing experiments buildson top of a real-time parallel simulation engine calledPRIME, which stands for a Parallel Real-time ImmersiveModelling Environment (PRIME, 2008). PRIME enablesreal-time large-scale network simulations and supportsseamless interactions with real distributed applications.

148 Y. Li et al.

PRIME is built on top of a parallel discrete-eventsimulation kernel that implements the Scalable SimulationFramework (SSF) (Cowie et al., 1999). The parallelsimulation kernel deals with synchronisation andcommunication issues between logical processes thatexecute simulation events in parallel, while PRIME itselfenables the notion of real time by incorporating packetsfrom external application as real-time events. In orderto meet the deadline requirement of the real-time events,PRIME employs a greedy priority-based schedulingalgorithm – real-time events are assigned with a higherpriority to ensure their timely processing.

PRIME can interact with a large number of clients,which may be traffic sources, sinks, or other networkentities like routers. It is important tobeable to incorporatea large number of such clients and yet remain transparentto the applications running on the clients. There areseveral ways to incorporate real applications as part ofthe simulation environment, which include using packetcapturing techniques (such as libpcap, IP table, and IPtunnel), preloading dynamic libraries, and modifying thebinary executables, and so forth. PRIME instead usesan open systems approach based on a Virtual PrivateNetwork (VPN) solution (Liu et al., 2007). We customiseVPN to function as a gateway that bridges traffic betweenthe clients and the simulated network. Clients establishconnection to the simulation gateway. Traffic generated bythe clients and destined for the virtual network is directedby the modified VPN through the gateway to the real-timenetwork simulator.

We illustrate this approach using an example. Supposewe have two BGP routers which need to build upthe peering relationship with each other. VPN sets uptwo TUN interfaces with IP addresses of 10.10.0.2 and10.10.1.8 for one router, similarly 10.10.1.9 and 10.10.1.20for another router. The corresponding simulated nodes

have the same IP address configuration, as shown inFigure 1. One BGP speaker first sends an OPENmessage with destination IP address of 10.10.1.9 viaits interface 10.10.1.8, then the packet is sent to themodified VPN server at the simulation gateway througha VPN connection. The simulation gateway forwardsthe packet to PRIME via a dedicated TCP connection.In PRIME, the packet is treated as a real-time eventand injected into the simulation event queue. PRIMEsimulates the packet being forwarded on the virtualnetwork as if it was created by the simulated node withthe same IP address 10.10.1.8. Upon reaching the node10.10.1.9, the packet will be exported from simulationengine, travelling in the reverse direction via the simulationgateway back to the external router with the IP address10.10.1.9.

A distinct advantage of this approach is that theemulation infrastructure does not require special hardwareto set up. It is also secure, a merit inherited directlyfrom the underlying VPN implementation, and scalable,since clients need not be co-located within the simulationinfrastructure. Further, multiple simulation gateways canbe used simultaneously to accommodate larger trafficcapacity between the clients and the real-time networksimulator. In order to produce accurate results, however,the emulation infrastructure needs a tight couplingbetween the emulated entities (i.e., the clients) and thereal-time simulator. In particular, the segment betweenthe clients and the real-time network simulator needs toinclude low-latency links. To maintain potentially hightraffic throughput demand, this segment must also havesufficient bandwidth for tunnelling traffic through theemulation infrastructure.

In the next section, we develop techniques that allowthe above real-time simulation infrastructure to scale upwhen used specifically for network routing experiments.

Figure 1 The emulation infrastructure based on VPN (see online version for colours)

Real-time network simulation support for scalable routing experiments 149

3 Scalable routing infrastructure

The availability of open-source router platforms, such asXORP, Zebra, and Quagga, has simplified the task ofresearchers, who can now prototype and evaluate routingprotocols with relative ease. To support experiments ona large-scale network consisting of many routers withmultiple traffic sources and sinks, we propose to integratethe open-source router platforms with a real-time networksimulation infrastructure.

3.1 Virtual router

In order to scale up the experiment in an easy waywe adopt virtualisation technology to host multiplerouter instances. Virtualisation helps maintain isolatednamespaces, each of which manages the virtual resourcesnecessary for executing a router instance. We thereforecall each instance a virtual router. Specifically, fourtypes of network resources shall be provided withina virtual router: network sockets, network interfaces,forwarding table, and loopback device. Routing protocols,e.g., BGP, RIP, and OSPF, need to send and receivepackets via standard network sockets (TCP, UDP, andraw sockets) in order to establish peering connectivityand exchanging routing information. Network interfacesand the forwarding table together represent the networkforwarding engine of the virtual router on which routingprotocols are run.Anetwork loopback device is sometimesused by the routing protocols to communicate amongseparate processes (e.g., see XORP, 2008).

In particular, we investigate four popular virtualisationtechnologies: Xen (Barham et al., 2003; OpenVZ, 2008),Linux-VServer (Soltesz et al., 2007) andVRF (2008).Aswecan see in Table 1, all four types of network resources areprovided in Xen and OpenVZ, while Linux-VServer andVRFhave only partial network virtualisation support. Forthis reason, we focus only on Xen and OpenVZ in ourexperiment studies.

Table 1 Network virtualisation in Xen, OpenVZ,Linux-VServer and Linux-VRF

Xen OpenVZ VServer VRF

Network sockets � � � �Network interfaces � � �Forwarding table � � �Loopback device � �

We also examined the scalability aspects of these networkvirtualisation choices. We found that the network stackin Xen (version 3.0.4) becomes unstable under hightraffic load with 32 or more domains. This phenomenonis consistent with similar observations in other studies(Padala et al., 2007; Bhatia et al., 2008). As a result, wechooseOpenVZ as the platform for our large-scale routingexperiments.

3.2 Forwarding plane offloading

PRIME provides real-time simulation capabilities andan emulation infrastructure that can seamlessly integratemultiple clients, may they be routers or otherwise.Since the routers must be emulated outside PRIME onclient machines where they can run the real routingsoftware directly, every packet travelling along its pathfrom the source to the destination must be exportedto each intermediate router for forwarding decisions,and subsequently imported back into the PRIMEsimulation engine. Thus, the forwardingoperation for eachpacket at each hop incurs a substantial I/O overhead.Consequently, the overall overhead will significantlyimpact the performance of the infrastructure, especiallyin large-scale routing experiments. To avoid this problem,we propose a forwarding plane offloading approach,which moves the packet forwarding functions from therouter software to the simulation engine. Since packetforwarding operations are carried out entirely withPRIME, we can eliminate the I/O overhead associatedwith communicating bulk-traffic back and forth betweenthe router software and the real-time simulator.

This approach, however conceptually straightforward,raises two key design questions. First, the forwardingtable for a specific virtual node used by the simulatorfor internal forwarding decisions must be synchronisedwith the actual forwarding table, which is updated bythe routing protocols running within the correspondingexternal router instance. The synchronisation is critical toensure correct forwardingoperationswithin the simulationengine. Second, the simulation engine must be instantlyinformed of any network interface reconfigurationsperformed at the external router instance. For instance,a network administrator may create virtual networkinterfaces or reset the MTU of an existing networkinterface; in such cases, the corresponding simulated routernode must be kept consistent with the changes.

3.3 The XORP architecture

In our current implementation, we combine theopen-source XORP router software with PRIME toprovide a scalable platform for conducting routingexperiments. Two reasons prompted our choice of XORP.First, XORP has the most up-to-date implementationsof routing protocols, including those commonly usedprotocols, such as BGP, OSPF, and RIP; it is operationis consistent with real router software and it is awell-established experimental platform for routingprotocol design and evaluation by the routing protocolresearch community. Second, XORP implements aForwarding Engine Abstract (FEA) layer, which splitsthe routing and forwarding plane as two separate layerswithin its implementation, thus considerably simplifyingour task of offloading the latter to PRIME.

The primary role of FEA is to shield XORPprocesses from concerns of forwarding plane operations,so that arbitrary forwarding mechanisms can be employed

150 Y. Li et al.

via plug-ins. FEAhas threemain components: ForwardingInformation Base (FIB) manager, interface manager,and I/O manager. These components are depicted inFigure 2, along with the corresponding XORP processesthat interact with FEA. The FIB manager receivesforwarding table updates from the Routing InformationBase (RIB) process, which arbitrates route updates amongrouting protocols (e.g., BGP or OSPF). The FIB managerpropagates the necessary changes (such as inserting ordeleting routing entries) to the underlying forwardingengine. Currently, XORP can choose either the traditionalUnix kernel (via Netlink) or the Click modular softwarerouter (Kohler et al., 2000) for forwarding. Access tothe forwarding plane is performed by the correspondingplug-ins.

Figure 2 Offloading the forwarding plane to the PRIMEsimulation environment (see online versionfor colours)

The router manager process can also issue interfaceconfiguration requests to the FEA interface manager,according to a pre-specified router configuration file orreal-time input from the system administrator via acommand-line interface. FEA interprets and executes theserequests on behalf of the underlying forwarding engine.Individual routing processes can register with FEA to benotified of changes in the interface configuration. The I/Omanager also provides an interface for all the routingprotocols to communicate routing information (such asOSPF’s Link State Update messages) with their peers onother router instances.

3.4 XORP forwarding plane offloadingimplementation

Figure 2 describes our architecture for forwarding planeoffloading with XORP. We create an additional XORP

forwarding plane plug-in within FEA, which we callthe PRIME Agent. It consists of two components: thePRIMESocket and the InterfaceAgent. Both componentsmaintain a common command channel with the PRIMEsimulator for transferring forwarding information updates(via the PRIME Socket) and interface configurationrequests (via the Interface Agent) to the correspondingrouter node in PRIME. This effectively results in theforwarding functionality being offloaded from XORP toPRIME; traffic on the simulated network can thus be keptwithin simulation.

The interaction between the PRIME Agent and thesimulationmechanism ismanaged as follows. The PRIMESocket provides primitives tomanage the forwarding tablein PRIME, including those that add or remove a singleforwarding entry as well as the entire forwarding table.The PRIME Socket accepts a command from the FIBmanager, packages the command into a control packet,and sends it through the command channel. We designatean unused address as the destination address of thecontrol packets, so that they can be distinguished fromregular data packets when reaching the simulator throughthe emulation infrastructure. At the PRIME-end of thecommand channel, the control packets are translated intosimulation events and presented to the emulation protocolsession at the corresponding router node. The emulationprotocol session interprets the control commands andexecutes them by modifying the forwarding table on therouter node as directed by XORP. The procedure ofinterfacemanagement is essentially the same. The interfacemanager can modify the network interface configurationat the corresponding router node through primitivesprovided by the Interface Agent of the FEA plug-in.

Since forwarding table updates and interfacemanagement requests are propagated to the simulationengine, the forwarding decisions inside simulation aresynchronised with the XORP instances, except whenthe update commands are in transit. (We investigatethe possible discrepancy in Section 4.3). The routingmessages created by the XORP routing protocolinstances are treated as foreground traffic from theperspective of PRIME and they are forwarded throughthe simulation engine with the aid of the emulationinfrastructure as described in Section 2. This mechanismis enabled by independent data channels on top of VPNautomatically.

4 Experiments

We conducted three sets of experiments to evaluate ourinfrastructure for performance, accuracy, and scalability.The initial set of experiments (Section 4.1) were controlledmicrobenchmarks that allow fine-grained evaluation ofthe proposed infrastructure, both with and withoutforwarding plane offloading to the simulator. Next, wecarried out an intra-domain routing experiment of areal network topology (Section 4.2) to demonstrate thecapability of our infrastructure. Finally we conducted

Real-time network simulation support for scalable routing experiments 151

scalability studies to explore how large an experiment canbe supported by our routing infrastructure.

4.1 Microbenchmarks

Since PRIME needs to interact with the XORP instancesand simulate the virtual network in real time, thetiming accuracy is paramount in obtaining trustworthyexperimental results. This means that the infrastructure(both PRIME and the external XORP router instances)should keep up with the wall-clock time, receivereal-time events promptly and output them withoutmissing their real-time deadlines. The purpose of ourmicrobenchmark-based evaluation was to illustrate thatthe proposed infrastructure can support scalable andreal-time routing experiments with the forwarding planeoffloading technique.

We used an extensible network model formicrobenchmarking, as shown in Figure 3. Node S and Cacted as the server and client, respectively; the serverinitiated constant-bit-rate UDP traffic to the client. Theintermediate nodes, R1, R2, . . . , Rn, were routers. Eachrouter node had one corresponding XORP instancerunning outside PRIME. We configured each XORPinstance to run a OSPFv2 protocol.

Figure 3 Model topology used for microbenchmarkexperiments (see online version for colours)

For the microbenchmark experiment we adopted Xenvirtual machines (Barham et al., 2003) to host PRIME andXORP instances. In Xen terminology, each domain is avirtual machine running on the host machine. Domain 0is the first domain created when the host is powered onand has privileges to access physical machine resources.It also provides an interface to manage other domains(called Domain U’s) run unprivileged.

Wesetup theXenenvironmentonaDELLOPTIPLEX745workstationwith Intel Core 2Duo 2.4GHz processorsand 4 GB memory. Both the host OS and guest OSran Debian Linux 4.0 distributions with kernel version2.6.16.33. PRIME and the simulation gateway ran in XenDomain 0 and each Domain U contained one XORPinstance. We wrote a set of scripts to support rapiddeployment of an arbitrary network experiment scenario.

To evaluate performance of the simulationinfrastructure and its impact on timing fidelity,

we measured round-trip delay and throughput as theprimary metrics for evaluation. We evaluated twosettings of the simulation infrastructure implementation:offloading-disabled (in which the forwardingplane offloading optimisation was not used) vs.offloading-enabled (in which the optimisation wasemployed).

Figure 4 depicts the round-trip delay measured byping between the client and the server as we increasedthe number of intermediate router hosts running XORP.The propagation delay of individual links were setto be 10 ms each. Consequently, the ideal round triptime would be (2 × 10 × n), where n is the number ofintermediate routers. We varied the UDP backgroundtraffic between the server and the client to be either 1Mb/sor 2Mb/s.Whenwe disabled forwarding plane offloading,the observed round-trip times deviated from the idealcase, especially when the number of intermediate XORPinstances was large. The two curves representing the caseswith offloading disabled overlap. The observed valuesdeviated by asmuch as 30%with 32XORP instances.With2 Mb/s offered load, the variation of the round-trip timewas slightly larger. Note that the link bandwidths wereset to 100 Mb/s, which was substantially larger than theoffered load, andconsequently shouldnotnoticeably affectthe round-trip times in the ideal case. The observed valueswhen offloading was enabled matched almost exactly withthe ideal values, with less than 0.2% error, regardless of theoffered load.

Figure 4 Impact on round-trip delays

Figure 5 depicts the effect of the simulation infrastructureoverhead on traffic throughput between the client andservernodes, againaswevaried thenumberof intermediateXORP router instances. The throughput is calculated bythe total size of data transferred from the server to the clientdivided by the time since the download request is made bythe client. The throughput decreased slightly (even in theidea case) with more router instances and therefore longerpath between the server and the client. With offloadingdisabled, the observed throughput was lower than thatof the ideal case, becoming more apparent with moreintermediate XORP instances. With offloading enabled,

152 Y. Li et al.

the observed throughput deviated only slightly (less than3% with 32 XORP instances). As expected, since the datadownload trafficwas routedwithin simulation, rather thanbeing forwarded through the emulation infrastructure,we achieved much better accuracy.

Figure 5 Impact on throughput

4.2 An intra-domain routing experiment

In this section we describe an intra-domain routingexperiment as a case study to demonstrate that a realisticrouting experiment can be achieved with the integration ofPRIME and XORP. The experiment was originally usedby VINI (Bavier et al., 2006) to demonstrate its capabilityto conduct elaborate routing tests on the PlanetLab. Theexperiment consisted of a realistic Abilene network modelwith 11 routers, as shown in Figure 6. We configuredthe model using the real topology, with link delays andbandwidths obtained from the observatory data availableatAbileneObservatoryWebSite (AbileneNetwork, 2008).In particular, our model reflected a snapshot of theAbilene backbone on 14, January 2008 at 07:45:11 UTC.We also configured each XORP instance to run OSPFv2protocol. We used the same Xen virtual machineenvironment as described in Section 4.1 for the experimentsetup.

Figure 6 The Abilene network (see online version for colours)

The purpose of the experiment is to observe theconvergence of OSPF and its effect on data traffic.We injected a link failure, followed by a recovery betweenthe routers at Chicago and Indianapolis (via the XORPcommand-line interface). We measured the effect on theround-trip time and data throughput between Denver andChicago. Figure 7 shows the round-trip time measured byping. When the link went down at 14 s, OSPF instancesrecomputed the routes resulting a diversion of the trafficfrom the upper links to the lower ones. The round-tripdelays were increased from around 36ms to around 60ms.The linkwas later repairedat 43 s into the experiment.Afterthe routes converged, OSPF re-established the shortestpath between the two cities.

Figure 7 OSPF route convergence indicated by ping RTT

We also used iperf to initiate a TCP transfer fromDenverto Chicago in a separate run and analysed the tcpdumpresults. Figure 8 shows the number of bytes received atthe receiver end (i.e., the router at Chicago). The TCPtransfer was interrupted when link failure occurred at 18 sand then resumed when an alternative route was found(at 32 s). At 50 s, the link was repaired, the shortest pathwas re-established, and a better throughput was achieved(demonstrated by a steeper slope in the figure).

Figure 8 TCP transfer during link failure and recovery

Real-time network simulation support for scalable routing experiments 153

4.3 Scalability studies

The objective of our scalability studies is to explore howlarge an experiment can be supported by our routinginfrastructure with the virtual router technique. Thenetwork model for this set of experiments consisted ofmultiple Autonomous Systems (ASes), each being anidentical medium-size network called the campus network(shown in Figure 9). The campuses were connected toforma ring-like topology. This campus networkmodelwasused by DARPA NMS program as the baseline networkmodel for studying simulation performance (Nicol, 2008).In particular, the campus network consisted of 508 endhosts and 30 router nodes.We set the delay and bandwidthof all links between the routers to 10 ms and 100 Mbps,respectively. Each router node had one correspondingXORP virtual router running outside of PRIME. Thevirtual routers were configured to adopt OSPFv2 andBGP4 as IGP andEGP respectively. EachAS installed oneborder router acting as a BGP speaker and peering withtwo BGP speakers in neighbouring ASes.We scaled up theexperiment by increasing the number of ASes.

Figure 9 A campus network represents one AS, where therouter R0 in Net0 acts as the border router of the AS

We used two DELL PowerEdge 1900 servers for thisexperiment, which were connected with a 100 MbpsEthernet. Each machine had an Intel Quad-Core Xeon2.0 GHz CPU and 8 GB memory. We set up OpenVZenvironment on one machine, which hosted all the XORPvirtual routers as well as the simulation gateway. PRIMEran on the other machine. We wrote a set of scriptsto support automatic deployment of arbitrary networkexperiment scenarios. In particular, we started the XORPinstances sequentially one after another using a script.In the case of eight campuses, the entire initialisation couldtake about 25 min.

Table 2 shows the memory consumption on the XORPmachine as we increased the number of virtual routers.The memory usage depends on two main factors: oneis the fixed memory consumption associated with theinitialisation of each virtual router, which leads to a costproportional to the number of virtual routers included inthe study; the other is the cost due to routing informationmaintained at each virtual router. The latter requiresan O(m × p × n2) memory consumption, where n is thenumber of virtual routers involved in the study, p is theaverage number of prefixes associated with each virtualrouter, and m is the memory consumed per prefix in theXORP RIB. As a result of these two factors, we observe asuper-linear increase in the memory consumption.

Table 2 Memory usage (in MB) by XORP instances

# VRs 30 90 150 210 240

Memory 385.6 1570.6 3640.6 6295.6 7879.9

Figure 10 shows the number of OSPF and BGP packetswhich passed through the routing infrastructure inan experiment consisting of eight campuses, i.e., with240 virtual routers. The sampling interval was 10 s.We can clearly observe the periodic OSPF link staterefreshing behaviour that occurred at 30-min intervals(with significantly higher traffic through the routinginfrastructure). In the default setting, OSPF requires thateach Link State Advertisement (LSA) be refreshed every30 min. That is, the OSPF router that originates an LSAneeds to increase the LSA’s link state sequence numberand re-flood the network every 1800 s. Since we startedthe routers sequentially during a period of approximately25 min, we observe that the impact from this refreshingbehaviourwas spread out accordingly.We used the defaultconfiguration for the BGP speakers that exported alllocal IP prefixes. Consequently, the OSPF link staterefreshing triggered BGP updates to be sent to BGP peers.Thus, the BGP traffic fluctuated similarly with the OSPFtraffic.

Figure 10 Routing packets seen by the emulation gatewaywith 240 XORP instances

154 Y. Li et al.

Figure 11 shows the CPU utilisation on the XORPmachine in the same run. The CPU utilisation variedconsistently with the behaviour of routing protocols.We notice that the peakCPUutilisationwas less than 25%.We can conclude that memory, instead of CPU, was thecritical resource constraint for scaling up the experiment.

Figure 11 CPU utilisation with 240 XORP instances

We also measured the latency of the emulationinfrastructure,which is themajor contributor to the latencyoverhead for synchronising forwarding informationbetween PRIME and XORP instances. We enabled theLOOPBACK option in PRIME, in which case the readerthread immediately forwards the packets to the writerthread upon receiving them. Then we used ping tosend 52-byte packets (i.e., the same size as the theforwarding information update packets) from one virtualrouter to another. Since the packets actually traversethe emulation infrastructure four times during a roundtrip, we can easily calculate the latency imposed by theemulation infrastructure. The result shows an averagelatency of 0.11 ms, which confirms our belief that nosignificant forwarding skew between PRIME and XORPinstances couldoccurwithour forwardingplaneoffloadingtechnique.

5 Related work

Most current real-time network simulators are based onexisting network simulators added with the capabilitiesof interacting with real applications. Examples includeNSE (Fall, 1999), IP-TNE (Bradford et al., 2000), MaSSF(Liu et al., 2003), and Maya (Zhou et al., 2004). NSE is anemulation extension of the popularns-2 simulator (Breslauet al., 2000) augmented with the support for connectingwith real applications and scheduling real-time events.IP-TNE is the first simulator we know that adopts parallelsimulation techniques for large-scale network emulations.MaSSF is based on the DaSSF simulator (Liu and Nicol,2008) with support for the grid computing environment.Maya is an emulation extension of the commercialQualNet simulator (ScalableNetworkTechnologies, 2008)

for wireless mobile networks. ROSENET (Gu andFujimoto, 2007) is an emulation framework that separatesnetwork simulation that provides high-fidelity networkresource estimates from low-fidelity emulation trafficforwarding. To some extent, this is similar in conceptto our approach of separating routing and forwardingfunctions.

Other alternatives for conducting experimentalnetwork research are emulation and physical testbeds.EmuLab (White et al., 2002) is a notable emulation toolthat consists of a large number of networked computersintegrated and coordinated to present a virtual networkenvironment. EmuLab currently supports hundreds ofprojects with thousands of users running over 18,000experiments per year. PlanetLab (2008) is a collectionof machines distributed across the internet and usedby researchers to develop new network services. Thesemachines run a customised operating system that allowsthe computing and networking resources to be sharedamong multiple experimentations that may take placesimultaneously. A critical component of this type ofphysical testbeds is the live traffic.

Our work bears resemblance with VINI (Bavier et al.,2006). VINI is a virtual network infrastructure thatenables realistic and controlled network experiment overPlanetLab. By using virtualisation technology, researchersare able to create a flexible virtual network topology.The implementation of VINI runs XORP instances withinUML virtual machines and click for packet forwarding.We drew inspiration from the VINI design. Our schemeis different, however, in that we focus on enabling realnetwork simulation to achieve realism, scalability, andflexibility. Our approach allows a large number of XORPinstances beyond what can be supported by PlanetLab.

6 Conclusions and future work

A sound infrastructure allowing scalable routingexperiments is a key enabler for the design anddevelopment of network routing protocols. We proposea novel infrastructure that combines a real-time networksimulation engine with an emulated implementation ofopen-source XORP router software to provide a realistic,scalable, and flexible testbed for routing experiments.Offloading the forwarding plane from theXORP instancesto the simulator allows scalability, by confining bulktraffic to be forwarded more efficiently within thesimulator without exposing it to the external routers.An experimental evaluation of this infrastructure, withmicrobenchmark tests and a case study, demonstratesthe effectiveness of our approach in minimising overheadand consequently improving accuracy and scalability. Thescalability studies show that our routing infrastructurecan support for a large scale routing experiment with asmany as 240 virtual routers running on a single physicalmachine. We also report the memory is the bottleneck formultiplexing a large number of virtual routers by usingvirtualisation technology.

Real-time network simulation support for scalable routing experiments 155

Our current infrastructure for scalable routingexperiments has limitations. For example, thesynchronisation updates between the XORP protocolinstances and their corresponding router nodes withinPRIME are only one-directional in our current prototypeimplementation; updates are only propagated fromthe emulated XORP routers to the simulation engine.As immediate future work, we will allow informationexchange in both directions so that we will be able to dealwith more complex requirements in protocol operation.

Acknowledgement

This research is supported in part by National ScienceFoundation grants CNS-0546712, CNS-0836408, andHRD-0317692. We would like to thank Eric Johnson atFIU SCIS for the helpful discussions. We would also liketo thank the anonymous reviewers for their constructivecomments.

References

Abilene Network (2008) Abilene Observatory SummaryData View, http://abilene.internet2.edu/observatory/data-views.html

Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T.,Ho, A., Neugebauer, R., Pratt, I. and Warfield, A. (2003)‘Xen and the art of virtualization’, Proceedings of the19th ACM Symposium on Operating Systems Principles(SOSP’03), Bolton Landing, New York, USA, pp.164–177.

Basu, A. and Riecke, J.G. (2001) ‘Stability issues in OSPFrouting’, Proceedings of the ACM SIGCOMM Conferenceon Applications, Technologies, Architectures, andProtocols for Computer Communications (SIGCOMM’01),San Diego, California, USA, pp.225–236.

Bauer, D., Yuksel, M., Carothers, C. and Kalyanaraman, S.(2006) ‘A case study in understanding OSPF and BGPinteractions using efficient experiment design’, Proceedingsof the 20th Workshop on Principles of Advanced andDistributed Simulation (PADS’06),Washington,DC,USA,pp.158–165.

Bavier, A., Feamster, N., Huang, M., Peterson, L. andRexford, J. (2006) ‘In VINI veritas: realistic andcontrolled network experimentation’, Proceedings ofthe ACM SIGCOMM Conference on Applications,Technologies, Architectures, and Protocols for ComputerCommunications (SIGCOMM’06), Pisa, Italy, pp.3–14.

Bhatia, S., Motiwala, M., Muhlbauer, W., Valancius, V.,Bavier, A., Feamster, N., Peterson, L. andRexford, J. (2008)Performance Evaluation of Virtualization Technologiesfor Server Consolidation, Technical Report GT-CS-07-10,Georgia Tech Computer Science, Atlanta, USA.

Bradford, R., Simmonds, R. and Unger, B. (2000) ‘A paralleldiscrete event IP network emulator’, Proceedings of the8th International Symposium on Modeling, Analysis andSimulation of Computer and Telecommunication Systems(MASCOTS’00), pp.315–322.

Breslau, L., Estrin, D., Fall, K., Floyd, S., Heidemann, J.,Helmy, A., Huang, P., McCanne, S., Varadhan, K., Xu, Y.and Yu, H. (2000) ‘Advances in network simulation’,IEEE Computer, Vol. 33, No. 5, pp.59–67.

Chim, T.W. and Yeung, K.L. (2004) ‘Time-efficient algorithmsfor BGP route configuration’, Proceedings of the IEEEInternational Conference on Communications (ICC’04),Paris, France, pp.1197–1201.

Cowie, J., Nicol, D. and Ogielski, A. (1999) ‘Modeling the globalinternet’, Computing in Science and Engineering, Vol. 1,No. 1, pp.42–50.

Dimitropoulos, X.A. and Riley, G.F. (2004) ‘Large-scalesimulation models of BGP’, Proceedings of IEEEInternational Symposium on Modeling, Analysis, andSimulation of Computer and Telecommunication Systems(MASCOTS’04), Volendam, The Netherlands, pp.287–294.

Fall, K. (1999) ‘Network emulation in the vint/ns simulator’,Proceedings of the 4th IEEE Symposium on Computers andCommunications (ISCC’99), pp.244–250.

Goodell, G., Aiello, W., Griffin, T.G., Ioannidis, J.,McDaniel, P. and Rubin, A. (2003) ‘Working aroundBGP: an incremental approach to improving security andaccuracy of interdomain routing’, Proceedings of the10th Annual Network and Distributed System SecuritySymposium (NDSS’03), San Diego, California, USA,pp.150–161.

Gu, Y. and Fujimoto, R. (2007) ‘Applying parallel anddistributed simulation to remote network emulation’,Proceedings for the 2007 Winter Simulation Conference,Washington DC, USA, pp.1328–1336.

Kohler, E.,Morris, R., Chen, B., Jannotti, J. andKaashoek,M.F.(2000) ‘The click modular router’, ACM Transactions onCompauter Systems, Vol. 18, No. 8, pp.263–297.

Liu, J. (2006) ‘Packet-level integration of fluid TCP modelsin real-time network simulation’, Proceedings of the 2006Winter Simulation Conference (WSC’06), pp.2162–2169.

Liu, J. (2008) ‘A primer for real-time simulation of large-scalenetworks’, Proceedings of the 41st Annual SimulationSymposium (ANSS’08), Ottawa, Canada, pp.85–94.

Liu, J., Mann, S., Vorst, N.V. and Hellman, K. (2007)‘An open and scalable emulation infrastructure forlarge-scale real-time network simulations’, Proceedingsof the 26th Annual IEEE Conference on ComputerCommunications, Joint Conference of the IEEEComputer and Communications Societies (INFOCOM’07)MiniSymposium, pp.2476–2480.

Liu, J. and Nicol, D.M. (2008) Dartmouth Scalable SimulationFramework (DaSSF), http://www.cis.fiu.edu/˜liux/research/projects/dassf/index.html

Liu, X. and Chien, A.A. (2004) ‘Realistic large-scale onlinenetwork simulation’, Proceedings of the SuperComputingConference (SC’04), Pittsburgh, USA, pp.18–31.

Liu,X., Xia,H. andChien,A.A. (2003) ‘Network emulation toolsformodeling grid behavior’,Proceedings of 3rd IEEE/ACMInternational Symposium on Cluster Computing and theGrid (CCGrid’03), Tokyo, Japan, pp.18–26.

Mahajan, R., Wetherall, D. and Anderson, T. (2002)‘Understanding BGP misconfiguration’, Proceedingsof the ACM SIGCOMM Conference on Applications,Technologies, Architectures, and Protocols forComputer Communications (SIGCOMM’02), Pittsburgh,Pennsylvania, USA, pp.3–16.

156 Y. Li et al.

Nicol, D. (2008) DARPA NMS Baseline Network Topology,http://www.ssfnet.org/Exchange/gallery/baseline/index.html

Nordstrom, O. and Dovrolis, C. (2004) ‘Beware of BGPattacks’, ACM Computer Communications Review, Vol. 34,No. 2, pp.1–8.

Obradovic, D. (2002) ‘Real-time model and convergence timeof BGP’, Proceedings of the 21st Annual Joint Conferenceof the IEEE Computer and Communications Societies(INFOCOM’02), New York City, USA, pp.893–901.

OpenVZ (2008) OpenVZ, http://openvz.org/

Padala, P., Zhu, X., Wang, Z., Singhal, S. and Shin, K.G. (2007)Performance Evaluation of Virtualization Technologiesfor Server Consolidation, Technical Report HPL-2007-59,HP Laboratories.

PlanetLab (2008) Planetlab: An Open Platorm for Developing,Deploying, and Accessing Planetary-scale Services,http://www.planet-lab.org/

PRIME (2008) http://www.cis.fiu.edu/prime/

Quagga (2008) Quagga Routing Suite, http://www.quagga.net/

Scalable Network Technologies (2008) http://scalablenetworks.com/

Shaikh, A., Dube, R. and Varma, A. (2002) ‘Avoiding instabilityduring graceful shutdown of OSPF’, Proceedings of the21st Annual Joint Conference of the IEEE Computer andCommunications Societies (INFOCOM’02),NewYorkCity,USA, pp.883–892.

Shaikh, A. and Greenberg, A. (2001) ‘Experience in blackboxOSPF measurement’, Proceedings of the SIGCOMMInternetMeasurementWorkshop, SanFrancisco,California,USA, pp.113–125.

Shaikh, A., Kalampoukas, L., Dube, R. and Varma, A.(2000) ‘Routing stability in congested networks:experimentation and analysis’, Proceedings of theACM SIGCOMM Conference on Applications,Technologies, Architectures, and Protocols for ComputerCommunications (SIGCOMM’00), Stockholm, Sweden,pp.163–174.

Soltesz, S., Potzl, H., Fiuczynski, M.E., Bavier, A.and Peterson, L. (2007) ‘Container-based operatingsystem virtualization: a scalable, high-performancealternative to hypervisors’, Proceedings of the 2nd ACMSIGOPS/EuroSys European Conference on ComputerSystemsof (EuroSys’07), Lisboa, Portugal, pp.3–16.

VRF (2008) Linux Virtual Routing and Forwarding (VRF),http://sourceforge.net/projects/linux-vrf/

White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad, S.,Newbold, M., Hibler, M., Barb, C. and Joglekar, A.(2002) ‘An integrated experimental environment fordistributed systems and networks’, Proceedings of the5th Symposium on Operating Systems Design andImplementation (OSDI’02), pp.255–270.

XORP (2008) XORP, http://www.xorp.org/

Zebra (2008) GNU Zebra, http://www.zebra.org/

Zhou, J., Ji, Z., Takai, M. and Bagrodia, R. (2004) ‘Maya:integrating hybrid network modeling to the physical world’,ACM Transactions on Modeling and Computer Simulation,Vol. 14, No. 2, pp.149–169.