A real-time network simulation infrastructure based on OpenVPN

The Journal of Systems and Software xxx (2008) xxx–xxx

Contents lists available at ScienceDirect

The Journal of Systems and Software

journal homepage: www.elsevier .com/ locate/ jss

ARTICLE IN PRESS

A real-time network simulation infrastructure based on OpenVPN q

Jason Liu *, Yue Li, Nathanael Van Vorst, Scott Mann, Keith HellmanSchool of Computing and Information Sciences, Florida International University, Miami, 11200 SW 8th Street, ECS 261B, FL 33199, USADepartment of Mathematical and Computer Sciences, Colorado School of Mines, Golden, CO 80401, USA

a r t i c l e i n f o

Article history:Received 6 February 2008Received in revised form 4 August 2008Accepted 4 August 2008Available online xxxx

Keywords:Network simulationNetwork emulationReal-time simulationVirtual private network

0164-1212/$ - see front matter Published by Elsevdoi:10.1016/j.jss.2008.08.015

q This article is an extended version of early workMiniSymposium (Liu et al., 2007). Part of this wowas at the Colorado School of Mines.

* Corresponding author. Tel./fax: +1 305 348 16E-mail addresses: [email protected] (J. Liu), yue

ieee.org (N.V. Vorst), [email protected] (S.(K. Hellman).

URL: http://www.cis.fiu.edu/~liux/ (J. Liu).

Please cite this article in press as: Liu, J.doi:10.1016/j.jss.2008.08.015

a b s t r a c t

We present an open and flexible software infrastructure that embeds physical hosts in a simulated net-work. In real-time network simulation, where real-world implementations of distributed applicationsand network services can run together with the network simulator that operates in real-time, real net-work packets are injected into the simulation system and subject to the simulated network conditionscomputed as a result of both real and virtual traffic traversing the network and competing for networkresources. Our real-time simulation infrastructure has been implemented based on Open Virtual PrivateNetwork (OpenVPN), modified and customized to bridges traffic between the physical hosts and the sim-ulated network. We identify the performance advantages and limitations of our approach via a set ofexperiments. We also present two interesting application scenarios to show the capabilities of thereal-time simulation infrastructure.

Published by Elsevier Inc.

1. Introduction

Over the years, we have seen a major shift in the global networktraffic in response to emerging ‘‘killer apps” of the Internet. Forexample, subject to interpretation, one recent study shows thatpeer-to-peer (P2P) applications currently contribute over 60% ofthe total Internet traffic (CacheLogic, 2005). During the sameperiod, we have also seen a growth of other applications, such asvoice-over-IP (VoIP), Internet teleconferencing, and multimediadata streaming, all having the potential to become the nextdominating traffic over the global network. The landscape of com-puter networks changes rapidly as a result of the interplay of manysocial and technological factors of highly unpredictable nature.This poses a considerable challenge to researchers designing thenext-generation high-performance networking and software infra-structures to satisfy the growing demand of these distributedapplications. The success of advancing critical network technolo-gies, to a large extent, depends on the available tools that can effec-tively prototype, analyze, and evaluate new designs and new ideas.Traditionally, networking research has relied on a variety of tools,ranging from physical testbeds (Barford and Landweber, 2003;

ier Inc.

published at the 2007 INFOCOMrk was done when Dr. Jason Liu

[email protected] (Y. Li), vanvorst@

Mann), [email protected]

et al., A real-time network sim

Peterson et al., 2002), through analytical models (Liu et al.,2004), to simulation and emulation (Breslau et al., 2000; Vahdatet al., 2002; Guruprasad et al., 2005). In addition, advanced parallelnetwork simulation tools, such as SSFNet (Cowie et al., 1999),GTNetS (Riley, 2003), and ROSSNet (Yaun et al., 2003), aim atextending the simulator’s capability to execute extremely largenetwork models and has helped improve our understanding ofthe global-scale network phenomena.

As the scale of the network we are able to simulate increases,however, one cannot underestimate the importance of simulationvalidation, which seems particularly elusive for large-scale net-work models. There have been efforts (e.g., Fall, 1999; Simmondset al., 2000; Song et al., 2000) in augmenting the network simula-tion with emulation capabilities, which we call real-time networksimulation (Liu, 2008). With the emulation extension, a networksimulator operating in real-time can run together with real-worlddistributed applications and network services. These applicationsand services generate real packets, which are injected into the sim-ulation system. Packet delays and losses are calculated as a func-tion of the simulated network condition. To a certain extent,real-time network simulation not only alleviates the burden ofdeveloping separate models for applications in simulation, but alsoincreases the confidence level of network simulation as real sys-tems are included in the network model.

Large-scale real-time network simulation requires simulationbe able to characterize the behavior of a network—potentially withmillions of network entities and with realistic traffic load. In addi-tion, the simulation must be scalable and able to keep up with thewall-clock time. To this end, parallel discrete-event simulationtechniques can be applied to increase the simulation event

ulation infrastructure based on OpenVPN, J. Syst. Software (2008),

mailto:[email protected]






http://www.cis.fiu.edu/~liux/

http://www.sciencedirect.com/science/journal/01641212

http://www.elsevier.com/locate/jss

2 J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx

ARTICLE IN PRESS

processing capabilities and to reduce the possibility of misseddeadlines (Simmonds and Unger, 2003). The multi-scale modelingtechniques have also been investigated that can selectively includecoarse-grain traffic models to effectively reduce the time complex-ity of a large-scale network simulation (Nicol and Yan, 2004; Zhouet al., 2004; Liu, 2006).

Scalability also implies high performance of the emulationinterface for real applications to interact with the simulator. Thisis an area where we have not seen significant progress being made.Consider a scenario, shown in Fig. 1, where collaborative and geo-graphically distributed research teams conduct a remote scientificexperiment over the network. The experiment involves physicalexperimental instruments, supercomputing facilities, and large-scale data storage centers. A high-performance software infrastruc-ture is in place to seamlessly provide coordination between theexperimental facilities and the distributed research teams. Fromthe perspective of network design, our goal is to use a large-scalevirtual network testbed (presumably run at a supercomputing cen-ter) to connect these experimental sites in order to evaluate bothsoftware and networking support for this large-scale distributedscience application. For networking research, the experimentaldata—including, for example, intermediate results that need to betransfered and stored, data streams for remote visualization, andcontrol flows for remote experiment steering—provides a realistictraffic load, invaluable for testing new network protocols andnetwork services. In this scenario, multiple entry points to the net-work simulator must be provided to facilitate real-time interactioninitiated from different geographical locations and with differenttraffic demand.

This article presents a real-time simulation infrastructure thatfacilitates interaction between the real applications and the net-work simulation. This effort is part of the PRIME (Parallel Real-timeImmersive network Modeling Environment) project, designed toprovide a self-sustained large-scale virtual network environmentfor researchers of distributed systems and networks (Liu, 2008).PRIME is derived from an interactive network simulator calledRINSE (Liljenstam et al., 2005) and aims at addressing severalimportant issues, including:

� Simulation scalability and real-time performance guarantee. Weenvision that using parallel simulation and multi-scale modelingtechniques and combining with novel real-time scheduling algo-

Fig. 1. A PRIME

Please cite this article in press as: Liu, J. et al., A real-time network simdoi:10.1016/j.jss.2008.08.015

rithms can significantly improve the timeliness of the interac-tive network simulator.

� Dynamic configuration and continuous network operation. Thevirtual network testbed is expected to accommodate multipleexperiments running either simultaneously or in succession withpotentially different network configurations without disruption.

� Scalable interaction and effortless integration with real applica-tions. We need to provide a real-time simulation infrastructurefor the network simulator run on supercomputers to dynami-cally interface with a large number of real applications.

This article focuses on the last issue. In particular, we propose asimulation infrastructure using a Virtual Private Network (VPN)server farm as the simulation gateway for distributed applicationsto dynamically connect to the real-time network simulator. Realdistributed applications run on computers configured as VPN cli-ents, which automatically forward network packets generated bythe applications and targeted the virtual network to the VPN serv-ers. A user-level daemon process run at each simulation gatewaymanages emulation connections to the simulator and redirectsthe traffic from the VPN servers to the corresponding simulationprocesses via TCP connections. At the simulation site, the networkpackets are translated into simulation events and then injectedinto the simulator. On the reverse path, packets arriving at a virtualnode in the virtual network are translated into real network pack-ets and sent to the simulation gateway via TCP. The user-level dae-mon process receives the packets and then forwards the packets tothe VPN server, which sends the packets to the client machine thatcorresponds to the virtual node in simulation and runs realapplications.

Using VPN as the simulation gateway between real applicationsand the simulator brings several benefits:

(1) The simulation gateway is placed outside the firewall of thesupercomputing center where we run the large-scale net-work simulation. The gateway is necessary because directemulation connections to the simulator are typically blockedby the firewall. The gateway provides a natural rendezvouspoint between the real distributed applications and thesimulator.

(2) On a client machine, each VPN connection is implementedthrough a network tunnel device assigned with a proper IP

application.


J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx 3

ARTICLE IN PRESS

address. We use an automatically generated VPN configura-tion file to customize the IP routing table to allow networktraffic targeting the virtual IP address space to be forwardedvia the VPN connection. Since in most cases the tunneldevice is treated as a regular network interface, this processis therefore transparent to the application we intend toemulate.

(3) The VPN server is capable of handling a large number of VPNconnections. Allowing dynamic behavior is importantbecause we expect the virtual network testbed to run for along period of time. Individual emulation connections maycomparatively have a shorter life span.

(4) Several VPN servers can be used to balance the traffic goingin and coming out of the real-time simulator. A clientmachine emulating a real application can choose to connectto a VPN server according to its geographical location and/orthe quality-of-service demand. One can also adopt a loadbalancing strategy to select a simulation gateway in a VPNserver farm.

(5) VPN provides support for enterprise-scale configurations,such as fine-grain access controls, authentication, andencryption. It also provides fault-tolerance measures, suchas fail-over. Data compression is also used for packet trans-mission through the VPN connection for better performance.We inherit all these features automatically.

(6) VPN packages are in general highly portable. For example,OpenVPN, the implementation we use for the simulationgateway, is open source and freely available on Windows,Mac OS X, and most Unix platforms (Yonan, 2008).

The remainder of this article is organized as follows. In Section2, we describe previous work related to our approach. Section 3presents our real-time simulation infrastructure in detail. Weimplemented a prototype of this system to study the feasibilityof such an approach. To demonstrate the performance of this sim-ulation infrastructure and assess its limit, we conducted prelimin-ary experiments. We report the results in Section 4. We also usedthis real-time simulation infrastructure to interact with existingnetwork routing protocols and a real network device designed forcontent filtering and cyber-security applications. Section 5 pre-sents our case studies. We conclude this article with a discussionof future work in Section 6.

2. Related work

Advanced emulation techniques can now provide full-scaleemulation capabilities for large-scale networks (Vahdat et al.,2002; Zheng and Ni, 2003; White et al., 2002. For example, Model-Net (Vahdat et al., 2002) multiplexes real network applications atthe peripheral of the system called ‘‘edge nodes”. Multiple applica-tions run unmodified on each edge node and, by setting up explicitroutes, send traffic through the emulation core that runs on paral-lel computers. Network operations are emulated using a time-dri-ven approach to calculate packet delays and losses. At its core,ModelNet uses an IP firewall (ipfw) rule to intercept incomingpackets and a kernel module is then used to simulate packet for-warding in the network. The ModelNet approach is appropriatefor a closed emulation system, where edge nodes are configuredstatically as an integral part of the system. A careful setup of thephysical testbed with a tight coupling between the edge nodesand the core can achieve a very high throughput allowing the emu-lation to scale up to tens of thousands of network entities. In com-parison, our real-time simulation infrastructure is open andflexible, and allows dynamic connections to the virtual networktestbed—the association between real applications and the simu-


lated network is ad hoc and potentially transient. More important,aside from conducting traffic generated by the emulated applica-tions, our network simulator can simulate additional traffic inthe background providing a more realistic and yet controllable net-work condition.

Bradford et al. gave a survey of techniques for importing andexporting real network packets, such as using raw sockets andthe pcap library, that allow real-time interactions with the net-work simulator (Bradford et al., 2001). In MaSSF (Liu and Chien,2006), applications can also run unmodified together with the sim-ulator. This is accomplished by intercepting communication-re-lated and time-sensitive systems calls, either through the use oflinker wrapper functions or with the replacement of dynamic link-ing libraries. To enable interaction with the emulation system,these techniques invariably require the applications to receivesome special treatment, either at compile/link time, which is prob-lematic if one does not have access to the source code, or at runtime, which tends to be difficult to implement and maintain. Fur-thermore, these techniques do not address issues, such as scalabil-ity and load balancing, which we regard as important for a large-scale emulation system.

OpenVPN is an open-source VPN solution implemented basedon IP tunneling, i.e., using TUN/TAP devices to encapsulate IP pack-ets inside UDP (Yonan, 2008). Previously we used IP tunneling toconnect a wireless ad hoc network simulator with the real imple-mentation of routing protocols to validate wireless models (Liuet al., 2005). NCTUns is a network simulator that uses IP tunnelingto connect with real-life network protocols on the same physicalhost (Wang et al., 2007). VINI uses OpenVPN to connect with theend users for network experimentation on PlanetLab (Bavieret al., 2006).

Our approach extends the previous effort in the RINSE simulator,which also allows real-time interactions with real applications pre-sumably running on geographically distributed client machines(Liljenstam et al., 2005). RINSE prescribes a blueprint for a large-scale emulation system. In particular, the design of RINSE is the firstwe know to designate a data server to connect the client applica-tions with the real-time simulator. By using a database, the servercan perform authentication and event bookkeeping for client appli-cations accessing the simulated network. The real-time simulationinfrastructure we describe in this article presents a viable approachto large-scale network emulation, which not only supports an effi-cient and flexible pathway to the real-time simulator, but also pro-vides a scalable solution for managing client applications.

3. The real-time simulation infrastructure

The real-time simulation infrastructure consists of three majorcomponents: the I/O threads collocated with the simulator, thesimulation gateways, and the client machines running real applica-tions. An example of the real-time simulation infrastructure isillustrated in Fig. 2.

(1) The I/O threads are collocated with the real-time networksimulator run on parallel computers (most likely in a super-computing center behind a firewall). The I/O threads areused by the real-time simulator to import real networkpackets and export simulated packets. The threads are con-nected via TCP connections to the simulation gateways, con-figured to tunnel network packets to and from theapplications running on client machines.

(2) Each simulation gateway runs a modified OpenVPN serverand a daemon process called ssfgwd, which is responsiblefor shunting IP packets between the virtual network andthe client applications.


Firewall

VPN Client

VPN Client

VPN Client

VPN Client

VPN Client

I/O threads

parallelmachines

virtual network10.0.0.0/16

10.0.0.14

10.0.0.14

10.0.1.2

10.0.1.2

VPN Server

VPN Server

ssfgwd

ssfgwd

TCP connections

simulation gateway

simulation gateway

Fig. 2. The real-time simulation infrastructure.

TCP UDP

IP

MAC

PHY PHY

EmulationSession

MAC

I/O Agent

SimulationProcess

virtual host

writerthread

readerthread

VPN Server

VPN Server

ssfgwd

ssfgwd

simulation gateway

simulation gateway

TCP UDP

IP

MAC

PHY PHY

EmulationSession

MAC

virtual host

Fig. 3. Connecting the simulator and the simulation gateway.


ARTICLE IN PRESS

(3) The client machines each run one or several OpenVPN cli-ents configured to connect to a designated simulation gate-way. There is a one-to-one mapping between a clientmachine and an emulated virtual node inside the simulation.OpenVPN set up the IP addresses for the client machineusing the same IP addresses of the corresponding virtualnode. OpenVPN also set up the IP routing table at the clientmachine automatically for the client applications to forwardtraffic to the virtual network through the VPN connection.

In the remainder of this section, we describe the details of thesethree components, as well as a mechanism to emulate routers withmultiple network interfaces.

3.1. The real-time network simulator and the I/O threads

PRIME extends the SSFNet network simulator for real-timelarge-scale network simulations. SSFNet was developed based ona simulation kernel that implements the Scalable SimulationFramework (SSF) for high-performance parallel and distributeddiscrete-event simulation (SSF Research Network, 2008). The sim-ulator has been augmented to allow real-time interactions withexternal network applications.

A virtual network is partitioned among parallel processors to besimulated in parallel. Each processor participating in the parallelnetwork simulation spawns two threads in addition to the simula-tion process assigned to this processor for event processing. An I/Oagent in the simulation process is responsible for both exportingand importing packets. A writer thread takes the packets exportedfrom the simulation process and forwards them to a simulationgateway, which then delivers the packets to the corresponding cli-ent machine running real applications. A reader thread acceptspackets from the client applications through a simulation gatewayand inserts the packets into the simulator.

The I/O agent and the I/O threads use established SSF functionsdesigned specifically to support emulation, including both export-ing simulation events and importing real network packets (seeLiljenstam et al., 2005 for more details). Each emulated virtualnode in the simulation contains an emulation protocol session thatintercepts packets at the IP layer. Upon receiving a simulated pack-et deemed to be sent out to the client application, the emulation


session forwards the event to the designated I/O agent in the samesimulation process (on the same processor), which exports thepacket to the writer thread by translating the simulated packetto a real IP packet. The write thread sends the IP packet to the sim-ulation gateway via TCP. On the reverse path, when the readerthread receives an IP packet, it translates the packet to a simulationevent and presents it to the I/O agent inside the simulator. The I/Oagent then forwards it to the emulation protocol session on thecorresponding virtual host, which pushes the packet down thesimulated protocol stack. This procedure is illustrated in Fig. 3.

3.2. The simulation gateway

The simulation gateway is a critical component of the real-timesimulation infrastructure, residing between the network simulatorand the client applications run on distributed machines. We set upan OpenVPN server at each simulation gateway managing incom-ing connections from the client machines. There can be multiplesimulation gateways for balancing the traffic load or for differenti-



ARTICLE IN PRESS

ated services (for example, each simulate gateway could offer a dif-ferent quality of service guarantee for its connected clients). Ateach gateway, traffic to and from the client applications are han-dled by OpenVPN. We defer the details of the OpenVPN setup tothe next section.

In addition to running the OpenVPN server that manages theOpenVPN clients, we create another process at the simulation gate-way to handle traffic to and from the real-time network simulator.The ssfgwd daemon is a process responsible for shunting IP pack-ets between the simulator’s I/O threads and the OpenVPN server. Itmaintains a separate TCP connection with each of the I/O threadson the parallel machines running the simulation. Packets to be in-serted into the real-time simulator are sent from ssfgwd via a ded-icated TCP connection to the reader thread associated with thesimulation process on the parallel machines. Packets exportedfrom the real-time simulator are sent from the writer thread viaanother TCP connection to ssfgwd. In either case, it is importantto have the TCP connections initiated from the simulator, sincethe simulator is expected to run at the back-end of a supercom-puter, which is normally situated behind a firewall and/or cannotbe determined a priori. It is quite possible that incoming TCP con-nections to the supercomputer are blocked by the firewall rules.

The type of the TCP connections to a simulation gateway is dis-tinguished by the data subsequently sent from the I/O threads viathe TCP connections. A one-byte identifier is used to label a con-nection from either a reader thread or a writer thread. In addition,the IP addresses of all virtual hosts to be emulated by the corre-sponding simulation process are subsequently sent by the readerthread. The ssfgwd process records this list of IP addresses andcreates a mapping from the IP addresses to the correspondingTCP connection to the reader thread that sends these addresses. La-ter, ssfgwd uses this mapping to forward traffic from the clientmachines to the correct simulation processes.

In order to support emulation of hosts with multiple networkinterfaces (for reasons which we discuss in Section 3.4), we modi-fied the VPN server to spawn the ssfgwd process directly and com-municate with it using the standard Unix pipes. An IP packetreceived by the VPN server from one of its clients is preceded withthe client’s IP address before it is sent to ssfgwd via the pipe. TheIP address is used by ssfgwd to deliver the packet to the corre-sponding simulation process. Recall that a mapping from the IP ad-dress to the simulation process that contains the client’s virtualhost is established immediately after the TCP connection is madeby the reader thread. Once the TCP connection is located, the pack-et is sent via the TCP connection to the designated reader thread atthe simulator. The packet is subsequently inserted into the proto-col stack of the virtual node that carries the same IP address of theclient machine that initiates this packet. In such a way, the packetappears as if it were generated directly by the virtual host. Simi-larly, when an IP packet emanating from a virtual host is sent tothe simulation gateway through the TCP connection establishedby the writer thread, the packet is preceded with an IP addressidentifying the virtual host (more precisely, a network interfaceon the virtual host) that exports the packet. The ssfgwd processsends both the IP address and the packet via the pipe to the Open-VPN server, which forwards the packet to the corresponding Open-VPN client with the IP address. In such a way, the packet arrives atthe client machine as if it receives the packet directly from a phys-ical network.

Our scheme supports the use of multiple simulation gatewaysto alleviate the traffic load placed otherwise on a single gatewayforming a bottleneck. Such load balancing decisions can be madeeither statically or dynamically. All virtual nodes in the virtual net-work can be partitioned and assigned to different simulation gate-ways at the configuration time. The entire emulation traffic istherefore divided among the simulation gateways and a better


throughput is anticipated. Alternatively, the clients may dynami-cally choose among a set of simulation gateways at the time ofits connection. For example, the IP Virtual Server (IPVS) can beused to implement a rather sophisticated load balancing schemeat a front-end server, which subsequently redirect services to acluster of servers at the back-end (IP virtual server, 2008). In othersituations, the clients may choose to connect to a gateway that cansatisfy a particular quality-of-service demand. This allows us todifferentiate client traffic to and from the real-time simulator.The quality-of-service demand could be as simple as finding a sim-ulation gateway that is geographically close to the client, or morecomplicated as to find a simulation gateway that can sustain trafficwith certain throughput and latency requirements. In our imple-mentation, we adopted the OpenVPN’s simple load-balancingscheme in which clients can choose randomly to connect to a ser-ver from a set of simulation gateways at the time of connection. Inany case, the I/O threads at the simulator must make separate TCPconnections to all simulation gateways. We implemented mecha-nisms to allow a client each time to connect to a different gateway;traffic from the simulator will be routed to the client machinethrough the gateway with which the client is currently engaged.

3.3. The VPN connections

OpenVPN was chosen to allow applications to dynamically con-nect to the simulation gateway and to emulate network interfaceson the client machines. We chose OpenVPN in our implementation,since it is publicly available and runs on most operating systems.

OpenVPN uses the TUN/TAP interface provided by the system.As shown in Fig. 4, each OpenVPN instance creates a virtual net-work device (named tun0 in this case). The virtual network devicehas two end-points: one as a regular network interface with an as-signed IP address and the other with a file interface through whichone can apply read and write operations. We assign the virtual net-work interface with an IP address that matches the IP address ofthe emulated network interface in simulation and add an entryto the kernel forwarding table to direct all traffic to the virtual IPaddress space to be sent through the virtual network device. In thisway, the client machine will assume the proper identity as applica-tions running on this machine can transparently forward traffic tothe simulated network. IP packets sent to the virtual networkinterface are read by OpenVPN through the file interface. Subse-quently, OpenVPN applies proper data compression and encryptionto the packets before sending them via UDP to the OpenVPN serverrunning at the simulation gateway. In the opposite direction, UDPpackets received from the remote OpenVPN server are decryptedand decompressed before the packets are written out the virtualnetwork interface. Thus the packets arrive at the application as ifthey come directly from the virtual network interface.

To set up the real-time simulation infrastructure, one runs amodified OpenVPN server at the simulation gateway using anOpenVPN server configuration file automatically generated fromthe virtual network specification. Each OpenVPN client connectsto a chosen simulation gateway using a separate client configura-tion file, also generated automatically beforehand. The procedurefor creating these configuration files is illustrated in Fig. 5, wherethe original components for setting up parallel network simulationexecution are shown on the left and the shaded components to theright are added to support the real-time simulation infrastructure.Same with SSFNet, one specifies the virtual network in PRIMEusing a simple configuration script written in the Domain Model-ing Language (DML). A DML network description includes thetopology of the network (i.e., sub-networks, hosts, routers, andlinks connecting network interfaces in the hosts and routers), thenetwork protocols running at each host or router, as well as thetraffic traversing the network (see SSF Research Network, 2008


tun0 eth0

OpenVPNClient

Application(e.g., iperf)

userkernel

10.0.0.14 131.94.130.11

tun0 eth0

OpenVPNClient

Application(e.g., iperf)

userkernel

10.0.1.2 131.94.130.15

eth0

OpenVPNServer

ssfgwd

PRIMESimulator

131.94.130.19

Simulation Gateway

Client Machine

Client Machine

Fig. 4. OpenVPN uses virtual network interfaces for tunneling application traffic.

DML Network Description

DML Auxiliary Information

DML Partitioning Information

SSFNet

VPN Server ConÞguration

VPN Client ConÞguration

VPNServer

VPNClient

dmlenv

dmlpart

vpnscript

Fig. 5. OpenVPN configurations generated automatically from DML network specification.


ARTICLE IN PRESS

for details). The DML network description is then processed by autility program called dmlenv, which automatically assigns IP ad-dresses to all network interfaces defined in the virtual network.dmlenv generates another DML file which contains the result ofthe IP address assignment, as well as other auxiliary informationrelated to setting up the network simulation, including the map-ping of communication channels between simulation entities andthe alignment of hosts and routers to form simulation processes.The auxiliary information will also be given to dmlpart, anotherutility program that partitions the virtual network and generatesyet another DML file describing the assignment of the simulationprocesses to parallel processors. All these DML files will be neededto start the SSFNet network simulator.

We provide another utility program called vpnscript to auto-matically generate the configuration files needed for setting up theOpenVPN servers and clients. We use a server configuration file toset up each simulation gateway and a client configuration file foreach emulated network interface. Routers and hosts in the virtualnetwork with multiple network interfaces require multiple Open-VPN connections, which we discuss in the next section. The vpn-

Please cite this article in press as: Liu, J. et al., A real-time network sidoi:10.1016/j.jss.2008.08.015

script first creates a certificate authority used by the OpenVPNservers to validate the encrypted incoming client connections.The script then generates a pair of public and private keys for eachOpenVPN server or client. Finally, for convenience, vpnscriptputs all files needed to start an OpenVPN server or client in a com-pressed archive file to be distributed separately. For each simula-tion gateway, the archive includes the private key of theOpenVPN server and the public keys of all OpenVPN clients man-aged by this simulation gateway (each corresponding to an emu-lated network interface). Also included in the archive is amapping from public keys to IP addresses. The mapping will beused by the OpenVPN server to assign IP addresses to incoming cli-ent connections, which are distinguished by the clients’ publickeys. For each OpenVPN client, the archive includes the privatekey of the client and the public keys of all designated simulationgateways. Another script is included in these archives to automatethe start of the OpenVPN servers and clients.

We also added a plug-in module to the OpenVPN server to up-date the network simulator of the connection status of the clientmachines. When a client establishes a VPN connection, the plug-

mulation infrastructure based on OpenVPN, J. Syst. Software (2008),

Fig. 6. Emulation of a router with multiple network interfaces.

1 Note that the changes only apply to the OpenVPN server.


ARTICLE IN PRESS

in code will be run to send a notification packet to the emulationprotocol session on the corresponding virtual node. Upon receivingthis notification, the virtual node is then allowed to export packetsto the client machine. Similarly, when a client machine disconnectsfrom the server, a notification is sent from the plug-in to the sim-ulator so that simulated packets to the virtual host will be droppedat the simulator side (rather than being carried all the way to thesimulation gateway and then getting dropped).

3.4. IP forwarding

Each VPN connection at the client machine corresponds to anemulated network interface inside the simulation. For hosts withmultiple network interfaces, multiple VPN connections are needed.It would not be necessary to modify the VPN server if these virtualnodes were simply end hosts. In this case, packets arrived at thesimulation gateway from a client machine could be sent to thesimulation process and the corresponding virtual host accordingtheir source IP addresses. Similarly, packets arrived at the simula-tion gateway from the simulator could be distinguished by theirdestination IP addresses and sent to the client machines via thecorresponding VPN connections.

This scheme, however, does not work if one wants to emulate arouter (e.g., to test a real routing protocol implementation). Con-sider an example shown in Fig. 6, where we emulate the router Bin a virtual network of three nodes. Since router B has two networkinterfaces with IP addresses 10.10.0.5 and 10.11.0.1, we run twoseparate OpenVPN clients on the client machine. Suppose thatnode A pings node C. An IP packet with a source address of10.10.0.6, a destination address of 10.11.0.2, and a payload of anICMP ECHO-REQUEST message reaches the network interface10.10.0.5 and is then exported to the simulation gateway as ex-pected. The ssfgwd process cannot determine how to forwardthe packet to the client machine by inspecting the packet alone.

To overcome this problem, each packet sent between the net-work simulator and the simulation gateway is preceded with anIP address indicating the network interface with which the packetis associated. For example, when exporting the ICMP packet ar-


rived at the network interface 10.10.0.5, the writer thread sendsthe address 10.10.0.5 ahead of the packet to the simulation gate-way. The OpenVPN server is modified to use this leading IP addressto differentiate traffic to the client machines,1 in this case, forward-ing the ICMP packet to the client machine labeled as 10.10.0.5through the corresponding VPN connection. The client machinetherefore receives the ICMP packet from the logical network inter-face 10.10.0.5, exactly as it happens in the simulated network. Sup-pose the forwarding table of the client machine is set up so that thepacket will be sent out immediately through the logical networkinterface 10.11.0.1. The OpenVPN server, upon receiving this pack-et, will attach an IP address 10.11.0.1 to the packet before sendingit onward to the ssfgwd process and subsequently to the simula-tor. The reader thread at the simulator uses this leading IP addressto dispatch the packet to the corresponding virtual host. The ICMPpacket is then sent down the protocol stack of router B, out fromvirtual network interface 10.11.0.1, continuing its journey on thevirtual network.

4. Experiments

4.1. Throughput and Latency

We conducted several experiments to evaluate the throughputand latency of our real-time simulation infrastructure. As men-tioned in Section 3, the simulation infrastructure consists of threecomponents on the server side: VPN server, ssfgwd, and I/Othreads. Each packet imported into (or exported from) the real-time simulator goes through these three components in sequence.To evaluate the throughput and the latency induced by each com-ponent, we added the components one at a time in the experi-ments and analyzed the associated overhead.

We first measured the overhead of OpenVPN by running the ori-ginal OpenVPN server with the IP forwarding option enabled. Thus,client machines connected to the same OpenVPN server can simply


Table 1Latency in milliseconds when various components of the real-time simulationinfrastructure are included

Minimum Average Maximum Standarddeviation

Through OpenVPN server 0.201 0.269 0.399 0.021Through OpenVPN server

and ssfgwd

0.209 0.294 0.449 0.023

Through all components 0.366 0.442 0.576 0.029


ARTICLE IN PRESS

communicate with each other via the UDP tunnels set up by Open-VPN; the server functions simply as a traffic stop between the cli-ents. To expose the overhead induced by ssfgwd, we added aLOOPBACK option in its implementation. Packets generated byone client and forwarded to the OpenVPN server and then tossfgwd are directly sent back to the OpenVPN server and finallydelivered to the other client. In a similar fashion, we added aLOOPBACK option to the real-time simulator. Packets sent fromone client via the OpenVPN server, ssfgwd, the reader thread,and finally reached the real-time simulator are immediately ex-ported to the writer thread, meandering through ssfgwd and theOpenVPN server to the other client.

We set up the experiment on a Xen virtual machine environ-ment Barham et al., 2003. The host machine is a DELL OPTIPLEX745 workstation with Intel Core 2 Duo 2.4 GHz processors and3 GB memory. Xen is a high-performance virtual machine monitor;we use Xen to minimize the overhead caused by transmitting net-work packets between physical hosts, which would depend on thephysical interconnect. In Xen terminology, each domain is a virtualmachine running on the host machine. Domain 0 is the first do-main created when the host is powered on and has privileges to ac-cess physical machine resources. It also provides an interface tomanage other user domains (called Domain U’s).

Both the host OS and guest OS ran Debian Linux 4.0 distribu-tions with kernel version 2.6.16.33. We put both the simulationgateway and the network simulator in domain 0 and created twouser domains each containing a client, where we used iperf Iperf– the TCP/UDP bandwidth measurement tool, 2008 to generateUDP traffic at a certain rate and ran ping to measure the round-trip delays. We varied the traffic load from 50 Mb/s to 170 Mb/s.Fig. 7 shows the throughput achieved between the clients.

If the traffic only goes through the OpenVPN server, thethroughput can reach as much as 145 Mb/s. A major part of theoverhead is from data encryption and compression, in addition tothe overhead incurred by IP tunneling over UDP. If we allow thetraffic to go through ssfgwd, the maximum throughput dropsdown to 125 Mb/s, mainly due to the overhead from contextswitching and memory copying. Once we include the real-timesimulator, which means that all three components of the simula-tion infrastructure are fully engaged in packet forwarding, thethroughput drops further down to 90 Mb/s. Although this speedis sufficient for a common 100 Mb/s Ethernet network setup, it isobvious that OpenVPN is not a proper choice to emulate faster net-work connections. OpenVPN is better suited for managing remoteemulation connections that have lower throughput demands.

50

60

70

80

90

100

110

120

130

140

150

60 80 100 120 140 160

Thro

ughp

ut (M

b/s)

Offered Load (Mb/s)

OpenVPN ServerOpenVPN Server and ssfgwdAll Components

Fig. 7. Throughput achieved when various components of the real-time simulationinfrastructure are included.


We used ping to measure the latency of the real-time simula-tion infrastructure; the results are shown in Table 1. The tableshows the round-trip times between the two clients as traffic goesthrough the OpenVPN server, the OpenVPN server and ssfgwd,and then all three components, respectively. As expected the la-tency increases when more components are involved in the path.Overall, it sums to a total of 0.442 ms on average if the packetsare forwarded through the entire simulation infrastructure.

4.2. Accuracy evaluation

We use another set of experiments to assess the impact on theemulation accuracy as the real-time simulation infrastructureinevitably introduces overheads when network traffic is sent be-tween the applications running on the client machines and thereal-time simulator. The overhead manifested as packet delaysand losses is not part of the simulated network and could becomea major contributor to emulation errors.

In our particular experiment setup, the machines running theOpenVPN clients and the simulation gateway were both AMD Ath-lon64 machines with a 2.2 GHz processor and 2 GB memory. Themachines were connected via a gigabit switch. We chose two loca-tions to run the network simulator. One is a 23-node Linux clusterlocated on campus (each node equipped with a Pentium IV 3.2 GHzCPU and 512 KB memory). The campus network is a 100 Mb/sEthernet. The other location is the IBM terascale supercomputernamed DataStar (consisting of 272 8-way p655+ and 7 32-wayp690 compute nodes) at the San Diego Supercomputer Center atUCSD (2008).

The simulated network is shaped like a dumbbell as shown inFig. 8, which has been commonly used to evaluate TCP congestioncontrol algorithms. There are N nodes aligned on either side of thedumbbell connected by two routers. A server node on one side ofthe dumbbell is sending traffic via TCP to the corresponding clientnode on the other side of the dumbbell. The delay of the link con-necting a router with one of its adjacent client or server nodes isset to be 100 ms and its bandwidth is 100 Kb/s. The link connectingthe two routers in the middle of the network has a delay of 300 msand a bandwidth of (100 � N) Kb/s. In our experiments, we emu-lated the router R1, which is located on the server side of the dumb-bell network. On the client machine, we simultaneously ran (N + 1)OpenVPN clients, each corresponding to a network interface of rou-ter R1. We also enabled IP forwarding on the client machine and setthe kernel routing table accordingly so that the Linux systemwould forward packets to and from the virtual tunnel devices (cre-ated by OpenVPN). We use this dumbbell network specifically toexpose the overhead of the real-time simulation infrastructure.By changing the number of TCP client/server pairs (N) we can varythe amount of network traffic traversing the real-time simulationinfrastructure.

Before the TCP transfers began, we first ran ping separatelyfrom the machine running the simulator and the machine runningthe VPN clients to the simulation gateway to measure the round-trip time of the segments between the simulator and the simulationgateway (represented as x in Table 2) and between the simulation


Fig. 8. Emulation of a dumbbell-shaped network.

Table 2Round-trip times measured in milliseconds

Local Linux cluster SDSC supercomputer

x 0.210 ± 0.064 46.674 ± 3.383y 0.042 ± 0.007 0.045 ± 0.007z 200.794 ± 0.889 247.197 ± 2.827� 0.39% 23.59%d 0.529 0.464

0 %

20 %

40 %

60 %

80 %

100 %D

ata

Tran

sfer

red

0 5 10 15 20 25 30 35 40Time (in seconds)

Sim, N=1,2,4,8,16,32,64N=128 (10 runs)

Fig. 9. Emulation of TCP data transfers on local Linux cluster.


ARTICLE IN PRESS

gateway and the client machine (represented as y). In the meantime, the client machine, on behalf of router R1, started pingingone of the simulated server nodes (say, S1, with the result being rep-resented as z). Let z0 be the simulated round-trip time between rou-ter R1 and S1 when R1 is not emulated (200.013 ms in this case). Wecalculate the emulation error to be � = |z � z0|/z0. Also, the differencein the round-trip times d = z � z0 � x � y reflects the delay overheadin addition to the imposed network delays of the two segments be-tween the client machine and the gateway and between the gate-way and the simulator. This delay overhead includes, for example,the time it takes to import and export simulation events and totransport packets between ssfgwd and the OpenVPN server insidethe simulation gateway. The baseline measurement reported inTable 2 was collected during a relative network quiescence. The re-sults shown are averages of 10 separate runs. In both case, the delayoverhead accounts for only about 1/2 ms on average, which coin-cides with the previous experiment.

Next we set up all server nodes in the dumbbell network to eachsend a file of 300 KB simultaneously over TCP to their client coun-terparts.2 We varied the number of client/server pairs, thus theamount of network traffic traversing the real-time simulationinfrastructure, during the experiment. Figs. 9 and 10 show theaggregate data transferred over all TCP connections as we ran thenetwork simulator on the local Linux cluster and at the SDSCsupercomputing center, respectively. Scaled up to 64 TCP connec-tions, the emulation on the local cluster produced results almostindistinguishable from the expected behavior. The telltale staircaseshape of the curve on the left of the figure is typical of the TCP data

2 A small jitter was added to avoid the artificial synchrony among the multiple TCPtransfers.


transfer. The maximum height of staircase shows that there wereat most 64 segments in flight at the peak transfer, which can betranslated to an aggregate throughput of 32 Mb/s for all 64 TCPconnections. As we increased N to 128, the performance diverged.We show the amount of data transferred during the first 40 sec-onds or so for each of the 10 runs on the right side of Fig. 9. Asthe traffic demand went beyond the capacity of the real-time sim-ulation infrastructure (and the 100 Mb/s campus network), signif-icant and unpredictable packet losses occurred causing the TCPcongestion control to reduce the send rates accordingly—a phe-nomenon that would not have occurred in the simulated network.The emulation result in these cases deviates from the expectednetwork behavior.

Fig. 10 shows the result of the network simulator running onthe SDSC supercomputer. Even with a single TCP connection, theoverall throughput decreased markedly to 88% of the expected va-lue, which is mainly due to the significant round-trip time betweenthe simulator and the client machine (with an average of about47 ms). More TCP connections further exacerbate this problem asadditional traffic further lengthens the delays experienced by thepackets crossing the real-time simulation infrastructure. Different


0%

20%

40%

60%

80%

100 %

0 5 10 15 20 25

Dat

a Tr

ansf

erre

d

Time (in seconds)

SimN=1N=2N=4N=8N=16N=32

Fig. 10. Emulation of TCP data transfers on SDSC supercomputer.

to the other campus

0:0 1:0

0:2

0:1net 0 net 1


ARTICLE IN PRESS

from the previous scenario, the latency—rather than packet loss—plays a major role in admitting the emulation errors. As shownin Fig. 11, the throughput reduced to 57% for 32 TCP connections,placing the outcome of the emulation poignantly different fromthe expected behavior.

We should emphasize here that the exact results from theseexperiments are unimportant since they merely reflect a particularsetup of our emulation system. The experiments, however, showthat the losses and delays experienced by the packets as they travelthrough the real-time simulation infrastructure can have a signifi-cant impact on the emulation accuracy. On the one hand, to avoidthe damaging effect from network latency depicted in Fig. 10, thesimulator should be placed close to both the simulation gatewayand the client machines. This is especially important if we are tosupport applications that are sensitive to network latencies (suchas TCP). On the other hand, with a tight coupling provided betweenthe simulator, the gateway, and the client machines, the latencyoverhead is insignificant. More important, our experiments con-firm that the throughput of the real-time simulation infrastructurecan scale up close to the network’s 100 Mb/s physical capacity.

We also need to emphasize that, although it is for certain thatsufficient bandwidth must be provided to allow the system to sus-tain further emulation traffic in demanding scenarios, in caseswhere no reference to the wall-clock time is required (such asthe first case study in the following section), we can allow the

50%

55%

60%

65%

70%

75%

80%

85%

90%

1 2 4 8 16 32Accu

racy

of T

hrou

ghpu

t (95

% C

onfid

ence

)

Number of TCP Connections

Fig. 11. Accuracy of TCP throughput using SDSC supercomputer.


real-time simulator to proportionally slow down with respect toreal-time. That is, we fix the simulation time advancement to onlya constant fraction of the real time; we make the simulation timeto advance by f 6 1 s for each wall-clock second. In relative terms,this essentially increases the packet processing capability of thesimulation gateways and the client machines, as well as the capac-ity of the physical network connections—by a factor of 1/f. The abil-ity to throttle simulation clock plays an important role in one ofour case studies, described in the next section.

5. Case studies

In this section we present two case studies involving real net-work applications using our real-time simulation infrastructure.

5.1. Content routing and distribution

For the first case study we use the real-time simulation infra-structure to test a network device, named Content Aware PolicyEngine (CAPE), developed at Northrop Grumman. CAPE allowsone to examine, search, log, and modify the content of the packetsthat pass through the device, which can be configured using a sim-ple language. CAPE has both hardware and software implementa-tions. For this case study we used a software version that runson a Linux box. The software, written in python, uses the Berkleypacket filter to snatch packets from the logical interfaces and thenapplies rules to the packets, possibly modifying them as specifiedin the configuration language before forwarding the resultingpackets onward. CAPE uses IP chains to drop the original packetsto prevent the host OS from processing them unintentionally.

In this study we use CAPE for content routing and distribution.We set up a simulation network that consists of two sub-networkswith a total of 60 routers and 1016 end hosts. This network is ascaled-down version of the baseline network model from the DAR-PA NMS program that has been used for large-scale simulationstudies. Each sub-network (shown in Fig. 12) has a server cluster(in net 1) that acts as the traffic source. Also, in net 2 and net 3,there are a total of 12 local area network clouds, each with 42 cli-ent hosts acting as the traffic sink. To populate the network with

server

LAN with 42 hosts

CAPE

2:0

4 5

3:0

2:52:4

2:32:2

2:1

2:6

3:1

3:2 3:3

1:1

net 2

net 3

Fig. 12. The DARPA NMS campus network instrumented with CAPE.


2.5%

3%

3.5%

4%

4.5%

5%

160 80 40 20 105

Avg

Pkt L

oss

Rat

e (9

5% C

onfid

ence

)

Number of Clients (N)

simemu/forward

emu/replicate

Fig. 13. Using CAPE content distribution to reduce packet loss.


ARTICLE IN PRESS

sufficient background traffic, at each LAN, we designated 320 fluidTCP flows, 2 packet TCP flows, and 2 packet UDP flows, each down-loading data from a randomly selected server from both campuses(see Liu, 2006 for more detail on the fluid TCP model). The applica-tion we focus on is streaming video, as a sequence of 1 KB UDP dat-agrams sent at a constant rate. We randomly chose N clients in net2 at one campus network to each request a 100 KB/s video streamfrom a server randomly chosen from the other campus network.We placed the CAPE device between the routers labeled 0:0 and4 (in Fig. 12) so that it could intercept the streaming traffic. Wemeasured the overall average packet loss rate of all clients as anindication of the receiving video quality.

We ran three tests. As the baseline, we took out the CAPE deviceand simply ran the simulation. In the second test, we emulated theCAPE device but programmed the device to forward all the trafficwithout inspecting the content. In the third test we configuredthe CAPE device to allow only one outstanding request from theclients to reach the server and the rest were cached inside the CAPEdevice. When the server started the video stream back to the de-vice, CAPE replicated the video stream to all other clients. In thisway, the network path between the CAPE device and the serverwas only burdened with one video stream and better network per-formance was expected.

Seattle

Sunnyvale

Los Angeles

DenverKan

Fig. 14. Abilene backbone


We ran the experiments on the same AMD machines with giga-bit connections shown in the previous section. We varied thenumber of clients N from 5 to 160, resulting an aggregate trafficof 4–128 Mb/s. We found the python-based software-based CAPEdevice was unable to process packets at a rate higher than 4 Kpackets per second without significantly dropping them. Therefore,for experiments with more than 40 clients, the speed of the real-time simulator was throttled down to accommodate the slow pro-cessing of the CAPE device. During the experiment, we also foundthe device was unable to handle more than 160 client connectionsdue to a bug in the software-based CAPE device. Fig. 13 shows that,as expected, the packet loss rate (averaged over 10 runs) increaseswith the increase of clients when content distribution is disabled.The results from simulation and emulation are close. When repli-cation is activated in CAPE, however, the loss rate remains stable,implying that the packet losses occur primarily at the network seg-ment between the CAPE device and the streaming video server.

5.2. Intra-domain routing

In our second case study, we use the real-time simulation infra-structure to investigate network routing. Specifically, we testXORP, which is an open-source software router platform for re-search and development (Handley et al., 2003). XORP supportsimplementations of common Internet routing protocols, includingBGP4+, OSPFv2, OSPFv3, RIPv2, RIPng, and several multicast rout-ing protocols. We modified XORP to support split routing and for-warding: routing protocols are run by XORP on the client machinesand the resulting routes are used to construct forwarding tablesused by the corresponding routers in simulation to make packetforwarding decisions (Li et al., 2008).

Here we study intro-domain routing using OSPFv2. We used atest scenario representing the Abilene backbone network, whichconsists of eleven routers at major cities in the Unite State (seeFig. 14). The test case is modified from the one developed originallyby Bavier et al. for evaluating VINI (Bavier et al., 2006), a virtualnetwork testbed for supporting experimentation on the PlanetLab.

We used fluid models to represent traffic on the backbone net-work (Liu, 2006). The number of TCP flows between the routerswas set to be proportional to the link utilization shown on the Indi-ana University Abilene Network Operations Center Weathermap(Abilene backbone network, 2008). Since the fluid traffic is calcu-lated inside the real-time network simulator, the fluid background

Houston

sas City

Atlanta

IndianapolisWashington, DC

New YorkChicago

routers running OSPFv2.



ARTICLE IN PRESS

traffic is not exposed to XORP instances, although the traffic flowsare forwarded as the result of routing. The purpose of the experi-ment is to observe the convergence of OSPF and its effect on datatraffic. We injected a link failure, followed by a recovery betweenthe routers at Chicago and Indianapolis (via the XORP commandshell). We measured the effect on the round-trip time and datathroughput between Denver and Chicago. Fig. 15 shows theround-trip time measured by ping. The link went down at 20and recovered at 50 s. During the convergence period OSPF in-stances recomputed the routes resulting the round-trip delayschanged briefly to around 60 ms. The reason is that the routersat the two cities had inconsistent routing states during the OSPFconvergence period; the requesting and responding ICMP packetstraversed different paths. After the routes converged, OSPF re-established the shortest path between the two cities.

We also used iperf to initiate a TCP transfer from Denver toChicago. We analyzed the tcpdump results obtained at the net-work interface at Chicago connecting to Indianapolis. Fig. 16 showsthe number of bytes received. We do not see any TCP activity whenlink failure occurred between 20 and 50 s, since traffic to Chicagowas re-routed through New York, where we did not run tcpdump.After link was repaired, the traffic resumed via the original path,yet with less throughput between 50 and 63 s. This is due to a lar-ger RTT during the OSPF convergence period (as shown in Fig. 15).

30

40

50

60

70

80

90

0 10 20 30 40 50 60 70 80

Ping

Rou

nd-T

rip T

ime

(ms)

Time (seconds)

Fig. 15. OSPF route convergence.

Fig. 16. TCP transfer during link failure and recovery.


At 63 s, the routes converged, the shortest path was re-established,and a better throughput was achieved (which is demonstrated by asteeper slope in the figure).

6. Conclusions and future work

An important aspect of our real-time simulation infrastructureis the use of an existing VPN framework, thus allowing us to inheritits important properties to derive a flexible, portable, scalable, andsecure implementation of a simulation gateway between the net-work applications and the real-time simulator. One could imple-ment the simulation gateway directly using the TUN/TAP virtualdevice interface. OpenVPN provides a solid and portable imple-mentation, where clients can use standard software to connect tothe real-time simulation system. In addition, OpenVPN providesdata encryption, which is particularly desirable for a long-termemulation deployment. Other packet capturing schemes exist,which include, for example, IP filters and kernel modifications.These alternative solutions would arrive at the same functionalityas OpenVPN, yet with a better performance (both in terms of smal-ler latency and larger throughput). These schemes, however, wouldimpact portability and flexibility of the simulation infrastructure.

Our study shows that the real-time simulation gateway usingOpenVPN is a viable approach, provided that the connectivity be-tween the client machines running real applications, the simula-tion gateway, and the real-time simulator has sufficiently highbandwidth and low latency to avoid significant emulation errors.The significance of such errors depends on the application. Forexample, for delay-sensitive applications, it is necessary to placethe client machines, the gateway, and the simulator within thesame high-performance low-latency network. We are currentlyinvestigating machine virtualization technologies for running thenetwork simulation and applications on the same host in orderto support more efficient connectivity between the simulator andthe applications. Such solutions would require the network appli-cations to move closer to the real-time simulator. The applicationsand the simulator can run on guest OSes (i.e., virtual machines, orVMs). Inter-VM communication can facilitate faster connection be-tween the applications and the simulator. Such technique willcomplement the OpenVPN solution, which works in more generalsituations beyond virtual machines, especially when the simulatorand the applications are running at different geographical loca-tions, in which case the OpenVPN solution would be most effectivefor latency-tolerant applications.

Our implementation allows client machines to choose ran-domly among multiple simulation gateways (i.e., a VPN serverfarm) to connect to the simulator. Such choices can be altered dur-ing the simulation. For example, in our implementation, a clientmachine can disconnect at run time and then reconnect to the sim-ulator via another simulation gateway. There are other criteria be-sides load balancing that can be used by client machines to selectamong multiple simulation gateways. For example, client connec-tions can be differentiated by the bandwidth and latency require-ments: a client running applications with a more stringentquality-of-service requirement should be assigned to a simulationgateway located closer to the real-time simulator and thereforewith higher bandwidth and lower latency.

Real-time simulation can interact directly with real implemen-tations of network protocols, network services, and distributedapplications. This approach bypasses the cumbersome and error-prone model development phase. Comparing with other major net-working research tools, real-time network simulation combinesthe advantages of both network simulation—with better scalabilityat modeling large-scale network behaviors and better flexibility atanswering what-if questions—and network emulation, which



ARTICLE IN PRESS

provides better accuracy by including real traffic and real imple-mentations of network applications.

Acknowledgments

This research is supported in part by the National Science Foun-dation Grants CNS-0546712, CNS-0836408, and HRD-0317692. Theauthors would like to thank the San Diego Supercomputing Center(SDSC) for allowing access to the computing resources and Nor-throp Grumman Corporation for the CAPE device prototype. Wealso want to thank Eric Johnson and Raju Rangaswami at FIU fortheir constructive suggestions and help with the experiments.

References

Abilene backbone network, 2008. <http://abilene.internet2.edu/>.Barford, P., Landweber, L., 2003. Bench-style network research in an Internet

instance laboratory. ACM SIGCOMM Computer Communication Review 33 (3),21–26.

Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I.,Warfield, A., 2003. Xen and the art of virtualization. In: Proceedings of the 19thACM Symposium on Operating Systems Principles (SOSP’03), pp. 164–177.

A. Bavier, N. Feamster, M. Huang, L. Peterson, J. Rexford, In VINI veritas: realistic andcontrolled network experimentation, in: SIGCOMM, 2006, pp. 3–14.

Bradford, R., Simmonds, R., Unger, B., 2001. Packet reading for network emulation.In: Proceedings of the Nineth International Symposium in Modeling, Analysisand Simulation of Computer and Telecommunication Systems (MASCOTS’01),pp. 150–157.

Breslau, L., Estrin, D., Fall, K., Floyd, S., Heidemann, J., Helmy, A., Huang, P., McCanne,S., Varadhan, K., Xu, Y., Yu, H., 2000. Advances in network simulation. IEEEComputer 33 (5), 59–67.

CacheLogic Inc., 2005. Peer-to-peer in 2005, <http://www.cachelogic.com/research/index.php>.

Cowie, J.H., Nicol, D.M., Ogielski, A.T., 1999. Modeling the global Internet.Computing in Science and Engineering 1 (1), 42–50.

Fall, K., 1999. Network emulation in the Vint/NS simulator. In: Proceedings of the4th IEEE Symposium on Computers and Communications (ISCC’99), pp. 244–250.

Guruprasad, S., Ricci, R., Lepreau, J., 2005. Integrated network experimentationusing simulation and emulation. In: Proceedings of the First InternationalConference on Testbeds and Research Infrastructures for the DEvelopment ofNeTworks and COMmunities (TRIDENTCOM’05), pp. 204–212.

Handley, M., Hodson, O., Kohler, E., 2003. Xorp: An open platform for networkresearch. Computer Communication Review 33 (1), 53–57.

Iperf – the TCP/UDP bandwidth measurement tool, 2008. <http://dast.nlanr.net/projects/Iperf/>.

IP virtual server (IPVS), 2008. <http://kb.linuxvirtualserver.org/wiki/IPVS>.Li, Y., Liu, J., Rangaswami, R., 2008. Real-time network simulation support for

scalable routing experiments. In: Proceedings of the 22nd Workshop onPrinciples of Advanced and Distributed Simulation (PADS), pp. 23–30.

Liljenstam, M., Liu, J., Nicol, D.M., Yuan, Y., Yan, G., Grier, C., 2005. RINSE: the real-time interactive network simulation environment for network securityexercises. In: Proceedings of the 19th Workshop on Parallel and DistributedSimulation (PADS’05), pp. 119–128.

Liu, J., 2006. Packet-level integration of fluid TCP models in real-time networksimulation. In: Proceedings of the 2006 Winter Simulation Conference(WSC’06), pp. 2162–2169.

Liu, J., 2008. A primer for real-time simulation of large-scale networks. In:Proceedings of the 41st Annual Simulation Symposium (ANSS’08), pp. 307–323.

Liu, X., Chien, A., 2006. Realistic large-scale online network simulation.International Journal of High Performance Computing Applications 20 (3),383–399.

Liu, Y., Presti, F.L., Misra, V., Towsley, D.F., Gu, Y., 2004. Scalable fluid models andsimulations for large-scale IP networks. ACM Transactions on Modeling andComputer Simulation (TOMACS) 14 (3), 305–324.

Liu, J., Yuan, Y., Nicol, D.M., Gray, R.S., Newport, C.C., Kotz, D., Perrone, L.F., 2005.Empirical validation of wireless models in simulations of ad hoc routingprotocols. SIMULATION: Transactions of The Society for Modeling andSimulation International 81 (4), 307–323.


Liu, J., Mann, S., Vorst, N.V., Hellman, K., 2007. An open and scalable emulationinfrastructure for large-scale real-time network simulations. In: Proceedings ofthe 2007 IEEE INFOCOM MiniSymposium, pp. 2471–2475.

Liu, J. 2008. The PRIME research, <http://www.cis.fiu.edu/prime/>.Nicol, D.M., Yan, G., 2004. Discrete event fluid modeling of background TCP traffic.

ACM Transactions on Modeling and Computer Simulation (TOMACS) 14 (3),211–250.

Peterson, L., Anderson, T., Culler, D., Roscoe, T., 2002. A blueprint for introducingdisruptive technology into the Internet. In: Proceedings of the First Workshopon Hot Topics in Networking (HotNets-I).

Riley, G.F., 2003. The Georgia Tech network simulator. In: Proceedings of the ACMSIGCOMM Workshop on Models, Methods and Tools for Reproducible NetworkResearch (MoMeTools’03), pp. 5–12.

San Diego Supercomputer Center at UCSD, 2008. SDSC DataStar user guide, <http://www.sdsc.edu/user_services/datastar/>.

Simmonds, R., Unger, B.W., 2003. Towards scalable network emulation. ComputerCommunications 26 (3), 264–277.

Simmonds, R., Bradford, R., Unger, B., 2000. Applying parallel discrete eventsimulation to network emulation. In: Proceedings of the 14th Workshop onParallel and Distributed Simulation (PADS’00), pp. 15–22.

Song, H., Liu, X., Jakobsen, D., Bhagwan, R., Zhang, X., Taura, K., Chien, A., 2000. TheMicroGrid: a scientific tool for modeling computational grids. ScientificProgramming 8 (3), 127–141.

SSF Research Network, 2008. Modeling the global Internet, <http://www.ssfnet.org/>.

Vahdat, A., Yocum, K., Walsh, K., Mahadevan, P., Kostic, D., Chase, J., Becker, D., 2002.Scalability and accuracy in a large scale network emulator. In: Proceedings ofthe Fifth Symposium on Operating Systems Design and Implementation(OSDI’02), pp. 271–284.

Wang, S.Y., Chou, C.L., Lin, C.C., 2007. The design and implementation of the NCTUnsnetwork simulation engine. Simulation Modelling Practice and Theory 15 (1),57–81.

White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad, S., Newbold, M., Hibler, M.,Barb, C., Joglekar, A. 2002. An integrated experimental environment fordistributed systems and networks. In: Proceedings of the Fifth Symposium onOperating Systems Design and Implementation (OSDI’02), pp. 255–270.

Yaun, G.R., Bauer, D., Bhutada, H.L., Carothers, C.D., Yuksel, M., Kalyanaraman, S.,2003. Large-scale network simulation techniques: examples of TCP and OSPFmodels. ACM SIGCOMM Computer Communication Review 33 (3), 27–41.

Yonan, J., 2008. OpenVPN – an open source SSL VPN solution, http://www.openvpn.net/.

Zheng, P., Ni, L.M., 2003. EMPOWER: a network emulator for wireline and wirelessnetworks. In: Proceedings of the 2003 IEEE INFOCOM, vol. 3, pp. 1933–1942.

Zhou, J., Ji, Z., Takai, M., Bagrodia, R., 2004. MAYA: integrating hybrid networkmodeling to the physical world. ACM Transactions on Modeling and ComputerSimulation (TOMACS) 14 (2), 149–169.

Jason Liu is an Assistant Professor in the School of Computing and InformationSciences at Florida International University. His research focuses on parallel dis-crete-event simulation, performance modeling and simulation of computer systemsand communication networks. He received a B.A. in Computer Science from BeijingPolytechnic University in China in 1993, an M.S. in Computer Science from Collegeof William and Mary in 2000, and a Ph.D. in Computer Science from DartmouthCollege in 2003.

Yue Li is a postdoctoral research associate in the School of Computing and Infor-mation Sciences at Florida International University. He received his Ph.D. degree inComputer Science from Xi’an JiaoTong University in 2005. He joined TsinghuaNational Laboratory for Information Science and Technology as a postdoc from 2005to 2007. His research interests include network simulation and emulation, highperformance simulation and multimedia networking.

Nathanael Van Vorst is currently enrolled in the Ph.D. program at the School ofComputing and Information Sciences at Florida International University. Hereceived a Masters degree in Computer Science from the Colorado School of Minesin 2007 and a Bachelors degree in Electrical Engineering from Colorado State Uni-versity in 2003.

Scott Mann is currently a Sr Software Engineer at Aztek Networks, Inc, Boulder, COdesigning and implementing VoIP/TDM gateways and emergency standalone sys-tems. He received a Masters in Mathematics from New Mexico State University in1982 and a Bachelors in Mathematics from University of CA, Santa Barbara in 1980.

Keith Hellman is currently enrolled in the Ph.D. Program at the Colorado School ofMines. He received his Masters degree in Computer Science from the same insti-tution in 2006 and a B.S. in Mathematics from the University of Chicago in 1990.


http://abilene.internet2.edu/

http://www.cachelogic.com/research/index.php

http://www.cachelogic.com/research/index.php

http://dast.nlanr.net/projects/Iperf/

http://dast.nlanr.net/projects/Iperf/

http://kb.linuxvirtualserver.org/wiki/IPVS

http://www.cis.fiu.edu/prime/

http://www.sdsc.edu/user_services/datastar/

http://www.sdsc.edu/user_services/datastar/

http://www.ssfnet.org/

http://www.openvpn.net/

http://www.openvpn.net/