Realistic Performance Analysis of WSN Protocols Through ...inside.mines.edu/fs_home/qhan/research/publication/pe-wasun2010.pdfRealistic Performance Analysis of WSN Protocols Through

Realistic Performance Analysis of WSN Protocols ThroughTrace Based Simulation∗

Alan Marchiori, Lin Guo, Josh Thomas, and Qi HanDepartment of Mathematical and Computer Sciences

Colorado School of Mines1500 Illinois Street, Golden, CO, 80401

{amarchio, lguo, jthomas, qhan}@mines.edu

ABSTRACTIt is a difficult endeavor to realistically evaluate the perfor-mance of wireless sensor network (WSN) protocols. Genericnetwork simulators are often used, but they tend to relyon synthetic models. Because WSN performance can be af-fected by many subtle features, these simulators lack a cer-tain level of realism. The most realistic performance assess-ment is to implement the protocol in question and observeits performance in a real world deployment. This approach,however, is time consuming, costly, and makes the directcomparison of various protocols nearly impossible. We be-lieve there exists a need to evaluate the real-world perfor-mance of WSN protocols in a controlled and repeatable fash-ion. To that end, we enable trace based WSN simulation byfirst enhancing an existing WSN profiler that automates thecollection of connectivity traces and the generation of sta-tistical link properties. We then present a trace based WSNsimulator built on the discrete event simulator SimPy usingthe standard Python. The use of the high level languagePython allows new WSN protocols to be rapidly prototypedand evaluated under the real-world conditions captured bythe WSN profiler. To validate the premise that our simula-tion results closely model the real-world performance of thesame protocol, we present a thorough performance analysisof the modern WSN collection tree protocol (CTP). Our ap-proach enables the creation and use of a WSN trace databasecollected from various deployment environments. Such adatabase could be used to both fairly and more realisticallybenchmark existing WSN protocols and provide timely feed-back on the real-world performance of protocols still in thedevelopment process.

Categories and Subject DescriptorsC.4 [Performance of Systems]: Modeling Techniques;C.2.1 [Network Architecture and Design]: Wireless Com-munications

∗This work is supported in part by NSF CNS-0855060.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.PE-WASUN’10, October 17–18, 2010, Bodrum, Turkey.Copyright 2010 ACM 978-1-4503-0276-0/10/10 ...$10.00.

General TermsExperimentation, Performance, Verification

Keywordsperformance analysis, trace based simulation, SimPy, exper-imental validation

1. INTRODUCTIONPerformance evaluation of network protocols is a difficult

subject. In the domain of wireless sensor networks (WSNs)these challenges are made more significant by the plethora ofspecialized software and hardware components used. ManyWSN researchers focus on building physical systems and as aresult focus on empirical performance metrics. These met-rics are useful to describe the performance of the overallsystem but make comparing two protocols difficult. In ad-dition, these results cannot be repeated because of changingenvironmental conditions.

On the other hand, general purpose network simulatorssuch as ns-21and OMNeT++ 2 have also been employed bythe WSN community. These simulators provide high perfor-mance simulation of network protocols. However, to provideuseful results many parameters need to be specified. Theseinclude simulation area, node density, radio model, noisemodel, etc. These parameters are used by synthetic modelsto simulate communications. As a result, users can readilyevaluate protocols under a variety of network conditions andachieve repeatable results. However, these methods are un-likely to accurately predict performance of a given protocolunder real world conditions.

The drawbacks of these two existing approaches illustratethe need for a hybrid approach. Experimental performancemetrics capture all of the real world effects (antenna orien-tation, manufacturing variances, environmental noise, etc.),however, they inhibit the direct comparisons of various pro-tocols. The general purpose network simulators allow pro-tocols to be evaluated under repeatable conditions, but dueto the subtleties of wireless communication lack a certainamount of realism. A hybrid approach would capture in-formation about real world WSN connectivity which couldthen be used directly by the network simulator.

Our approach is to use trace based simulation. Tracebased simulation simply collects network connectivity infor-mation from an actual WSN deployment or testbed and then

1http://www.isi.edu/nsnam/ns/2http://www.omnetpp.org/

the simulator makes packet delivery decisions based on theactual connectivity data. This process, therefore, has twodistinct steps: 1) collect trace data; and then 2) simulatecommunications using the collected data. To collect con-nectivity data we have extended and improved the TestbedProfiler tool [7]. For simulation we have selected the dis-crete event simulator SimPy3 as a base for developing ourown WSN simulation tool. We could have extended ns-2 orOMNeT++, however, we believe that using the Python lan-guage allows users to rapidly prototype potential WSN pro-tocols without the added overhead and structure imposed bya complex network simulation package. Both of these toolsare available to the community via our website4.

By making these tools available and validating our hybridapproach, we hope to encourage other WSN researchers tocontribute and use WSN connectivity traces in the perfor-mance analysis of WSN protocols. Inspiration for our work ispartially a response to the Connectivity project5 at the Uni-versity of California at Berkeley. The connectivity projectis a repository for connectivity traces gathered from vari-ous real-world wireless sensor network deployments. Theirgoal is to gather connectivity traces from a variety of envi-ronments (indoors, outdoors, in a forest, with and withoutinterference, etc.) to be used as a benchmark for commu-nication protocols. The repository currently contains twoconnectivity traces. Our contributions allow researchers toquickly and easily make use of and/or submit data to therepository.

The contributions made by this paper are threefold:

1. An enhanced WSN Profiler tool for the automation ofcollecting WSN traces and generating statistical con-nectivity information.

2. An extension of SimPy enabling the rapid prototypingand evaluation of WSN protocols using connectivitytraces.

3. Validation of our approach using the modern WSN col-lection tree protocol (CTP) [2].

Section 2 describes the WSN profiler in detail and presentsits automated visualization capabilities. Section 3 describesour approach to trace based WSN simulation using SimPy.Section 4 defines the procedure we use to validate our sim-ulations using CTP and provides the results of our perfor-mance evaluation of CTP. Section 5 contrasts our approachto the existing simulation tools. Our final conclusions arepresented in Section 6.

2. WSN PROFILERTo obtain connectivity profiles for the wireless sensor net-

work topologies used in our evaluation of CTP, we haveadapted an application suite called TestbedProfiler [7] ini-tially developed by Chad Metcalfe, Tracy Camp, and MichaelColagrosso at the Colorado School of Mines. This utilityreveals the underlying connectivity of a wireless sensor net-work by systematically recording and then analyzing thereception of packets that are broadcast by each node in thenetwork.

3http://simpy.sourceforge.net/4http://thor.mines.edu/5http://wsn.eecs.berkeley.edu/connectivity/

In its original form, TestbedProfiler provides a number ofuseful connectivity statistics, such as packet delivery ratio(PDR), average RSSI and LQI values, and average neighborcount. It also generates simple connectivity charts that indi-cate the presence or absence of connections between nodes–but not the quality of those connections. Seeing this as alimitation of a useful tool, we modified TestbedProfiler toproduce two additional types of graphs. The first is a spa-tial representation of the network that shows connections be-tween nodes in terms of their estimated transmission count(i.e., ETX [1]), which is a measure of the quality of a connec-tion (see Figure 1). The ETX of a link is the total numberof packet transmissions and retransmissions required to senda packet back and forth across the link one time, assumingthat each node in the route retransmits the packet until it issuccessfully received. The second type of graph is generatedfor each node and shows the PDR both to and from everyother node in the network (see Figures ?? and ??).

Because we see this tool as being useful beyond the scopeof simple testbed deployments, we refer to our enhancedversion simply as ‘WSN Profiler’. In this section, we firstprovide a brief overview of its components and then describeeach in detail.TinyOS application: The TinyOS application runs on the

node hardware itself. It transmits bursts of packetsto elicit connectivity information from its neighbors,which, running the same code, report details of eachpacket they receive to a central server.

Profile Server: The Profile Server, which is linked to thenodes via a serial connection, controls the executionof the profile experiment. It tasks each node to beginits transmissions in turn and also logs incoming packetdata for later processing.

Analysis and Visualization Scripts: These scripts are acollection of scripts that are run post-experiment onthe resulting data to provide average statistics and avisualization of network connectivity.

2.1 TinyOS ApplicationAs mentioned, TestbedProfiler makes use of a TinyOS ap-

plication (TestbedProfilerC) that runs directly on the nodesthat comprise the network in question. TestbedProfilerC hastwo context-dependent roles, which can be conceptualized assender and receiver.

In its role as sender, the application lies dormant until itreceives a command to broadcast, which comes in the formof a serial message from the profile server. Once instructed,the node broadcasts the specified number of packets at reg-ular intervals using its radio and a simple periodic timer.The program is wired to the CC2420 radio stack, making itdeployable only on nodes that utilize the CC2420 hardware.

TestbedProfilerC also functions passively as a receiver.When a node hears a broadcast packet sent from one of itsneighbors, it reports reception of the message to the profileserver. The report contains the sender’s ID, the recipient’sID, the packet sequence number, and the RSSI and LQI val-ues for the received message. Because the report is sent viathe node’s serial connection to the profile server, it providesa reliable means of data delivery that does not interfere withthe radio experimentation.

(a) Testbed in a Grid topology (b) Testbed in a Clustered topology

1.0

1.5

2.0

2.5

3.0

(c) ETX values of links in a Grid topology1.0

1.5

2.0

2.5

3.0

(d) ETX values of links in a Clustered topology

Figure 1: The network configuration (top) and visualization outputs showing the computed link quality interms of ETX (bottom).

2.2 Profile ServerThe Profile Server is a computer running TestbedProfiler–

a command-line Java program responsible for controlling theexecution of the experiment. It must be connected to allnodes that will be profiled via a serial connection (USB, forexample).

Once run, the program has two main responsibilities. First,it must sequentially task each node to broadcast, ensuringthat only one node is broadcasting at a given time. Sec-ond, it must listen for incoming packets from the nodes andwrite them to a trace file for later processing and analysis.Each packet reception event adds a single line–consisting ofreceiver ID, source ID, packet sequence number, RSSI, andLQI–to the trace file. The receiver ID is the ID of the nodethat received the packet; the source ID is that of the packet’ssender. The sequence number of the packet is simply a run-ning zero-based count of the number of packets sent by thecurrently transmitting node. The RSSI and LQI values forthe packet are retrieved directly from the receiving node’sradio hardware and provide a means to quantify the per-ceived strength of the reception event.

The Java interface has been updated to provide the usergreater control over the details of the profile collection, al-lowing it to better reflect the WSN application he or she in-tends to deploy. The command to broadcast originally con-tained only the desired number of iterations and the powerlevel at which to transmit; radio channel and message in-terval were hard-coded on the nodes to channel 15 and to50ms, respectively, and packets of sizes 25, 50, 75, and 100bytes were sent for each iteration. Now, however, the user isalso able to specify the radio channel on which to broadcast,the interval between the transmission of one message to thenext, and the size of the packet to be sent. Additionally, asingle invocation of the profiler originally repeated the sameprocedure for a range of power levels, but now one invocation

corresponds to one power level. This prevents the minglingof disparate data and simplifies automation (e.g. using shellscripting).

2.3 Analysis and Visualization ScriptsWSN Profiler includes a number of scripts that help make

sense of the raw connectivity data gathered by the profileexperiment. A pair of python scripts were included with theoriginal version of TestbedProfiler. The first, CreateProfile,parses a data profile and calculates statistics for each nodein the network. This includes a summary of its connectivityto other nodes with packet delivery rates, average RSSI andLQI values, and average neighbor count. The second script,CreateDigraph, generates a simple connectivity graph thatshows connections between nodes. However, this graph doesnot indicate the quality of the connections. This means thatno distinction is made on the graph between a link in which100% of packets were successfully delivered and one in which0.01% of packets were delivered.

Aware that this is a significant limitation, we expandedthe post-experiment output to include two additional graphs,both of which convey both quantity and quality of links. Thefirst new type of graph (see Figures 1c and 1d) provides a2-dimensional representation of nodes in the network andindicates connections by lines between nodes. The color ofeach line is based on the ETX of the link that line repre-sents, normalized to the color scale shown on the right sideof the graph. An ETX of one, represented by dark green,indicates a high-quality connection in which no packets arelost. Conversely, an ETX of infinity would represent the ab-sence of a connection and is not shown. The user can specifythe upper ETX limit of what is considered a true connec-tion; for Figures 1c and 1d, we use an upper ETX ceiling ofthree, meaning that links with an ETX of more than threeare not shown. This type of graph is excellent at quickly

2 3 4 5 6 7 8 9 11

12

13

14

15

16

17

18

19

21

22

23

24

25

26

27

28

29

31

32

33

34

35

36

37

38

39

Node ID

0.00

0.25

0.50

0.75

1.00

Pack

et

Deliv

ery

Rati

o

Node 1 - Packet Delivery/Reception Rates

Packets FromPackets To

2 3 4 5 6 7 8 91

01

11

21

31

41

51

61

71

81

92

02

12

22

32

42

52

62

72

82

93

03

13

23

33

43

53

6

Node ID

0.00

0.25

0.50

0.75

1.00

Pack

et

Deliv

ery

Rati

o



1 2 3 4 5 6 7 8 9 11

12

13

14

15

16

17

18

19

21

22

24

25

26

27

28

29

31

32

33

34

35

36

37

38

39

Node ID

0.00

0.25

0.50

0.75

1.00

Pack

et

Deliv

ery

Rati

o



Grid

1 2 3 4 5 6 7 8 91

01

11

21

31

41

51

61

71

81

92

02

12

22

42

52

62

72

82

93

03

13

23

33

43

53

6

Node ID

0.00

0.25

0.50

0.75

1.00

Pack

et

Deliv

ery

Rati

o



Clustered

Figure 2: Visualization outputs showing per-node PDR graphs for Node 1 (upper left-hand corner) and Node23 (second from the right in the bottom row) in each topology shown in Figure 1b.

conveying the degree of connectivity in the entire networkand for identifying potentially problematic links.

The second type of graph shows node-specific connectivityinformation and is automatically generated for every nodein the network (see Figure 2). It shows both incoming andoutgoing packet delivery ratios for the node relative to everyother node in the network. A PDR of one for packets fromthe node to a second node indicates that the second nodereceived every packet sent by the first; likewise, a PDR of onefor packets to the node means that the node received everypacket sent by the other. A PDR of zero, of course, meansthe opposite. These graphs are very useful in identifyinghow well a node is connected to the rest of the network,the quality of a specific link, and asymmetric (e.g. one-way)connections.

3. WSN SIMULATION IN SIMPYOur WSN simulator, named WsnSimPy, is based on the

discrete event simulator SimPy. SimPy is an object-oriented,process-based discrete-event simulator written in Python.Each process generates and is notified of events througha scheduler. A good overview of SimPy can be found in[6]. However, a detailed understanding of SimPy is not re-quired because our WSN extensions abstract most of theSimPy logic with our own WSN-specific interfaces (e.g. Ra-

dio.Transmit(packet)). In our simulation each node is im-plemented as a distinct SimPy process. SimPy maintains thescheduler and dispatches events to processes at the appro-priate time. The scheduler executes in a single thread andas a result all simulated processes are non-preemptive. Thissimplifies development and debugging because execution isdeterministic and no lock/unlock operations are required.

Simulation of a WSN may seem challenging at first butthere are just a few issues to be resolved. At the most basiclevel the simulator must allow multiple nodes to send andreceive packets. To accomplish this, we first create a sin-gle network object, then create several node objects. Eachnode receives a reference to the network object and duringinitialization calls a registration method so that the network

is informed of all nodes. The node objects all have a receive

method and the network has a transmit method. Any nodecan call the transmit method and the network will generatereceive events on the appropriate nodes at the appropriatetime. To produce an accurate simulation, we must ensurethat the selection of the receiving nodes and the timing ofthe receive events are realistic. This must take into accountthe half-duplex nature of the radio and the possibility ofcollisions.

We have developed two version of WsnSimPy: WsnSimPy(trace), and WsnSimPy (synthetic). WsnSimPy (trace) usestrace data collected from the WSN Profiler to simulate com-munications; and WsnSimPy (synthetic) uses a syntheticradio model to simulate communications. We will explainthe synthetic radio model in the next section. The architec-ture of WsnSimPy (trace) is shown in Figure 3; here node1 transmits a packet. The simulated network then queriesthe trace data to determine the receiving nodes–in this casenodes 2 and 3 are indicated. A packet reception event isthen scheduled for these two nodes at some time in the fu-ture (depending on the size of the packet). Each node iscomprised of a single SimPy process (the radio) containingseveral Python objects (the networking stack and applica-tion). The radio has a transmit queue but no receive queue.This is because when a packet receive event is signaled theradio layer immediately passes the packet up the networkstack. A transmit queue is needed to serialize sequentialpacket transmissions through the radio. If this queue lengthis exceeded, the radio layer rejects (drops) packets from theupper layers. These upper layers may or may not performtheir own queuing depending on the networking stack beingsimulated.

To simulate a packet transmission, the source node firstchecks if it has sent a packet recently (a configurable parame-ter, we use < 32 milliseconds). If so, it uses the next sequen-tial sequence number in the trace data (wrapping aroundfrom the end to start if needed) and signals the start of apacket reception on each node specified in the trace. If not,it selects a new random sequence number from the trace and

Node 2

Radio

TxQ

Link Layer

Network Layer

...

Application

TraceData

Node 3

Radio

TxQ

Link Layer

Network Layer

...

Application

Node 1

Radio

TxQ

Link Layer

Network Layer

...

Application

DetermineRecipients

TX RX

now() SimPy Event List

ComputeDelay

Figure 3: The architecture of WsnSimPy (trace).Node 1 initiates a transmission creating a TX event.When the transmission begins, the trace data is usedto select recipients and the transmission delay iscomputed. Receive (RX) events are scheduled forall selected nodes (in this case, nodes 2 and 3) atthe computed time.

uses this data to identify recipients. This procedure main-tains temporal artifacts in the trace data when appropriateand removes any temporal correlation when not appropriate.

To properly simulate the timing of events, a delay is in-serted before each packet delivery is signaled. We are sim-ulating an IEEE 802.15.4 radio with a data rate of 250kbits per second, therefore, we can use this to compute de-lays based on estimated packet size (32µsec per byte). Overthe short communication range of the IEEE 802.15.4 radios,the propagation delays are well below 1 µsec, therefore, thetotal delay is dominated by the transmission delay and wedo not account for propagation delay. We also do not ac-count for processing delays because of its application-specificnature. However, processing delays could be easily addedby the following line in the node’s receive event: yield

hold,self,<simulated processing time in µ sec>.All transmissions are serialized through the a simulated

half-duplex radio. This means a node that is transmittingfails to receive any messages sent until the simulated trans-mission delay has elapsed. The reverse also holds; a nodethat is receiving cannot transmit a new message until thecurrent reception is complete.

Collisions are detected independently on each receivingnode. This allows the possibility of a collision at one nodewhile other nodes successfully receive the packets. After apacket reception has occurred, a collision window is insertedwith a small delay (which is a configurable parameter, weuse 32 µsec). If another packet reception begins within thiscollision window we discard all collided packets. This ap-proach assumes an idealized MAC layer that can perfectlydetect and reschedule transmissions that may occur outsideof the collision window. This is only a simple approximationof collisions, however, our results (Section 4) indicate thatthis approximation is reasonably accurate in practice. Wecaution that this approximation is less appropriate for high

data rates. To accurately simulate high data rate applica-tions, a more realistic MAC layer would be needed.

4. SAMPLE PERFORMANCE ANALYSISIn this section we present a sample performance analy-

sis of the Collection Tree Protocol (CTP) [2]. The goalis to demonstrate that the performance metrics computedby WsnSimPy (trace) are more realistic than other simu-lation methods. For the other methods, we use TOSSIM6

and WsnSimPy (synthetic).TOSSIM provides a scalable andhigh fidelity simulation of a complete TinyOS sensor net-work. We compare each of these methods to results obtainedfrom our 36 node testbed under two configurations shown inFigure 1. The first configuration is an approximately uni-form grid. The second is a clustered topology where there isa cluster of nodes in each corner and fewer nodes elsewhere.To simulate a physically larger network we use a reducedtransmitter power level (-25 dBm) which yields an averagehop count of 3 to 3.5 in clustered topology and an averageneighbor count of 12. In the following, we explain the radiomodel used in our simulator before presenting experimentalresults.

4.1 Synthetic Radio ModelThe synthetic radio model used in WsnSimPy (synthetic)

uses the signal to noise ratio (SNR) [10] as in TOSSIM toselect recipients. The model uses the ratio of received signalstrength to noise to determine which nodes should receivepackets. The received signal strength is indicated by theRSSI (Received Signal Strength Indicator). A node will suc-cessfully receive a packet if its RSSI is greater than the envi-ronmental noise floor; otherwise, the packet will be dropped.

To simulate noise, We use the CPM noise model. Previousresearch reveals that CPM [3] accurately models the envi-ronmental noise in wireless sensor networks by using histor-ical noise values to predict future noise values. To improvethe quality of our evaluations, we collected a noise tracefor our wireless sensor network testbed. This was done us-ing a simple TinyOS application that collected backgroundnoise by sampling the radio’s RSSI reading 10 times per sec-ond in the absence of any active radio transmissions. Eachreading was then reported to a central PC via the mote’sserial connection in a fashion identical to that described forpacket-reception reports in WSN Profiler. Installed on all36 motes, this program allowed us to collect more than 1.3million individual noise readings over the course of an hour.The resulting trace provided us with a realistic noise modelfor use in our CTP evaluations.

RSSI is affected by the distance between the sender andthe receiver, and the transmission power level of the sender.A node using a higher transmission power level will createa correspondingly higher signal strength at the receiver. Inaddition, different network topologies may also lead to dif-ferent RSSI due to the radio interference and propagationeffects. RSSI can be computed as the product of transmis-sion signal power and link gain. In our model, the transmis-sion signal power of each node is assumed to be a constant.Link gain reflects the link connectivity between node pairs,i.e., it indicates physical signal attenuation when a packet istransmitted from one node to the other.

6http://www.cs.berkeley.edu/~pal/research/tossim.html

Table 1: Experimental results across four environments with different link layer modelsTest Environment Radio Model No. of Nodes Tx. Period PDR No. of Beacons/Node No. of DuplicatesTOSSIM SNR 3 8s 100% 13 3WsnSimPy SNR 3 8s 100% 10 3WsnSimPy Trace data 3 8s 100% 10 0TestBed Real hardware 3 8s 100% 2 0

To collect realistic link gain values in the testbed, we usedata from the WsnProfiler. If no packet was received at aparticular node, the link gain between this node pair is as-sumed to be a very large negative value which will causeevery packet to be dropped in the radio model; otherwise,the link gain between the node pair is the RSSI value aver-aged over the entire trace.

4.2 CTP ValidationTo validate our implementation of CTP in WsnSimPy and

make sure our implementation of CTP is correct, we testedCTP in a small chain network consisting of three nodes.The end node of the chain is configured as root node, andthe other two nodes are configured as tree nodes. Each treenode periodically generates and transmits its own data pack-ets and also forwards data packets from its neighbor. Du-plicate packets are observed in the root node. This is dueto the aggressive transmission policy used by CTP whereCTP retransmits a packet at most 32 times if the node failsto send out the packet or the node does not receive an ac-knowledgement. Both WsnSimPy (synthetic) and TOSSIMuse the same link gain and same noise trace dataset, sothe underlying radio model has the same network topol-ogy. Table 1 summarizes the experimental results. Thecomparable results across all three simulative environments(TOSSIM, WsnSimPy (synthetic), and WsnSimPy (trace))indicate that our implementation of CTP and synthetic ra-dio model in WsnSimPy is valid. The simulation results arealso comparable to those from the test bed, hence validatingour experimental methodology.

4.3 CTP Performance AnalysisTo evaluate the accuracy of WsnSimPy (trace), we stud-

ied the overall performance of CTP across several environ-ments including both simulations and testbed experiments.CTP is known as a highly reliable and effective protocol forlow data rate applications. We choose PDR (Packet DeliverRatio), number of beacons per node, churn (number of par-ent changes) per node as the metrics for testing of CTP’sreliability and effectiveness. PDR measures the reliability;number of beacons per node measures the effectiveness; andchurn indicates the stability of routing and topology sincea churn event is triggered when a node switches to a newparent and reestablishes the path to the sink node. In ourexperiments, we placed the root node in the corner of thetestbed to get longer routes.

In the Grid topology shown in Figure 1a, each node sendsa packet every eight seconds and each experiment is runfor 30 minutes. As indicated in Figure 4, all four scenar-ios show the similar trends in all three different metrics.We also observe that among all three simulative environ-ments, WsnSimPy (trace) delivers performance closest to

the testbed results. This implies that our trace-based simu-lator is well designed and our hybrid approach is promising.

To explore CTP performance in an irregular network, weconducted the same set of experiments in a Clustered topol-ogy shown in Figure 1b and the results are presented in Fig-ure 5. We observe that the results from WsnSimPy (trace)differ from testbed results by less than 5% in all three met-rics. However, the results from WsnSimPy (synthetic) di-verge significantly from the testbed results, i.e., by morethan 20% in all three metrics. Therefore, we can concludethat our trace based simulator WsnSimPy delivers more re-alistic results comparing against other simulators. One ofthe possible justifications for this is as follows. The under-lying link gain in WsnSimPy (synthetic) only captures theaverage RSSI between the node pairs, and the RSSI varianceamong the different packet transmissions is not reflected inthe link gain values. The trace data reveals that more varia-tion of RSSI occurs in clustered topology than it does in gridtopology. Without considering the large variation of real linkgain, the synthetic radio model is very different from the ac-tual radio propagation behaviour in test bed with clusteredtopology.

We have further studied CTP performance when data ratevaries. Figure 6 shows the results in a clustered topol-ogy. In addition to PDR, we use two other metrics. Oneis path length, the average number of hops a data packettakes. It indicates the ability of CTP to explore the networkdepth. The other is Cost/Path Length, the average trans-mission per link. It indicates the effort of delivering a datapacket. We again observe that the results from WsnSimPy(trace) are the closest to testbed results in all three metrics.TOSSIM renders lower PDR and longer path length, indicat-ing that the routing tree is less effective and might be unsta-ble. Similarly, WsnSimPy (synthetic) results in lower PDRand higher cost, implying that packet loss is over-estimatedin the SNR radio model. Figure 7 shows the results in a gridtopology. All three simulative environments show the similarPDR and Cost/Path Length which is close to testbed results.The Path length from the simulative environments divergesignificantly from the testbed results. Compared with theother two simulators, WsnSimPy (trace) delivers more re-alistic results. TOSSIM and WsnSimPy (synthetic) againshow longer path length, indicating that the estimated SNRradio model is worse than the real radio propagation. Com-bined the results from grid topology and clustered topology,we conclude that the synthetic SNR radio model well pre-dicts the radio propagation in grid topology, but is far dif-ferent from the real radio propagation in clustered topology.

In summary, CTP achieves comparable reliability and ef-fectiveness in an irregular network as in a grid network ina testbed. Both TOSSIM and WsnSimPy (synthetic) per-form well in grid networks, but perform poorly in irregularnetworks. The use of a synthetic radio model by both sim-

0.98

0.985

0.99

0.995

1

1.005

1.01

5 10 15 20 25 30

PD

R

Duration (minute)

TestbedWsnSimPy(trace)

WsnSimPy(synthetic)TOSSIM

5

10

15

20

25

30

35

5 10 15 20 25 30

No.o

f beacons/n

ode

Duration (minute)



0

0.5

1

1.5

2

2.5

3

3.5

5 10 15 20 25 30

Churn

Duration (minute)



Figure 4: CTP performance in a Grid topology

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

5 10 15 20 25 30

PD

R

Duration (minute)



0

50

100

150

200

250

300

350

5 10 15 20 25 30

No.o

f beacons/n

ode

Duration (minute)



0

10

20

30

40

50

60

70

5 10 15 20 25 30

Churn

Duration (minute)



Figure 5: CTP performance in a Clustered topology

0.2

0.4

0.6

0.8

1

1.2

1.4

1 2 3 4 5 6 7 8

PD

R

Inter-Packet Interval (second)



2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8

Path

Length




10

20

30

40

50

60

1 2 3 4 5 6 7 8

Cost/P

ath

Length




Figure 6: CTP performance with different data rates in a Clustered toplogy.

0.8

0.85

0.9

0.95

1

1.05

1.1

1 2 3 4 5 6 7 8

PD

R




3

4

5

6

7

1 2 3 4 5 6 7 8

Path

Length




1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1 2 3 4 5 6 7 8

Cost/P

ath

Length




Figure 7: CTP performance with different data rates in a Grid toplogy.

ulators most likely contributed to the poor performance inirregular networks. In contrast, WsnSimPy (trace) matchesvery closely to the testbed results in both grid and irregularnetworks. This is a strong indication that our trace-basedsimulator WsnSimPy (trace) is a very promising tool thathas the potential to provide the most realistic results com-paring to the other simulators using synthetic radio models.

5. RELATED WORKSimulation has long been used to explore, explain, and

evaluate wireless sensor networks and their behaviors. As aresult, multiple simulation tools have been developed. Theycan be split into two main types: 1) general purpose networksimulators (e.g., ns-2, OMNeT++); and 2) WSN-specifichardware/OS simulators (e.g., TOSSIM [5], Avrora [11],Cooja [8]). We next briefly highlight the primary differencesbetween these approaches and ours.

The standard for network simulation is considered by manyto be ns-2. ns-2 is an object oriented discrete event simula-tor written in C++. Two of the main design goals were highperformance and scalability. As a result the learning curveto ns-2 is quite steep. This discourages researchers fromWSNs from using it. Our simulator, on the other hand,provides a WSN-centric view of the network without thecomplex structure imposed by ns-2.

OMNeT++ is a more contemporary discrete event simu-lator used for general purpose network simulation. Part ofthe motivation to develop OMNeT++ was to provide a moreuser friendly interface than ns-2. Although OMNeT++ doesnot have native WSN extensions, they have been added asa module called Castalia7. In [9] the authors say: “Thesimulator, named Castalia, boasts the most accurate wire-less channel and radio models for WSN found in currentliterature. These models are capturing some essential ex-perimental findings. This does not guarantee though thatthe simulator will behave similarly with a real deploymentat the high level (i.e., the protocol or application level).”This highlights perfectly the differences from our approach.Specifically, we are interested in simulating the high levelperformance characteristics of WSN protocols.

The hardware/OS specific simulators such as TOSSIM,Avrora, and Cooja are all focused on simulating the behaviorof the individual sensor nodes. As a result the communica-tions models used tend to focus on validating functionalityand not evaluating performance. In [4] one can find thiswarning: “TOSSIM’s primary goal is to provide a high fi-delity simulation of TinyOS applications. For this reason, itfocuses on simulating TinyOS and its execution, rather thansimulating the real world. While TOSSIM can be used tounderstand the causes of behavior observed in the real world,it does not capture all of them, and should not be used forabsolute evaluations.” This warning applies equally to allsimulators in this category. As with the general purposenetwork simulators, our goal is to do exactly what the cur-rent methods cannot–enable absolute evaluations of WSNprotocols.

7http://castalia.npc.nicta.com.au/

6. CONCLUSIONIn this paper we have introduced our improved WSN pro-

filing tool and trace based WSN simulator built on SimPy,WsnSimPy. The profiler automates the process of collect-ing and analyzing connectivity information in WSN deploy-ments. The output generated by the profiler can then beused for trace based simulation of WSN protocols. Usingthe collection tree protocol, we demonstrate that the simu-lated performance based on trace data was more represen-tative of the real world performance than results based onsynthetic radio models. Because the trace based simulationsare repeatable, this approach enables different protocols tobe evaluated under identical network conditions. We alsobelieve that enabling realistic performance analysis of WSNprotocols from the high level language, Python, is an invalu-able tool to be used in the protocol design process. In ourfuture work we plan to continue the development of thesetools and to create a benchmark set of WSN profiles to facil-itate the fair comparison of WSN protocols under a varietyof network conditions.

7. REFERENCES[1] D. S. J. De Couto, D. Aguayo, J. Bicket, and R. Morris. A

high-throughput path metric for multi-hop wireless routing.In Proceedings of the 9th ACM International Conferenceon Mobile Computing and Networking (MobiCom), 2003.

[2] O. Gnawali, R. Fonseca, K. Jamieson, D. Moss, andP. Levis. Collection tree protocol. In Proceedings of the 7thACM Conference on Embedded Networked Sensor Systems(SenSys), 2009.

[3] H. Lee, A. Cerpa, and P. Levis. Improving wirelesssimulation through noise modeling. IPSN 2007, January2007.

[4] P. Levis and N. Lee. TOSSIM: A simulator for TinyOSnetworks. www.cs.berkeley.edu/~pal/pubs/nido.pdf,2003.

[5] P. Levis, N. Lee, M. Welsh, and D. Culler. TOSSIM:accurate and scalable simulation of entire TinyOSapplications. In Proceedings of the 1st internationalconference on Embedded networked sensor systems(SenSys), 2003.

[6] N. Matloff. A discrete-event simulation course based on theSimPy language. http://heather.cs.ucdavis.edu/~matloff/simcourse.html,accessed 2010.

[7] C. Metcalf. TOSSIM live: Towards a testbed in a thread.Master’s thesis, Colorado School of Mines, 2007.

[8] F. Osterlind, A. Dunkels, J. Eriksson, N. Finne, andT. Voigt. Cross-level simulation in COOJA. In Proceedingsof the European Conference on Wireless Sensor Networks(EWSN), Poster/Demo session, 2007.

[9] H. Pham, D. Pediaditakis, and A. Boulis. From simulationto real deployments in wsn and back. In IEEEInternational Symposium on a World of Wireless, Mobileand Multimedia Networks (WoWMoM), 2007.

[10] T. Rusak and P. Levis. Investigating a physically-basedsignal power model for robust low power wireless linksimulation. In Proceedings of the 11th internationalsymposium on modeling, analysis and simulation ofwireless and mobile systems (MSWiM), 2008.

[11] B. L. Titzer, D. K. Lee, and J. Palsberg. Avrora: scalablesensor network simulation with precise timing. InProceedings of the 4th international symposium onInformation processing in sensor networks (IPSN), 2005.

Documents

Realistic Performance Analysis of WSN Protocols Through ...inside.mines.edu/fs_home/qhan/research/publication/pe-wasun2010.pdfRealistic Performance Analysis of WSN Protocols Through