Chapter 5

Experiments

This chapter presents the methodology used to solve the problem of optimizing thepeer-to-peer information transfer in VANET networks by using VDTP protocol. Theaim of this chapter is to provide all information necessary to enable the repetitionof the experiments carried out in this Thesis. The chapter is organized as follows:Section 5.1 gives an overview about the experiments. Then, Section 5.2 shows thesoftware and hardware tools used for both, the development and the executions.Finally, Section 5.3 presents the metrics used to compare the performance of thedi!erent algorithms.

5.1 Optimization Framework

The optimization framework for this problem is composed by two main parts: anoptimization algorithm and a simulation procedure. The optimization part is car-ried out by (independently) one of the five algorithms described in Section 3.3. Allof them are specially adapted to find optimal (or cuasi-optimal) solutions in con-tinuous search spaces (which is the case in this work). The simulation process is away of assigning a quantitative quality value to the factors regulating VDTP, thusleading to optimal configurations of this protocol tailored to a given scenario. Thisprocedure is carried out by means of the ns-2.31 [98] simulator (see Section 2.6) inwhich we have implemented the VDTP protocol for transferring files in VANETs.

In each optimization algorithm, the evaluation of the solutions is carried out bymeans of the simulation component. As Figure 5.1 illustrates, when a given algo-rithm generates a new solution it is immediately used for configuring the VDTP ina given simulation. After the simulation, ns-2 returns the global information about

73

CHAPTER 5. EXPERIMENTS

the transmission time required for transferring files, the number of lost packets gen-erated during the simulation, and the amount of data exchanged between vehicles.This information is used to compute the fitness function presented in Section 4.3.

Figure 5.1: Optimization framework for VDTP configuration in VANETs, where thealgorithms invoke the ns-2 simulator for each solution evaluation.

5.1.1 Instances: VANET Scenarios

One of the most important and critical task to solve the problem is to obtain themetrics necessary to evaluate the performance of a particular VDTP configuration.This evaluation has been carried out by using the simulator VanetMobiSim/Ns-2(see Section 2.6).

In order to obtain results close to the real world, we have defined two simulationVANET scenarios (instances) from real areas of Málaga in Spain (see Figure 5.2),creating an urban and a highway scenarios. We have defined this two distinct sce-narios since the characteristics of the movement of vehicles for these two scenariosare di!erent enough to a!ect the transfer of files. For example, in urban areas thevehicles density is higher and these vehicles travel at lower speeds than in interurbanenvironments, increasing the likelihood that the transfers are carried out successfullyin urban than in the highways. Therefore, we can analyze in both scenarios the be-havior and performance of the compared algorithms, as well as the di!erences in theresulting VDTP configurations in terms of communication e"ciency. Furthermore,we can compare these automatically generated configurations against the ones usedin the real experiments by human experts of CARLINK Consortium [26, 25].

74

5.1. OPTIMIZATION FRAMEWORK

a) Urban scenario map

b) Highway scenario map

Figure 5.2: Selected areas of Málaga for our VANET simulations.

The urban instance covers an area of 120,000 m2 (Figure 5.2.a) including build-ings and semaphores. Whereas that the highway scenario covers a stretch of 1 kmwith two directions (Figure 5.2.b) without buildings and semaphores. In both sce-narios there are 30 vehicles moving through the roads, ten of those are trying todownload a file that is stored in other ten vehicles, which is equivalent to simulateten di!erent transfers on the same scenario. Vehicles on the urban scenario travelat speeds between 30 km/h and 50 km/h. However, vehicles on highway roads doso at speeds ranging from 80 km/h and 110 km/h (see Figure 5.3).

The vehicles try to transfer 1 Mbyte (1,024 Kbytes) files. However, the transfermay be of less data, if the communication is rejected (see Section 4.2.1). Thecommunications between the vehicles is conditioned by the use of IEEE 802.11bstandard network interfaces [53]. To obtain more realistic results, we have used thereal characteristics of the PCMCIA PROXIM ORiNOCO - Classic Gold1 networkinterface which has been connected to an omnidirectional antenna with a gain of 7dBi. Other features of the protocols used to deploy the VANET are summarized inTable 5.1.

1ORiNOCO Classic Gold PC Card: http://www.proxim.com/products/wifi/client/goldpccard/

75

CHAPTER 5. EXPERIMENTS

a) Urban scenario representation

b) Highway scenario representation

Figure 5.3: Málaga areas representation by using VanetMobiSim for their simulation.

Table 5.1: VANET instance specification

Parameter ValueNumber of vehicles 30Number of file transfers 10Link Layer: Network Interface PROXIM ORiNOCO - Classic Gold (802.11b)Link Layer: antenna gain 7 dBi (Omnidirectional)Routing Protocol DSR Protocol [69]Transport Protocol DSR Protocol [69]Application Protocol VDTP Protocol [23]

Using a simulator for the task of evaluating solutions introduces a new issue tokeep in mind, the algorithm runtime. The simulation requires a high computationtime, increasing the execution time of algorithms critically. In order to resolve thisproblem we have executed the algorithms using a cluster to parallelize the di!erentindependent runs (see Section 5.2).

76

5.1. OPTIMIZATION FRAMEWORK

5.1.2 Algorithms Parameter Settings

In order to compare the performance of the five studied algorithms solving OFTCproblem we have executed them on an equal footing. For this reason in our exper-iments, they have the same Stop Condition: 1,000 solution evaluations (VANETsimulations) per run. Therefore, the population based algorithms (PSO, DE, GA,and ES) were configured with 20 individuals, performing 50 generational steps(20 individuals ! 50 generations = 1, 000 evaluations); and SA performs 1,000generations. In turn, the comparison is done from 30 independent runs performedof each of the algorithms to compare more representative results.

The parameterization of the SA and PSO algorithms used to solve the OFTCproblem are presented in tables 5.2 and 5.3, respectively.

Table 5.2: Parameterization of SA optimization algorithm.Parameter Symbol ValueNumber of Independent Runs 30Number of Generations 1,000Temperature Decay ! 0.8Markov Chain Size 5

Table 5.3: Parameterization of PSO optimization algorithm.Parameter Symbol ValueNumber of Independent Runs 30Number of Generations 50Population Size 20Local Coe"cient "1 2·rand(0,1)Social Coe"cient "2 2·rand(0,1)Inertia Weigh w 0.5

For the other algorithms, we have to configure some di!erent parameters aswhich types of selection/replacement methods they use. The implemented geneticalgorithm selects the parents by using two di!erent methods: the first progenitoris selected by using tournament selection and the second one by ranking selection.DE and ES selects both parents randomly. These three algorithms replace eachindividual after generation asynchronously. Other configuration parameters for thealgorithms GA, ES, and DE are specified in Table 5.4.

77

CHAPTER 5. EXPERIMENTS

Table 5.4: Parameterization of GA, ES, and DE optimization algorithms.Algorithm Parameter Symbol Value

Number of Independent Runs 30All Number of Generations 50

Population Size 20

GA Crossover Probability Pcru 0.8Mutation Probability Pmut 0.2

ES Crossover Probability Pcru 0.9Mutation Probability Pmut 0.1

DE Crossover Probability Cr 0.9Mutation Factor µ 0.1

5.2 Software and Hardware Tools

This section presents the distinct tools necessary for the implementation and exe-cution carried out in our experiments.

5.2.1 MALLBA Library

The development of all algorithms used in this project has been carried out by usingMALLBA [4], a C++ based framework of metaheuristics for solving optimizationproblems. This framework provides a set of skeletons for combinatorial optimizationthat can be extend easily. MALLBA o!ers for each algorithm a skeleton formed by aset of classes that interact each other to solve the optimization problem instantiated.Skeletons are implemented by a set of required and provided classes that representan abstraction of the entities participating in the resolution method.

• Provided Classes implement the basic functionality of the algorithm, thatis, the internal aspects of the skeleton in a problem-independent way.

• Required Classes present the specific behavior and information of the prob-lem. The interface of these classes is unique and pre-established to allowinteraction with the provided classes. This part of the skeleton has to beimplemented from the problem objective.

Although the skeleton may vary according to the algorithm, all the skeletonscoincide with a core set of classes. The UML diagram (Figure 5.4) presents theClass Diagram of the core of MALLBA.

78

5.2. SOFTWARE AND HARDWARE TOOLS

Required Classes

Figure 5.4: Common design of Mallba skeletons.

In this Thesis we have developed all classes needed to use the optimizationalgorithms presented before, however we have not implemented Solver_Lan andSolver_Wan since we have not solved this problems using parallel implementations.

5.2.2 Optimizing VDTP by using VanetMobiSim/Ns-2

The selected simulator, VanetMobiSim/Ns-2, allows the users to add new featuresas protocols or functionalities. This characteristic is important since VDTP is notincluded in this simulator. In order to solve this problem, ns-2 has been extendedwith VDTP protocol definition. This has been carried out developing the protocolusing the programming languages OTcl and C++ (see Section 2.6.2). OTcl hasbeen used to define the packet protocol identifier and the VDTP default values forparameters settings. For fhis purpose we have modified two files: ns-packet.tcland ns-default.tcl (see Listing 5.1), respectively. In C++ has been developedthe protocol operation, as well as the link between C++ and Tcl interpreter of thesimulator. The implementation in C++ is based on developing di!erent classes thatinherit from some classes provided by ns-2 (see Figure 5.5).

Listing 5.1: ns-default.tcl code modificationAppl i cat ion /VDTP set chunk_size_ 25600 # 256 KbytesAppl i cat ion /VDTP set f i l e_s i z e_ 1048576 # 1 MbyteAppl i cat ion /VDTP set max_attempts_ 8 # 8 at temptsAppl i cat ion /VDTP set max_waitingtime_ 8 # 8 seconds

79

CHAPTER 5. EXPERIMENTS

Figure 5.5: Classes developed to simulate VDTP in ns-2.

This classes implement di!erent aspects necessary to simulate VDTP by usingns-2 simulator:

• VDTPappClass: It is the interface between the Tcl simulation and the protocoldefinition.

• VDTPData: It implements the header of VDTP packet and the management ofits fields.

• VDTPDataS: It defines the whole VDTP packet and the interaction betweenthe upper and lower levels of this protocol.

• AskPacketTimer: Necessary to get the times and timers used to send FIRPand DRP packets (see Section 4.2.1).

• AckTimer: Necessary to get the times and timers used to send FIRQ and DRQpackets (see Section 4.2.1).

• VDTPapp: It implements the whole VDTP protocol operation.

80

5.2. SOFTWARE AND HARDWARE TOOLS

In turn, we have added a new functionality to ns-2, which it is used to returnthe metrics needed for evaluating the fitness function of a given configuration. Afterthe implementation of all these classes, the simulator had to be compiled. Then,we were able to use the VDTP protocol for transferring files in VANET networks.Therefore, ns-2 were ready to be used in the optimization process (see Figure 5.1).

5.2.3 Parallel Executions by using Condor

One of the main problems associated with use of simulation to evaluate the fitnessfunction is the required execution time. The execution of our experiments has beencarried out by using PC machines with Pentium IV 2.4GHz core, 1GB of RAM,and O.S Linux Fedora core 6. The average execution time needed to simulate thescenarios defined in the previous sections is 4.33 seconds. If the algorithms run 1,000generations and we execute 30 independent runs, the execution time per algorithm is36.08 hours (see Equation 5.1). If that is multiplied by five (number of algorithms)the time needed to get results is over 180 hours. In this computation, we have takeninto account just the fitness evaluation time, we have not considered the computationtimes of all tasks that are running by the optimization algorithms.

timeexperiment = 4.33! 1000! 30 = 129900 seconds = 36, 08 hours (5.1)

We have therefore decided to execute each of the 30 independent runs in parallelbut independently, since the parallelism here is used to decrease the time of exper-imentation. The executions were carried out on a cluster of 250 PC machines witha Pentium IV 2.4 GHz, 1GB of RAM, and OS Linux Fedora Core 6.

As the execution of each each algorithm separately require in average 36,08 hours,we have used a High-Throughput Computing (HTC) environment. Years ago, su-percomputers were used to address the problems that required a lot of resources fora long time. Such systems are capable of dealing with problems that require highcomputation power. However, just the institutions with high budgets might use thissystems because they had very high prices. Nowadays, a collection of interconnectedheterogeneous nodes (machines) can be used to execute tasks in distributed manner.These nodes represent resources, which can be simple personal computers. There-fore, in place of supercomputers, we can use multiple cheap PCs, interconnected viaa network, and simply installing a software that is responsible for distributing thejobs between the di!erent computers.

81

CHAPTER 5. EXPERIMENTS

In this Thesis we have chosen Condor [132] to create the HTC environment.Condor is a freely available project at the University of Wisconsin-Madison, it is de-signed to encapsulate and run large collections of distributed computing resourceswith the ultimate aim of providing more access to available computing power. Con-dor specializes in giving users the ability to run huge numbers of tasks over longperiods of time. When users submit jobs, Condor searches for and finds the availablemachines on the entire network and executes the work on these machines. Condorhas the ability to detect whether a machine that was running a job becomes unavail-able for a while. In addition, the users can set a checkpoint and migrate their worktasks to di!erent machines that are idle or available. The main features of Condorare [16]:

• Starting with version 6.9.5, Condor is released with the Apache License ver-sion 2.0.

• No special programming is required to use Condor. It is able to run normalUNIX programs, only requiring the user to relink, not recompile them orchange any code.

• The local execution environment is preserved for remotely executing processes.Users do not have to worry about moving data files to remote workstationsbefore executing programs there.

• Condor software is responsible for locating and allocating idle workstations.Its users do not have to search for idle machines, nor are they restricted tousing machines only during a static portion of the day.

• Owners of workstations have complete priority over their own machines.

• Users of Condor may be assured that their jobs will eventually complete.

• Measures have been taken to assure owners of workstations that their filesys-tems will not be touched by remotely executing jobs.

• Condor does its work completely outside the kernel, and is compatible withBerkeley 4.2 and 4.3 UNIX kernels and many of their derivatives. Users donot have to run a custom operating system to get the benefits of Condor.

82

5.3. USED METRICS TO COMPARE RESULTS

5.3 Used Metrics to compare Results

The main objective of this Thesis is to solve the problem of optimizing the peer-to-peer transfer of information in VANET networks. For this purpose we have used fivedi!erent metaheuristic algorithms (SA, GA, DE, PSO, and ES). Therefore, in thiswork we have also set the objective of evaluating the performance of these di!erentmetaheuristic techniques in solving the problem, in order to decide which of thealgorithms is best suited to this kind of problems.

We analyze the obtained results separately in the two previously defined sce-narios (urban and highway). The metrics used to compare the performance of thedi!erent algorithms are based on the study of the final values of the objective func-tion for each of the 30 independent runs. For our work we have used the mean,standard deviation, median, minimum, and maximum values obtained forthe di!erent algorithms. As a minimization problem, the configuration that obtainsthe minimum fitness value (see Section 4.3) is the optimal. Also, we have used theFriedman test for a comparison of algorithms with statistical significance.

In addition, we have compared the performance of the optimal VDTP config-uration returned for each of the algorithms. In order to do so, we have simulatedVANET instances with the returned configurations in the median value of fitnessminimum of 30 independent runs. Also, we have compared the execution time spentby each algorithm to complete the entire process of finding the optimal VDTP con-figuration. All this study is presented in the next chapter.

83

CHAPTER 5. EXPERIMENTS

84