Using Dependability Benchmarks to Support ISO/IEC SQuaRE

Jesus Friginal, David de Andres, Juan-Carlos RuizSTF-ITACA, Universitat Politecnica de Valencia,

Campus de Vera s/n, 46022, SpainEmail: {jefrilo, ddandres, jcruizg}@disca.upv.es

Regina MoraesSchool of Technology, University of Campinas - UNICAMP

Campus I Limeira, BrazilEmail: regina@ft.unicamp.br

AbstractThe integration of Commercial-Off-The-Shelf(COTS) components in software has reduced time-to-marketand production costs, but selecting the most suitable compo-nent, among those available, remains still a challenging task.This selection process, typically named benchmarking, requiresevaluating the behaviour of eligible components in operation,and ranking them attending to quality characteristics. Mostexisting benchmarks only provide measures characterising thebehaviour of software systems in absence of faults ignoringthe hard impact that both accidental and malicious faults haveon software quality. However, since using COTS to build asystem may motivate the emergence of dependability issuesdue to the interaction between components, benchmarkingthe system in presence of faults is essential. The recentISO/IEC 25045 standard copes with this lack by consideringaccidental faults when assessing the recoverability capabilitiesof software systems. This paper proposes a dependabilitybenchmarking approach to determine the impact that faults(noted as disturbances in the standard) either accidental ormalicious may have on the quality features exhibited bysoftware components. As will be shown, the usefulness of theapproach embraces all evaluator profiles (developers, acquirersand third-party evaluators) identified in the ISO/IEC 25000SQuaRE standard. The feasibility of the proposal is finallyillustrated through the benchmarking of three distinct softwarecomponents, which implement the OLSR protocol specification,competing for integration in a wireless mesh network.

Keywords-Dependability Benchmarking; Quality Evaluation;ISO/IEC SQuaRE;

I. INTRODUCTION

In a globalised context where software vendors from allover the world, compete for offering the most innovative,reliable, and cheapest products, reducing the time-to-marketand development costs of software is an essential issue.This situation has promoted the use and integration ofreusable third-party components, known as Commercial-Off-the-Shelf (COTS) components, in software products [1].

The use of COTS components imposes a set of qualityconsiderations related to risks and uncertainties [2] thatshould be taken into account when integrating them in soft-ware solutions. Such risks derive from the lack of knowledgeand documentation that one may have about the COTScomponents structure, development process and availableAPIs. Given such considerations and the huge variety ofpotential COTS components found today in the market foreach specific need, the selection of the right alternative,

among those available, remains a complex and critical task,since a wrong choice may result in an unaffordable set ofeconomical, reputation and/or human losses [3].

Avoiding such wrong decisions is not easy and requiresthe consideration of performability requirements for all soft-ware components. This means that, in addition to conformwith their functional requirements, software componentsmust exhibit minimum levels of performance and depend-ability features to justify their quality. For any componentmeeting its functional specification, the greater its performa-bility, the better its quality. This motivates the importance ofevaluating the ability of each software component to meetits functional and performance specifications in absence,but also in presence of disturbances (faults and attacks)that may affect its regular behaviour during execution. Itis worth noting that this issue goes beyond the mere test ofthe recoverability properties exhibited by a component. Itencompasses the idea of including the notion of robustnessin the concept of software quality as suggested in [4].

Dependability benchmarking is a well-known techniqueto evaluate the impact of faults over the various aspectsconsidered for establishing the quality of software products,ranging from operating systems, to databases and webservers [5]. The basic idea is to define an evaluation processthat (i) proposes a set of performance and dependabilitymeasures to characterise the quality of a particular softwarecomponent, named the benchmark target, (ii) identifies theexecution profile and experimentation setup to deploy, (iii)establishes the experimental procedure to follow, the timeavailable for experimentation, and the procedure to retrieve,from the system under benchmarking, the measurementsrequired to deduce the performance and dependability mea-sures initially established, and finally (iv) provides guide-lines to exploit such measures in order to establish a rankingamong evaluated targets to select the most appropriate onefor the considered solution. Fault and attack injection arebasic techniques underpinning this type of benchmarking[6]. They are used to emulate during experimentation theoccurrence of disturbances and observe the resulting systemreaction, while keeping the reproducibility of experimentsand minimising the level of spatial and temporal intrusive-ness induced in the target system.

The evaluation framework established by dependability

2011 17th IEEE Pacific Rim International Symposium on Dependable Computing

978-0-7695-4590-5/11 $26.00 2011 IEEEDOI 10.1109/PRDC.2011.13

28

benchmarks is today partially integrated in the ISO/IECSystems and software Quality Requirements and Evaluation(SQuaRE) standard, which defines an evaluation module, theISO/IEC 25045, dealing with the assessment of the recov-erability of software systems in the presence of accidentalfaults. Although, the ISO/IEC 25010 module underlines theimportance of checking the security of software products,no attackloads are considered today in the current versionof the ISO/IEC 25045 standard module. On the other hand,the assessment process described in the module only makessense when the evaluated system provides fault tolerance,i.e. implements mechanisms to recover from the occurrenceof faults. The interest of dependability benchmarks in otherevaluation contexts, like those identified in the ISO/IEC25040 module (development, acquisition and external eval-uation), are not currently explored so far by the standard.

The present paper defends the interest of enriching theevaluation processes described in the SQuaRE 25040 stan-dard module with dependability benchmarks establishinghow to assess the quality of software components at runtimeattending to, among other criteria, performability considera-tions. To cope with such objective, Section II performs a crit-ical analysis of the ISO/IEC 25045, identifies its limitationsand establishes the interest of dependability benchmark-ing for the different evaluation processes specified in theISO/IEC 25040 standard module. Then, section III presentsthe various conceptual and technical modifications that oneshould introduce in such evaluation processes to exploit thepotentials of dependability benchmarks for software qualityevaluation, specially when a COTS must be selected. Thefeasibility of the approach is illustrated through the casestudy provided in section IV. Finally, section V reports onlessons learnt and provides conclusions.

II. CRITICAL ANALYSIS OF ISO/IEC 25045

ISO/IEC 25000 (SQuaRE) joins together the ISO/IEC9126 and 14598 standards, conserving the core of theproposed quality models and the guidelines that are relevantfor the software product quality evaluation. It defines asoftware quality model that has been used as a reference forevaluating the software (i) internal quality (static attributes ofa software product satisfy stated needs), (ii) external quality(a software product enables the behaviour of a system tosatisfy stated needs), and (iii) quality in use (a product canbe used to meet users needs with effectiveness, efficiency,freedom from risk and satisfaction). In addition, SQuaREcomplements the previous standard with some amendments,in particular, recognising the importance of security anddependability aspects [7].

Following this trend, the recently released ISO/IEC 25045standard [8] has been incorporated into the SQuaRE seriesto provide the basis for the quality evaluation under theviewpoint of recoverability (a product can recover affecteddata and re-establish the state of the system in the event

of a failure). This characteristic is of prime importance forevaluating the fault-tolerance capability of components andsystems, and thus the ISO/IEC 25045 proposes a disturbanceinjection methodology for its characterisation. This method-ology is structured in three phases: (i) the execution of theworkload, addressed to exercise the system in absence offaults (baseline phase), (ii) the execution of the very sameworkload but now in the presence of the faultload (testphase), oriented to detect and exploit system vulnerabilities,and (iii) the comparison of the set of measures obtained inboth baseline and test phases to assess the recoverabilityproperties of the system.

Although these standards represent a step forward towardsthe inclusion of dependability characteristics in the qualityof software products, there still exists a number of consider-ations that limits the completeness of the proposed method-ologies. Dependability benchmarking approaches may com-plement these standards to widen their scope, beyond themere testing of the recoverability properties of fault-tolerantsoftware, to support the performability evaluation of evennon-critical software products. The limitations that could betackled by adapting the exiting concepts from the depend-ability benchmarking domain include:

1) Recoverability measures: The ISO/IEC 25045 standardonly relies on two different measures (resiliency ratiobetween the throughput obtained in absence and presenceof faults, and autonomic recovery index degree of au-tomation in the system response against a threat) in order toevaluate the recoverability capabilities of components andsystems. However, research in the field of recoverabilityperformed from the dependability benchmarking viewpoint[9], highlights the importance of also characterising therecoverability of a system through its ability to detect,diagnose, repair, and restore the normal behaviour of thesystem. For instance, in case of suffering malicious attacks,the software product with the lowest detection and diagnosistimes could be able to react faster and, thus, limit the effectof these disturbances in the system. Likewise, low repairand restoration times are interesting to reduce the systemdowntime. Hence, recoverability attributes should compriseall those measures that may support the software productselection for fault-tolerant systems.

2) Fault injection for quality evaluation: Despite faultinjection being a well-known technique that has provenits usefulness in a wide range of software engineeringdomains [10], neither old ISO/IECs 9126 and 14598 northe recent ISO/IEC SQuaRE considered its use until theappearance of ISO/IEC 25045. However, its application hasbeen restricted to just the evaluation of the recoverabilitycharacteristics of the system. As dependability benchmark-ing states, the potential of fault injection goes beyond thisparticular dimension and is an indispensable tool to assessthe quantitative degradation of the system behaviour inpresence of disturbances. For instance, although different

29

products may achieve similar recoverability levels, the oneobtaining better capacity, resource consumption, or timingbehaviour in presence of disturbances is probably the bestsuited component for integration. Accordingly, the qualityevaluation process will benefit from applying fault injectiontechniques for the assessment of other reliability subcharac-teristics defined in ISO/IEC SQuaRE beyond recoverability,and other quality (sub)characteristics related to performanceefficiency and security.

3) Categories of faults: The ISO/IEC 25045 standardjust focuses on common high-level operational faults andevents such as unexpected shutdown, resource contention,loss of data, load resolution and restart failures (as depictedin Figure 1 with a black wide dashed line) . However, asshown in Figure 1, the disturbance scope in the context ofdependability benchmarking comprises not only accidentalfaults, but also malicious ones (attacks). Indeed, the consid-eration of attacks in the dependability evaluation process iscurrently of prime importance, as components and systemsare nowadays designed to ease their interconnection andfacilitate the exchange of data. Hence, the proposed faultinjection process must include also an attackload to evaluatethe impact of attacks on product quality and enable thedetection of existing vulnerabilities.

4) Statistical significance of evaluation: The statisticalsignificance of quality evaluation processes is an importantfactor when validating the conformity of the results. Forthis reason, ISO/IEC 25045 proposes executing the baselinethree times and considers it acceptable if the statisticalsignificance among these three runs does not differ morethan 5%. Although this procedure may be sound for thebaseline phase, as the workload execution in absence ofdisturbance could probably be repeatable, this is not thecase for the test phase. Even when executing the verysame workload, results may greatly differ when distinct

Faults considered in ISO/IEC 25045Development Operational

Faults

Internal Internal External

Human-made Human-madeNatural Natural Natural

Hardware SoftwareHardwareSoftware Hardw HardwHardw

MalMal NonMalicious

NonMalicious

NonMaliciousMalMal

NonMalicious

NonMal

NonMal

NonMal

NonDel

NonDel NonDel

NonDel

NonDel

NonDel

NonDel

Del Del DelDelDel Del Del Del

Acc Acc Acc Acc Acc Acc Acc Acc AccAcc AccInc Inc Inc Inc Inc Inc Inc Inc

Per Per Per Per Per Per Per Per Per Per Per Per Tr Per Tr Per Tr Per Tr Per Tr Per Tr Per Tr Per TrTr TrTrTr

Development Faults Physical Faults Interaction Faults11Mal: Malicious Del: Deliberate Acc: Accidental Inc: Incompetence Per: Permanent Tr: Transient

Persistence

Capability

Intent

Objective

Dimension

Phenomenological cause

System boundries

Phase of creation or occurrence

Figure 1. Disturbances in computing systems (adapted from [11])

types of faults and attacks are injected in different locations(injection points) and moments (injection times). Existingresearch on dependability benchmarking could be usefulto determine the required number of experiments to beperformed, according to the eligible targets and the selectedfault- and attackload, to ensure a certain level of consistencyand stability on results that may vary depending on thesystem characteristics.

5) Interpretation of measures: The results of the ap-plication of the ISO/IEC 25045 standard should be pre-sented as a report summarising the impact of faults inthe system, giving a special attention to those weaknessesthat should be improved. However, lessons learnt fromdependability benchmarking experience show that analysingthe huge amount of information resulting from the numberof considered measures, faults, attacks, and scenarios, isquiet complex. That is why the systematic use of exist-ing methodologies for guiding the interpretation of resultswhen considering multidimensional measures, as presentedin different dependability benchmarking studies, could bequite convenient for the comparison and selection of COTSsoftware components.

As can be seen, dependability benchmarking concepts,methodologies and approaches, may be adapted to addressthe limitations of the ISO/IEC SQuaRE standard, beyondthe mere assessment of the system recoverability, to supportthe evaluation, comparison and selection of COTS softwarecomponents and systems in presence of disturbances.

III. DEPENDABILITY BENCHMARKING APPROACH

Basically, a dependability benchmark is characterisedthrough different dimensions [5], which define (i) the ex-pected benchmark user and target system, (ii) the purposeof each considered measure and how to interpret them,(iii) how to configure the system under benchmarking and(iv) how to exercise the target component to experimentallyobtain the measurements required to deduce the proposedmeasures. Our proposal focuses on adapting all these notionsto introduce them into the general evaluation process definedin ISO/IEC 25040. The different stages of this dependabilitybenchmarking approach and the activities deployed on eachof them, depicted in Figure 2, are next detailed.

A. Establishment of the evaluation requirements

The proposed benchmarking methodology does not intendto automate the task of system operators when selectinga proper component at integration time, it rather tries tosupport and guide the comparison and selection of COTScomponents meeting the defined system requirements.

It must be noted that the software product quality re-quirements must also take into account the occurrence offaults and attacks in the system. Accordingly, acceptablelimits (quality thresholds) for performance degradation orincreased resource consumption, for instance, should also

30

!

"

"

#

$%&'"()*+,-.,

!%

%!%%

$%%&'")*+,-.

Figure 2. Stages defined in our dependability benchmarking approach

be defined for systems that may continue working on adegraded mode.

Following this reasoning, and as COTS components areusually provided as black-boxes, this dependability bench-marking approach will determine the impact of faults andattacks on the functionality provided by COTS components(coarse-grained viewpoint), rather than focusing on internalaspects.

B. Evaluation specification

The evaluation specification defines the most appropriatemeasures to evaluate the components requirement and theexperimental conditions (experimental profile) under whichthe experimentation will be performed.

1) Measures: Benchmark representativeness depends, toa large extent, on the set of measures selected to characterisethe system behaviour and are, therefore, dependent on theevaluator needs. In this sense, our methodology relies on asubset from the 8 characteristics and 31 subcharacteristicsof the software quality model defined by ISO/IEC 25010.As focusing on the impact of faults and attacks on thesystem behaviour, those quality characteristics related tothe system operation, like performance efficiency, and thoserelated to its dependability, like reliability and security, havebeen considered (see Table I). In practice, these abstract(sub)characteristics are quantified via their association with(one or more) measurable attributes, whose value is obtainedfrom measurement functions, as Figure 3 depicts.

Table IQUALITY CHARACTERISTICS AND SUBCHARACTERISTICS CONSIDERED

Performance efficiency Reliability SecurityTime behaviour Maturity ConfidentialityResource utilization Availability IntegrityCapacity Fault tolerance Non-repudiation

Recoverability AccountabilityAuthenticity

2) Decision criteria definition and application: Oncemeasures have been computed, it is necessary to determinetheir degree of goodness within the applicative context bymeans of decision criteria defined in terms of numericalthresholds. As a wide amount of measures could hinderthe results analysis, and so the components comparisonand selection, our proposal considers the aggregation ofmeasures as a valuable approach to complement the resultsobtained from evaluation and represent them in a friendlyand easy-to-interpret way.

The Logic Score of Preference (LSP) [12], which has beensuccessfully used for the quantitative quality evaluation of awide variety of software engineering products (ranging fromdatabases to web browsers), is a fuzzy-logic-based techniquethat computes the global score (S) of a component throughmeasures aggregation via Equation 1.

S = (k

i=1

wisri )

1r (1)

In Equation 1, each si represents an elementary score(referred to as elementary preference) obtained by each(sub)characteristic. Different criterion functions detail howto quantitatively evaluate each (sub)characteristic to es-tablish an equivalence between its value and the systemrequirements within a 0-to-100 scale. Additionally, eachelementary preference is associated to a weight wi whichdetermines its importance in the evaluation context and k isthe amount of elementary scores considered in the aggrega-tion, where

ki=1 wi = 1. The power r is a mathematical

artifice, described in detail in [12], aiming at satisfying thelogical relationship required by the evaluator for the differentelementary scores within the same aggregation block.

Software ProductQuality

QualityCharacteristics

QualitySub-Characteristics

Quality Measures

Quality MeasureElements

MeasurementFunction

indicate

indicate

are applied to

generates

composed of

composed of

Software ProductQuality

QualityCharacteristics

QualitySub-Characteristics

Quality Measures

Quality MeasureElements

MeasurementFunction

indicate

indicate

are applied to

generates

composed of

composed of

Software

Figure 3. Relationship between quality characteristics and measures(extracted from ISO/IEC 25020)

31

Finally, in order to complement the analytical results ofLSP, we propose the use of graphical approaches, like Kiviator radar diagrams [13], to represent the different measuresin an easy-to-interpret footprint.

3) Work-, fault- and attack-load: The representativenessof results obtained throughout the proposed benchmarkingprocess will depend, among other things, on the selectionof a workload that matches as close as possible the realoperation conditions of the final system. As real workloads(applications used in real environments) are really hard toobtain, and the representativeness of synthetic workloads isdoubtful from a dependability viewpoint, we propound theuse of carefully designed realistic workloads (based on realapplications) to obtain a suitable degree of representative-ness.

Likewise, the different disturbances a system may besubjected to during benchmarking should reflect as much aspossible those that may affect the system during its normaloperation in the applicative context it is deployed on (e.g.,web servers, databases, communication networks, etc). Typ-ically, only those accidental faults and stressful conditionsexperienced by systems in the field, which constitute thefaultload, were considered in the standard. However, thegreat impact that misuse and malicious attacks may haveon the expected behaviour of the system, has led as toconsider the necessity of incorporating an attackload intothe proposed benchmarking approach.

C. Evaluation design, execution and provision of results

The evaluation design must consider practical aspects,like the experimental representativeness, observability andintrusiveness, to successfully deploy the evaluation process.Among these aspects, determining the proper amount ofexperiments that should be performed to increase the con-fidence on the obtained results is of paramount importancebut has been obviated by the standard.

Our proposal relies on Equation 2, which is typically usedin statistical testing, to determine the number of single dis-turbances (N) to be injected for each considered fault/attackmodel. P is the lowest probability for a single disturbanceto affect a given element. Q reflects the probability ofstatistically targeting that particular element at least onceduring experimentation.

N(Q,P ) =ln(1Q)ln(1 P ) (2)

In case that it is not feasible to perform the requirednumber of experiments, due to its huge number or its longduration, we recommend that the number of experimentsperformed should be bounded by the manpower involvedand the time available for experimentation.

Once designed, the experimental evaluation can be per-formed following the proposed four-fold approach depictedin Figure 4.

Check phaseInitialisation phase

Set-up

Experiment 1 Experiment n

Warm-upWorkload generation

Monitoring

...Baseline phase

Experiment 1 Experiment n...Test phase

Set-up Warm-upWorkload generation

Fault injectionMonitoring

Figure 4. Experimental procedure

During the initialisation phase, the component underbenchmarking is configured and deployed within the system,which is also properly configured.

After this, the baseline phase, known as Golden Run inthe dependability domain [5], starts. It consists of a numberof successive experiments in which the system executesthe selected workload, in absence of disturbances, and thecomponent activity is monitored. Each experiment presentsan initial set-up time, required to restore the original stateof the system, followed by a warm-up time, devoted to leadthe system to a stable state.

The test phase corresponds to the execution of the work-load in the presence of the disturbance-load to evaluate theimpact of these disturbances on the system behaviour. Itsexecution is identical to the baseline phase, but introducingthe fault/attack into the system at a particular location ata given time. To ease the analysis of results, our approachconsiders that a test phase is related to one particular typeof fault/attack, thus requiring additional test phases in caseof considering more disturbances.

At the end of the experimentation, the check phase anal-yses all the log files generated with information collectedwhile monitoring the system activity. In order to reducethe degree of intrusiveness on measurements, all this in-formation will be offline processed, filtered and correlated,to extract the measures required to quantify the systembehaviour, their average values and variability indicatorssuch as the standard deviation or confidence intervals.

To ease the comparison process once the different eligiblecandidates have been evaluated, these results will be accom-panied by their LSP global score and their correspondingKiviat graphs.

IV. CASE STUDY

Wireless multi-hop (also known as ad hoc) networks arespontaneous, dynamic and self-managed networks compris-ing all sort of (mobile) devices which collaborate to maintainnetwork connectivity without any centralised infrastructure,thus saving the time and money required to deploy it.Routing protocols are essential software components incharge of establishing communication routes among nodes.Due to the nature of these kind of networks, it is useless toevaluate the quality of available routing protocols (more than50 specifications) during the network operation in absenceof disturbances, since the open wireless communication

32

Table IIMEASURES

!

"

#

#

#

$

#$

%$ %$ %$ %$ %$ %$ %$ %$ & ' ( ) ) * ) )

+

+

+

+

+

+

+

+

,

,

,

,

"

-,.

+-,

.

/0-,

.-/. /-/. /-/.

-*.

-*.

medium and the absence of fixed infrastructure (certifica-tion authorities) make them really sensitive to accidentalfaults (e.g. ambient noise or low battery) and maliciousattacks (e.g. jellyfish or tampering). Hence, the feasibilityand usefulness of the proposed dependability benchmarkingapproach will be shown through a case study involving theselection of the most suitable routing protocol implementa-tion to be deployed for a given ad hoc network.

A. Evaluation requirements

The purpose of this case study is the pilot deployment ofa simple ad hoc network within the Department of SystemsData Processing and Computers of our university (UPV) asa previous step to its deployment on scenarios where thequick, easy, and low-cost Internet connectivity is today anecessity. Prior experiences developed by members of thesame department in African schools can be consulted in [14].

As previously stated, system integrators must carefullyselect the most suitable routing protocol to provide qualitycommunications without delays nor interruptions. olsrd,developed by the most active and wider community de-voted to the development of open-source routing protocols(www.olsr.org), is the most extended implementation of thepopular Optimized Link State Routing (OLSR) protocol.Accordingly, three different versions of olsrd have beenselected as benchmarks targets: v.0.4.10 (released in 2006),v.0.5.6 (released in 2008) and current v.0.6.0 (released in2010). Their only dependencies are the standard C librariesand they all used the default configuration.

The system under benchmarking comprises 16 wirelessnodes: six (6) HP 530 laptops (1.46 GHz Intel CeleronM410 processor, 512 MB of RAM, internal IEEE 802.11b/gBroadcom WMIB184G wireless card, 4 Li-Ion cells battery(2000 mAh)) running an Ubuntu 7.10 distribution, and ten(10) Linksys routers (200 MHz MIPS processor, 16 MBof RAM, IEEE 802.11b/g Broadcom BCM5352 antenna)running a WRT distribution (White Russian). A similardeployment can be found in [15] to provide low-cost Internetaccess to an actual residential area.

Finally, taking into account the selected applicative con-text, the ad hoc routing protocol quality requirements will bedefined in terms of time behaviour (fast communications),resource utilisation (low resource consumption, battery inour case), and capacity (transferred information) for per-formance efficiency, availability (communication should bepossible most of the time) and recoverability (communica-tion must recover from unexpected events) for reliability,and integrity (transferred information must be right) forsecurity. To determine the requirements for each selectedsubcharacteristic, we would follow those defined in [16]for a voting application addressed to enable lecturers toget feedback from audience in auditoriums or conventioncentres. According to this information, an availability overa 90% will be excellent whereas below a 55% would be inthe bounds for an acceptable communication. The commonlimits of a medium quality communication with respectto its time behaviour range from 40ms to 400ms [17].As far as evaluating the rest of selected subcharacteristicswas not a requirement in [17], we will set their qualityrequirements as follows: acceptable transfer rate between110Kpbs and 180Kpbs, packet integrity among 95% and99%, route integrity between 20% and 50%, a resourceutilisation ranging from 11J to 15J , a recoverability among20s and 40s and route resiliency between 55% and 90%according to the behaviour experimented by network nodesin a previous fault-free experimentation.

B. Measures selection

The different subcharacteristics that have been previouslyselected should be quantified by means of a number ofdifferent measures. According to the applicative context ofthis case study, recoverability will be characterised throughpacket resilience and route restoration, whereas integritywill be defined in terms of packet integrity and routeintegrity. However, the time behaviour, resource utilisation,capacity, and availability, will be quantified by just onemeasure: route delay, energy consumption, route throughput,and route availability, respectively. The purpose, definition,

33

si =

0,

, Xmin

100,

Xminai Xmax Xmin

100

ai min

Xmax

ai Xmax

i

i

i

i

i i i

,Xmax ai

Xmax Xmin

ai min

Xmin Xmax

ai Xmax

100,

0,

100si =

i

i

i

i

i i i

(a) Increasing criterion function (b) Decreasing criterion function

X

X

< ai < < ai