SOQ: A Service-Oriented Quorum-Based Protocol for Resilient Real-TimeCommunication in Partitionable Networks

Bo Zhang and Binoy RavindranECE Dept., Virginia Tech, Blacksburg, VA 24061, USA

{alexzbzb,binoy}@vt.edu

Abstract

We consider efficient real-time communication mecha-nisms for applications in unreliable and partitionable net-works. Utilizing quorum systems, we present a quorum-based protocol called SOQ to let nodes update and queryservice information to a selected set of servers (a quorum).Due to the intersection property of quorums, nodes can ob-tain latest updated information by simply accessing a quo-rum. To make the protocol adaptive to network partitions,we propose update/query triggering mechanisms to deter-mine when nodes trigger updates/queries. A quorum accessstrategy for nodes to judiciously select a quorum to accessis designed so that the probability that a query returns thelatest service information is maximized. We give in-depthanalysis of the protocol, including the communication over-head, load and availability relationship of quorum systemsand timeliness analysis of distributed applications. Our ex-perimental studies show that SOQ is resilient to networkpartitions without incurring large overheads.

1 Introduction

Designing efficient real-time communication mecha-nisms in unreliable networks, e.g. mobile ad-hoc networks(MANETs) is very challenging. Due to the mobility and dy-namic characteristics of such networks, existing real-timecommunication solutions under fixed infrastructure-basednetworks cannot provide satisfying performance. The de-sired mechanism needs to provide end-to-end timing assur-ance for distributed real-time applications in such networks.

One of the unique challenges in these networks is par-titioning, which is the breakdown of a connected networktopology into two or more separate, unconnected topolo-gies. In fixed infrastructure-based networks, the occurrenceof network partition is highly unlikely and only possible ifbig parts of the infrastructure fail simultaneously.

Our design of real-time communication protocol for dis-tributed applications is based on the concept of a quorum

system, which is a family of subsets that always intersect.For example, consider a network of N nodes. We select nnodes (n � N ) from the network as servers. The remain-ing nodes are ordinary nodes or clients. A quorum systemis formed by the selected servers. Each update by a client issent to a subset of servers (a quorum). Similarly, each queryis also sent to a quorum. Due to the intersection propertyof quorum systems, there is high probability that at leastone server will keep the latest information of the previousnode. Thus, the appropriate destination can be discovered.The challenge, however, is to design efficient update/querymechanism as well as quorum access strategy so that theprobability of a hit (a query returning the latest record of anode) is maximized even when the network is partitioned.

SOQ performs efficiently both when the network isconnected and when it is partitioned. Successive up-dates/queries by a node need not be sent to the same subsetof servers. This strategy can efficiently balance the load onthe servers. On the other hand, because the set of reach-able servers changes with time, a node has to determine thereachable subset of servers before each update/query basedon its recent information about server reachability.

We make the following major contributions in this paper:(1) We design a service-oriented quorum-based proto-

col: SOQ for resilient real-time communication of dis-tributed applications in partitionable network;

(2) We design an efficient update/query triggering pol-icy and quorum access strategy for SOQ to improve the per-formance of the protocol;

(3) We analyze the protocol from various aspects includ-ing its communication overhead, load and availability re-lationship of quorum systems and timeliness analysis fordistributed applications. The bound of the load of quorumincurred by SOQ is determined by partial quorum availabil-ity. The expected delay for a distributed application is de-termined by the quorum placement and access strategy; and

(4) Our experimental studies show that when the proba-bility of network partition is large enough, the query accu-racy and hit probability drops significantly, and the load ofthe quorum system arises dramatically.

2008 14th IEEE Pacific Rim International Symposium on Dependable Computing

978-0-7695-3448-0/08 $25.00 © 2008 IEEE

DOI 10.1109/PRDC.2008.53

192

The rest of the paper is organized as follows: In Sec-tion 2, we discuss the related work. Section 3 discussesquorum preliminaries. We present the system model in Sec-tion 4. The protocol is described in Section 5. We analyzethe protocol in Section 6. In Section 7 we present our ex-perimental studies. We conclude in Section 8.

2 Related Work

Quorum system is a basic tool for reliable agreement indistributed systems [1, 12, 14]. Researchers have also de-signed quorum systems in a dynamic environment, e.g. [4,8]. To apply quorum systems in partitioned networks, Her-lihy [7] presented dynamic quorum adjustment method forpartitioned data. This method permitted an object’s quo-rum to be adjusted dynamically in response to failures andrecoveries. A transaction that is unable to progress usingone set of quorums may switch to another, more favorableset, and transactions in different partitions may progress us-ing different sets. Karumanchi et al. [10] proposed strate-gies that use local knowledge about the reachablility to ju-diciously select quorums in partitionable mobile ad hoc net-works. They designed an update/query protocol to let nodesupdate their locations when needed.

Quorum system is widely used in ad hoc networks. Oneof the most popular applications is implementing locationservice. In [4], a uniform random quorum system is usedfor mobility management. Nodes form a virtual backbone.When a node moves, it updates its location with one quorumcontaining the nearest backbone node. Each source nodethen queries the quorum containing its nearest backbone forthe location of the destination. Luo et al. [11] present aProbabilistic quorum system for ad hoc networks (Pan), acollection of protocols for the reliable storage of data in mo-bile ad hoc networks. A gossip-based protocol is designedfor quorum access and an asymmetric quorum constructionis applied. These work has noticed the highly dynamic andunpredictable topology changes in ad hoc networks. Ourwork differs from them in that we focus on quorum-basedprotocol in network partitions.

To the best of our knowledge, our work is the first oneto apply quorum systems for distributed real-time appli-cations. Han et al. [5] [6] present gossip-baseds protocolto counter network failures and message losses. In thiswork, the possibility of network partitions is “coarsely"handled by lumping the probability of message losses (dueto partitions, among other things) in one of the input vari-ables of their analysis. Our work is based on our pastwork [16], where we designed a quorum-based gossip pro-tocol based on [5]. In [16], we applied quorums to limit therange of each gossip round and thereby reduce the messageoverhead, resulting in improved timeliness behavior of dis-tributed tasks. Based on this work, in this paper, we design

a quorum based update/query protocol which is adaptive tonetwork partitions and do not need to gossip the query inthe network. In this way, the message overhead is highlyreduced and timeliness behavior is further improved.

3 Quorum Preliminaries

Given a universe U of elements, a quorum system Q ={Q1, ..., Qm} on U is a family of subsets of U such that anytwo quorums Qi and Qj have a non-empty intersection [2].Quorum systems are widely used in distributed systems forachieving mutual exclusion, consistent data replication, anddissemination of information. In typical quorum-based al-gorithms, each client accesses the system by accessing allthe elements in some quorum Qi belonging to Q. The in-tersection property ensures that any Qi would suffice to op-erate on behalf of the system. Usually, the client chooses Qi

from a probability distribution p : Q → [0, 1] over Q; thisp is called the access strategy for the quorum system. Theaccess strategy is typically chosen based on two concerns:the load and availability of the quorum system.

Definition 1. Given a quorum system Q on U , the load onu ∈ U of access strategy p is ΣQi�up(Qi). The load ofthe quorum system is the load of the most heavily loadedelement u ∈ U : LQ = maxu∈U ΣQi�up(Qi).

Definition 2. Given a quorum system Q on U and a clientvi, the availability of the quorum system is defined as theratio of the number of quorums reachable from vi to thetotal number of quorums.

Since the accesses of one quorum by clients (which arethemselves nodes in the network) have to be implementedby messages sent along the network, the performance ofquorum-based systems now crucially depends on the delaysintroduced by these accesses [3]. In fact, we would like thelogical quorums Qi ∈ Q to be mapped to closely clusteredphysical nodes in the network so that we do not incur largedelays in trying to reach far-flung parts of the network.

We introduce the concept of Grid quorum systems forour discussion, which is defined as follows:

Definition 3. On a universe U of k2 elements, a Grid quo-rum system is composed of k2 elements laid out on a k byk square grid M , and each quorum Q from Q is formed bytaking all the elements from some row and some column ofM . Hence each quorum has 2k− 1 elements, and there arek2 quorums in Q.

The uniform access strategy yields the optimal load forthe Grid. For a Grid quorum system of n elements, thisstrategy will lead to the optimal load of O( 1√

n). [13]

193

4 Models

Application Model. We model a distributed applica-tion as T = {t1, t2, ...tn}, where ti is called an activity.An activity is composed of a sequence of sections, i.e.,ti = [si

1, si2, ...s

im]. Sections of an activity must be exe-

cuted sequentially, i.e., si1 ≺ si

2 ≺ ... ≺ sim. A section con-

stitutes the portion of the service’s execution on a node. Itis invoked on a service requestor, and finished on a serviceprovider. An activity’s most recent section — the activity’scurrent execution locus — is called the activity’s head, andthe node hosting the head is called the activity’s head node.The head of a activity is the only section that is active.

When a section is created on its service requestor, it doesnot know the identity of its service provider. Hence, the ser-vice requestor needs to discover the service provider whichprovides the requested service in the network. Such a sec-tion involves three steps: (1) service requestor queries; (2)service provider replies; and (3) service executions. The se-quence is repeated in a particular order, and a set of suchsequences that runs concurrently forms an application.

For example, in our previous mobile chess game sce-nario, a chess game held between two players is an activ-ity. A set of such concurrent activities, i.e., a chess gametournament, forms a distributed online application.

Network Model. We assume a network of N nodes. Aset of servers is selected from the network and a quorumsystem Q = {Q1, ..., Qm} is constructed. The sets Qi isconstructed a priori and every node knows the membershipof these sets by broadcast at the start of the protocol. Givenn servers, it is possible to form quorums of size O(

√n).

A basic unicast routing protocol such as DSR [9] isassumed to be available for packet transmission betweennodes. MAC-layer packet scheduling is assumed using aCSMA/CA-like protocol (e.g., IEEE 802.11). We assumethat node clocks are synchronized (e.g., [15]).

We assume that nodes may fail by crashing, links mayfail transiently or permanently, and messages may be lost.Network partitions can occur, which is the breakdown ofthe connected network topology into two or more separate,unconnected topologies. When the network is partitioned,nodes in different partitions cannot communicate.

5 The SOQ Protocol

5.1 Basic Protocol

The basic protocol of SOQ consists of three parts: ser-vice information update, service information query, andsection execution. For each section of an activity, the ser-vice requestor discovers the appropriate service providerbased on service update/query protocol. We design up-date/query protocol for service information update and

query, and section execution protocol for service executedon the service provider.

Service Information Update: When a node vi wishesto update its service information, it timestamps the da-tum with its clock value. The format of the datum is< ID(i), addr(i), service(i), timestamp(i) >. The IDfield is unique for each node, e.g., the MAC address, whichis used to identify the node and doesn’t change. Theaddr field contains the network address the node designatedfrom the network. When the network partitions or mergeswith other networks, the network address of the node maychange. The service field represents the current availableservice that the node provides. For example, an idle clientin the mobile chess game may provide “join a new game"service. The service field may change depending on thestatus of the client.

When an update is triggered on a node, it selects a quo-rum Qi from the set of quorums and sends an UPDATEmessage, timestamped with its local clock value, to allservers in the quorum. The servers, on receiving the UP-DATE message, overwrite their old copy of the data itemwith the new copy. If they do not have an old copy of thatdata item, they simply add the information received in themessage to their database.

Let an UPDATE message be first sent to quorum Qi.Later, let another UPDATE, for the same data item, be sentto quorum Qj . Then, all servers in the set Qi − Qj haveoutdated versions of the data item, while all servers in Qj

have the latest version of the data item.Service Information Query: When a query is triggered

on a node, it selects a quorum Qj and sends a QUERY mes-sage to all servers in the quorum. When a server receives aQUERY for a specific service and has a copy of it, the serversends a REPLY containing the information along with thetimestamp associated with the datum. Otherwise, the serversends a NULL reply. By receiving all the REPLY messages,the service requestor selects the value of the datum with thegreatest timestamp.

As two quorums always intersect in the fixed network,the set of queried servers is bounded to contain at least oneserver that belonged to the quorum that received the latestupdate. Hence, each query returns the latest value of thequeried data item.

Section Execution: When the current service requestorselects the data, it extracts the addr field of the data andsends that node the request to execute the service. Therequested node, upon receiving the request, will checkwhether it can execute the requested service. If so, it ex-ecutes it and sends back the acknowledgement. If not, italso sends back the acknowledgement that it cannot pro-vide the requested service. The service requestor will trig-ger the service information query again to discover the ser-vice provider. The section is successfully executed until the

194

requested service is executed on the service provider.

5.2 Update/Query Triggering Mechanism

In unreliable and partitionable networks, we design ap-propriate update/query triggering mechanisms for SOQ soas to mitigate the impact of network partitions.

Update Triggering: The aim for nodes updating serviceinformation is to propagate the information such that othernodes, on querying for information, obtain as recent a ver-sion as possible. We propose following three policies totrigger updates: (1) The service type provided by the nodehas changed since the last update; (2) The network addressof the node has changed since the last update; and (3) Acertain number of links incident on it have been establishedor broken since the last update

The first policy is very straight-forward. The service in-formation must be updated whenever it changes. For thesecond strategy, when the network partitions, the networkaddress of the client may change. Thus, an update is trig-gered to send the server its new network address so that itcan be communicated with other nodes. The last update pol-icy is motivated by the observation that in ad-hoc networks,the frequency of updates should reflect the dynamism exhib-ited by the network. It is an attempt to capture the relativechange in topology.

Query Triggering: The queries will only be sent by ser-vice requestors. We propose following three policies to trig-ger queries: (1) A section is invoked on a service requestor;(2) No available service update information received sincethe last query; and (3) The selected service provider cannotbe reached, or doesn’t provide the service any more.

When a section of an activity is invoked, a query is trig-gered to discover the proper service provider. When there isno service information matched in the servers that it queries,there are two possibilities: (1) There is no node provid-ing the requested service in the service requestor’s networkpartition; or (2) The proper service provider exists, but thequeried servers do not have the latest information for somereason. The second phenomenon seems to contradict thequorum intersection property. However, it is quite possiblein partitioned networks. For example, let the node update itsservice to Qi, and a service requestor sends the query to Qj .Hence, the updated information will be saved on Qi ∩ Qj .If those servers belong to another network partition, the ser-vice requestor will not get the correct information. Thus,the second policy is applied to trigger a query again.

The third policy is applied in the worst case: the selectedservice provider cannot be reached or does not provide theservice. This is because the service provider is selected bystale information. The selected node is either located in an-other network partition, or cannot provide the requested ser-vice. Thus, the service requestor has to re-trigger a query.

5.3 Quorum Access Strategy

From the previous discussion, we observe that in a parti-tionable network, not all the servers in the selected quorumare reachable from the node that sends the update or query.Thus, it is crucial to design an efficient quorum access strat-egy to maximize the probability of a hit — i.e., a queryreturning the latest information about a node.

Let node vi select quorum Qi to update the service infor-mation. If some elements of Qi are in a different partitionfrom vi at the time of the update, they will not receive theupdate. We denote the set of servers which receive the up-date as Q

′i. Subsequently, let another node vj select a quo-

rum Qj to query the service information, and we denote theset of servers which receive the query as Q

′j . Thus, there is

a possibility that the query may not return any information,or return stale information.

To alleviate this problem, we need to select quorums tomaximize the hit probability. The idea is to select quorumsso that the sets Qi−Q

′i and Qj−Q

′j are as small as possible.

The smaller these sets, the greater the probability of the setof reachable queried servers intersecting with the set thatreceived the latest update.

We use the concept of disqualified list from [10] to selectservers for updates and queries. Node vi maintains a dis-qualified list, DQLi, containing servers that vi believes areunreachable. The node selects servers for updates/querieson the basis of DQLi’s composition.

Initially, DQLi is empty. We propose the followingsteps for updates and queries based on the disqualified list:(1) Node vi first eliminates all quorums that have at least

one node in DQLi;(2) If there are quorums remaining, one of them is ran-

domly selected and messages are sent to servers in thisquorum;

(3) If all quorums have at least one node in DQLi, thenquorums having the smallest number of elements inDQLi are located. One of them is randomly selectedand messages are sent to servers in this quorum;

(4) If at least one server sends an acknowledgement in thecase of an update, or a non-NULL time-stamped valuein the case of a query, the operation is said to succeed;and

(5) If a reply is not received from a server within Ttimeout,node vi concludes that this server is not reachable andadds it to the DQLi. Usually we set Ttimeout =Ω(E[δ(vi, vj)]), where δ(vi, vj) is the round-trip timebetween nodes in the network. Once a server has beenadded to DQLi, it stays in it for a disqualified duration,δDQL. At the end of δDQL, the server is removed fromDQLi. The value of δDQL can be adjusted to keepbalance between updating disqualified list in time andreasonable communication overhead.

195

6 Protocol Analysis

6.1 Communication Overhead

Given n servers, there exists quorum formation schemesthat give quorums of size O(

√n). Even if an update and

query requires a constant number of retries before success,the communication complexity is O(

√n). As n is usually

much smaller than the total number of nodes N , the com-munication overhead is reasonable.

6.2 Quorum Load and Availability

The load of the servers in the quorum system describesthe probability that a server is selected in the given accessstrategy. The load of the most heavily loaded element eval-uates the access strategy. When the quorum system is fullyavailable, i.e., no partition exists in the quorum system,SOQ’s quorum access strategy transforms to uniform ac-cess strategy and the load of all elements is O( 1√

n). When

the quorum system partitions, the load of the servers willincrease due to the decreased availability of quorums. Thefollowing examples reveal the relationship between loadand availability of Grid quorum system.

Figure 1. Server placement in 4 × 4 Grid quo-rum system with least quorum availability

An example of worst case of quorum availability of 4×4Grid quorum system is shown in Figure 1. Elements markedby “P" represent servers in other network partitions at thetime the node selects quorums to send update/query. Shadedelements represent the set of servers nodes can select to ac-cess according to our quorum access strategy. In the quorumsystem with ε such servers, Figure 1 illustrates the place-ment of these servers in the Grid quorum system whichresults in least quorum availability. In this scenario,

√n

servers in other partitions are adequate to make each quo-rum contain one element in other network partitions, andnodes in this network partition cannot find a quorum fromwhich it can access all its elements. By SOQ’s quorum ac-cess strategy, clients will randomly choose a quorum to ac-cess. Specifically, for a Grid quorum system of n servers, ifwe use ε to denote the number of servers in the disqualifiedlist of the service provider, We have following lemmas:

Lemma 1. The number of elements in other network parti-tions and the size of the quorum system determine the lowerbound of the availability of Grid quorum systems.

Proof. In the worst case, when ε ≤ √n−1, if we use AQ to

denote the availability of Grid quorum system Q, we have:

AQ =No. of available quorums

Total No. of quorums=

(√

n − ε)2

n(1)

When ε ≥ √n,

AQ = 0 (2)

Hence, in a quorum system where ε servers belong toother network partitions, the lower bound of quorum avail-ability is provided by Equations 1 and 2.

Based on the quorum access strategy of SOQ, the loadincurred by this server placement is:

LQ =2(√

n − ε′) − 1

(√

n − ε′)2, ε

′= ε mod

√n (3)

This server placement, however, does not incur the heav-iest load among all possible situations. In fact, when ε =i√

n, i = 0, 1, 2..., each quorum under this placement con-tains i unreachable servers. In this case, the quorum accessstrategy becomes the uniform access strategy.

Figure 2. Server placement in 4 × 4 Grid quo-rum system with heaviest quorum load

On the other hand, the placement incurring the heaviestload does not necessarily imply the worst quorum availabil-ity, as shown in Figure 2. Even if there is always at least1 quorum available, the load of the quorum will increaseto 1 when the number of unreachable servers increases. Inthis scenario, at least

√n − 1 unreachable servers incur the

heaviest load of the quorum system.

Lemma 2. The number of elements in other network parti-tions and the size of the quorum system determine the upperbound of the load of Grid quorum systems.

Proof. Based on the quorum access strategy of SOQ, whenε ≤ √

n − 1, we have:

LQ =2(√

n − ε) − 1(√

n − ε)2(4)

196

When ε ≥ √n − 1,

LQ = 1. (5)

Hence, in a quorum system where ε servers belong to othernetwork partitions, the upper bound of the load of the quo-rum system is provided by Equations 4 and 5.

When ε ≤ √n − 1, the availability of this placement is

the same as that given by Equation 1. When ε ≥ √n, we

have AQ = 1n . Similar to the previous case, this placement

does not incur the worst quorum availability. The placementalways yields at least one quorum available, which impliesthat clients can always find a quorum in which all elementscan be accessed.

Thus, we note that there exists some relationship be-tween the availability and the load of the quorum system.We can extend the definition of quorum availability to par-tial quorum availability to deduct this relationship:

Definition 4. Partial Quorum Availability: For a quorumsystem with quorum size k, partial quorum availability is avector A = {A0, A1, ..., Ak−1}, where Ai represents theportion of quorums where k − 1 elements are available.

From this definition, we see that∑

0≤i≤k−1 Ai = 1.With this definition, we have the following theorem:

Theorem 3. The lower and upper bounds of the load of theGrid quorum system operating on SOQ is determined by thepartial availability and the size of the quorum system.

Proof. By SOQ’s quorum access strategy, the set of quo-rums with the least number of unreachable elements is cho-sen. For Grid quorum systems, we can therefore derivethe relationship between load and partial quorum availabil-ity: LQ = ai+aj−1

Ai∗n , where i∗ = arg minAi=0 i; aiaj =Ai∗n; 1 ≤ ai, aj ≤ √

n − 1.Note that ai and aj are integer factors of Ai∗n. Thus,

the load of the quorum can have several possibilities basedon the knowledge of partial quorum availability. This is be-cause, we only know the number of reachable quorums, butdo not know the configuration of these quorums. For exam-ple, if Ai∗n = 12, we have no idea whether these quorumsare formed by 3 columns and 4 rows or 2 columns and 6rows. Generally speaking, different quorum configurationscan result in different loads.

Although the load of the quorum cannot be determinedthrough the knowledge of quorum availability, we canlower-bound and upper-bound it. Note that ai and aj areintegers. Thus, we have: 2√aiaj� ≤ ai + aj ≤ ai∗ + aj∗ ,where ai∗ = maxai≤√

n−1 ai. Note that aiaj = Ai∗n.Hence, the load of the quorum system with the quorum ac-cess strategy is bounded by:

2√Ai∗n − 1�Ai∗n

≤ LQ ≤ ai∗ + aj∗ − 1Ai∗n

(6)

6.3 Timeliness Analysis

The performance of SOQ protocol depends on howfast the application is executed. Hence, the expectedtime needed for section executions significantly affects theglobal performance. We have the following theorem:

Theorem 4. The expected delay incurred on each section isbounded by three factors: (1) The expected delay for a ser-vice requestor to access quorums; (2) The expected numberof query attempts for a hit; and (3) The expected number ofqueries returning stale information.

Proof. Using Di to denote the delay incurred on ser-vice provider vi, we have Di =

∑qi

j=1 δ(vi, Qj) +∑hi

k=1 δ(vi, vk), where δ(vi, Qj) denotes the access delayfrom node vi to quorum Qj . qi is the number of query at-tempts for a hit, and hi is the number of times vi selectinga service provider.

Nodes access a quorum by reaching all its elementsδ(vi, Qj) = maxu∈Q δ(vi, u). Then, the expected delay(under specific quorum access strategy p) for vi to accessQ is Δ(vi) =

∑Q∈Q p(Q)δ(vi, Q).

In the best case, the service requestor gets the latestservice information by just sending one query. Thus, theoptimal delay of node vi is D0

i = δ(vi, Qo) + δ(vi, vo).Thus, the penalty delay incurred on the service requestor is:D1

i =∑qi−1

j=1 δ(vi, Qj) +∑hi−1

k=1 δ(vi, vk).Note that qi −1 is the number of query attempts before a

hit, and hi−1 is the number of times that a service requestorselects a service provider based on stale information. So wehave qi ≥ hi. By the quorum intersection property and thetriangle inequality, we have δ(vi, vk) ≤ Δ(vi) + Δ(vk).Thus, the delay incurred on service provider vi is boundedby Di ≤

∑qi

j=1 δ(vi, Qj) + hiΔ(vi) +∑hi

k=1 Δ(vk).By taking expectations over quorum access strategy p:

E(Di) ≤ E(qi + hi)Δ(vi) +E(hi)∑

k=1

Δ(vk). (7)

Hence, E(Di) is bounded by Δ(vi), E(qi) and E(hi).

Δ(vi) is determined by the network topology and quo-rum system placement. When the servers are selected andthe quorum system is determined, it solely depends on thenetwork topology. E(qi) and E(hi) depend on the specificquorum access strategy. The SOQ quorum access strategymaximizes the probability of intersection between a queryand an update based on the local knowledge of reachability.

From Equation 7, the expected delay for activity ti isE(Di) =

∑sj∈ti

E(Dj). Hence, the expected delay for an

application T is E(T ) = maxti∈T E(Di).

197

7 Experimental Studies

We conducted simulation experiments to evaluate the ac-curacy of the query, the hit probability, and the load of thequorum system. We considered a network with 200 mobilenodes randomly distributed in a round region with radius50 length units. Nodes were allowed to move within thisregion. Some nodes were selected as servers to form a Gridquorum system. We varied the number of servers to form4× 4, 6× 6, and 8× 8 Grid quorum systems. We simulatednode mobility by setting the duration for which a node staysat a location to be exponentially distributed with mean 0.5time units. After this duration, the node randomly selectsanother location within the round region that is at most 10length units away from its current location.

We considered 8 types of service, and each type of ser-vice is provided by 4 randomly selected nodes in the net-work. The duration for which a node provides a servicewas exponentially distributed with mean 1 time unit. In ourexperiments, we set δDQL = 0.25 time units.

0 5 10 15 20 25 300

1

2

3

4

5

6

7

8

9

10

Wireless Communication Range

No.

of Q

uerie

s

4*4 Grid6*6 Grid8*8 Grid

Figure 3. Number of Attempts for a Query

The average number of attempts for a successful querywith different wireless communication ranges and differ-ent Grid quorum system sizes is shown in Figure 3. Thedata points were obtained from 100 successful queries. Asuccessful query returns a non-NULL data containing therequested service information. When the wireless commu-nication range is equal to 25, 20, and 15 length units, just1-2 attempts are only needed for a successful query, regard-less of the applied Grid quorum system. This is because,for these wireless communication ranges, the probability ofnetwork partitioning is relatively low. Even if the networkpartitions, the partitions may get merged quickly. However,the number of attempts increases dramatically when therange equals 10 and 5 time units. For these values, the prob-ability of network partitioning increases significantly andmost of the queries return NULL data. For three differentsizes of Grid quorum systems, when the wireless communi-cation range is large (e.g., 15, 20, 25), the average numberof attempts are almost the same. When the communication

range is small (e.g., 5, 10), 4 × 4 Grid quorum needs moreattempts for a successful query because the probability thatno server exists in a network partition increases. Thus, theGrid quorum system of smaller size suffers more than quo-rum systems of larger size.

0 5 10 15 20 25 300

2

4

6

8

10

12

14

16

18

20

Wireless Communication Range

No.

of Q

uerie

s

4*4 Grid6*6 Grid8*8 Grid

Figure 4. Number of Attempts for a Hit

Under the same condition, Figure 4 shows the averagenumber of query attempts for a hit, calculated from 100 hits.A successful query does not imply a hit due to the existenceof stale information. When all queried servers return staleservice information or NULL data, the querying node willselect the stale information with the highest timestamp. Inthis case, the query is successful but the selected target isnot a proper service provider. In other words, a “success-ful" query doesn’t necessarily return “correct" data. Hence,the average number of attempts for a hit is larger for a suc-cessful query, as shown in Figure 4. Similar to Figure 3,4× 4 Grid quorum system suffers the most from increasingprobability of network partitioning among the three Gridquorum systems.

23

45

6

5

10

15

20

250

20

40

60

80

100

No. of sections

Wireless Communication Range

Suc

cess

Rat

io (

%)

Figure 5. Success Ratio of Distributed Tasks

Figure 5 illustrates the relationship between the successratio of distributed activities and different wireless commu-nication ranges. The success ratio of an activity is the prob-ability that an activity is successfully executed in a desig-nated time period d. In our experiments we set d = 100

198

time units. Distributed activities are composed of differentnumber of sections. In our experiments, we set the numberof sections from 2 to 6, under 5 different values of wirelesscommunication range. Success ratio is calculated from 100tasks with the same parameters. The figure shows that thetask success ratio decreases when the wireless communica-tion range decreases and the number of sections increases.When network partitioning occurs frequently, e.g., when thewireless communication range equals 5 or 10 length units,the success ratio drops dramatically. Generally, the successratio suffers more from decreasing wireless communicationrange than from increasing number of sections.

0 5 10 15 20 25 300

10

20

30

40

50

60

70

80

90

100

Wireless Communication Range

Load

(%

)

4*4 Grid6*6 Grid8*8 Grid

Figure 6. Load of the Grid quorum system

We measured the load of different sizes of Grid quorumsystems. Figure 6 shows this simulation result. The datapoint is the mean value of 100 queries. The figure illus-trates that a quorum system of larger size implies a smallerload, which verifies the analytical result. As we expected,the load of the server increases significantly when the prob-ability of network partitioning increases. This is because,when network partitioning occurs, less number of serversare available in one network partition, which increases theprobability that a server is selected to access.

8 Conclusions and Future work

In this paper, we designed the SOQ protocol for real-timecommunication in unreliable and partitionable networks.We show that SOQ is resilient for real-time communicationswithout incurring large communication overhead. Further,the load of quorum systems incurred by SOQ is determinedby the availability of the system, and that the delay neededfor an application execution is determined by the quorumsystem placement and access strategy. When the probabil-ity of network partition is large enough, the query accuracyand hit probability drops significantly, and the load of thequorum system increases dramatically. We also observe thatthe larger size of quorum systems improves the query accu-racy and hit probability when network partitions occur and

reduces the load of quorum systems. On the other hand, itwill also increase the communication overhead.

Our work can be extended immediately by relaxing theassumption on the (known) selection of servers. Anotherassumption that we made is that the quorum placement isknown, which represents the map from elements of quo-rums to physical nodes in the network. This assumption,however, can be also relaxed. Proper quorum placementcan be determined to minimize the average delay for nodesto access quorums.

References

[1] A. E. Abbadi, D. Skeen, and F. Cristian. An efficient, fault-tolerant protocol for replicated data management. In ACMSIGACT/SIGMOD PODS, pages 215–229, 1985.

[2] Y. Amir and A. Wool. Optimal availability quorum systems:theory and practice. Infor. Process. Lett., 65(5):223–228,1998.

[3] A. Gupta, B. M. Maggs, F. Oprea, and M. K. Reiter. Quorumplacement in networks to minimize access delays. In ACMPODC, pages 87–96, 2005.

[4] Z. Haas and B. Liang. Ad hoc mobility management withuniform quorum systems. IEEE/ACM Transactions on Net-working, 7(2):409–418, 1999.

[5] K. Han, B. Ravindran, and E.D.Jensen. Rtg-l: Dependablyscheduling real-time distributable threads in large-scale, un-reliable networks. In IEEE PRDC, 2007.

[6] K. Han, B. Ravindran, and E.D.Jensen. Rtmg: Schedulingreal-time dstributable threads in large-scale, unreliable net-works with low message overhead. In IEEE ICPADS, 2007.

[7] M. Herlihy. Dynamic quorum adjustment for partitioneddata. ACM TODS, 12(2):170–194, June 1987.

[8] S. Jajodia and D. Mutchler. Dynamic voting algorithms formaintaining the consistency of a replicated database. ACMTODS, 15(2):230–280, June 1990.

[9] D. B. Johnson, D. A. Maltz, and J. Broch. Dsr: The dy-namic source routing protocol for multihop wireless ad hocnetworks. In Ad Hoc Networking, pages 139–172. Addison-Wesley, 2001.

[10] G. Karumanchi, S. Muralidharan, and R. Prakash. Informa-tion dissemination in partitionable mobile ad hoc networks.In IEEE SRDS, 1999.

[11] J. Luo, J.-P. Hubaux, and P. T. Eugster. Pan: providing re-liable storage in mobile ad hoc networks with probabilisticquorum systems. In MobiHoc, pages 1–12, 2003.

[12] D. Makhi and M. Reiter. Byzantine quorum system. Dis-tributed Computing, 11(4):203–213, Octobers 1998.

[13] M. Naor and U. Wieder. Scalable and dynamic quorum sys-tems. In ACM PODC, 2003.

[14] M. Naor and A. Wool. The load, capacity, and availability ofquorum systems. SIAM Journal on Computing, 27(2):423–447, April 1998.

[15] K. Romer. Time synchronization in ad hoc networks. InMobihoc, pages 173–182, 2001.

[16] B. Zhang, K. Han, B. Ravindran, and E.D.Jensen. Rtqg:Real-time quorum-based gossip protocol for unreliable net-works. In ARES, pages 564–571, 2008.

199