A Data Sharing Facility for Mobile Ad-Hoc Emergency and Rescue Applications

Thomas Plagemann, Ellen Munthe-Kaas, Katrine S. Skjelsvik, Matija Puzar,

Vera Goebel University of Oslo, Department of Informatics

{plageman, ellenmk, katrins, matija, goebel}@ifi.uio.no

Ulrik Johansen, Joe Gorman

SINTEF {ulrik.johansen,

joe.gorman} @sintef.no

Santiago Pérez Marín Telefonica

spmarin@tid.es

Abstract

Efficient information sharing is very important for emergency and rescue operations. Mobile Ad-Hoc Networks (MANETs) are often the only network environment for such operations. We have developed the MIDAS Data Space (MDS) to transparently share information among rescue applications in such environments. To achieve the required level of availability for important information, MDS performs optimistic replication. The problems caused by optimistic replication, like consistency management, are not solved by standard solutions; instead we employ tailor-made solutions for emergency and rescue applications.

Keywords: Emergency and rescue operations, mobile ad-hoc networks, tailor-made middleware, shared data space.

1. Introduction and background

Organizations involved in rescue operations of recent disasters have recognized that “the flow of information throughout the disaster cycle is crucial for effective humanitarian operations” [1]. It is obvious that data networks, like Mobile Ad-Hoc Networks (MANETs), can help to increase the efficiency of the collaborative work of the rescue personnel. In order to enable programmers to develop useful applications for the emergency and rescue domain, a kind of middleware layer for information sharing over MANETs is needed to hide most of the peculiarities of MANETs and to allow the programmer to focus on the application domain issues. It is the goal of this paper to show how we address this need in the MIDAS project [2] with the MIDAS Data Space (MDS). The core

purpose of MDS is to provide applications running on multiple nodes and at different locations, with a mechanism to share information by inserting data to and retrieving data from a shared data space. To application programmers, MDS is supposed to resemble a relational database, with functions to insert data to and retrieve data from shared tables. In the design we take advantage of properties of the application domain and provide a solution that targets a highly specialized ad-hoc networking system.

The challenging research problems that need to be solved by MDS, have two different sources: the networking environment and the application requirements. The main problem with the networking environment in emergency and rescue operations, is that no assumptions can be made about existing infrastructure in the emergency area. Therefore, MDS must work over MANETs that are formed by the devices brought into the emergency area and have the following properties: (1) bandwidth is limited, and (2) the network can be prone to frequent and/or long term network partitions. One obvious issue for applications is that rescue teams with personnel from different organizations, like policemen, firemen, physicians, and paramedics, typically arrive independently at the scene and need to share information. Some of this information might be mission critical and should be continuously available to all applications, even in the advent of network partitions; other information might have lower availability requirements. Furthermore, it is important that it is possible after the operation to trace in detail what happened, especially in case any (human) failure happened. Therefore, data that is important for auditing, must not be deleted or altered during the event.

The characteristics of the networking environment and the requirements on information availability, impose to apply optimistic replication within MDS.

27th International Conference on Distributed Computing Systems Workshops (ICDCSW'07)0-7695-2838-4/07 $20.00 © 2007

While replication increases the availability of information, it also introduces the problem of consistency management and thereby increased network traffic. Since bandwidth is the core bottleneck in MANETs, the tradeoff between high availability and high costs of consistency management must be well balanced. This does not only include the number of replicas to be used, but also their location and decisions about whether data should be moved, e.g. to the nodes where it is consumed.

While the emergency and rescue networking environment and application domain introduce several hard problems for MDS, they also have some properties that allow simple solutions for generally hard problems: (1) all rescue team members have a natural incentive to collaborate and share resources, (2) governmental bodies can define a priori collaboration rules for the different rescue organizations, (3) trust among the different team members can be achieved a priori [3], and (4) an emergency and rescue event is typically very short term, i.e., in the range of several hours (up to a few days at most).

In the remainder of this paper we show in Section 2 how we leverage these properties to solve some fundamental problems for MDS, including consistency management, common sense of time, and how to convey what kind of information is for share. Furthermore, we present in Section 3 the detailed design of MDS. The main goal of this design is to separate different concerns, like consistency management protocols from replication placement decisions. We need to experimentally study and compare various solutions for the different concerns, because earlier studies have shown that the performance of different solutions heavily depends on the network properties, including mobility and density, and the application work load [4]. Since there are up to now no real-life mobility traces or application workloads from emergency and rescue operations available, it is not possible to finally determine the best protocols for MDS. Therefore, we believe that the open and flexible design of MDS and the description of how to solve some fundamental issues for MDS, provide an important contribution to the state-of-the-art even if we do not provide any quantitative performance results in this paper.

2. Fundamental issues

There are four fundamental issues that have a strong impact on the entire design of MDS:

• achieving a common sense of time, • consistency management in disrupted net-

works,

• life time of data, and • the problem of conveying information about

what kinds of shareable data exist. In the following sections, we discuss these issues,

give some background information, and explain how we address them in MIDAS.

2.1 Common sense of time

MDS timestamps each table record for logging and auditing purposes. However, it should be noted that one cannot maintain a global clock nor a perfect synchronization between all clocks of all devices at all times in the MIDAS environment. The reason is that network dynamics might lead to shorter or longer term network partitions. There is always a certain drift between device clocks; to control this, synchronization protocols can be used. In addition, if some devices have GPS receivers and the environment does not prevent the receiving of GPS signals, they might synchronize devices in their communication range with the GPS time.

However, the clocks cannot be synchronized across network partitions. Therefore, we aim in MIDAS for a “common sense of time”, but do not rely on perfectly synchronized clocks. We assume that it is possible to synchronize the different clocks just before the rescue operation with some external and globally accessible timing service, thus at the start of the operation the difference between the clocks is rather small. For instance, with Internet access the Network Time Protocol (NTP) [5] could be used. NTP achieves accuracy in the range of a few milliseconds (ms). The rescue operations do not last for very long, typically in the order of hours to at most a few days. During this time, the clock drift will be typically (substantially) lower than a few seconds (based on reported numbers). Applications can then take into account the estimated clock drift.

2.2 Consistency and life time of data

The unit of shareable data is relational tables. Ideally, if the same unit is accessed simultaneously by different MDS users, it is available for all and with identical contents. In practice however, data may be unavailable or users perceive different contents due to e.g. network disruption and limitations in processing resources in the network and at nodes. To explicitly differentiate between the concept of using the same virtual data and accessing a particular copy of a table, we use the terms Virtual Table and Table Instance.

For MDS users, MDS is at any time just a set of Virtual Tables in use. A Virtual Table is materialized

27th International Conference on Distributed Computing Systems Workshops (ICDCSW'07)0-7695-2838-4/07 $20.00 © 2007

by one or multiple Table Instances. Figure 1 illustrates this relationship.

MDS user

Logical MDS VT1 VT2 VTn

TI1 TI1 TI1TIn TI2 TIn

Physical MDS

Node A Node B Node X

MDS API

VTx: Virtual Table X TIx: Instance of Virtual Table x

Figure 1. Virtual Tables and Table Instances in the logical and physical MDS

Table Instances are replicated within MDS to increase availability and improve performance. One of the central tasks of MDS is to manage the creation and storage of Table Instances and maintain consistency between them. However, in the MIDAS environment it is in general not possible to guarantee consistency of any kind since write operations cannot be propagated across network partitions. Only if there is full connectivity between all MIDAS nodes, it is possible to aim for consistency. Moreover, consistency protocols are often very expensive with respect to resources they consume, and in general establish hard states within the network, which cause many problems in the presence of mobility and network partitions. Having realized this, it is important to note the following:

(i) Unresolvable inconsistency between two Table Instances, e.g., when two network partitions with respective Instances of a Virtual Table remerge, can only occur if table records are updated in-place. If versioned data is used, i.e., each update or delete creates a new version of the record, consistency is easily obtained by a simple synchronization action where missing records are added to each Table Instance.

(ii) The MDS requirements to logging and auditing imply that data cannot be deleted; it must be stored until it can be analyzed or replayed after the rescue operation [6].

(iii) The life span of MDS is short. Therefore the amount of data that might be produced during the life time of MDS, is limited and should not impose a problem with respect to storage space. Even for diskless devices several Gigabytes of storage space is a realistic assumption. In the future, we can count on using several hundreds of Gigabytes on a simple laptop

and even Terabytes of storage space in the MIDAS environment. In addition, the MIDAS approach has adopted the fundamental principles of peer-to-peer systems, in that each peer contributes some of its resources to the system. Thus MDS inherits the P2P systems’ property of self-scaling, which basically states that the higher the load in the system, the more resources are available.

Therefore, we argue that a very efficient and elegant solution for MDS is to internally avoid update-in-place. Instead, we substitute user-initiated update and delete operations with an append operation. In this way we do not delete data during the short life span of MDS and reduce the consistency problem to a synchronization problem. The goal of synchronization in MDS is to synchronize all Table Instances as fast as possible under the given connectivity and resource constraints. Plagemann et al. [7] provides a more detailed description of the consistency management in MDS.

2.3 Common data schema

The goal of providing a virtual data space for information sharing, is to simplify the application programmer’s task. In this context, it should be noted that MDS supports information sharing for groups of applications that run independently in the network. This raises the question of how MDS can provide different applications with a common understanding of the meaning of the data that they are willing to share. Basically, there are two approaches to solving this problem: (1) a common standard for the naming and structure of tables is introduced, or (2) the middleware provides services for analyzing the semantics of the shared units, e.g. based on an ontology. One example for the second approach can be found in the Ad-Hoc InfoWare project [8], [9]. Experiences with this approach show that it requires solving many quite hard research problems. In contrast, using a common standard is in line with the understanding of the application domain. Since each application is developed for a particular Service, it is natural to include in the Service definition also a standard for the tables that could be used for the Service. In other words, the MDS schema is part of the Service definition. Therefore, table structures are known to application programmers, and the corresponding schema definition is passed to MDS instances when they are started. During the running phase, the schema cannot be changed.

27th International Conference on Distributed Computing Systems Workshops (ICDCSW'07)0-7695-2838-4/07 $20.00 © 2007

3. Architecture of MDS

The MDS architecture has been developed using the design principle of “separation of concerns” to enable independent development and evaluation of the different protocols that address these concerns. The resulting architecture is illustrated in Figure 2 and comprises the following components: The Local Storage is a third party local relational database to actually store data, i.e., Table Instances; while the Global MetaData Manager (GMDM) represents a distributed data dictionary to keep track of on which nodes which Table Instances are stored. The original decision of where to store Table Instances is made by the Data Allocator. The Data Synchronizer makes sure that modifications are propagated to all Instances. The Subscription Manager allows applications to subscribe to changes in Tables, and the Query Analyzer analyses application requests and forwards them to the local store or a remote node. The Access Control component rejects all unauthorized application requests.

Figure 2. MDS architecture overview, with component dependencies

All components are important in the MIDAS project, but due to space restrictions, we focus our further discussion on those components that are the most challenging ones for MDS, i.e., GMDM, Data Synchronizer and Data Allocator. The design of these components is strongly influenced by the networking environment and application requirements. 3.1 Global MetaData Manager

The GMDM maintains information about what kind of data is available and where, i.e., Virtual Tables and the locations of their Instances. The functionality of GMDM bears some resemblance to the data dictionary

in database systems, by its maintaining data about (the shared) data, i.e., metadata.

In dynamic mobile ad-hoc networks it is impossible to keep a complete global view of all Table Instances at every node at all times, due to frequent network partitions and merges. We need to keep the amount of metadata and the frequency of metadata updates at a reasonable level. This is done as follows: (1) Each node that stores a Table Instance, maintains the full metadata description of it. (2) Every node maintains for each Virtual Table the identities of remote nodes that store Instances of it.

Thus, if an MDS user requests information about a Virtual Table, it can consult its local GMDM. If the Table Instance is stored locally, the MDS user obtains the data from the Local Storage. If the Table Instance is stored remotely, the GMDM can furnish the MDS user with a list of nodes that store it.

For the purpose of supporting metadata synchronization in particular and lazy synchronization in general, GMDM implements its own synchronization trigger to initiate lazy synchronization of metadata. It is necessary to perform experiments to find out what kind of algorithms to use for GMDM synchronization and how to define the criteria for the synchronization trigger.

3.2 Data Allocator

It is the aim of the Data Allocator component to perform proper replication management of Table Instances. Multiple Instances of a Virtual Table can increase system performance and the availability of the Virtual Table. However, this does not come for free. Modify operations have to be propagated to all Table Instances in order to keep them synchronized. Thus, the Data Allocator must not unilaterally increase the number of Table Instances, but also reduce it if necessary, to keep the synchronization costs acceptable.

The Data Allocator is always invoked when the first record is inserted into a Virtual Table, triggering the creation of at least one Table Instance. In most cases, the Table Instance will be allocated on the node where the insert operation has been invoked, i.e., in the Local Storage. However, there might be cases where there is not sufficient storage space left to create the Table Instance locally. Another reason not to (only) store a Table Instance locally, is the importance of the Virtual Table for other nodes, predefined in the Service description.

To identify situations when new Instances should be created or old Instances deleted, it is worthwhile to analyze the advantages and disadvantages of

27th International Conference on Distributed Computing Systems Workshops (ICDCSW'07)0-7695-2838-4/07 $20.00 © 2007

replication. Disadvantages include higher overall storage need, consumption of network bandwidth to move Table Instances to their destination, and especially synchronization costs. Synchronization is necessary when one Table Instance is changed. This change has to be propagated to all other Table Instances of the Virtual Table, causing more traffic the more Table Instances there are. This also implies that creating multiple Instances of Virtual Tables that are frequently changed, should be considered very carefully.

The most important advantage of having multiple Instances, is increased availability. In case of network partitions, a well-designed Data Allocator is crucial for enabling nodes in different partitions to access a Virtual Table. Table Instances have to be allocated to nodes in such a way that at least one Table Instance exists in each network partition. Another advantage is increased locality, i.e., the Table Instance is close to the application that accesses the Virtual Table. Locality improves access time to Virtual Tables and can reduce network traffic because it reduces the need for sending operations through the network. However, there is a trade-off between the additional costs of synchronization and moving of Table Instances, and the reduced costs of better locality.

The Data Allocator instances on different nodes coordinate themselves to avoid concurrent replication management activities, i.e., creation of new Table Instances or removal of Table Instances that are triggered by a change in the network conditions. For each Virtual Table, the node responsible for allocation is the one having the lowest node id of all nodes hosting an Instance of this Virtual Table.

3.3 Data Synchronizer

By introducing versioning instead of updating or deleting data, we reduce the consistency problem to a synchronization problem. Instead of maintaining a certain order on read and write operations it is “only” necessary to keep all data versions in all Table Instances. It is the aim of the Data Synchronizer to synchronize all Table Instances as fast as possible under the given network conditions and resource restrictions. To minimize overhead on network traffic, we only transfer information about changes to a table.

There are two fundamental approaches which both need to be supported within MDS.

Eager synchronization: When a Table Instance is modified, the Data Synchronizer propagates the change to all known nodes that also carry an Instance of the Virtual Table.

Lazy synchronization: When during a network merge, two nodes come within each other’s reach, they compare the contents of the Table Instances they store. To do this efficiently, each table might be summarized by a hash value. In case two Instances of a Virtual Table differ, they are synchronized by exchanging the missing data. In MIDAS, we will also study other synchronization triggers that could combine multiple hops and minimum disconnection time between such nodes.

We refer to the first Table Instance modification caused by an insert, update or delete operation as the original change because it triggers the Data Synchronizer. The information about the location of Table Instances is managed by the GMDM. However, there is no guarantee that the GMDM knows about all Table Instances. For example, a new Table Instance might have been created in another network partition. During the time span these two partitions exist, multiple insert, update, and delete record operations might be performed. When the two partitions merge, it takes some time until the GMDMs know about the existence and location of this Table Instance. During this time span it is also not possible for the Data Synchronizer to propagate changes to the Instance, because it does not know about it.

Eager synchronization is the fastest solution for synchronization, but consumes more resources than lazy synchronization.

Our solution supports eventual consistency in the sense that if no more insert, update, or delete record operations towards a Virtual Table are invoked within a network partition, and the set of nodes in the network partition remains unchanged sufficiently long, all the Table Instances in the partition will eventually get identical contents. It should be noted that eventual consistency is the strongest consistency model that can be supported in MIDAS for the given environments and requirements.

4. Conclusion and future work

MDS aims to resemble a relational data space to transparently share information over MANETs for emergency and rescue applications. It is the goal of the MIDAS project and the MDS to shield application programmers as good as possible from all peculiarities caused by MANETs, like connection loss, network partitioning and merging, missing global clock, etc. Therefore, these issues have do be addressed within MDS. To achieve the necessary degree of availability for important information, MDS performs optimistic replication. However, the drawback of replication is

27th International Conference on Distributed Computing Systems Workshops (ICDCSW'07)0-7695-2838-4/07 $20.00 © 2007

the complexity and cost of consistency management over MANETs.

To solve these hard problems, we leverage particular characteristics of the emergency and rescue domain: (1) all rescue team members have a natural incentive to collaborate and share resources, (2) governmental bodies can define a priori collaboration rules for the different rescue organizations, (3) trust among the different team members can be achieved a priori, and (4) an emergency and rescue event is typically very short term.

This insight allows us to avoid update in-place operations and to introduce instead only new versions of records. This is turn reduces consistency management to synchronization management, and also naturally supports the requirement of logging and auditing after the event.

The area of mobile information systems and the use of optimistic replication has been an active research area. The most related systems to MDS in this sense are Bayou [10] and Roam [11]. Bayou supports eventual consistency. It lets mobile users replicate a database, modify it, and synchronize lazily with other replicas. Roam is a system for nomadic users. Replicas are clustered into peer-based dynamic wards. Synchronization is performed between pairs of ward members. Neither Bayou nor Roam perform eager synchronization, thus our requirement to high availability of data is not met.

The design of MDS is derived by applying the principle of separation of concerns. There are many different approaches respective protocols for the different tasks in MDS, like synchronization management, data allocation, and global metadata management. These protocols, e.g., eager vs. lazy synchronization, often have very different properties, and which one is “best” is entirely depending on the networking environment and the application workload. Synthetic mobility models are not sufficient to reflect real-world behavior; on the other hand, real world traces and workloads from emergency and rescue operations are not available. Therefore, it is important to have such an open design of MDS.

As part of our ongoing work, we implement the first prototype of MDS and will evaluate solutions for the different MDS components not only with standard mobility models, but also with traces extracted from the RoboCupRescue Simulation Project [12] which provides a generic urban disaster simulation environment. We consider these traces as close as possible to a real rescue operation. In the long term, we

are also interested to collect together with rescue organizations real-life traces and to use them for the evaluation of MDS.

Acknowledgments. The MIDAS project is funded by the IST Programme of the European Commission, contract no. 27055.

10. References

[1] International Federation of Red Cross and Red Crescent Societies (IFRC), World Disasters Report 2005, Oct. 2005, http://www.ifrc.org/publicat/wdr2005/index.asp [2] J. Gorman, “The MIDAS Project: Interworking and Data Sharing”, Interworking 2006, Santiago, Chile, January 2007. [3] M. Puzar, J. Andersson, T. Plagemann, and Y. Roudier, “SKiMPy: A Simple Key Management Protocol for MANETs in Emergency and Rescue Operations”, 2nd European Workshop on Security and Privacy in Ad hoc and Sensor Networks (ESAS 2005), Visegrad, Hungary, July, 2005. [4] K.S. Skjelsvik, V. Goebel, and T. Plagemann, “A Configurable Event Notification Service for MANETs”, 3rd MiNEMA Workshop, Leuven, Belgium, February, 2006. [5] D. Mills, “Network Time Protocol (Version 3) Specification, Implementation and Analysis”, Request for Comments: 1305, Internet Engineering Task Force, May 1992.[6] E. Klintskog, P. Gründler, K. Skjelsvik, and S. Pérez Marín, “Proof-of-Concept Application Requirements”, MIDAS Deliverable D5.3, Nov. 2006. [7] T. Plagemann, E. Munthe-Kaas, and V. Goebel, “Reconsidering Consistency Management in Shared Data Spaces for Emergency and Rescue Applications”, BTW-MDM 2007, Aachen, Germany, March 2007. [8] T. Plagemann et al., “Middleware Services for Information Sharing in Mobile Ad-hoc Networks - Challenges and Approach”, Workshop on Challenges of Mobility, IFIP TC6 World Computer Congress 2004, Toulouse, France, Aug. 2004. [9] E. Munthe-Kaas et al., “Mobile Middleware for Rescue and Emergency Scenarios”, in: Mobile Middleware, P. Bellavista, A. Corradi (Eds.), CRC Press, Sept. 2006. [10] D.B. Terry et al., “Managing Update Conflicts in Bayou, a Weakly Connected Replicated Storage System”, Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (SOSP’05), Copper Mountain, Colorado, USA, Dec. 1995, pp. 172-182. [11] D. Ratner, P. Reiher, and G.J. Popek, “Roam: A Scalable Replication System for Mobility”, Mobile Network Applications, Vol. 9, No. 5, Oct. 2004, pp. 537-544. [12] The RoboCupRescue simulation project, 2005, http://www.robocup2005.org/roborescue.

27th International Conference on Distributed Computing Systems Workshops (ICDCSW'07)0-7695-2838-4/07 $20.00 © 2007