IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED …lingliu/papers/2012/tpds-Remo.pdf · test hypotheses on system behavior [1]. Similarly, the data rate and buffer occupancy in each

Resource-Aware Application State MonitoringShicong Meng, Student Member, IEEE, Srinivas Raghav Kashyap,

Chitra Venkatramani, and Ling Liu, Senior Member, IEEE

Abstract—The increasing popularity of large-scale distributed applications in datacenters has led to the growing demand of distributed

application state monitoring. These application state monitoring tasks often involve collecting values of various status attributes from a

large number of nodes. One challenge in such large-scale application state monitoring is to organize nodes into a monitoring overlay

that achieves monitoring scalability and cost effectiveness at the same time. In this paper, we present REMO, a REsource-aware

application state MOnitoring system, to address the challenge of monitoring overlay construction. REMO distinguishes itself from

existing works in several key aspects. First, it jointly considers intertask cost-sharing opportunities and node-level resource constraints.

Furthermore, it explicitly models the per-message processing overhead which can be substantial but is often ignored by previous

works. Second, REMO produces a forest of optimized monitoring trees through iterations of two phases. One phase explores cost-

sharing opportunities between tasks, and the other refines the tree with resource-sensitive construction schemes. Finally, REMO also

employs an adaptive algorithm that balances the benefits and costs of overlay adaptation. This is particularly useful for large systems

with constantly changing monitoring tasks. Moreover, we enhance REMO in terms of both performance and applicability with a series

of optimization and extension techniques. We perform extensive experiments including deploying REMO on a BlueGene/P rack

running IBM’s large-scale distributed streaming system—System S. Using REMO in the context of collecting over 200 monitoring tasks

for an application deployed across 200 nodes results in a 35-45 percent decrease in the percentage error of collected attributes

compared to existing schemes.

Index Terms—Resource-aware, state monitoring, distributed monitoring, datacenter monitoring, adaptation, data-intensive

Ç

1 INTRODUCTION

RECENTLY, we have witnessed a fast growing set of large-scale distributed applications ranging from stream

processing [1] to applications [2] running in Cloud datacen-ters. Correspondingly, the demand for monitoring thefunctioning of these applications also increases substantially.Typical monitoring of such applications involves collectingvalues of metrics, e.g., performance related metrics, from alarge number of member nodes to determine the state of theapplication or the system. We refer to such monitoring tasksas application state monitoring. Application state monitoring isessential for the observation, analysis, and control ofdistributed applications and systems. For instance, datastream applications may require monitoring the data receiv-ing/sending rate, captured events, tracked data entities,signature of internal states, and any number of application-specific attributes on participating computing nodes toensure stable operation in the face of highly bursty workloads[3] [4]. Application provisioning may also require continu-ously collecting performance attribute values such as CPU

usage, memory usage, and packet size distributions fromapplication-hosting servers [5].

One central problem in application state monitoring isorganizing nodes into a certain topology where metricvalues from different nodes can be collected and delivered.In many cases, it is useful to collect detailed performanceattributes at a controlled collection frequency. As anexample, fine-grained performance characterization infor-mation is required to construct various system models and totest hypotheses on system behavior [1]. Similarly, the datarate and buffer occupancy in each element of a distributedapplication may be required for diagnosis purposes whenthere is a perceived bottleneck [3]. However, the overhead ofcollecting monitoring data grows quickly as the scale andcomplexity of monitoring tasks increase. Hence, it is crucialthat the monitoring topology should ensure good monitor-ing scalability and cost effectiveness at the same time.

While a set of monitoring-topology planning approacheshave been proposed in the past, we find that these approachesoften have the following drawbacks in general. First of all,existing works either build monitoring topologies for eachindividual monitoring task (TAG [6], SDIMS [7], PIER [8], joinaggregations [9], REED [10], operator placement [11]), or use astatic monitoring topology for all monitoring tasks [11]. Thesetwo approaches, however, often produce suboptimal mon-itoring topologies. For example, if two monitoring tasks bothcollect metric values over the same set of nodes, using onemonitoring tree for monitoring data transmission is moreefficient than using two, as nodes can merge updates for bothtasks and reduce per-message processing overhead. Hence,multimonitoring-task-level topology optimization is crucialfor monitoring scalability.

Second, for many data-intensive environments, monitor-ing overhead grows substantially with the increase of

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 23, NO. X, XXXXXXX 2012 1

. S. Meng is with the Georgia Institute of Technology, 37975 GT Station,Atlanta, GA 30332. E-mail: [email protected].

. S.R. Kashyap is with Google Inc., 1600 Amphitheatre Parkway MountainView, CA 94043. E-mail: [email protected].

. C. Venkatramani is with the IBM TJ Watson Research Center, 19 SkylineDrive, Hawthorne, NY 10532. E-mail: [email protected].

. L. Liu is with the Georgia Institute of Technology, KACB 3340, GeorgiaTech, 266 Ferst Drive, Atlanta, GA 30332-0765.E-mail: [email protected].

Manuscript received 26 June 2011; revised 5 Dec. 2011; accepted 9 Feb. 2012;published online 29 Feb. 2012.Recommended for acceptance by J. Zhang.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number TPDS-2011-06-0426.Digital Object Identifier no. 10.1109/TPDS.2012.82.

1045-9219/12/$31.00 � 2012 IEEE Published by the IEEE Computer Society

monitoring tasks and deployment scale [7] [12]. It isimportant that the monitoring topology should be resourcesensitive, i.e., it should avoid monitoring nodes spendingexcessive resources on collecting and delivering attributevalues. Unfortunately, existing works do not take node-levelresource consumption as a first-class consideration. Thismay result in overload on certain nodes which eventuallyleads to monitoring data loss. Moreover, some assumptionsin existing works do not hold in real-world scenarios. Forexample, many works assume that the cost of updatemessages is only related with the number of values withinthe message, while we find that a fixed per-messageoverhead is not negligible.

Last but not the least, application state monitoring tasksare often subject to change in real-world deployments [13].Some tasks are short term by nature, e.g., ad hoc taskssubmitted to check the current system usage [14]. Othertasks may be frequently modified for debugging, e.g., a usermay specify different attributes for one task to understandwhich attribute provides the most useful information [13].Nevertheless, existing works often consider monitoringtasks to be static and perform one-time topology optimiza-tion [10] [11]. With little support for efficient topologyadaptation, these approaches would either produce sub-optimal topologies when using a static topology regardlessof changes in tasks, or introduce high adaptation cost whenperforming comprehensive topology reconstruction for anychange in tasks [15].

In this paper, we present REMO, a resource-awareapplication state monitoring system, that aims at addres-sing the above issues. REMO takes node-level availableresources as the first class factor for building a monitoringtopology. It optimizes the monitoring topology to achievethe best scalability and ensures that no node would beassigned with excessive monitoring workloads for theiravailable resources.

REMO employs three key techniques to deliver cost-effective monitoring topologies under different environ-ments. In the conference version [15] of this paper, weintroduced a basic topology planning algorithm. This algorithmproduces a forest of carefully optimized monitoring trees fora set of static monitoring tasks. It iteratively explores cost-sharing opportunities among monitoring tasks and refinesthe monitoring trees to achieve the best performance giventhe resource constraints on each node. One limitation of thebasic approach is that it explores the entire search space foran optimal topology whenever the set of monitoring tasks ischanged. This could lead to significant resource consump-tion for monitoring environments where tasks are subject tochange. In this journal paper, we present an adaptive topologyplanning algorithm which continuously optimizes the mon-itoring topology according to the changes of tasks. Toachieve cost effectiveness, it maintains a balance between thetopology adaptation cost and the topology efficiency, andemploys cost-benefit throttling to avoid trivial adaptation.To ensure the efficiency and applicability of REMO, we alsointroduce a set of optimization and extension techniques in thispaper. These techniques further improve the efficiency ofresource-sensitive monitoring tree construction scheme, andallow REMO to support popular monitoring features such asIn-network aggregation and reliability enhancements.

We undertake an experimental study of our system andpresent results including those gathered by deploying

REMO on a BlueGene/P rack (using 256 nodes booted intoLinux) running IBM’s large-scale distributed streamingsystem—System S [3]. The results show that our resource-aware approach for application state monitoring consistentlyoutperforms the current best known schemes. For instance,in our experiments with a real application that spanned up to200 nodes and about as many monitoring tasks, using REMOto collect attributes resulted in a 35-45 percent reduction inthe percentage error of the attributes that were collected.

To our best knowledge, REMO is the first system thatpromotes resource-aware methodology to support andscale multiple application state monitoring tasks in large-scale distributed systems. We make three contributions inthis paper.

. We identify three critical requirements for large-scale application state monitoring: the sharing ofmessage processing cost among attributes, meeting node-level resource constraints, and efficient adaptation towardmonitoring task changes. Existing approaches do notaddress these requirements well.

. We propose a framework for communication-efficientapplication state monitoring. It allows us to optimizemonitoring topologies to meet the above threerequirements under a single framework.

. We develop techniques to further improve theapplicability of REMO in terms of runtime efficiencyand supporting new monitoring features.

The rest of the paper is organized as follows: Section 2identifies challenges in application state monitoring. Section3 illustrates the basic monitoring topology constructionalgorithm, and Section 4 introduces the adaptive topologyconstruction algorithm. We optimize the efficiency of REMOand extend it for advanced features in Sections 5 and 6. Wepresent our experimental results in Section 7. Section 8describes related works and Section 9 concludes this paper.

2 SYSTEM OVERVIEW

In this section, we introduce the concept of application statemonitoring and its system model. We also demonstrate thechallenges in application state monitoring, and point outthe key questions that an application state monitoringapproach must address.

2.1 Application State Monitoring

Users and administrators of large-scale distributed applica-tions often employ application state monitoring for observa-tion, debugging, analysis, and control purposes. Eachapplication state monitoring task periodically collects valuesof certain attributes from the set of computing nodes overwhich an application is running. We use the term attributeand metric interchangeably in this paper. As we focus onmonitoring topology planning rather than the actual produc-tion of attribute values [16], we assume values of attributesare made available by application-specific tools or manage-ment services. In addition, we target at datacenter-likemonitoring environments where any two nodes can commu-nicate with similar cost (more details in Section 3.3). Formally,we define an application state monitoring task t as follows:

Definition 1. A monitoring task t ¼ ðAt;NtÞ is a pair of sets,where At �

Si2Nt

Ai is a set of attributes and Nt � N is a set

2 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 23, NO. X, XXXXXXX 2012

of nodes. In addition, t can also be represented as a list of node-attribute pairs ði; jÞ, where i 2 Nt; j 2 At.

2.2 The Monitoring System Model

Fig. 1 shows the high-level model of REMO, a system wedeveloped to provide application state monitoring function-ality. REMO consists of several fundamental components:

Task manager takes state monitoring tasks and re-moves duplication among monitoring tasks. For instance,monitoring tasks

t1 ¼ ðfcpu utilizationg; fa; bgÞ and

t2 ¼ ðfcpu utilizationg; fb; cgÞ

have duplicated monitored attribute cpu utilization on nodeb. With such duplication, node b has to send cpu utilizationinformation twice for each update, which is clearly un-necessary. Therefore, given a set of monitoring tasks, the taskmanager transforms this set of tasks into a list of node-attribute pairs and eliminates all duplicated node-attributepairs. For instance, t1 and t2 are equivalent to the listfa-cpu utilization, b-cpu utilizationg and fb-cpu utilization,c-cpu utilizationg, respectively. In this case, node-attributepair fb-cpu utilizationg is duplicated, and thus, is eliminatedfrom the output of the task manager.

Management core takes deduplicated tasks as input andschedules these tasks to run. One key subcomponent of themanagement core is the monitoring planner which deter-mines the interconnection of monitoring nodes. For simpli-city, we also refer to the overlay connecting monitoringnodes as the monitoring topology. In addition, the manage-ment core also provides important support for reliabilityenhancement and failure handling. Data collector providesa library of functions and algorithms for efficiently collect-ing attribute values from the monitoring network. It alsoserves as the repository of monitoring data and providesmonitoring data access to users and high-level applications.Result processor executes the concrete monitoring opera-tions including collecting and aggregating attribute values,triggering warnings, etc.

In this paper, we focus on the design and implementa-tion of the monitoring planner. We next introduce monitor-ing overhead in application state monitoring which drivesthe design principles of the monitoring planner.

2.3 Monitoring Overhead and Monitoring Planning

On a high level, a monitoring system consists of nmonitoring nodes and one central node, i.e., data collector.Each monitoring node has a set of observable attributes

Ai ¼ fajjj 2 ½1;m�g. Attributes at different nodes but withthe same subscription are considered as attributes of thesame type. For instance, monitored nodes may all havelocally observable CPU utilization. We consider an attributeas a continuously changing variable which outputs a newvalue in every unit time. For simplicity, we assume allattributes are of the same size a and it is straightforward toextend our work to support attributes with different sizes.

Each node i, the central node or a monitoring node, has acapacity bi (also referred to as the resource constraint ofnode i) for receiving and transmitting monitoring data. Inthis paper, we consider CPU as the primary resource foroptimization. We associate each message transmitted in thesystem with a per-message overhead C, and, the cost oftransmitting a message with x values is C þ ax. This costmodel is motivated by our observations of monitoringresource consumption on a real-world system which weintroduce next.

Our cost model considers both per-message overheadand the cost of payload. Although other models mayconsider only one of these two, our observation suggeststhat both costs should be captured in the model. Fig. 2shows how significant the per-message processing overheadis. The measurements were performed on a BlueGene/Pnode which has a quad core 850 MHz PowerPC processor.The figure shows an example monitoring task where nodesare configured in a star network where each nodeperiodically transmits a single fixed small message to a rootnode over TCP/IP. The CPU utilization of the root nodegrows roughly linearly from around 6 percent for 16 nodes(the root receives 16 messages periodically) to around68 percent for 256 nodes (the root receives 256 messagesperiodically). Note that this increased overhead is due to theincreased number of messages at the root node and not dueto the increase in the total size of messages. Furthermore,the cost incurred to receive a single message increases from0.2 to 1.4 percent when we increase the number of values inthe message from 1 to 256. Hence, we also model the costassociated with message size as a message may contain alarge number of values relayed for different nodes.

In other scenarios, the per-message overhead could betransmission or protocol overhead. For instance, a typicalmonitoring message delivered via TCP/IP protocol has amessage header of at least 78 bytes not including applica-tion-specific headers, while an integer monitoring data isjust 4 bytes.

As Fig. 3 shows, given a list of node-attribute pairs, themonitoring planner organizes monitoring nodes into aforest of monitoring trees where each node collects valuesfor a set of attributes. The planner considers the aforemen-tioned per-message overhead as well as the cost of attributes

MENG ET AL.: RESOURCE-AWARE APPLICATION STATE MONITORING 3

Fig. 1. A high-level system model.

Fig. 2. CPU usage versus increasing message number/size.

transmission (as illustrated by the black and white bar in theleft monitoring tree) to avoid overloading certain monitor-ing nodes in the generated monitoring topology. Inaddition, it also optimizes the monitoring topology toachieve maximum monitoring data delivery efficiency. Asa result, one monitoring node may connect to multiple trees(as shown in Figs. 3 and 4c). Within a monitoring tree T ,each node i periodically sends an update message to itsparent. As application state monitoring requires collectingvalues of certain attributes from a set of nodes, such updatemessages include both values locally observed by node iand values sent by i’s children, for attributes monitored byT . Thus, the size of a message is proportional to the numberof monitoring nodes in the subtree rooted at node i. Thisprocess continues upward in the tree until the messagereaches the central data collector node.

2.4 Challenges in Monitoring Planning

From the users’ perspective, monitoring results should beas accurate as possible, suggesting that the underlyingmonitoring network should maximize the number of node-attribute pairs received at the central node. In addition,such a monitoring network should not cause the excessiveuse of resource at any node. Accordingly, we define themonitoring planning problem (MP) as follows:

Problem Statement 1. Given a set of node-attribute pairs formonitoring � ¼ f!1; !2; . . . ; !pg where !q ¼ ði; jÞ, i 2 N ,j 2 A, q 2 ½1; p�, and resource constraint bi for each associatednode, find a parent fði; jÞ; 8i; j, where j 2 Ai such that node iforwards attribute j to node fði; jÞ that maximizes the totalnumber of node-attribute pairs received at the central node andthe resource demand of node i, di, satisfies di � bi; 8i 2 N .

NP-completeness. When restricting all nodes to onlymonitor the same attribute j, we obtain a special case of themonitoring planning problem where each node has at mostone attribute to monitor. As shown by Kashyap et al. [17],this special case is an NP-complete problem. Consequently,the monitoring planning problem is an NP-complete

problem, since each instance of MP can be restricted to thisspecial case. Therefore, in REMO, we primarily focus onefficient approaches that can deliver reasonably goodmonitoring plan.

We now use some intuitive examples to illustrate thechallenges and the key questions that need to be addressedin designing a resource-aware monitoring planner. Fig. 4shows a monitoring task involving 6 monitoring nodeswhere each node has a set of attributes to deliver (asindicated by alphabets on nodes). The four examples (a), (b),(c), and (d) demonstrate different approaches to fulfill thismonitoring task. Example (a) shows a widely used topologyin which every node sends its updates directly to the centralnode. Unfortunately, this topology has poor scalability,because it requires the central node to have a large amount ofresources to account for per-message overhead. We refer tothe approach used in example (a) as the star collection.Example (b) organizes all monitoring nodes in a single treewhich delivers updates for all attributes. While this mon-itoring plan reduces the resource consumption (per-messageoverhead) at the central node, the root node now has to relayupdates for all node-attribute pairs, and again facesscalability issues due to limited resources. We refer to thisapproach as one-set collection. These two examples suggestthat achieving certain degree of load balancing is importantfor a monitoring network.

However, load balance alone does not lead to a goodmonitoring plan. In example (c), to balance the trafficamong nodes, the central node uses three trees, each ofwhich delivers only one attribute, and thus achieves a morebalanced workload compared with example (b) (one-setcollection) because updates are relayed by three root nodes.However, since each node monitors at least two attributes,nodes have to send out multiple update messages instead ofone as in example (a) (star collection). Due to per-messageoverhead, this plan leads to higher resource consumption atalmost every node. As a result, certain nodes may still fail todeliver all updates and less resources will be left over foradditional monitoring tasks. We refer to the approach inexample (c) as singleton-set collection.

The above examples reveal two fundamental aspects ofthe monitoring planning problem. First, how to determine thenumber of monitoring trees and the set of attributes on each? Thisis a nontrivial problem. Example (d) shows a topology whichuses one tree to deliver attribute a, b, and another tree todeliver attribute c. It introduces less per-message overheadcompared with example (c) (singleton-set collection) and is amore load-balanced solution compared with example (b)(one-set collection). Second, how to determine the topology for


Fig. 3. An example of monitoring planning.

Fig. 4. Motivating examples for the topology planning problem.

nodes in each monitoring tree under node-level resource con-straints? Constructing monitoring trees subject to resourceconstraints at nodes is also a nontrivial problem and thechoice of topology can significantly impact node resourceusage. Example (e) shows three different trees. The startopology (upper left), while introducing the least relayingcost, causes significant per-message overhead at its root. Thechain topology (upper right), on the contrary, distributes theper-message overhead among all nodes, but causes the mostrelaying cost. A “mixed” tree (bottom) might achieve a goodtradeoff between relaying cost and per-message overhead,but it is determine its optimal topology.

3 THE BASIC REMO APPROACH

The basic REMO approach promotes the resource awaremultitask optimization framework, consisting of a twophase iterative process and a suite of multitask optimizationtechniques. At a high level, REMO operates as a guided localsearch approach, which starts with an initial monitoringnetwork composed of multiple independently constructedmonitoring trees, and iteratively optimizes the monitoringnetwork until no further improvements are possible. Whenexploring various optimization directions, REMO employscost estimation to guide subsequent improvement so thatthe search space can be restricted to a small size. Thisguiding feature is essential for the scalability of large-scaleapplication state monitoring systems.

Concretely, during each iteration, REMO first runs apartition augmentation procedure which generates a list ofmost promising candidate augmentations for improving thecurrent distribution of monitoring workload among mon-itoring trees. While the total number of candidate augmen-tations is very large, this procedure can trim down the sizeof the candidate list for evaluation by selecting the mostpromising ones through cost estimation. Given the gener-ated candidate augmentation list, the resource-aware evalua-tion procedure further refines candidate augmentations bybuilding monitoring trees accordingly with a resource-aware tree construction algorithm. We provide more detailsto these two procedures in the following discussion.

3.1 Partition Augmentation

The partition augmentation procedure is designed toproduce the attribute partitions that can potentially reducemessage processing cost through a guided iterative process.These attribute partitions determine the number of mon-itoring trees in the forest and the set of attributes each treedelivers. To better understand the design principles of ourapproach, we first briefly describe two simple but mostpopular schemes, which essentially represent the state-of-the-art in multiple application state monitoring.

Recall that among example schemes in Fig. 4, one scheme(example (c)) delivers each attribute in a separate tree, andthe other scheme (example (b)) uses a single tree to deliverupdates for all attributes. We refer to these two schemes as theSingleton-set partition scheme (SP) and the One-set partition(OP) scheme, respectively. We use the term “partition”because these schemes partition the set of monitoredattributes into a number of nonoverlapping subsets andassign each subsets to a monitoring tree.

Singleton-set partition. Specifically, given a set ofattributes for collection A, singleton-set partition schemedivides A into jAj subsets, each of which contains a distinctattribute in A. Thus, if a node has m attributes to monitor, itis associated with m trees. This scheme is widely used inprevious work, e.g., PIER [8], which constructs a routingtree for each attribute collection. While this schemeprovides the most balanced load among trees, it is notefficient, as nodes have to send update messages for eachindividual attribute.

One-set partition. The one-set partition scheme uses thesetA as the only partitioned set. This scheme is also used in anumber of previous work [11]. Using OP, each node cansend just one message which includes all the attributevalues, and thus, saves per-message overhead. Neverthe-less, since the size of each message is much larger comparedwith messages associated with SP, the correspondingcollecting tree cannot grow very large, i.e., contains limitednumber of nodes.

3.1.1 Exploring Partition Augmentations

REMO seeks a middle ground between these extremesolutions—one where nodes pay lower per-message over-head compared to SP while being more load-balanced andconsequently more scalable than OP. Our partition augmen-tation scheme explores possible augmentations to a givenattribute partition P by searching for all partitions that areclose to P in the sense that the resulting partition can becreated by modifying P with certain predefined operations.

We define two basic operations that are used to modifyattribute set partitions.

Definition 2. Given two attribute sets APi and AP

j in partitionP , a merge operation over AP

i and APj , denoted as AP

i ffl APj ,

yields a new set APk ¼ AP

i [APj . Given one attribute set AP

i

and an attribute �, a split operation on APi with regard to �,

denoted as APi . �, yields two new sets AP

k ¼ APi � �.

A merge operation is simply the union of two setattributes. A split operation essentially removes oneattribute from an existing attribute set. As it is a specialcase of set difference operation, we use the set differencesign (�) here to define split. Furthermore, there is norestriction on the number of attributes that can be involvedin a merge or a split operation. Based on the definition ofmerge and split operations, we now define neighboringsolution as follows:

Definition 3. For an attribute set partition P , we say partitionP 0 is a neighboring solution of P if and only if either9AP

i ; APj 2 P so that P 0 ¼ P �AP

i �APj þ ðAP

i ffl APj Þ, or

9APi 2 P; � 2 AP

i so that P 0 ¼ P �APi þ ðAP

i . �Þ þ f�g.

A neighboring solution is essentially a partition obtainedby make “one-step” modification (either one merge or onesplit operation) to the existing partition.

Guided partition augmentation. Exploring all neighbor-ing augmentations of a given partition and evaluating theperformance of each augmentation is practically infeasible,since the evaluation involves constructing resource-con-strained monitoring trees. To mitigate this problem, we usea guided partition augmentation scheme which greatlyreduces the number of candidate partitions for evaluation.


The basic idea of this guided scheme is to rank candidatepartitions according to the estimated reduction in the totalcapacity usage that would result from using the newpartition. The rationale is that a partition that provides alarge decrease in capacity usage will free up capacity formore attribute value pairs to be aggregated. Following this,we evaluate neighboring partitions in the decreased orderof their estimated capacity reduction so that we can find agood augmentation without evaluating all candidates. Weprovide details of the gain estimation in the appendixwhich can be found on the Computer Society DigitalLibrary at http://doi.ieeecomputersociety.org/10.1109/TPDS.2012.82. This guided local-search heuristic is essentialto ensuring the practicality of our scheme.

3.2 Resource-Aware Evaluation

To evaluate the objective function for a given candidatepartition augmentation m, the resource-aware evaluationprocedure evaluates m by constructing trees for nodesaffected by m and measures the total number of node-attribute value pairs that can be collected using these trees.This procedure primarily involves two tasks. One isconstructing a tree for a given set of nodes without exceedingresource constraints at any node. The other is for a nodeconnected to multiple trees to allocate its resources todifferent trees.

3.2.1 Tree Construction

The tree construction procedure constructs a collection treefor a given set of nodes D such that no node exceeds itsresource constraints while trying to include as many nodesas possible into the constructed tree. Formally, we definethe tree construction problem as follows:

Problem Statement 2. Given a set of n vertices, each has xiattributes to monitor, and resource constraint bi, find a parentvertex pðiÞ; 8i, so that the number of vertices in theconstructed tree is maximized subject to the followingconstraints where ui is the resource consumed at vertex i forsending update messages to its parent

1. For any vertex i in the tree,P

pðjÞ¼i uj þ ui � bi.2. Let yi be the number of all attribute values transmitted

by vertex i. We have yi ¼ xi þP

pðjÞ¼i xj.3. According to our definition, ui � C þ yi � a.

The first constraint requires that the resource spent onnode i for sending and receiving updates should not exceedits resource constraint bi. The second constraint requires anode to deliver its local monitored values as well as valuesreceived from its children. The last constraint states that thecost of processing an outgoing message is the combination ofper-message overhead and value processing cost. The treeconstruction problem, however, is also NP-complete [17]and we present heuristics for the tree-construction problem.

To start with, we first discuss two simple tree construc-tion heuristics

Star. This scheme forms “star”-like trees by givingpriority to increasing the breadth of the tree. Specifically, itadds nodes into the constructed tree in the order of decreasedavailable capacity, and attaches a new node to the node withthe lowest height and sufficient available capacity, until nosuch nodes exist. STAR creates bushy trees and consequently

pays low relay cost. However, owing to large node degrees,the root node suffers from higher per-message overhead, andconsequently, the tree cannot grow very large.

Chain. This scheme gives priority to increasing theheight of the tree, and constructs “chain”-like trees. CHAIN

adds nodes to the tree in the same way as STAR does exceptthat it tries to attach nodes to the node with the highest heightand sufficient available capacity. CHAIN creates long treesthat achieve very good load balance, but due to the numberof hops each message has to travel to reach the root, mostnodes pay a high relay cost.

STAR and CHAIN reveal two conflicting factors incollection tree construction—resource efficiency and scal-ability. Minimizing tree height achieves resource efficiency,i.e., minimum relay cost, but causes poor scalability, i.e.,small tree size. On the other hand, maximizing tree heightachieves good scalability, but degrades resource efficiency.The adaptive tree construction algorithm seeks a middle-ground between the STAR and CHAIN procedures in thisregard. It tries to minimize the total resource consumption,and can trade off overhead cost for relay cost, and viceversa, if it is possible to accommodate more nodes bydoing so.

Before we describe the adaptive tree constructionalgorithm, we first introduce the concept of saturated treesand congested nodes as follows:

Definition 4. Given a set of nodes N for tree construction andthe corresponding tree T which contains a set of nodesN 0 � N , we say T is saturated if no more nodes d 2ðN �N 0Þ can be added to T without causing the resourceconstraint to be violated for at least one node in T . We refer tonodes whose resource constraint would be violated if d 2ðN �N 0Þ is added to T as congested nodes.

The adaptive tree construction algorithm iterativelyinvokes two procedures, the construction procedure and theadjusting procedure. The construction procedure runs theSTAR scheme which attaches new nodes to low-level existingtree nodes. As we mentioned earlier, STAR causes thecapacity consumption at low-level tree nodes to be muchheavier than that at other nodes. Thus, as low-level treenodes become congested we get a saturated tree, theconstruction procedure terminates and returns all congestednodes. The algorithm then invokes the adjusting procedure,which tries to relieve the workload of low level tree nodes byreducing the degree of these nodes and increasing the heightof the tree (similar to CHAIN). As a result, the adjustingprocedure reduces congested nodes and makes a saturatedtree unsaturated. The algorithm then repeats the construct-ing-adjusting iteration until no more nodes can be added tothe tree or all nodes have been added.

3.3 Discussion

REMO targets at datacenter-like environments where theunderlying infrastructure allows any two nodes in thenetwork can communicate with similar cost, and focuses onthe resource consumption on computing nodes rather thanthat of the underlying network. We believe this setting fitsfor many distributed computing environments, even whencomputing nodes are not directly connected. For instance,


communication packets between hosts located in the samerack usually pass through only one top-of-rack switch,while communication packets between hosts located indifferent racks may travel through longer communicationpath consisting of multiple switches or routers. Thecorresponding overhead on communication endpoints,however, is similar in these two cases as packet forwardingoverhead is outsourced to network devices. As long asnetworks are not saturated, REMO can be directly appliedfor monitoring topology planning.

when the resource consumption on network devicesneeds to be considered, e.g., networks are bottleneckresources, REMO cannot be directly applied. Similarly, forenvironments where internode communication requiresnodes to actively forward messages, e.g., peer-to-peeroverlay networks and wireless sensor networks, theassumption of similar cost on communication endpointsdoes not hold as longer communication paths also incurhigher forwarding cost. However, REMO can be extendedto handle such changes. For example, its local searchprocess can incorporate the forwarding cost in the resourceevaluation of a candidate plan. We consider such extensionfor supporting such networks as our future work.

4 RUNTIME TOPOLOGY ADAPTION

The basic REMO approach works well for a static set ofmonitoring tasks. However, in many distributed comput-ing environments, monitoring tasks are often added,modified or removed on the fly for better informationcollection or debugging. Such changes necessitate theadaptation of monitoring topology. In this section, westudy the problem of runtime topology adaptation forchanges in monitoring tasks.

4.1 Efficient Adaptation Planning

One may search for an optimal topology by invoking theREMO planning algorithm every time a monitoring task isadded, modified or removed, and update the monitoringtopology accordingly. We refer to such an approach asREBUILD. REBUILD, however, may incur significantresource consumption due to topology planning computa-tion as well as topology reconstruction cost (e.g., messagesused for notifying nodes to disconnect or connect),especially in datacenter-like environments with a massivenumber of mutable monitoring tasks undergoing relativelyfrequent modification.

An alternative approach is to introduce minimumchanges to the topology to fulfill the changes of monitoringtasks. We refer to such an approach as DIRECT-APPLY orD-A for brevity. D-A also has its limitation as it may resultin topologies with poor performance over time. Forinstance, when we continuously add attributes to a taskfor collection, D-A simply instructs the corresponding treeto deliver these newly added attribute values until somenodes become saturated due to increased relay cost.

To address such issues, we propose an efficient adaptiveapproach that strikes a balance between adaptation cost andtopology performance. The basic idea is to look for newtopologies with good performance and small adaptationcost (including both searching and adaptation-related

communication cost) based on the modification to monitor-ing tasks. Our approach limits the search space totopologies that are close variants of the current topologyin order to achieve efficient adaptation. In addition, it rankscandidate adaptation operations based on their estimatedcost-benefit ratios so that it always performs the mostworthwhile adaptation operation first. We refer to thisscheme as ADAPTIVE for brevity.

When monitoring tasks are added, removed or modified,we first applies D-A by building the corresponding trees withthe tree building algorithm introduced in Section 3 (nochanges in the attribute partition). We consider the resultingmonitoring topology after invoking D-A as the base topology,which is then optimized by our ADAPTIVE scheme. Notethat the base topology is only a virtual topology plan stored inmemory. The actual monitoring topology is updated onlywhen the ADAPTIVE scheme produces a final topology plan.

Same as the algorithm in Section 3, the ADAPTIVEscheme performs two operations, merging and splitting,over the base topology in an iterative manner. For eachiteration, the ADAPTIVE scheme first lists all candidateadaptation operations for merging and splitting, respec-tively, and ranks the candidate operations based onestimated cost effectiveness. It then evaluates mergingoperations in the order of decreasing cost effectiveness untilit finds a valid merging operation. It also evaluates splittingoperations in the same way until it finds a valid splittingoperations. From these two operations, it chooses one withthe largest improvement to apply to the base topology.

Let T be the set of reconstructed trees. To ensureefficiency, the ADAPTIVE scheme considers only mergingoperations involving at least one tree in T as candidatemerging operations. This is because merging two trees thatare not in T is unlikely to improve the topology. Otherwise,previous topology optimization process would haveadopted such a merging operation. The evaluation of amerging operation involves computationally expensive treebuilding. As a result, evaluating only merging operationsinvolving trees in T greatly reduces the search space andensures the efficiency and responsiveness (changes ofmonitoring tasks should be quickly applied) of theADAPTIVE scheme. For a monitoring topology with ntrees, the number of possible merging operations isC2n ¼

nðn�1Þ2 , while the number of merging operations

involving trees in T is jT j � ðn� 1Þ which is usuallysignificantly smaller than nðn�1Þ

2 as T << n for mostmonitoring task updates. Similarly, the ADAPTIVE schemeconsiders a splitting operation as a candidate operation onlyif the tree to be split is in T .

The ADAPTIVE scheme also minimizes the number ofcandidate merging/splitting operations it evaluates forresponsiveness. It ranks all candidate operations andalways evaluates the one with the greatest potential gainfirst. To rank candidate operations, the ADAPTIVE schemeneeds to estimate the cost effectiveness of each operation.We estimate the cost effectiveness of an operation based onits estimated benefit and estimated adaptation cost. Theestimated benefit is the same as gðmÞ we introduced inSection 3. The estimated adaptation cost refers to the cost ofapplying the merging operation to the existing monitoringtopology. This cost is usually proportional to the number of


edges modified in the topology. To estimate this cost, weuse the lower bound of the number of edges that wouldhave to be changed.

4.2 Cost-Benefit Throttling

The ADAPTIVE scheme must ensure that the benefit ofadaption justifies the corresponding cost. For example, amonitoring topology undergoing frequent modification ofmonitoring tasks may not be suitable for frequent topologyadaptation unless the corresponding gain is substantial. Weemploy cost-benefit throttling to apply only topologyadaptations whose gain exceeds the corresponding cost.Concretely, when the ADAPTIVE scheme finds a validmerging or splitting operation, it estimates the adaptationcost by measuring the volume of control messages neededfor adaptation, denoted by Madapt. The algorithm considersthe operation cost effective if Madapt is less than a thresholddefined as follows:

ThresholdðAmÞ ¼ ðTcur �minfTadj;i; i 2 AmgÞ � ðCcur � CadjÞ;

where Am is the set of trees involved in the operation, Tadj;iis the last time tree i being adjusted, Tcur is the current time,Ccur is the volume of monitoring messages delivered in unittime in the trees of the current topology, and Cadj is thevolume of monitoring messages delivered in unit time inthe trees after adaptation. ðTcur �minfTadj;i; i 2 AmgÞ essen-tially captures how frequently the corresponding trees areadjusted, and ðCcur � CadjÞ measures the efficiency gain ofthe adjustment. Note that the threshold will be large ifeither the potential gain is large, i.e., ðCcur � CadjÞ is large, orthe corresponding trees are unlikely to be adjusted due tomonitoring task updates, i.e., ðTcur �minfTadj;i; i 2 AmgÞ islarge. Cost-benefit throttling also reduces the number ofiterations. Once the algorithm finds that a merging orsplitting is not cost-effective, it can terminate immediately.

5 OPTIMIZATION

The basic REMO approach can be further optimized toachieve better efficiency and performance. In this section,we present two techniques, efficient tree adjustment andordered resource allocation, to improve the efficiency ofREMO tree construction algorithm and its planningperformance, respectively.

5.1 Efficient Tree Adjustment

The tree building algorithm introduced in Section 3iteratively invokes a construction procedure and an adjust-ing procedure to build a monitoring tree for a set of nodes.One issue of this tree building algorithm is that it generateshigh-computation cost, especially its adjusting procedure. Toincrease the available resource of a congested node dc, theadjusting procedure tries to reduce its resource consumptionon per-message overhead by reducing the number of itsbranches. Specifically, the procedure first removes thebranch of dc with the least resource consumption. We usebdc to denote this branch. It then tries to reattach nodes in bdcto other nodes in the tree except dc. It considers thereattaching successful if all nodes of bdc is attached to thetree. As a result, the complexity of the adjusting procedure isOðn2Þ where n is the number of nodes in the tree.

We next present two techniques that reduces thecomplexity of the adjusting procedure. The first one, branchbased reattaching, reduces the complexity to OðnÞ byreattaching the entire branch bdc instead of individual nodesin bdc . It trades off a small chance of failing to reattaching bdcfor considerable efficiency improvement. The secondtechnique, Subtree-only searching, reduces the reattachingscope to the subtree of dc, which considerably reducessearching time in practice (the complexity is still OðnÞ). Thesubtree-only searching also theoretically guarantees thesearching completeness.

5.1.1 Branch-Based Reattaching

The above adjusting procedure breaks branches into nodesand moves one node at a time. This per-node-reattachingscheme is quite expensive. To reduce the time complexity, weadopts a branch-based reattaching scheme. As its namesuggests, this scheme removes a branch from the congestednode and attaches it entirely to another node, instead ofbreaking the branch into nodes and performing the reattach-ing. Performing reattaching in a branch basis effectivelyreduces the complexity of the adjusting procedure.

One minor drawback of branch-based reattaching is thatit diminishes the chance of finding a parent to reattach thebranch when the branch consists of many nodes. However,the impact of this drawback is quite limited in practice. Asthe adjusting procedure removes and reattaches thesmallest branch first, failing to reattach the branch suggeststhat nodes of the tree all have limited available resource. Inthis case, node-based reattaching is also likely to fail.

5.1.2 Subtree-Only Searching

The tree adjusting procedure tries reattaching the prunedbranch to all nodes in the tree except the congested nodedenoted as dc. This adjustment scheme is not efficient as itenumerates almost every nodes in the tree to test if thepruned branch can be reattached to the node. It turns outthat testing nodes outside dc’s subtree is often unneces-sary. The following theorem suggests that testing nodeswithin dc’s subtree is sufficient as long as the resourcedemand of the node failed to add is higher than that of thepruned branch.

Theorem 1. Given a saturated tree T outputted by theconstruction procedure, df the node failed to add to T , acongested node dc and one of its branches bdc , attaching bdc to anynode outside the subtree of dc causes overload, given that theresource demand of df is no larger than that of bdc , i.e.,udf � ubdc .

Proof. If there exists a node outside the subtree of dc,namely do, that can be attached with bdc without causingoverload, then adding df to do should have succeeded inthe previous execution of the construction procedure, asudf � ubdc . However, T is a saturated tree when addingdf , which leads to a contradiction. tuHence, we improve the efficiency of the original tree

building algorithm by testing all nodes for reattaching onlywhen the resource demand of the node failed to add ishigher than that of the pruned branch. For all other cases,the algorithm performs reattaching test only within thesubtree of dc.


5.2 Ordered Resource Allocation

For a node associated with multiple trees, determining howmuch resource it should assign to each of its associated trees isnecessary. Unfortunately, finding the optimal resourceallocation is difficult because it is not clear how muchresource a node will consume until the tree it connects to isbuilt. Exploring all possible allocations to find the optimal oneis clearly not an option as the computation cost is intractable.

To address this issue, REMO employs an efficient on-demand allocation scheme. Since REMO builds the monitor-ing topology on a tree-by-tree sequential basis, the on-demand allocation scheme defers the allocation decision untilnecessary and only allocates capacity to the tree that is aboutto be constructed. Given a node, the on-demand allocationscheme assigns all current available capacity to the tree that iscurrently under construction. Specifically, given a node iwithresource bi and a list of trees each with resource demand dij,the available capacity assigned to tree j is bi �

Pj�1k¼1 dij. Our

experiment results suggest that our on-demand allocationscheme outperforms several other heuristics.

The on-demand allocation scheme has one drawbackthat may limit its performance when building trees withvery different sizes. As on-demand allocation encouragesthe trees constructed first to consume as much resource asnecessary, the construction of these trees may not invokethe adjusting procedure which saves resource consumptionon parent nodes by reducing their branches. Consequently,resources left for constructing the rest of the trees is limited.

We employ a slightly modified on-demand allocationscheme that relieves this issue with little additionalplanning cost. Instead of not specifying the order ofconstruction, the new scheme constructs trees in the orderof increasing tree size. The idea behind this modification isthat small trees are more cost efficient in the sense that theyare less likely to consume much resource for relaying cost.By constructing trees from small ones to large ones, theconstruction algorithm pays more relaying cost for betterscalability only after small trees are constructed. Ourexperiment results suggest the ordered scheme outperformsthe on-demand scheme in various settings.

6 EXTENSIONS

Our description of REMO so far is based on a simplemonitoring scenario where tasks collect distributed valueswithout aggregation or replication under a uniform valueupdating frequency. Real-world monitoring, however, oftenposes diverse requirements. In this section, we present threeimportant techniques to support such requirements inREMO. The In-network-aggregation-aware planning techni-que allows REMO to accurately estimate per-node resourceconsumption when monitoring data can be aggregatedbefore being passed to parent nodes. The reliabilityenhancement technique provides additional protection tomonitoring data delivery by producing topologies thatreplicate monitoring data and pass them through distinctpaths. The heterogeneous-update-frequency supportingtechnique enables REMO to plan topologies for monitoringtasks with different data updating frequencies by correctlyestimating per-node resource consumption of such mixedworkloads. These three techniques introduce little to no extraplanning cost. Moreover, they can be incorporated into

REMO as plugins when certain functionality is required bythe monitoring environment without modifying the REMOframework.

6.1 Supporting In-Network Aggregation

In-network aggregation is important to achieve efficientdistributed monitoring. Compared with holistic collection,i.e., collecting individual values from nodes, In-networkaggregation allows individual monitoring values to becombined into aggregate values during delivery. Forexample, if a monitoring task requests the SUM of certainmetric m on a set of nodes N , with In-network aggregation,a node can locally aggregate values it receives from othernodes into a single partial sum and pass it to its parent,instead of passing each individual value it receives.

REMO can be extended to build monitoring topology formonitoring tasks with In-network aggregation. We firstintroduce a funnel function to capture the changes ofresource consumption caused by In-network aggregation.Specifically, a funnel function on node i, fnlmi ðgm; nmÞ,returns the number of outgoing values on node i for metricm given the In-network aggregation type gm and thenumber of incoming values nm. The corresponding resourceconsumption of node i in tree k for sending update messageto its parent is,

uik ¼ C þX

m2Ai\APk

a � fnlmi ðgm; nmÞ; ð1Þ

where Ai \APk is the set of metrics node i needs to collect

and report in tree k, a is the per value overhead and C is theper message overhead. For SUM aggregation, the corre-sponding funnel function is fnlmi ðSUMm; nmÞ ¼ 1 becausethe result of SUM aggregation is always a single value.Similarly, for TOP10 aggregation, the funnel function isfnlmi ðTOP10m; nmÞ ¼ minf10; nmg. For holistic aggregationwe discussed earlier, fnlmi ðHOLISTICm; nmÞ ¼ nm. Hence,(1) can be used to calculate per-node resource consumptionfor both holistic aggregation and In-network aggregation inthe aforementioned monitoring tree building algorithm.Note that it also supports the situation where one treeperforms both In-network and holistic aggregation fordifferent metrics.

Some aggregation functions such as DISTINCT, however,are data dependent in terms of the result size. For example,applying DISTINCT on a set X of 10 values results in a setwith size ranging from 1 to 10, depending how manyrepeated values A contains. For these aggregations, wesimply employ the funnel function of holistic aggregation foran upper bound estimation in the current implementation ofREMO. Accurate estimation may require sampling-basedtechniques which we leave as our future work.

6.2 Reliability Enhancements

Enhancing reliability is important for certain mission criticalmonitoring tasks. REMO supports two modes of reliabilityenhancement, same source different paths (SSDP) and differentsources different paths (DSDP), to minimize the impact of linkand node failure. The most distinct feature of the reliabilityenhancement in REMO is that the enhancement is done byrewriting monitoring tasks and requires little modificationto the original approach.


The SSDP mode deals with link failures by duplicating thetransmission of monitored values in different monitoringtrees. Specifically, for a monitoring task t ¼ ða;NtÞ requiringSSDP support, REMO creates a monitoring task t0 ¼ ða0; NtÞwhere a0 is an alias of a. In addition, REMO restricts that a anda0 would never occur in the same set of a partition P duringpartition augmentation, which makes sure messages updat-ing a and a0 are transmitted within different monitoring trees,i.e., different paths. Note that the degree of reliability can beadjusted through different numbers of duplications.

When a certain metric value is observable at multiplenodes, REMO also supports the DSDP mode. For example,computing nodes sharing the same storage observe the samestorage performance metric values. Under this mode, userssubmit monitoring tasks in the form of t ¼ ða;NidenticalÞwhere Nidentical ¼ Nðv1Þ; Nðv2Þ; . . . ; NðvnÞ. NðviÞ denotes theset of nodes that observe the same value vi and Nidentical is aset of node groups each of which observes the same value.Let k ¼ minfjNðviÞj; i 2 ½1; n�g. REMO rewrites t into kmonitoring tasks so that each task collects values for metrica with a distinct set of nodes drawn from NðviÞ; i 2 ½1; n�.Similar to SSDP model, REMO then constructs the topologyby avoiding any of the kmonitoring tasks to be clustered intoone tree. In this way, REMO ensures values of metrics can becollected from distinct sets of nodes and delivered throughdifferent paths.

6.3 Supporting Heterogeneous Update Frequencies

Monitoring tasks may collect values from nodes withdifferent update frequencies. REMO supports heteroge-neous update frequencies by grouping nodes based on theirmetric collecting frequencies and constructing per-groupmonitoring topologies. When a node has a single metric awith the highest update frequency, REMO considers thenode as having only one metric to update as other metricspiggyback on a. When a node has a set of metrics updatedat the same highest frequencies, denoted by Ah, it evenlyassigns other metrics to piggyback on metrics in of Ah.Similarly, REMO considers the node as having a set ofmetrics Ah to monitor as other metrics piggyback on Ah. Weestimate the cost of updating with piggybacked metrics fornode i by ui ¼ C þ a �

Pj freqj=freqmax where freqj is the

frequency of one metric collected on node i and freqmax isthe highest update frequency on node i.

Sometimes metric piggybacking cannot achieve theprecise update frequency defined by users. For example, ifthe highest update frequency on a node is 1/5 (msg/sec), ametric updated at 1/22 can at best be monitored at either 1/20 or 1/25. If users are not satisfied with such an approxima-tion, our current implementation separates these metrics outand builds individual monitoring trees for each of them.

7 EXPERIMENTAL EVALUATION

We undertake an experimental study of our system andpresent results including those gathered by deployingREMO on a BlueGene/P rack (using 256 nodes booted intoLinux) running IBM’s large-scale distributed streamingsystem—System S.

Synthetic data set experiments. For our experiments onsynthetic data, we assign a random subset of attributes to

each node in the system. For monitoring tasks, we generatethem by randomly selecting jAtj attributes and jNtj nodeswith uniform distribution, for a given size of attribute set Aand node set N . We also classify monitoring tasks into twocategories—1) small-scale monitoring tasks that are for asmall set of attributes from a small set of nodes, and 2)large-scale monitoring tasks that either involves manynodes or many attributes.

To evaluate the effectiveness of different topologyconstruction schemes, we measure the percentage ofattribute values collected at the root node with themonitoring topology produced by a scheme. Note that thisvalue should be 100 percent when the monitoring workloadis trivial or each monitoring node is provisioned withabundant monitoring resources. For comparison purposes,we apply relatively heavy monitoring workloads to keepthis value below 100 percent for all schemes. This allows usto easily compare the performance of different schemes bylooking at their percentage of collected values. Schemeswith higher percentage of collected values not only achievebetter monitoring coverage when monitoring resources arelimited, but also have better monitoring efficiency in termsof monitoring resource consumption.

Real system experiments. Through experiments in a realsystem deployment, we also show that the error in attributevalue observations (due to either stale or dropped attributevalues) introduced by REMO is small. Note that this errorcan be measured in a meaningful way only for a real systemand is what any “user” of the monitoring system wouldperceive when using REMO.

System S is a large-scale distributed stream processingmiddleware. Applications are expressed as dataflow graphsthat specify analytic operators interconnected by datastreams. These applications are deployed in the System Sruntime as processes executing on a distributed set of hosts,and interconnected by stream connections using transportssuch as TCP/IP. Each node that runs application processescan observe attributes at various levels such as at theanalytic operator level, System S middleware level, and theOS level. For these experiments, we deployed one suchSystem S application called YieldMonitor [18], that monitorsa chip manufacturing test process and uses statisticalstream processing to predict the yield of individual chipsacross different electrical tests. This application consisted ofover 200 processes deployed across 200 nodes, with 30-50attributes to be monitored on each node, on a BlueGene/Pcluster. The BlueGene is very communication rich and allcompute nodes are interconnected by a 3D Torus meshnetwork. Consequently, for all practical purposes, we havea fully connected network where all pairs of nodes cancommunicate with each other at almost equal cost.

7.1 Result Analysis

We present a small subset of our experimental results tohighlight the following observations amongst others. First,REMO can collect a larger fraction of node-attribute pairs toserve monitoring tasks presented to the system compared tosimple heuristics (which are essentially the state-of-the-art).REMO adapts to the task characteristics and outperformseach of these simple heuristics for all types of tasks andsystem characteristics, e.g., for small-scale tasks, a collection


mechanism with fewer trees is better while for large-scaletasks, a collection mechanism with more trees is better.Second, in a real application scenario, REMO also sig-nificantly reduces percentage error in the observed valuesof the node-attribute pairs required by monitoring taskswhen compared to simple heuristics.

Varying the scale of monitoring tasks. Fig. 5 comparesthe performance of different attribute set partition schemesunder different workload characteristics. In Fig. 5a, wherewe increase the number of attributes in monitoring tasks,i.e., increasing jAtj, our partition augmentation scheme(REMO) performs consistently better than singleton-set(SINGLETON-SET) and one-set (ONE-SET) schemes. Inaddition, ONE-SET outperforms SINGLETON-SET whenjAtj is relatively small. As each node only sends out onemessage which includes all its own attributes and thosereceived from its children, ONE-SET causes the minimumper-message overhead. Thus, when each node monitorsrelatively small number of attributes, it can efficientlydeliver attributes without suffering from its scalabilityproblem. However, when jAtj increases, the capacitydemand of the low-level nodes, i.e., nodes that are close tothe root, increases significantly, which in turn limits the sizeof the tree and causes poor performance. In Fig. 5b, wherewe set jAtj ¼ 100 and increase jNtj to create extremely heavyworkloads, REMO gradually converges to SINGLETON-SET, as SINGLETON-SET achieves the best load balanceunder heavy workload which in turn results in the bestperformance.

Varying the number of monitoring tasks. We observesimilar results in Figs. 5c and 5d, where we increase thetotal number of small-scale and large-scale monitoringtasks, respectively.

Varying nodes in the system. Fig. 6 illustrates theperformance of different attribute set partition schemeswith changing system characteristics. In Figs. 6a and 6b,where we increase the number of nodes in the system givensmall- and large-scale monitoring tasks, respectively, wecan see SINGLETON-SET is better for large-scale taskswhile ONE-SET is better for small-scale tasks, and REMO

performs much better than them in both cases, around90 percent extra collected node-attribute pairs.

Varying per-message processing overhead. To study theimpact of per-message overhead, we vary the C=a ratiounder both small- and large-scale monitoring tasks in Figs. 6cand 6d. As expected, increased per-message overhead hitsthe SINGLETON-SET scheme hard since it constructs a largenumber of trees and, consequently, incurs the largestoverhead cost while the performance of the ONE-SETscheme which constructs just a single tree degrades moregracefully. However having a single tree is not the bestsolution as shown by REMO which outperforms both theschemes as C=a is increased, because it can reduce thenumber of trees formed when C=a is increased.

Comparison of tree-construction schemes. In Fig. 7, westudy the performance of different tree constructionalgorithms under different workloads and system charac-teristics. Our comparison also includes a new algorithm,namely MAX_AVB, a heuristic algorithm used in TMON[17] which always attaches new node to the existing nodewith the most available capacity. While we vary differentworkloads and system characteristics in the four figures,our adaptive tree construction algorithm (ADAPTIVE)always performs the best in terms of percentage of collectedvalues. Among all the other tree construction schemes,STAR performs well when workload is heavy, as suggestedby Figs. 7a and 7b. This is because STAR builds trees withminimum height, and thus, pays minimum cost for


Fig. 5. Comparison of attribute set partition schemes under differentworkload characteristics.

Fig. 6. Comparison of attribute set partition schemes under differentsystem characteristics.

Fig. 7. Comparison of tree construction schemes under differentworkload and system characteristics.

relaying, which can be considerable when workloads areheavy. CHAIN performs the worst in almost all cases.While CHAIN provides good load balance by distributingper-message overhead in CHAIN-like trees, nodes have topay high cost for relaying, which seriously degrades theperformance of CHAIN when workloads are heavy (per-forms the best when workloads are light as indicated by theleft portions of both Figs. 7a and 7b). MAX_AVB schemeoutperforms both STAR and CHAIN given small workload,as it avoids over stretching a tree in breadth or height bygrowing trees from nodes with the most available capacity.However, its performance quickly degrades with increasingworkload as a result of relaying cost.

Real-world performance. To evaluate the performanceof REMO in a real-world application. we measure theaverage percentage error of received attribute values forsynthetically generated monitoring tasks. Specifically, wemeasure average percentage error between the snapshot ofvalues observed by our scheme and compare it to thesnapshot of “actual” values (that can be obtained bycombining local log files at the end of the experiment).Fig. 8a compares the achieved percentage error betweendifferent partition schemes given increasing number ofnodes. Recall that our system can deploy the applicationover any number of nodes. The figure shows that ourpartition augmentation scheme in REMO outperforms theother partition schemes. The percentage error achieved byREMO is around 30-50 percent less than that achieved bySINGLETON-SET and ONE-SET. Interestingly, the percen-tage error achieved by REMO clearly reduces when thenumber of nodes in the system increases. However,according to our previous results, the number of nodeshas little impact on the coverage of collected attributes. Thereason is that as the number of nodes increases, monitoringtasks are more sparsely distributed among nodes. Thus,each message is relatively small and each node can havemore children. As a result, the monitoring trees constructedby our schemes are “bushier,” which in turn reduces the

percentage error caused by latency. Similarly, we can seethat REMO gains significant error reduction compared withthe other two schemes in Fig. 8b where we compare theperformance of different partition schemes under increas-ing monitoring tasks.

Runtime adaptation. To emulate a dynamic monitoringenvironment with a small portion of changing tasks, wecontinuously update (modify) the set of running tasks withincreasing update frequency. Specifically, we randomlyselect 5 percent of monitoring nodes and replaces 50 percentof their monitoring attributes. We also vary the frequencyof task updates to evaluate the effectiveness of our adaptivetechniques.

We compare the performance and cost of four differentschemes: 1) DIRECT-APPLY (D-A) scheme which directlyapplies the changes in the monitoring task to the monitor-ing topology. 2) REBUILD scheme which always performsfull-scale search from the initial topology with techniqueswe introduced in Section 3. 3) NO-THROTTLE schemewhich searches for optimized topology that is close to thecurrent one with techniques we introduced in Section 4.4) ADAPTIVE scheme is the complete technique setdescribed in Section 4, which improves NO-THROTTLEby applying cost-benefit throttling to avoid frequenttopology adaptation when tasks are frequently updated.

Fig. 9a shows the CPU time of running differentplanning schemes given increasing task updating fre-quency. The X-axis shows the number of task updatebatches within a time window of 10 value updates. The Y -axis shows the CPU time (measured on a Intel CORE Duo2 2.26 GHz CPU) consumed by each scheme. We can seethat D-A takes less than 1 second to finish since it performsonly a single round tree construction. REBUILD consumesthe most CPU time among all schemes as it always exploresthe entire searching space. When cost-benefit throttling isnot applied the NO-THROTTLE scheme consumes lessCPU time. However, its CPU time grows with task updatefrequency which is not desirable for large-scale monitoring.With throttling, the adaptive schemeincurs even less CPUtime (1-3 s) as it avoids unnecessary topology optimizationfor frequent updates. Note that while the CPU timeconsumed by ADAPTIVE is higher than that of D-A, it isfairly acceptable.

Fig. 9b illustrates the percentage of adaptation cost overthe total cost for each scheme. Here, the adaptation cost ismeasured by the total number of messages used tonotifying monitoring nodes to change monitoring topology(e.g., one such message may inform a node to disconnectfrom its current parent node and connect to another parent


Fig. 8. Comparison of average percentage error.

Fig. 9. Performance comparison of different adaptation schemes given increasing task updating frequencies.

node). Similarly, the total cost of a scheme is the totalnumber of messages the scheme used for both adaptationand delivering monitoring data. REBUILD introducesthe highest adaptation cost because it always pursues theoptimal topology which can be quite different from thecurrent one. Similar to what we observed in Fig. 9a, NO-THROTTLE achieves much lower adaption cost comparedwith REBUILD does. ADAPTIVE further reduces adapta-tion cost which is very close to that of D-A, because cost-benefit throttling allows it to avoid unnecessary topologyoptimization when task updating frequency grows.

Fig. 9c shows the scheme-wise difference of the total cost(including both adaptation and data delivery messages).The Y -axis shows the ratio (percentage) of total costassociated with one scheme over that associated with D-A. REBUILD initially outperforms D-A as it producesoptimized topology which in turn saves monitoring com-munication cost. Nevertheless, as task updating frequencyincreases, the communication cost of adaptation messagesgenerated by REBUILD increases quickly, and eventuallythe extra cost in adaptation surpasses the monitoringcommunication cost it saves. NO-THROTTLE shows similargrowth of total cost with increasing task updating fre-quency. ADAPTIVE, however, consistently outperforms D-A due to its ability to avoid unnecessary optimization.

Fig. 9d shows the performance of schemes in terms ofcollected monitoring attribute values. The Y -axis shows thepercentage of collected values of one scheme over that of D-A. Note that the result we show in Fig. 9c is the generatedtraffic volume. As each node cannot process traffics beyondits capacity, the more traffic generated, the more likely weobserve miss-collected values. With increasing task updat-ing frequency, the performance of REBUILD degradesfaster than that of D-A due to its quickly growing cost intopology optimization (see Figs. 9b and 9c). On the contrary,both NO-THROTTLE and ADAPTIVE gain an increasingperformance advantage over D-A. This is because themonitoring topology can still be optimized with relativelylow adaptation cost with NO-THROTTLE and ADAPTIVE,but continuously degrades with D-A, especially with hightask updating frequency.

Overall, ADAPTIVE produces monitoring topologieswith the best value collection performance (Fig. 9d), whichis the ultimate goal of monitoring topology planning. Itachieves this by minimizing the overall cost of the topology(Fig. 9c) by only adopting adaptations whose gain outweighscost. Its searching time and adaptation cost, although slighterhigher than schemes such as D-A, is fairly small for allpractical purposes.

Optimization. Figs. 10a and 10b show the speedup ofour optimization techniques for the monitoring tree adjust-ment procedure, where the Y -axis shows the speedup ofone technique over the basic adjustment procedure, i.e., theratio between CPU time of the basic adjustment procedureover that of an optimized procedure. Because the basicadjustment procedure reattaches a branch by first breakingup the branch into individual nodes and performing a per-node-based reattaching, it takes considerably more CPUtime compared with our branch-based reattach and subtree-only reattach techniques. With both techniques combined,we observe a speedup at up to 11 times, which is especiallyimportant for large distributed systems. We also find thatthese two optimization techniques introduce little perfor-mance penalties in terms of the percentage of valuescollected from the resulting monitoring topology(<2%).

Figs. 11a and 11b compare the performance of differenttree-wise capacity allocation schemes, where UNIFORMdivides the capacity of one node equally among all trees itparticipates in, PROPORTIONAL divides the capacityproportionally according to the size of each tree, ON-DEMAND and ORDERED are our allocation techniquesintroduced in Section 5.2. We can see that both ON-DEMAND and ORDERED consistently outperform UNI-FORM and PROPORTIONAL. Furthermore, ORDEREDgains an increasing advantage over ON-DEMAND withgrowing nodes and tasks. This is because large number ofnodes and tasks cause one node to participate into trees withvery different sizes, where ordered allocation is useful foravoiding improper node placement, e.g., putting one node asroot in one tree (consuming much of its capacity) while it stillneeds to participate in other trees.

Extension. Fig. 12a compares the efficiency of basicREMO with that of extended REMO when given tasks thatinvolves In-network aggregation and heterogeneous updatefrequencies. Specifically, we apply MAX In-network aggre-gation to tasks so that one node only needs to send thelargest value to its parent node. In addition, we randomly


Fig. 10. Speedup of optimization schemes. Fig. 11. Comparison between resource allocation schemes.

Fig. 12. Performance of extension techniques.

choose half of the tasks and reduce their value updatefrequency by half. The Y -axis shows values collected byREMO enhanced with one extension technique, normalizedby values collected by the basic REMO. Note that collectedvalues for MAX In-network aggregation refer to valuesincluded in the MAX aggregation, and are not necessarilycollected by the root node.

The basic REMO approach is oblivious to In-networkaggregation. Hence, it tends to overestimate communicationcost of the monitoring topology, and prefers SINGLETON-SET-like topology where each tree delivers one or fewattributes. As we mentioned earlier, such topologiesintroduce high per-message overhead. On the contrary,REMO with aggregation-awareness employs funnel func-tions to correctly estimate communication cost and pro-duces more efficient monitoring topology. We observesimilar results between the basic REMO and REMO withupdate-frequency-awareness. When both extension techni-ques are combined, they can provide an improvement closeto 50 percent in terms of collected values.

Fig. 12b compares the efficiency of REMO with replicationsupport and two alternative techniques. The SINGLETON-SET-2 scheme uses two SINGLETON-SET trees to delivervalues of one attribute separately. The ONE-SET-2 schemecreates two ONE-SET trees, each of which connects all nodesand delivers values of all attributes separately. REMO-2 isour Same-Source-Different-Path technique with a replicationfactor of 2, i.e., values of each attribute are delivered throughtwo different trees. Compared with the two alternativeschemes, REMO-2 achieves both replication and efficiency bycombining multiple attributes into one tree to reduce per-message overhead. As a result, it outperforms both alter-natives consistently given increasing monitoring tasks.

8 RELATED WORK

Much of the early work addressing the design of distributedquery systems mainly focuses on executing single queriesefficiently. As the focus of our work is to support multiplequeries, we omit discussing these work. Research onprocessing multiple queries on a centralized data stream[19], [13], [20], [21] is not directly related with our workeither, as the context of our work is distributed streamingwhere the number of messages exchanged between thenodes is of concern.

A large body of work studies query optimization andprocessing for distributed databases (see [22] for a survey).Although our problem bears a superficial resemblance tothese distributed query optimization problems, our pro-blem is fundamentally different since in our problemindividual nodes are capacity constrained. There are alsomuch work on multiquery optimization techniques forcontinuous aggregation queries over physically distributeddata streams [21], [23], [24], [25], [19], [13], [20], [26]. Theseschemes assume that the routing trees are provided as partof the input. In our setting where we are able to choose frommany possible routing trees, solving the joint problem ofoptimizing resource-constrained routing tree constructionand multitask optimization provides significant benefitover solving only one of these problems in isolation asevidenced by our experimental results.

More recently, several work studies efficient data collec-tion mechanisms. CONCH [27] builds a spanning forest withminimal monitoring costs for continuously collecting read-ings from a sensor network by utilizing temporal and spatialsuppression. However, it does not consider the resourcelimitation at each node and per-message overhead as we did,which may limit its applicability in real-world applications.PIER [8] suggests using distinct routing trees for each queryin the system, in order to balance the network load, which isessentially the SINGLETON-SET scheme we discussed. Thisscheme, though achieves the best load balance, may causesignificant communication cost on per-message overhead.

9 CONCLUSIONS

We developed REMO, a resource-aware multitask optimiza-tion framework for scaling application state monitoring inlarge-scale distributed systems. The unique contribution ofthe REMO approach is its techniques for generating thenetwork of monitoring trees that optimizes multiple mon-itoring tasks and balances the resource consumption atdifferent nodes. We also proposed adaptive techniques toefficiently handle continuous task updates, optimizationtechniques that speedup the searching process up to a factorof 10, and techniques extending REMO to support advancedmonitoring requirements such as In-network aggregation.We evaluated REMO through extensive experiments includ-ing deploying REMO in a real-world stream processingapplication hosted on BlueGene/P. Our experimental resultsshow that REMO significantly and consistently outperformsexisting approaches.

ACKNOWLEDGMENTS

This work is partially supported by NSF grants from CISECyberTrust program, NetSE program, and CyberTrustCross Cutting program, and an IBM faculty award and anIntel ISTC grant on Cloud Computing.

REFERENCES

[1] N. Jain, L. Amini, H. Andrade, R. King, Y. Park, P. Selo, and C.Venkatramani, “Design, Implementation, and Evaluation of theLinear Road Bnchmark on the Stream Processing Core,” Proc.ACM SIGMOD Int’l Conf. Management of Data (SIGMOD), 2006.

[2] B. Hayes, “Cloud Computing,” Comm. ACM, vol. 51, no. 7, pp. 9-11, 2008.

[3] L. Amini, N. Jain, A. Sehgal, J. Silber, and O. Verscheure,“Adaptive Control of Extreme-Scale Stream Processing Systems,”Proc. IEEE 26th Int’l Conf. Distributed Computing Systems (ICDCS),2006.

[4] J. Borkowski, D. Kopanski, and M. Tudruj, “Parallel IrregularComputations Control Based on Global Predicate Monitoring,”Proc. Int’l Symp. Parallel Computing in Electrical Eng. (PARELEC),2006.

[5] K. Park and V.S. Pai, “Comon: A Mostly-Scalable MonitoringSystem for Planetlab,” Operating Systems Rev., vol. 40, no. 1, pp. 65-74, 2006.

[6] S. Madden, M.J. Franklin, J.M. Hellerstein, and W. Hong, “Tag: ATiny Aggregation Service for Ad-Hoc Sensor Networks,” Proc.Fifth Symp. Operating Systems Design and Implementation (OSDI),2002.

[7] P. Yalagandula and M. Dahlin, “A Scalable Distributed Informa-tion Management System,” Proc. SIGCOMM, pp. 379-390, 2004.

[8] R. Huebsch, B.N. Chun, J.M. Hellerstein, B.T. Loo, P. Maniatis, T.Roscoe, S. Shenker, I. Stoica, and A.R. Yumerefendi, “TheArchitecture of Pier: An Internet-Scale Query Processor,” Proc.Second Conf. Innovative Data Systems Research (CIDR), 2005.


[9] G. Cormode and M.N. Garofalakis, “Sketching Streams throughthe Net: Distributed Approximate Query Tracking,” Proc. 31st Int’lConf. Very Large Data Bases (VLDB), pp. 13-24, 2005.

[10] D.J. Abadi, S. Madden, and W. Lindner, “Reed: Robust, EfficientFiltering, and Event Detection in Sensor Networks,” Proc. 31st Int’lConf. Very Large Databases (VLDB), 2005.

[11] U. Srivastava, K. Munagala, and J. Widom, “Operator Placementfor In-Network Stream Query Processing,” Proc. ACM SIGMOD-SIGACT-SIGART Symp. Principles of Database Systems (PODS),pp. 250-258, 2005.

[12] C. Olston, B.T. Loo, and J. Widom, “Adaptive Precision Setting forCached Approximate Values,” Proc. ACM SIGMOD Int’l Conf.Management of Data (SIGMOD ’01), 2001.

[13] S. Krishnamurthy, C. Wu, and M.J. Franklin, “On-the-Fly Sharingfor Streamed Aggregation,” Proc. ACM SIGMOD Int’l Conf.Management of Data (SIGMOD ’06), pp. 623-634, 2006.

[14] S. Ko and I. Gupta, “Efficient On-Demand Operations in DynamicDistributed Infrastructures,” Proc. Second Workshop Large-ScaleDistributed Systems and Middleware (LADIS), 2008.

[15] S. Meng, S.R. Kashyap, C. Venkatramani, and L. Liu, “Remo:Resource-Aware Application State Monitoring for Large-ScaleDistributed Systems,” Proc. IEEE 29th Int’l Conf. DistributedComputing Systems (ICDCS), pp. 248-255, 2009.

[16] K. Marzullo and M.D. Wood, Tools for Constructing DistributedReactive Systems. Cornell Univ., 1991.

[17] S.R. Kashyap, D. Turaga, and C. Venkatramani, “Efficient Treesfor Continuous Monitoring,” 2008.

[18] D.S. Turaga, M. Vlachos, O. Verscheure, S. Parthasarathy, W. Fan,A. Norfleet, and R. Redburn, “Yieldmonitor: Real-Time Monitor-ing and Predictive Analysis of Chip Manufacturing Data,” 2008.

[19] R. Zhang, N. Koudas, B.C. Ooi, and D. Srivastava, “MultipleAggregations over Data Streams,” Proc. ACM SIGMOD Int’l Conf.Management of Data (SIGMOD), 2005.

[20] J. Li, D. Maier, K. Tufte, V. Papadimos, and P.A. Tucker, “NoPane, No Gain: Efficient Evaluation of Sliding-Window Aggre-gates over Data Streams,” ACM SIGMOD Record, vol. 34, no. 1,pp. 39-44, 2005.

[21] S. Madden, M.A. Shah, J.M. Hellerstein, and V. Raman,“Continuously Adaptive Continuous Queries over Streams,” Proc.ACM SIGMOD Int’l Conf. Management of Data (SIGMOD), 2002.

[22] D. Kossmann, “The State of the Art in Distributed QueryProcessing,” ACM Computing Surveys, vol. 32, no. 4, pp. 422-469,2000.

[23] R. Huebsch, M.N. Garofalakis, J.M. Hellerstein, and I. Stoica,“Sharing Aggregate Computation for Distributed Queries,” Proc.ACM SIGMOD Int’l Conf. Management of Data (SIGMOD), 2007.

[24] A. Silberstein and J. Yang, “Many-to-Many Aggregation forSensor Networks,” Proc. 27th Int’l Conf. Data Eng. (ICDE),pp. 986-995, 2007.

[25] S. Xiang, H.-B. Lim, K.-L. Tan, and Y. Zhou, “Two-Tier MultipleQuery Optimization for Sensor Networks,” Proc. 27th Int’l Conf.Distributed Computing Systems (ICDCS), p. 39, 2007.

[26] J. Borkowski, “Hierarchical Detection of Strongly ConsistentGlobal States,” Proc. Third Int’l Symp. Parallel and DistributedComputing/Third Int’l Workshop Algorithms, Models, and Tools forParallel Computing on Heterogeneous Networks (ISPDC/HeteroPar),pp. 256-261, 2004.

[27] A. Silberstein, R. Braynard, and J. Yang, “Constraint Chaining:On Energy-Efficient Continuous Monitoring in Sensor Net-works,” Proc. ACM SIGMOD Int’l Conf. Management of Data(SIGMOD), 2006.

Shicong Meng currently working toward thePhD degree in the College of Computing atGeorgia Tech. He is affiliated with the Center forExperimental Research in Computer Systems(CERCS) where he works with professor LingLiu. His research focuses on performance,scalability, and security issues in large-scaledistributed systems such as cloud datacenters.He is a student member of the IEEE.

Srinivas Raghav Kashyap received the PhDdegree in computer science from the Universityof Maryland at collage park in 2007. Currently,he is a software engineer at Google Inc. and wasa postdoctoral researcher at IBM T.J. WatsonResearch Center before moving to Google.

Chitra Venkatramani received the PhD degreein computer science from SUNY at Stony Brookin 1997, and was also engaged in variousprojects around multimedia data streaming andcontent distribution at IBM’s Watson labs. She isa research staff member and manager of theDistributed Streaming Systems Group at theIBM T.J. Watson Research Center. The focus ofher team is on research related to System S, asoftware platform for large scale, distributed

stream processing. Her team explores issues in high performance,large-scale distributed computing systems, including advanced compu-tation and communication infrastructures, fault tolerance, scheduling,and resource management for data-intensive applications. She holdsnumerous patents and publications and is also a recipient of IBM’sOutstanding Technical Achievement Award.

Ling Liu is a full professor in the School ofComputer Science at Georgia Institute of Tech-nology. There she directs the research programsin Distributed Data Intensive Systems Lab(DiSL), examining various aspects of dataintensive systems with the focus on perfor-mance, availability, security, privacy, and energyefficiency. He has served as general chair andPC chairs of numerous IEEE and ACM confer-ences in data engineering, distributed comput-

ing, service computing, and cloud computing fields and is a coeditor-in-chief of the 5 volume Encyclopedia of Database Systems (Springer).Currently, she is on the editorial board of several international journals.He has published more than 300 International journal and conferencearticles in the areas of databases, distributed systems, and internetcomputing. She is a recipient of the Best Paper Award of ICDCS 2003,WWW 2004, the 2005 Pat Goldberg Memorial Best Paper Award, and2008 international conference on Software Engineering and DataEngineering. Her current research is primarily sponsored by NSF,IBM, and Intel. She is a senior member of the IEEE.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


Documents

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED …lingliu/papers/2012/tpds-Remo.pdf · test hypotheses on system behavior [1]. Similarly, the data rate and buffer occupancy in each