JustRunIt: Experiment-Based Management of Virtualized Data Centers

Wei Zheng, Ricardo Bianchini, G. John Janakiraman, Jose Renato Santos, and Yoshio Turner

Department of Computer Science HP LabsRutgers University Hewlett-Packard Corporation

{wzheng,ricardob}@cs.rutgers.edu {joserenato.santos,yoshio.turner}@hp.com

AbstractManaging data centers is a challenging endeavor. State-of-the-art management systems often rely on analyticalmodeling to assess the performance, availability, and/orenergy implications of potential management decisionsor system congurations. In this paper, we argue thatactual experiments are cheaper, simpler, and more ac-curate than models for many management tasks. To sup-port this claim, we built an infrastructure for experiment-based management of virtualized data centers, calledJustRunIt. The infrastructure creates a sandboxed en-vironment in which experiments can be runon a verysmall number of machinesusing real workloads andreal system state, but without affecting the on-line sys-tem. Automated management systems or the system ad-ministrator herself can leverage our infrastructure to per-form management tasks on the on-line system. To evalu-ate the infrastructure, we apply it to two common tasks:server consolidation/expansion and evaluating hardwareupgrades. Our evaluation demonstrates that JustRunItcan produce results realistically and transparently, and benicely combined with automated management systems.

1 Introduction

Managing data centers is a challenging endeavor, es-pecially when done manually by system administrators.One of the main challenges is that performing many man-agement tasks involves selecting a proper resource allo-cation or system conguration out of a potentially largenumber of possible alternatives. Even worse, evaluatingeach possible management decision often requires under-standing its performance, availability, and energy con-sumption implications. For example, a common man-agement task is to partition the systems resources acrossapplications to optimize performance and/or energy con-sumption, as is done in server consolidation and virtualmachine (VM) placement. Another example is the eval-uation of software or hardware upgrades, which involves

determining whether application or system behavior willbenet from the candidate upgrades and by how much.Along the same lines, capacity planning is a commonmanagement task that involves selecting a proper systemconguration for a set of applications.

Previous efforts have automated resource-partitioningtasks using simple heuristics and/or feedback control,e.g. [1, 6, 18, 21, 28, 29]. These policies repeatedlyadjust the resource allocation to a change in system be-havior, until their performance and/or energy goals areagain met. Unfortunately, when this react-and-observeapproach is not possible, e.g. when evaluating softwareor hardware upgrades, these policies cannot be applied.

In contrast, analytical modeling can be used to auto-mate all of these management tasks. Specically, mod-eling can be used to predict the impact of the possiblemanagement decisions or system congurations on per-formance, availability, and/or energy consumption. Withthese predictions, the management system can makethe best decision. For example, researchers have builtresource-partitioning systems for hosting centers that usemodels to predict throughput and response time, e.g.[9, 27]. In addition, researchers have built systems thatuse models to maximize energy conservation in data cen-ters, e.g. [7, 13]. Finally, researchers have been build-ing models that can predict the performance of Internetapplications on CPUs with different characteristics [23];such models can be used in deciding whether to upgradethe server hardware.

Performance models are often based on queuing the-ory, whereas availability models are often based onMarkovian formalisms. Energy models are typicallybased on simple (but potentially inaccurate) models ofpower consumption, as a function of CPU utilization orCPU voltage/frequency. On the bright side, these modelsare useful in data center management as they provide in-sight into the systems behaviors, can be solved quickly,and allow for large parameter space explorations. Essen-tially, the models provide an efcient way of answering

what-if questions during management tasks.Unfortunately, modeling has a few serious shortcom-

ings. First, modeling consumes a very expensive re-source: highly skilled human labor to produce, calibrate,and validate the models. Second, the models typicallyrely on simplifying assumptions. For example, memory-less arrivals is a common assumption of queuing modelsfor Internet services [24]. However, this assumption isinvalid when requests come mostly from existing ses-sions with the service. Another common simplifyingassumption is the cubic relationship between CPU fre-quency and power consumption [7]. With advances inCPU power management, such as clock gating, the exactpower behavior of the CPU is becoming more complexand, thus, more difcult to model accurately. Third, themodels need to be re-calibrated and re-validated as thesystems evolve. For example, the addition of new ma-chines to a service requires queuing models to be cali-brated and validated for them.

Given these limitations, in this paper we argue thatactual experiments are a better approach than modelingfor supporting many management tasks. Actual experi-ments exchange an expensive resource (human labor) formuch cheaper ones (the time and energy consumed bya few machines in running the experiments). Moreover,they do not rely on simplifying assumptions or requirecalibration and validation. Thus, actual experiments arecheaper, simpler, and more accurate than models in theirability to answer what-if questions. We further arguethat the experiments can be performed in a exible, real-istic, and transparent manner by leveraging current virtu-alization technology.

To support our claims in a challenging environment,we built JustRunIt, an infrastructure for experiment-based management of virtualized data centers hostingmultiple Internet services. JustRunIt creates a sand-boxed environment in which experiments can be run ona small number of machines (e.g., one machine per tierof a service) without affecting the on-line system. Jus-tRunIt clones a small subset of the on-line VMs (e.g.,one VM per tier of the service) and migrates them to thesandbox. In the sandbox, JustRunIt precisely controlsthe resources allocated to the VMs, while offering thesame workload to them that is offered to similar VMson-line. Workload duplication is implemented by Jus-tRunIts server proxies. For exibility, the administra-tor can specify the resources (and the range of alloca-tions) with which to experiment and how long experi-ments should be run. If there is not enough time to runall possible experiments (i.e., all combinations of accept-able resource allocations), JustRunIt uses interpolationbetween actual experimental results to produce the miss-ing results but ags them as potentially inaccurate.

Automated management systems or the system admin-

istrator can use the JustRunIt results to perform manage-ment tasks on the on-line system. If any interpolatedresults are actually used by the system or administra-tor, JustRunIt runs the corresponding experiments in thebackground and warns the administrator if any experi-mental result differs from the corresponding interpolatedresult by more than a threshold amount.

To evaluate our infrastructure, we apply it to systemsthat automate two common management tasks: serverconsolidation/expansion and evaluation of hardware up-grades. Modeling has been used in support of both tasks[7, 24], whereas feedback control is only applicable forsome cases of the former [7]. JustRunIt combines nicelywith both systems. Our evaluation demonstrates thatJustRunIt can produce results realistically and transpar-ently, enabling automated management systems to per-form their tasks effectively. In fact, JustRunIt can pro-duce system congurations that are as good as those re-sulting from idealized, perfectly accurate models, at thecost of the time and energy dedicated to experiments.

The remainder of the paper is organized as follows.The next section describes JustRunIt in detail. Section3 describes the automated management systems that wedesigned for our two case studies. Section 4 presents ourevaluation of JustRunIt and the results of our case stud-ies. Section 5 overviews the related work. Finally, Sec-tion 6 draws our conclusions, discusses the limitations ofJustRunIt, and mentions our future work.

2 JustRunIt Design and Implementation

2.1 Target Environment

Our target environment is virtualized data centers thathost multiple independent Internet services. Each servicecomprises multiple tiers. For instance, a typical three-tierInternet service has a Web tier, an application tier, and adatabase tier. Each tier may be implemented by multipleinstances of a software server, e.g. multiple instances ofApache may implement the rst tier of a service. Eachservice has strict response-time requirements speciedin SLAs (Service Level Agreements) negotiated betweenthe service provider and the data center.

In these data centers, all services are hosted in VMsfor performance and fault isolation, easy migration, andresource management exibility. Moreover, each soft-ware server of a service is run on a different VM. VMshosting software servers from different services may co-locate on a physical machine (PM). However, VMs host-ing software servers from the same service tier are hostedon different PMs for high availability. All VMs havenetwork-attached storage provided by a storage server.

Figure 1: Overview of JustRunIt. X represents a re-sult obtained through experimentation, whereas I rep-resents an interpolated result. T represents an interpo-lated result that has been used by the management entity.

2.2 System InfrastructureFigure 1 shows an overview of the system infrastruc-ture of JustRunIt. There are four components: exper-imenter, driver, interpolator, and checker. The experi-menter implements the VM cloning and workload dupli-cation mechanism to run experiments. Each experimenttests a possible conguration change to a cloned soft-ware server under the current live workload. A congu-ration change may be a different resource allocation (e.g.,a larger share of the CPU) or a different hardware setting(e.g., a higher CPU voltage/frequency). The results ofeach experiment are reported as the server throughput,response time, and energy consumption observed underthe tested conguration.

The experiment driver chooses which experiments torun in order to efciently explore the conguration pa-rameter space. The driver tries to minimize the numberof experiments that must be run while ensuring that allthe experiments complete within a user-specied timebound. The driver and experimenter work together toproduce a matrix of experimental results in the cong-uration parameter space. The coordinates of the matrixare the conguration parameter values for each type ofresource, and the values recorded at each point are theperformance and energy metrics observed for the corre-sponding resource assignments.

Blank entries in the matrix are lled in by the interpo-lator, based on linear interpolation from the experimen-tal results in the matrix. The lled matrix is providedto the management entityi.e., the system administratoror an automated management systemfor use in decidingresource allocations for the production system.

If the management entity uses any of the interpolatedperformance or energy values, the checker invokes the

Figure 2: Virtualized data center and JustRunIt sandbox.Each box represents a VM, whereas each group of boxesrepresents a PM. W2, A2, and D2 mean Web, ap-plication, and database server of service 2, respectively.S A2 means sandboxed application server of service 2.

experimenter to run experiments to validate those val-ues. If it turns out that the difference between the ex-perimental results and the interpolated results exceeds auser-specied threshold value, then the checker notiesthe management entity.

We describe the design of each component of Jus-tRunIt in detail in the following subsections.

2.2.1 Experimenter

To run experiments, the experimenter component of Jus-tRunIt transparently clones a subset of the live produc-tion system into a sandbox and replays the live workloadto the sandbox system. VM cloning instantly brings thesandbox to the same operational state as the productionsystem, complete with fully warmed-up application-leveland OS-level caches (e.g., le buffer cache). Thus, testscan proceed with low startup time on a faithful replicaof the production system. By cloning only a subset ofthe system, JustRunIt minimizes the physical resourcesthat must be dedicated to testing. Workload replay to thesandbox is used to emulate the timing and functional be-havior of the non-duplicated portions of the system.

The use of JustRunIt in a typical virtualized data cen-ter is illustrated in Figure 2. The gure shows VMs ofmultiple three-tier services sharing each PM. Each ser-vice tier has multiple identically congured VMs placedon different PMs. (Note that VMs of one tier do not sharePMs with VMs of other tiers in the gure. Although Jus-tRunIt is agnostic to VM placement, this restriction onVM placement is often used in practice to reduce soft-ware licensing costs [18].) For simpler management, the

set of PMs in each tier is often homogeneous.The gure also shows one VM instance from each

tier of service 2 being cloned into the sandbox for test-ing. This is just an example use of JustRunIt; we canuse different numbers of PMs in the sandbox, as we dis-cuss later. Conguration changes are applied to the cloneserver, and the effects of the changes are tested by replay-ing live trafc duplicated from the production system.The sandbox system is monitored to determine the re-sulting throughput, response time, and energy consump-tion. The experimenter reports these results to the driverto include in the matrix described in Section 2.2. If ex-periments are run with multiple service tiers, a differentmatrix will be created for each tier.

Although it may not be immediately obvious, the ex-perimenter assumes that the virtual machine monitor(VMM) can provide performance isolation across VMsand includes non-work-conserving resource schedulers.These features are required because the experiments per-formed in the sandbox must be realistic representationsof what would happen to the tested VM in the produc-tion system, regardless of any other VMs that may beco-located with it. We can see this by going back to Fig-ure 2. For example, the clone VM from the applicationtier of service 2 must behave the same in the sandbox(where it is run alone on a PM) as it would in the produc-tion system (where it is run with A1, A3, or both), giventhe same conguration. Our current implementation re-lies on the latest version of the Xen VMM (3.3), whichprovides isolation for the setups that we consider.

Importantly, both performance isolation and non-work-conserving schedulers are desirable characteris-tics in virtualized data centers. Isolation simplies theVM placement decisions involved in managing SLAs,whereas non-work-conserving schedulers allow moreprecise resource accounting and provide better isolation[18]. Most critically, both characteristics promote per-formance predictability, which is usually more importantthan achieving the best possible performance (and ex-ceeding the SLA requirements) in hosting centers.Cloning. Cloning is accomplished by minimally ex-tending standard VM live migration technology [8, 16].The Xen live migration mechanism copies dirty memorypages of a running VM in the background until the num-ber of dirty pages is reduced below a predened thresh-old. Then VM execution is paused for a short time (tensof milliseconds) to copy the remaining dirty pages to thedestination. Finally, execution transfers to the new VM,and the original VM is destroyed. Our cloning mecha-nism changes live migration to resume execution on boththe new VM and the original VM.

Since cloning is transparent to the VM, the clone VMinherits the same network identity (e.g., IP/MAC ad-dresses) as the production VM. To avoid network address

conicts, the cloning mechanism sets up network addresstranslation to transparently give the clone VM a uniqueexternal identity exposed to the network while conceal-ing the clone VMs internal addresses. We implementedthis by extending Xens backend network device driver(netback) to perform appropriate address translationsand protocol checksum corrections for all network trafcto and from the clone VM.

The disk storage used by the clone VMs must alsobe replicated. During the short pause of the produc-tion system VM at the end of state transfer, the cloningmechanism creates a copy-on-write snapshot of the blockstorage volumes used by the production VM, and as-signs them to the clone VM. We implemented this us-ing the Linux LVM snapshot capability and by exportingvolumes to VMs over the network using iSCSI or ATAOver Ethernet. Snapshotting and exporting the storagevolumes incurs only a sub-second delay during cloning.Storage cloning is transparent to the VMs, which see log-ical block devices and do not know that they are access-ing network storage.

JustRunIt may also be congured not to perform VMcloning in the sandbox. This conguration allows it toevaluate upgrades of the server software (e.g., Apache),operating system, and/or service application (as long asthe application upgrade does not change the applicationsmessaging behavior). In these cases, the managemententity has to request experiments that are long enoughto amortize any cold-start caching effects in the sandboxexecution. However, long experiments are not a problem,since software upgrades typically do not have stringenttime requirements.Proxies. To carry out testing, the experimenter replayslive workload to the VMs in the sandbox. Two low-overhead proxies, called in-proxy and out-proxy, are in-serted into communication paths in the production sys-tem to replicate trafc to the sandbox. The proxies areapplication protocol-aware and can be (almost entirely)re-used across services that utilize the same protocols, aswe detail below. The in-proxy mimics the behavior of allthe previous tiers before the sandbox, and the out-proxymimics the behavior of all the following tiers. The localview of a VM, its cloned sandbox VM, and the proxiesis shown in Figure 3.

After cloning, the proxies create as many connectionswith the cloned VM as they have with the original VM.The connections that were open between the proxies andthe original VM at the time it was cloned will timeout atthe cloned VM. In fact, no requests that were active inthe original VM at the time of cloning get successfullyprocessed at the cloned VM.

The in-proxy intercepts requests from previous tiersto the tested VM. When a request arrives, the in-proxyrecords the request (Reqn in Figure 3) and its arrival

!"!

#$

%#&

%#&

Figure 3: Cloned VM and proxy data structures.

time (tn). The in-proxy forwards the request to the on-line production system and also sends a duplicate requestto the sandbox for processing. To prevent the sandboxsystem from running ahead of the production system,the transmission of the duplicate request is delayed bya xed time interval (it is sufcient for the xed timeshift to be set to any value larger than the maximum re-sponse time of the service plus the cloning overhead).Both systems process the duplicated requests and eventu-ally generate replies that are intercepted by the in-proxy.For the reply from the production system, the in-proxyrecords its arrival time (Tn) and forwards the reply backto the previous tier. Later, when the corresponding replyfrom the sandbox arrives, the in-proxy records its arrivaltime (tsn). The arrival times are used to measure theresponse times of the production and sandbox systems.

The production and sandbox VMs may need to sendrequests to the next tier to satisfy a request from the pre-vious tier. These (duplicated) requests are intercepted bythe out-proxy. The out-proxy records the arrival times(t0n) and content of the requests from the productionsystem, and forwards them to the next tier. The out-proxyalso records the arrival times (t0n) and content of thecorresponding replies, and forwards them to the produc-tion system. When the out-proxy receives a request fromthe sandbox system, it uses hash table lookup to nd thematching request that was previously received from theproduction system. (Recall that the matching request willcertainly have been received because the replay to thesandbox is time-shifted by more than the maximum re-sponse time of the service.) The out-proxy transmits therecorded reply to the sandbox after a delay. The delay isintroduced to accurately mimic the delays of the subse-quent tiers and is equal to the delay that was previouslyexperienced by the production system (t0n t0n) forthe same request-reply pair.

At the end of an experiment, the in-proxy reports thethroughput and response time results for the productionand sandbox systems. The throughput for each systemis determined by the number of requests successfully

served from the tiers following the in-proxy. The re-sponse time for each system is dened as the delay aftera request arrives to the in-proxy until its reply is received.Since out-proxies enforce that the delays of subsequenttiers are equal for the production and sandbox system,the difference of throughput and response time betweenthe production and sandbox systems is the performancedifference between the original VM and cloned VM.

The proxies can be installed dynamically anywhere inthe system, depending on which VMs the managemententity may want to study at the time. However, we haveonly implemented in-proxies and out-proxies for Weband application servers so far. Cross-tier interactions be-tween proxies, i.e. the communication between the out-proxy of the Web tier and the in-proxy of the applicationtier, occur in exactly the same way as the communicationbetween regular servers.

In future work, we plan to implement an in-proxy fordatabase servers by borrowing code from the Clustered-JDBC (C-JDBC) database middleware [5]. Briey, C-JDBC implements a software controller between a JDBCapplication and a set of DBMSs. In its full-replicationmode, C-JDBC keeps the content of the database repli-cated and consistent across the DBMSs. During experi-mentation, our in-proxy will do the same for the on-lineand sandboxed DBMSs. Fortunately, C-JDBC alreadyimplements the key functionality needed for cloning,namely the ability to integrate the sandboxed DBMSand update its content for experimentation. To completeour in-proxy, we plan to modify C-JDBC to record theon-line requests and later replay them to the sandboxedDBMS. We have modied C-JDBC in similar ways [17].Non-determinism. A key challenge for workload replayis to tolerate non-deterministic behavior in the produc-tion and sandbox systems. We address non-determinismin three ways. First, to tolerate network layer non-determinism (e.g., packet drops) the proxies replicateapplication-layer requests and replies instead of replicat-ing network packets directly.

Second, the replay is implemented so that the sand-boxed servers can process requests and replies in a dif-ferent order than their corresponding on-line servers;only the timing of the message arrivals at the sandboxedservers is guaranteed to reect that of the on-line servers.This ordering exibility tolerates non-determinism in thebehavior of the software servers, e.g. due to multithread-ing. However, note that this exibility is only acceptablefor Web and application-tier proxies, since requests fromdifferent sessions are independent of each other in thosetiers. We will need to enforce ordering more strictly inthe in-proxy for database servers, to prevent the origi-nal and cloned databases from diverging. Our in-proxywill do so by forcing each write (and commit) to executeby itself during experimentation only forcing a complete

ordering between all pairs of read-write and write-writeoperations; concurrent reads will be allowed to executein any order. We have successfully created this strict or-dering in C-JDBC before [17] and saw no noticeable per-formance degradation for one of the services we study inthis paper.

Third, we tolerate application-layer non-determinismby designing the proxies to be application protocol-aware (e.g., the Web server in-proxies understand HTTPmessage formats). The proxies embody knowledge ofthe elds in requests and replies that can have non-deterministic values (e.g., timestamps, session IDs).When the out-proxy sees a non-deterministic value ina message from the sandbox, the message is matchedagainst recorded messages from the production systemusing wildcards for the non-deterministic elds.

Our study of two services (an auction and a bookstore)shows that our proxies effectively tolerate their non-determinism. Even though some messages in these ser-vices have identical values except for a non-deterministiceld, our wildcard mechanism allows JustRunIt to prop-erly match replies in the production and sandbox systemsfor two reasons. First, all replies from the sandbox aredropped by the proxies, preventing them from disruptingthe on-line system. Second, using different replies due towildcard mismatch does not affect the JustRunIt resultsbecause the replies are equivalent and all delays are stillaccounted for.

We plan to study non-determinism in an even broaderrange of services. In fact, despite our promising ex-perience with the auction and bookstore services, sometypes of non-determinism may be hard for our proxies tohandle. In particular, services that non-deterministicallychange their messaging behavior (not just particularelds or the destination of the messages) or their loadprocessing behavior (e.g., via non-deterministic load-shedding) would be impossible to handle. For example, aservice in which servers may send an unpredictable num-ber of messages in response to each request cannot behandled by our proxies. We have not come across anysuch services, though.

2.2.2 Experiment Driver

Running experiments is not free. They cost time and en-ergy. For this reason, JustRunIt allows the managemententity to congure the experimentation using a simpleconguration le. The entity can specify the tier(s) withwhich JustRunIt should experiment, which experimentheuristics to apply (discussed below), which resources tovary, the range of resource allocations to consider, howmany equally separated allocation points to consider inthe range, how long each experiment should take, andhow many experiments to run. These parameters can di-

rectly limit the time and indirectly limit the energy con-sumed by the experiments, when there are constraints onthese resources (as in Section 3.1). When experimenttime and energy are not relevant constraints (as in Sec-tion 3.2), the settings for the parameters can be looser.

Based on the conguration information, the experi-ment driver directs the experimenter to explore the pa-rameter space within the time limit. The driver startsby running experiments to ll in the entries at the cor-ners of the result matrix. For example, if the experimentsshould vary the CPU allocation and the CPU frequency,the matrix will have two dimensions and four corners:(min CPU alloc, min CPU freq), (min CPU alloc, max CPUfreq), (max CPU alloc, min CPU freq), and (max CPU alloc,max CPU freq). The management entity must congureJustRunIt so at least these corner experiments can be per-formed. After lling in the corner coordinates, the driverthen proceeds to request experiments exactly in the mid-dle of the unexplored ranges dened by each resourcedimension. After those are performed, it recursively sub-divides the unexplored ranges in turn. This process isrepeated until the number of experiments requested bythe management entity have been performed or there areno more experiments to perform.

We designed two heuristics for the driver to use toavoid running unnecessary experiments along each ma-trix dimension. The two observations behind the heuris-tics are that: 1) beyond some point, resource additionsdo not improve performance; 2) the performance gainfor the same resource addition to different tiers will notbe the same, and the gains drop consistently and contin-ually (diminishing returns).

Based on observation 1), the rst heuristic cancels theremaining experiments with larger resource allocationsalong the current resource dimension, if the performancegain from a resource addition is less than a thresholdamount. Based on observation 2), the second heuristiccancels the experiments with tiers that do not producethe largest gains from a resource addition. As we addmore resources to the current tier, the performance gainsdecrease until some other tier becomes the tier with thelargest gain from the same resource addition. For ex-ample, increasing the CPU allocation on the bottlenecktier, say the application tier, will signicantly improveoverall response time. At some point, however, the bot-tleneck will shift to other tiers, say the Web tier, at whichpoint the driver will experiment with the Web tier andgain more overall response time improvement with thesame CPU addition.

2.2.3 Interpolator and Checker

The interpolator predicts performance results for pointsin the matrix that have not yet been determined through

experiments. For simplicity, we use linear interpolationto ll in these blanks, and we mark the values to indicatethat they are just interpolated.

If the management entity uses any interpolated results,the checker tries to verify the interpolated results by in-voking the experimenter to run the corresponding exper-iments in the background. If one of these backgroundexperimental results differs from the corresponding in-terpolated result by more than a user-specied thresholdvalue, the checker raises a ag to the management entityto decide how to handle this mismatch.

The management entity can use this information inmultiple ways. For example, it may recongure thedriver to run more experiments with the correspondingresources from now on. Another option would be to re-congure the range of allocations to consider in the ex-periments from now on.

2.3 DiscussionUses of JustRunIt. We expect that JustRunIt will be use-ful for many system management scenarios. For exam-ple, in this paper we consider resource management andhardware upgrade case studies. In these and other sce-narios, JustRunIt can be used by the management entityto safely, efciently, and realistically answer the samewhat-if questions that modeling can answer given thecurrent workload and load intensity.

Moreover, like modeling, JustRunIt can benet fromload intensity prediction techniques to answer questionsabout future scenarios. JustRunIt can do so because itsrequest replay is shifted in time and can be done at anydesired speed. (Request stream acceleration needs toconsider whether requests belong to an existing sessionor start a new session. JustRunIt can properly acceler-ate requests because it stores enough information aboutthem to differentiate between the two cases.) Section 6discusses how the current version of JustRunIt can bemodied to answer what-if questions about differentworkload mixes as well.

Although our current implementation does not imple-ment this functionality, JustRunIt could also be used toselect the best values for software tunables, e.g. the num-ber of threads or the size of the memory cache in Webservers. Modeling does not lend itself directly to thistype of management task. Another possible extensioncould be enabling JustRunIt to evaluate the correctnessof administrator actions, as in action-validation systems[15, 17]. All the key infrastructure required by these sys-tems (i.e., proxies, cloning, sandboxing) is already partof the current version of JustRunIt, so adding the abilityto validate administrator actions should be a simple ex-ercise. Interestingly, this type of functionality cannot beprovided by analytic models or feedback control.

Obviously, JustRunIt can answer questions and vali-date administrator actions at the cost of experiment timeand energy. However, note that the physical resourcesrequired by JustRunIt (i.e., enough computational re-sources for the proxies and for the sandbox) can be a verysmall fraction of the data centers resources. For exam-ple, in Figure 2, we show that just three PMs are enoughto experiment with all tiers of a service at the same time,regardless of how large the production system is. Evenfewer resources, e.g. one PM, can be used, as long as wehave the time to experiment with VMs sequentially. Fur-thermore, the JustRunIt physical resources can be bor-rowed from the production system itself, e.g. during pe-riods of low load.

In essence, JustRunIt poses an interesting tradeoff be-tween the amount of physical resources it uses, the ex-periment time that needs to elapse before decisions canbe made, and the energy consumed by its resources.More physical resources translate into shorter experi-ment times but higher energy consumption. For this rea-son, we allow the management entity to congure Jus-tRunIt in whatever way is appropriate for the data center.Engineering cost of JustRunIt. Building the JustRunItproxies is the most time-consuming part of its implemen-tation. The proxies must be designed to properly handlethe communication protocols used by services. Our cur-rent proxies understand the HTTP, mod jk, and MySQLprotocols. We have built our proxies starting from thepublicly available Tinyproxy HTTP proxy daemon [2].Each proxy required only between 800 and 1500 newlines of C code. (VM cloning required 42 new linesof Python code in the xend control daemon and the xmmanagement tool, whereas address translation required244 new lines of C code in the netback driver.) The vastmajority of the difference between Web and applicationserver proxies comes from their different communicationprotocols.

The engineering effort required by the proxies can beamortized, as they can be reused for any service basedon the same protocols. However, the proxies may needmodications to handle any non-determinism in the ser-vices themselves. Fortunately, our experience with theauction and bookstore services suggests that the effortinvolved in handling service-level non-determinism maybe small. Specically, it took one of us (Zheng) less thanone day to adapt the proxies designed for the auction tothe bookstore. This is particularly promising in that hehad no prior knowledge of the bookstore whatsoever.

One may argue that implementing JustRunIt may re-quire a comparable amount of effort to developing ac-curate models for a service. We have experience withmodeling the performance, energy, and temperature ofserver clusters and storage systems [4, 13, 12, 19] andlargely agree with this claim. However, we note that Jus-

tRunIt is much more reusable than models, across differ-ent services, hardware and software characteristics, andeven as service behavior evolves. Each of these factorsrequires model re-calibration and re-validation, whichare typically labor-intensive. Furthermore, for models tobecome tractable, many simplifying assumptions aboutsystem behavior (e.g., memoryless request arrivals) mayhave to be made. These assumptions may compromisethe accuracy of the models. JustRunIt does not requirethese assumptions and produces accurate results.

3 Experiment-based Management

As mentioned in the previous section, our infrastructurecan be used by automated management systems or di-rectly by the system administrator. To demonstrate itsuse in the former scenario, we have implemented sim-ple automated management systems for two commontasks in virtualized hosting centers: server consolida-tion/expansion (i.e., partitioning resources across the ser-vices to use as few active servers as possible) and evalu-ation of hardware upgrades. These tasks are currentlyperformed by most administrators in a manual, labor-intensive, and ad-hoc manner.

Both management systems seek to satisfy the servicesSLAs. An SLA often species a percentage of requeststo be serviced within some amount of time. Another pos-sibility is for the SLA to specify an average response time(over a period of several minutes) for the correspondingservice. For simplicity, our automated systems assumethe latter type of SLA.

The next two subsections describe the managementsystems. However, before describing them, we note thatthey are not contributions of this work. Rather, they arepresented simply to demonstrate the automated use ofJustRunIt. More sophisticated systems (or the admin-istrator) would leverage JustRunIt in similar ways.

3.1 Case Study 1: Resource ManagementOverview. The ultimate goal of our resource-management system is to consolidate the hosted servicesonto the smallest possible set of nodes, while satisfyingall SLAs. To achieve this goal, the system constantlymonitors the average response time of each service, com-paring this average to the corresponding SLA. Becauseworkload conditions change over time, the resources as-signed to a service may become insufcient and the ser-vice may start violating its SLA. Whenever such a vio-lation occurs, our system initiates experiments with Jus-tRunIt to determine what is the minimum allocation ofresources that would be required for the services SLAto be satised again. Changes in workload behavior of-ten occur at the granularity of tens of minutes or even

1. While 1 do2. Monitor QoS of all services3. If any service needs more resources or4. can use fewer resources5. Run experiments with bottleneck tier6. Find minimum resource needs7. If used any interpolated results8. Inform JustRunIt about them9. Assign resources using bin-packing heuristic

10. If new nodes need to be added11. Add new nodes and migrate VMs to them12. Else if nodes can be removed13. Migrate VMs and remove nodes14. Complete resource adjustments and migrations

Figure 4: Overview of resource-management system.

hours, suggesting that the time spent performing exper-iments is likely to be relatively small. Nevertheless, toavoid having to perform adjustments too frequently, thesystem assigns 20% more resources to a service than itsminimum needs. This slack allows for transient increasesin offered load without excessive resource waste. Sincethe resources required by the service have to be allocatedto it, the new resource allocation may require VM migra-tions or even the use of extra nodes.

Conversely, when the SLA of any service is being sat-ised by more than a threshold amount (i.e., the averageresponse time is lower than that specied by the SLA bymore than a threshold percentage), our system consid-ers the possibility of reducing the amount of resourcesdedicated to the service. It does so by initiating exper-iments with JustRunIt to determine the minimum allo-cation of resources that would still satisfy the servicesSLA. Again, we give the service additional slack in itsresource allocation to avoid frequent reallocations. Be-cause resources can be taken away from this service, thenew combined resource needs of the services may not re-quire as many PMs. In this case, the system determinesthe minimum number of PMs that can be used and im-plements the required VM migrations.Details. Figure 4 presents pseudo-code overviewing theoperation of our management system. The experimentswith JustRunIt are performed in line 5. The managementsystem only runs experiments with one software serverof the bottleneck tier of the service in question. Themanagement system can determine the bottleneck tier byinspecting the resource utilization of the servers in eachtier. Experimenting with one software server is typicallyenough for two reasons: (1) services typically balancethe load evenly across the servers of each tier; and (2)the VMs of all software servers of the same tier and ser-vice are assigned the same amount of resources at theirPMs. (When at least one of these two properties does nothold, the management system needs to request more ex-periments of JustRunIt.) However, if enough nodes canbe used for experiments in the sandbox, the system couldrun experiments with one software server from each tierof the service at the same time.

The matrix of resource allocations vs. response timesproduced by JustRunIt is then used to nd the minimumresource needs of the service in line 6. Specically, themanagement system checks the results in the JustRunItmatrix (from smallest to largest resource allocation) tond the minimum allocation that would still allow theSLA to be satised. In lines 7 and 8, the system informsJustRunIt about any interpolated results that it may haveused in determining the minimum resource needs. Jus-tRunIt will inform the management system if the inter-polated results are different than the actual experimentalresults by more than a congurable threshold amount.

In line 9, the system executes a resource assignmentalgorithm that will determine the VM to PM assignmentfor all VMs of all services. We model resource assign-ment as a bin-packing problem. In bin-packing, the goalis to place a number of objects into bins, so that we min-imize the number of bins. We model the VMs (and theirresource requirements) as the objects and the PMs (andtheir available resources) as the bins. If more than oneVM to PM assignment leads to the minimum number ofPMs, we break the tie by selecting the optimal assign-ment that requires the smallest number of migrations. Ifmore than one assignment requires the smallest numberof migrations, we pick the one of these assignments ran-domly. Unfortunately, the bin-packing problem is NP-complete, so it can take an inordinate amount of time tosolve it optimally, even for hosting centers of moderatesize. Thus, we resort to a heuristic approach, namelysimulated annealing [14], to solve it.

Finally, in lines 1014, the resource-allocation systemadjusts the number of PMs and the VM to PM assign-ment as determined by the best solution ever seen bysimulated annealing.Comparison. A model-based implementation for thismanagement system would be similar; it would simplyreplace lines 58 with a call to a performance modelsolver. Obviously, the model would have to have beencreated, calibrated, and validated a priori.

A feedback-based implementation would replace lines58 by a call to the controller to execute the experimentsthat will adjust the offending service. However, notethat feedback control is only applicable when repeatedlyvarying the allocation of a resource or changing a hard-ware setting does not affect the on-line behavior of theco-located services. For example, we can use feedbackcontrol to vary the CPU allocation of a service with-out affecting other services. In contrast, increasing theamount of memory allocated to a service may requiredecreasing the allocation of another service. Similarly,varying the voltage setting for a service affects all ser-vices running on the same CPU chip, because the cores incurrent chips share the same voltage rail. Cross-serviceinteractions are clearly undesirable, especially when they

1. For each service do2. For one software server of each tier3. Run experiments with JustRunIt4. Find minimum resource needs5. If used any interpolated results6. Inform JustRunIt about them7. Assign resources using bin-packing heuristic8. Estimate power consumption

Figure 5: Overview of update-evaluation system.

may occur repeatedly as in feedback control. The keyproblem is that feedback control experiments with theon-line system. With JustRunIt, bin-packing and nodeaddition/removal occur before any resource changes aremade on-line, so interference can be completely avoidedin most cases. When interference is unavoidable, e.g.the offending service cannot be migrated to a node withenough available memory and no extra nodes can beadded, changes to the service are made only once.

3.2 Case Study 2: Hardware UpgradesOverview. For our second case study, we built a manage-ment system to evaluate hardware upgrades. The systemassumes that at least one instance of the hardware beingconsidered is available for experimentation in the sand-box. For example, consider a scenario in which the host-ing center is considering purchasing machines of a modelthat is faster or has more available resources than thatof its current machines. After performing experimentswith a single machine of the candidate model, our systemdetermines whether the upgrade would allow servers tobe consolidated onto a smaller number of machines andwhether the overall power consumption of the hostingcenter would be smaller than it currently is. This infor-mation is provided to the administrator, who can makea nal decision on whether or not to purchase the newmachines and ultimately perform the upgrade.Details. Figure 5 presents pseudo-code overviewing ourupdate-evaluation system. The experiments with Jus-tRunIt are started in line 3. For this system, the ma-trix that JustRunIt produces must include informationabout the average response time and the average powerconsumption of each resource allocation on the upgrade-candidate machine. In line 4, the system determines theresource allocation that achieves the same average re-sponse time as on the current machine (thus guaranteeingthat the SLA would be satised by the candidate machineas well). Again, the administrator congures the systemto properly drive JustRunIt and gets informed about anyinterpolated results that are used in line 4.

By adding the extra 20% slack to these minimum re-quirements and running the bin-packing algorithm de-scribed above, the system determines how many newmachines would be required to achieve the current per-formance and how much power the entire center would

consume. Specically, the center power can be estimatedby adding up the power consumption of each PM in theresource assignment produced by the simulated anneal-ing. The consumption of each PM can be estimated byrst determining the base power of the candidate ma-chine, i.e. the power consumption when the machine ison but no VM is running on it. This base power shouldbe subtracted from the results in the JustRunIt matrix ofeach software server VM. This subtraction produces theaverage dynamic power required by the VM. Estimatingthe power of each PM then involves adding up the dy-namic power consumption of the VMs that would run onthe PM plus the base power.Comparison. Modeling has been used for this manage-ment task [7]. A modeling-based implementation for ourmanagement system would replace lines 26 in Figure 5with a call to a performance model solver to estimate theminimum resource requirements for each service. Basedon these results and on the resource assignment com-puted in line 7, an energy model would estimate the en-ergy consumption in line 8. Again, both models wouldhave to have been created, calibrated, and validated a pri-ori. In contrast, feedback control is not applicable to thismanagement task.

4 Evaluation

4.1 MethodologyOur hardware comprises 15 HP Proliant C-class bladesinterconnected by a Gigabit Ethernet switch. Each serverhas 8 GBytes of DRAM, 2 hard disks, and 2 Intel dual-core Xeon CPUs. Each CPU has two frequency points, 2GHz and 3 GHz. Two blades with direct-attached disksare used as network-attached storage servers. They ex-port Linux LVM logical volumes to the other blades us-ing ATA over Ethernet. One Gbit Ethernet port of everyblade is used exclusively for network storage trafc. Wemeasure the energy consumed by a blade by querying itsmanagement processor, which monitors the peak and av-erage power usage of the entire blade.

Virtualization is provided by XenLinux kernel 2.6.18with the Xen VMM [3], version 3.3. For improvingXens ability to provide performance isolation, we pinDom0 to one of the cores and isolate the service(s) fromit. Note, however, that JustRunIt does not itself im-pose this organization. As JustRunIt only depends onthe VMM for VM cloning, it can easily be ported to useVMMs that do not perform I/O in a separate VM.

We populate the blade cluster with one or more in-dependent instances of an on-line auction service. Todemonstrate the generality of our system, we also exper-iment with an on-line bookstore. Both services are or-ganized into three tiers of servers: Web, application, and

database tiers. The rst tier is implemented by ApacheWeb servers (version 2.0.54), the second tier uses Tomcatservlet servers (version 4.1.18), and the third tier uses theMySQL relational database (version 5.0.27). (For perfor-mance reasons, the database servers are not virtualizedand run directly on Linux and the underlying hardware.)We use LVS load balancers [30] in front of the Web andapplication tiers. The service requests are received bythe Web servers and may ow towards the second andthird tiers. The replies ow through the same path in thereverse direction.

We exercise each instance of the services using a clientemulator. The auction workload consists of a biddingmix of requests (94% of the database requests are reads)issued by a number of concurrent clients that repeatedlyopen sessions with the service. The bookstore workloadcomprises a shopping mix, where 20% of the requestsare read-write. Each client issues a request, receives andparses the reply, thinks for a while, and follows a linkcontained in the reply. A user-dened Markov model de-termines which link to follow. The code for the services,their workloads, and the client emulator are from the Dy-naServer project [20] and have been used extensively byother research groups.

4.2 JustRunIt OverheadOur overhead evaluation seeks to answer two ques-tions: (1) Does the overhead of JustRunIt (proxies, VMcloning, workload duplication, and reply matching) de-grade the performance of the on-line services? and (2)How faithfully do servers in the sandbox represent on-line servers given the same resources?

To answer these questions, we use our auction serviceas implemented by one Apache VM, one Tomcat VM,and MySQL. Using a larger instance of the service wouldhide some of the overhead of JustRunIt, since the proxiesonly instrument one path through the service. Each ofthe VMs runs on a different blade. We use one blade inthe sandbox. The two proxies for the Web tier run onone of the blades, whereas those for the application tierrun on another. The proxies run on their own blades topromote performance isolation for the auction service. Inall our experiments, the time shift used by JustRunIt is 10seconds behind the on-line service.Overhead on the on-line system? To isolate the over-head of JustRunIt on the on-line service, we experimentwith three scenarios: (1) Plain no proxies are installed;(2) ProxiesInstalled proxies are installed around theWeb and application servers, but they only relay the net-work trafc; and (3) JustRunIt proxies are installedaround the Web and application servers and perform allthe JustRunIt functionality.

Figures 6 and 7 depict the average throughput and re-

0200

400

600

800

1000

1200

1400

1600

1800

2000

300 600 900 1200 1500 1800Offered Load (reqs/s)

Thro

ugh

put (

reqs

/s)

PlainProxiesInstalledJustRunIt

Figure 6: Throughput as a function of offered load.

0

10

20

30

40

50

60

70

80

300 600 900 1200 1500 1800Offered Load (reqs/s)

Res

pon

se Ti

me

(ms)

PlainProxiesInstalledJustRunIt

Figure 7: Response time as a function of offered load.

0

5

10

15

20

25

30

35

25 50 75 100CPU Allocation

Res

pon

se Ti

me

(ms)

0

50

100

150

200

250

300

350

400

450Th

rou

ghpu

t (re

qs/s

)

Live RTSB RTLive TSB T

Figure 8: On-line and sandboxed performance as a functionof CPU allocation at offered load 500 requests/second.

0

20

40

60

80

100

120

140

160

25 50 75 100CPU Allocation

Res

pon

se Ti

me

(ms)

0

100

200

300

400

500

600

700

800

900

Thro

ugh

put (

reqs

/s)

Live RTSB RTLive TSB T

Figure 9: On-line and sandboxed performance as a functionof CPU allocation at offered load 1000 requests/second.

sponse time of the on-line service, respectively, as a func-tion of the offered load. We set the CPU allocation of allservers to 100% of one core. In this conguration, theservice saturates at 1940 requests/second. Each bar cor-responds to a 200-second execution.

Figure 6 shows that JustRunIt has no effect on thethroughput of the on-line service, even as it approachessaturation, despite having the proxies for each tier co-located on the same blade.

Figure 7 shows that the overhead of JustRunIt is con-sistently small (< 5ms) across load intensities. We arecurrently in the process of optimizing the implementa-tion to reduce the JustRunIt overheads further. However,remember that the overheads in Figure 7 are exagger-ated by the fact that, in these experiments, all applicationserver requests are exposed to the JustRunIt instrumenta-tion. If we had used a service with 4 application servers,for example, only roughly 25% of those requests wouldbe exposed to the instrumentation (since we only needproxies for 1 of the application servers), thus loweringthe average overhead by 75%.

Performance in the sandbox? The results above isolate

the overhead of JustRunIt on the on-line system. How-ever, another important consideration is how faithful thesandbox execution is to the on-line execution given thesame resources. Obviously, it would be inaccurate tomake management decisions based on sandboxed exper-iments that are not very similar to the behavior of theon-line system.

Figures 8 and 9 compare the performance of the on-line application server (labeled Live) to that of thesandboxed (labeled SB) application server at 500 re-quests/second and 1000 requests/second, respectively.In both gures, response times (labeled RT) andthroughputs (labeled T) are measured at the applica-tion servers in-proxy. Again, each result represents theaverage performance over 200 seconds.

As one would expect, the gures show that increasingthe CPU allocation tends to increase throughputs and re-duce response times. The difference between the offeredload and the achieved throughput is the 20% of requeststhat are served directly by the Web server and, thus, donot reach the application servers in-proxy. More inter-estingly, the gures clearly show that the sandboxed ex-ecution is a faithful representation of the on-line system,

010

20

30

40

50

60

1 2 3 4 5 6 7 8 9Time (minutes)

Res

pon

se Ti

me

(ms)

Service 0 RTService 1 RTService 2 RTService 3 RT

Figure 10: Server expansion using JustRunIt.

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9Time (minutes)

Res

pon

se Ti

me

(ms)

Service 0 RTService 1 RTService 2 RTService 3 RT

Figure 11: Server expansion using accurate modeling.

regardless of the offered load.The results for the Web tier also show the sandboxed

execution to be accurate. Like the application-tier re-sults, we ran experiments with four different CPU allo-cations, under two offered loads. When the offered loadis 500 reqs/s, the average difference between the on-lineand sandboxed results is 4 requests/second for through-put and 1 ms for response time, across all CPU alloca-tions. Even under a load of 1000 requests/second, the av-erage throughput and response time differences are only6 requests/second and 2 ms, respectively.

Our experiments with the bookstore service exhibit thesame behaviors as in Figures 6 to 9. The throughput isnot affected by JustRunIt and the overhead on the re-sponse time is small. For example, under an offered loadof 300 requests/second, JustRunIt increases the mean re-sponse time for the bookstore from 18 ms to 22 ms. For900 requests/second, the increase is from 54 ms to 58 ms.Finally, our worst result shows that JustRunIt increasesthe mean response time from 90 ms to 100 ms at 1500requests/second.

4.3 Case Study 1: Resource ManagementAs mentioned before, we built an automated resourcemanager for a virtualized hosting center that leveragesJustRunIt. To demonstrate the behavior of our manager,we created four instances of our auction service on 9blades: 2 blades for rst-tier servers, 2 blades for second-tier servers, 2 blades for database servers, and 3 bladesfor storage servers and LVS. Each rst-tier (second-tier)blade runs one Web (application) server from each ser-vice. Each server VM is allocated 50% of one core asits CPU allocation. We assume that the services SLAsrequire an average response time lower than 50 ms inevery period of one minute. The manager requested Jus-tRunIt to run 3 CPU-allocation experiments with any ser-vice that violated its SLA, for no longer than 3 minutesoverall. A 10th blade is used for the JustRunIt sand-

box, whereas 2 extra blades are used for its Web andapplication-server proxies. Finally, 2 more blades areused to generate load.

Figure 10 shows the response time of each service dur-ing our experiment; each point represents the average re-sponse time during the corresponding minute. We ini-tially offered 1000 requests/second to each service. Thisoffered load results in an average response time hover-ing around 40 ms. Two minutes after the start of theexperiment, we increase the load offered to service 0 to1500 requests/second. This caused its response time toincrease beyond 50 ms during the third minute of the ex-periment. At that point, the manager started JustRunItexperiments to determine the CPU allocation that wouldbe required for the services application servers (the sec-ond tier is the bottleneck tier) to bring response time backbelow 50 ms under the new offered load. The set of Jus-tRunIt experiments lasted 3 minutes, allowing CPU allo-cations of 60%, 80%, and 100% of a core to be tested.The values for 70% and 90% shares were interpolatedbased on the experimental results.

Based on the response-time results of the experiments,the manager determined that the application server VMsof the offending service should be given 72% of a core(i.e., 60% of a core plus the 20% of 60% = 12% slack).Because of the extra CPU allocation requirements, themanager decided that the system should be expanded toinclude an additional PM (a 15th blade in our setup). Topopulate this machine, the manager migrated 2 VMs to it(one from each PM hosting application server VMs). Be-sides the 3 minutes spent with experiments, VM cloning,simulated annealing, and VM migration took about 1minute altogether. As a result, the manager was able tocomplete the resource reallocation 7 minutes into the ex-periment. The experiment ended with all services satis-fying their SLAs.

Comparison against highly accurate modeling. Fig-ure 11 shows what the system behavior would be if the

resource manager made its decisions based on a highlyaccurate response-time model of our 3-tier auction ser-vice. To mimic such a model, we performed the Jus-tRunIt experiments with service 0 under the same offeredload of Figure 10 for all CPU allocations off-line. Theseoff-line results were fed to the manager during the exper-iment free of any overheads. We assumed that the model-based manager would require 1 minute of resource-usagemonitoring after the SLA violation is detected, before themodel could be solved. Based on the JustRunIt results,the manager made the same decisions as in Figure 10.

The gure shows that modeling would allow the sys-tem to adjust 2 minutes faster. However, developing, cal-ibrating, and validating such an accurate model is a chal-lenging and labor-intensive proposition. Furthermore,adaptations would happen relatively infrequently in prac-tice, given that (1) it typically takes at least tens of min-utes for load intensities to increase signicantly in realsystems, and (2) the manager builds slack into the re-source allocation during each adaptation. In summary,the small delay in decision making and the limited re-sources that JustRunIt requires are a small price to payfor the benets that it affords.

4.4 Case Study 2: Hardware UpgradeWe also experimented with our automated system forevaluating hardware upgrades in a virtualized hostingcenter. To demonstrate the behavior of our system, weran two instances of our auction service on the samenumber of blades as in our resource manager studyabove. However, we now congure the blades that runthe services to run at 2 GHz. The blade in the JustRunItsandbox is set to run at 3 GHz to mimic a more pow-erful machine that we are considering for an upgrade ofthe data center. We offer 1000 requests/second to eachservice. We also cap each application server VM of bothservices at 90% of one core; for simplicity, we do not ex-periment with the Web tier, but the same approach couldbe trivially taken for it as well.

During the experiment, the management system re-quested JustRunIt to run 4 CPU-allocation experimentsfor no longer than 800 seconds overall. (Note, though,that this type of management task does not have real-time requirements, so we can afford to run JustRunIt ex-periments for a much longer time.) Since each serveris initially allocated 90% of one core, JustRunIt is toldto experiment with CPU allocations of 50%, 60%, 70%,and 80% of one core; there is no need for interpolation.The management systems main goal is to determine (us-ing simulated annealing) how many of the new machineswould be needed to achieve the same response time thatthe services currently exhibit. With this information, theenergy implications of the upgrade can be assessed.

Based on the results generated by JustRunIt, the man-agement system decided that the VMs of both servicescould each run at 72% CPU allocations (60% of one coreplus 12% slack) at 3 GHz. For a large data center withdiverse services, a similar reduction in resource require-ments may allow for servers to be consolidated, whichwould most likely conserve energy. Unfortunately, ourexperimental system is too small to demonstrate theseeffects here.

4.5 SummaryIn summary, the results above demonstrate that the Jus-tRunIt overhead is small, even when all requests areexposed to our instrumentation. In real deployments,the observed overhead will be even smaller, since therewill certainly be more than one path through each ser-vice (at the very least to guarantee availability and fault-tolerance). Furthermore, the results show that the sand-boxed execution is faithful to the on-line execution. Fi-nally, the results demonstrate that JustRunIt can be effec-tively leveraged to implement sophisticated automatedmanagement systems. Modeling could have been appliedto the two systems, whereas feedback control is applica-ble to resource management (in the case of the CPU al-location), but not upgrade evaluation. The hardware re-sources consumed by JustRunIt amount to one machinefor the two proxies of each tier, plus as few as one sand-box machine. Most importantly, this overhead is fixedand independent of the size of the production system.

5 Related Work

Modeling, feedback control, and machine learningfor managing data centers. State-of-the-art manage-ment systems rely on analytical modeling, feedback con-trol, and/or machine learning to at least partially auto-mate certain management tasks. As we have mentionedbefore, modeling has complexity and accuracy limita-tions, whereas feedback control is not applicable to manytypes of tasks. Although machine learning is useful forcertain management tasks, such as fault diagnosis, it alsohas applicability limitations. The problem is that ma-chine learning can only learn about system scenarios andcongurations that have been seen in the past and aboutwhich enough data has been collected. For example, itapplies to neither of the tasks we study in this paper.Nevertheless, machine learning can be used to improvethe interpolation done by JustRunIt, when enough dataexists for it to derive accurate models.

JustRunIt takes a fundamentally different approach tomanagement; one in which accurate sandboxed experi-ments replace modeling, feedback control, and machinelearning.

Scaling down data centers. Gupta et al. [10] pro-posed the DieCast approach for scaling down a service.DieCast enables some management tasks, such as pre-dicting service performance as a function of workload,to be performed on the scaled version. Scaling is accom-plished by creating one VM for each PM of the serviceand running the VMs on an off-line cluster that is an or-der of magnitude smaller than the on-line cluster. Be-cause of the signicant scaling in size, DieCast also usestime dilation [11] to make guest OSes think that they arerunning on much faster machines. For a 10-fold scaledown, time dilation extends execution time by 10-fold.

DieCast and JustRunIt have fundamentally differentgoals and resource requirements. First, JustRunIt targetsa subset of the management tasks that DieCast does; thesubset that can be accomplished with limited additionalhardware resources, software infrastructure, and costs.In particular, JustRunIt seeks to improve upon model-ing by leveraging native execution. Because of time di-lation, DieCast takes excessively long to perform eachexperiment. Second, JustRunIt includes infrastructurefor automatically experimenting with services, as wellas interpolating and checking the experimental results.Third, JustRunIt minimizes the set of hardware resourcesthat are required by each experiment without affecting itsrunning time. In contrast, to affect execution time by asmall factor, DieCast requires an additional hardware in-frastructure that is only this same small factor smallerthan the entire on-line service.

Sandboxing and duplication for managing data cen-ters. A few efforts have proposed related infrastructuresfor managing data centers. Specically, [15, 17] consid-ered validating operator actions in an Internet service byusing request duplication to a sandboxed extension of theservice. For each request, if the replies generated by theon-line environment and by the sandbox ever differ dur-ing a validation period, a potential operator mistake isagged. Tan et al. [25] considered a similar infrastruc-ture for verifying le servers.

Instead of operator-action validation in a single, non-virtualized Internet service, our goal is to experimen-tally evaluate the effect of different resource allocations,parameter settings, and other potential system changes(such as hardware upgrades) in virtualized data centers.Thus, JustRunIt is much more broadly applicable thanprevious works. As a result, our infrastructure is quitedifferent than previous systems. Most signicantly, Jus-tRunIt is the rst system that may explore a large numberof scenarios that differ from the on-line system, while ex-trapolating results from the experiments that are actuallyrun, and verifying its extrapolations if necessary.

Selecting experiments to run. Previous works have pro-posed sophisticated approaches for selecting the experi-

ments to run when benchmarking servers [22] or opti-mizing their conguration parameters [26, 31]. Such ap-proaches are largely complementary to our work. Specif-ically, they can be used to improve experiment-basedmanagement in two ways: (1) automated managementsystems can use them to dene/constrain the parameterspace that JustRunIt should explore; or (2) they can beused as new heuristics in JustRunIts driver to eliminateunnecessary experiments.

6 Conclusions

This paper introduced a novel infrastructure forexperiment-based management of virtualized data cen-ters, called JustRunIt. The infrastructure enables an au-tomated management system or the system administra-tor to answer what-if questions experimentally duringmanagement tasks and, based on the answers, select thebest course of action. The current version of JustRunItcan be applied to many management tasks, including re-source management, hardware upgrades, and softwareupgrades.Limitations. There are three types of what-if ques-tions that sophisticated models can answer (by makingsimplifying assumptions and costing extensive human la-bor), whereas JustRunIt currently cannot. First, service-wide models can answer questions about the effect ofa service tier on other tiers. In the current version ofJustRunIt, these cross-tier interactions are not visible,since the sandboxed virtual machines do not communi-cate with each other.

Second, models that represent request mixes at a lowenough level can answer questions about hypotheticalmixes that have not been experienced in practice. Cur-rently, JustRunIt relies solely on real workload dupli-cation for its experiments, so it can only answer ques-tions about request mixes that are offered to the system.Nevertheless, JustRunIt can currently answer questionsabout more or less intense versions of real workloads,which seems to be a more useful property.

Finally, models can sometimes be used to spot perfor-mance anomalies, although differences between modelresults and on-line behavior are often due to inaccura-cies of the model. Because JustRunIt uses complete-statereplicas of on-line virtual machines for greater realism inits experiments, anomalies due to software server or op-erating system bugs cannot be detected.Future work. We plan to extend JustRunIt to al-low cross-tier communication between the sandboxedservers. This will allow the administrator to conguresandboxing with or without cross-tier interactions. Wealso plan to create infrastructure to allow experimenta-tion with request mixes other than those observed on-

line. The idea here is to collect a trace of the on-lineworkload offered to one server of each tier, as well as thestate of these servers. Later, JustRunIt could install thestates and replay the trace to the sandboxed servers. Dur-ing replay, the request mix could be changed by eliminat-ing or replicating some of the traced sessions. Finally, weplan to build an in-proxy for a database server, startingwith code from the C-JBDC middleware.

AcknowledgementsWe would like to thank our shepherd, John Dunagan, andthe anonymous reviewers for comments that helped im-prove this paper signicantly. This research was partiallysupported by Hewlett-Packard and by NSF grant #CSR-0509007.

References

[1] M. Aron, P. Druschel, and W. Zwaenepoel. ClusterReserves: A Mechanism for Resource Management inCluster-Based Network Servers. In Proceedings of theInternational Conference on Measurement and Modelingof Computer Systems, June 2000.

[2] Banu. Tinyproxy. http://www.banu.com/tinyproxy/,2008.

[3] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris,A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xenand the Art of Virtualization. In Proceedings of the 19thACM Symposium on Operating Systems Principles, Octo-ber 2003.

[4] E. V. Carrera, S. Rao, L. Iftode, and R. Bianchini. User-Level Communication in Cluster-Based Servers. In Pro-ceedings of the 8th International Symposium on High-Performance Computer Architecture, February 2002.

[5] E. Cecchet, J. Marguerite, and W. Zwaenepoel. C-JDBC: Flexible Database Clustering Middleware. In Pro-ceedings of the USENIX Annual Technical Conference,Freenix Track, June 2004.

[6] J. Chase, D. Anderson, P. Thackar, A. Vahdat, andR. Boyle. Managing Energy and Server Resources inHosting Centers. In Proceedings of the 18th Symposiumon Operating Systems Principles, October 2001.

[7] Y. Chen, A. Das, W. Qin, A. Sivasubramaniam, Q. Wang,and N. Gautam. Managing Server Energy and Opera-tional Costs in Hosting Centers. In Proceedings of theInternational Conference on Measurement and Modelingof Computer Systems, June 2005.

[8] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul,C. Limpach, I. Pratt, and A. Warfield. Live Migrationof Virtual Machines. In Proceedings of the InternationalSymposium on Networked Systems Design and Implemen-tation, 2005.

[9] R. Doyle, J. Chase, O. Asad, W. Jin, and A. Vahdat.Model-Based Resource Provisioning in a Web ServiceUtility. In Proceedings of the 4th USENIX Symposiumon Internet Technologies and Systems, March 2003.

[10] D. Gupta, K. Vishwanath, and A. Vahdat. DieCast: Test-ing Distributed Systems with an Accurate Scale Model.In Proceedings of the International Symposium on Net-worked Systems Design and Implementation, May 2008.

[11] D. Gupta, K. Yocum, M. McNett, A. C. Snoeren, G. M.Voelker, and A. Vahdat. To Infinity and Beyond: Time-Warped Network Emulation. In Proceedings of the Inter-national Symposium on Networked Systems Design andImplementation, May 2006.

[12] T. Heath, A. P. Centeno, P. George, L. Ramos, Y. Jaluria,and R. Bianchini. Mercury and Freon: Temperature Em-ulation and Management for Server Systems. In Proceed-ings of the 12th International Conference on Architec-tural Support for Programming Languages and Operat-ing Systems, October 2006.

[13] T. Heath, B. Diniz, E. V. Carrera, W. Meira Jr., andR. Bianchini. Energy Conservation in HeterogeneousServer Clusters. In Proceedings of the 10th ACM SIG-PLAN Symposium on Principles and Practice of ParallelProgramming, June 2005.

[14] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimiza-tion by Simulated Annealing. Science, 220(4598), 1983.

[15] K. Nagaraja, F. Oliveira, R. Bianchini, R. P. Martin, andT. D. Nguyen. Understanding and Dealing with OperatorMistakes in Internet Services. In Proceedings of the 6thUSENIX Symposium on Operating Systems Design andImplementation, December 2004.

[16] M. Nelson, B.-H. Lim, and G. Hutchins. Fast TransparentMigration for Virtual Machines. In Proceedings of theUSENIX Annual Technical Conference, April 2005.

[17] F. Oliveira, K. Nagaraja, R. Bachwani, R. Bianchini, R. P.Martin, and T. D. Nguyen. Understanding and Validat-ing Database System Administration. In Proceedings ofUSENIX Annual Technical Conference 2006, June 2006.

[18] P. Padala, K. G. Shin, X. Zhu, M. Uysal, Z. Wang,S. Singhal, A. Merchant, and K. Salem. Adaptive Controlof Virtualized Resources in Utility Computing Environ-ments. In Proceedings of EuroSys, March 2007.

[19] E. Pinheiro, R. Bianchini, and C. Dubnicki. Exploit-ing Redundancy to Conserve Energy in Storage Systems.In Proceedings of the International Conference on Mea-surement and Modeling of Computer Systems (SIGMET-RICS), June 2006.

[20] Rice University. DynaServer Project.http://www.cs.rice.edu/CS/Systems/DynaServer, 2003.

[21] K. Shen, H. Tang, T. Yang, and L. Chu. Integrated re-source management for cluster-based internet services. InProceedings of the 5th USENIX Symposium on OperatingSystems Design and Implementation, December 2002.

[22] P. Shivam, V. Marupadi, J. Chase, and S. Babu. CuttingCorners: Workbench Automation for Server Benchmark-ing. In Proceedings of the 2008 USENIX Annual Techni-cal Conference, June 2008.

[23] C. Stewart, T. Kelly, A. Zhang, and K. Shen. A Dollarfrom 15 Cents: Cross-Platform Management for InternetServices. In Proceedings of the USENIX Annual Techni-cal Conference, June 2008.

[24] C. Stewart and K. Shen. Performance Modeling and Sys-tem Management for Multi-component Online Services.In Proceedings of the International Symposium on Net-worked Systems Design and Implementation, May 2005.

[25] Y.-L. Tan, T. Wong, J. D. Strunk, and G. R. Ganger.Comparison-based File Server Verification. In Proceed-ings of the USENIX Annual Technical Conference, June2005.

[26] R. Thonangi, V. Thummala, and S. Babu. FindingGood Configurations in High-Dimensional Spaces: Do-ing More with Less. In Proceedings of the IEEE Interna-tional Symposium on Modeling, Analysis, and Simulationof Computer and Telecommunication Systems, September2008.

[27] B. Urgaonkar, P. Shenoy, A. Chandra, P. Goyal, andT. Wood. Agile Dynamic Provisioning of Multi-tier In-ternet Applications. ACM Transactions on Adaptive andAutonomous Systems, 3(1), March 2008.

[28] B. Urgaonkar, P. Shenoy, and T. Roscoe. ResourceOverbooking and Application Profiling in Shared Host-ing Platforms. In Proceedings of the 5th Symposium onOperating Systems Design and Implementation, Decem-ber 2002.

[29] T. Wood, P. J. Shenoy, A. Venkataramani, and M. S.Yousif. Black-box and Gray-box Strategies for VirtualMachine Migration. In Proceedings of the 4th Sympo-sium on Networked Systems Design and Implementation,April 2007.

[30] W. Zhang. Linux Virtual Server for Scalable NetworkServices. In Proceedings of the Linux Symposium, July2000.

[31] W. Zheng, R. Bianchini, and T. D. Nguyen. AutomaticConfiguration of Internet Services. In Proceedings of Eu-rosys, March 2007.