Power and Performance in I/O for Scientific Applications

K. Coloma A. Choudhary A. Ching W.K. LiaoCenter for Ultrascale Computing and Information Security

Northwestern University{kcoloma, choudhar, aching, wkliao}@ece.northwestern.edu

S. W. Son M. KandemirCSE Department

Pennsylvania State University{sson, kandemir}@cse.psu.edu

L. WardScalable Computing Systems Department

Sandia National Laboratorieslee@sandia.gov

Abstract

The I/O patterns of large scale scientific applica-tions can often be characterized as small, non-contiguous,and regular. From a performance and power perspec-tive, this is perhaps the worse kind of I/O for a disk. Two ap-proaches to mitigating the mechanical limitations of disksare write-back caches and software-directed power man-agement. Previous distributed caches are plagued by syn-chronization and scalability issues. The Direct AccessCache: DAChe system is a user-level distributed cachedthat addresses both these problems. Past work on manag-ing disk power during run time were effective, one shouldbe able to improve on those results by adopting a proac-tive scheme.

1. Introduction

Both transparent client-side caching and software-directed disk power management have an affect on I/O per-formance and power. While power consumption is not theprimary concern of write-back caching, write-back cachingnevertheless aggregates man smaller writes into largerones improving both performance and power. The pri-mary issue with client-side caching in large-scale systemsis scalably enforcing coherency. Coherency is of particu-lar concern in scientific applications, since many processesmay be accessing a single file. Because of this, the more re-laxed semantics of filesystems like NFS cannot be toleratedbecause of the counter-intuitive results they might pro-

duce. DAChe uses remote memory access and a scalablelock manager to maintain coherency and address applica-tions that must use independent I/O.

Software-directed power managment is, of course, moreconcerned with power, but extra performance is a wel-comed side-effect. Prior research and present practices in-clude spinning up and spinning down hard drives based onload at run time. One traditional way is to consider only twospeeds (TPM), while another, Dynamic RPM (DRPM) con-siders several incremental speeds. A more proactive way ofefficiently managing disk power consumption is to analyzecode and insert explicit disk speed calls.

In section 2 we describe the potential benefits and chal-lenges of file caching in large scale systems. Then in sec-tion 3, we describe its architecture and some initial perfor-mance results. Following in section 4, we discuss compilerdirected disk power management, before moving on to ourconlcuding remarks in section 5.

2. Client-side Caching Issues

The core of the cache coherency problem is ensuringglobally accessible data is up-to-date when used. Two pro-cesses,p0andp1, may read and cache some shared piece ofdata. Subsequently,p0 may write to the data. When thep1rereads the data, it will read directly from cache and not thenew data written byp0. This general issue arises at manylevels of the memory hierarchy, and is attacked from differ-ent angles depending sprecific features at each level.

In the context of I/O, caching can mask the latency of ac-cessing a local or even remote disk by allowing for quicker

reads and writes. Another optimization made possible byclient-side caching is write-behind in which small write datais buffered in the cache while computation continues. Theaggregated writes to one page are finally written to disk, canreduce the number of I/O calls.

In a parallel environment, there are often many moreclients than there are I/O servers. Client-side caching cannot only provide cached data quicker, but it also reduces theload on and contention for I/O server resources. Parallel en-vironments are similar to distributed ones in that there aremultiple clients accessing a single file system. In a parallelenvironment however, many processes often work on a sin-gle problem and concurrently access an output or input file.Allowing a single client at a time to access and cache a fileis unreasonable. Either a multiple client-side caches mustbe carefully maintained, or there should be know cachingat all. Maintaining a client-side cache in a parallel environ-ment is far more challenging than doing it in a distributedone.

Whenever considering client-side caching, it is neces-sary to also consider what kind of semantics to follow. Thestrict nature of POSIX consistency semantics make them ill-suited for parallel and distributed computing environments.Rigid enforcement of POSIX consistency in these environ-ments is usually at the expense of performance by eitherdisallowing efficient caching or centralizing cache manage-ment. MPI semantics are, by default, slightly more relaxed.After write completion, data must be visible to all processesin the same communicator. Unless atomic mode is explicitlyset, concurrent access to conflicting parts of a file are unde-fined. The tricky thing is that in atomic mode, an atomicaccess may be noncontiguous. If not for the last bit aboutatomic noncontiguous accesses, a POSIX compliant filesys-tem could provide all that MPI requires semantically.

Cache coherency provides fairly intuitive results. Withsequential consistency so hard to provide at the library orfile system level, this onus is better left to the applicationdeveloper who is much more well equiped to handle order-ing issues than any library could possibly be.

3. DAChe

DAChe uses one-sided communication, and can be archi-tecturally divided into 3 primary subsystems: cache meta-data, locking, and cache management. Figure 1 illustratesthe basic interactions between these subsystems. Its mod-ular implementation makes it quite easy to port and ex-periment with given that some basic RMA requirementsare met. One over-arching theme always under consider-ation during the design of DAChe is to keep all aspectsof cache management as decentralized as possible. A sec-ondary theme is minimization communication where pos-

8

ClientProcess

ClientProcess

ClientProcess

� � � �� � � �� � � �� � � � � � � �� � � �

� � � �� � � �� � � �� � � �

� � � �� � � �� � � �� � � �Cache Data

Metadata

Passive State

Cache DataMetadata

Passive State

Cache DataMetadata

Passive State

page0page3page6

page1page4

page8

page2page5

page7

filepage ids 0 1 2 3 4 5 6 7

Figure 1. DAChe architecture with passivemetadata and cache servers on each client.Metadata is striped across clients, but a filepage can be cached on any client.

sible. The main tenet of DAChe is that only a single copyof any file page can reside on at most one process. Thissingle-copy rule ensures cache coherency and removes thetask of maintaining state for replicated data. Another way tothink of it is that there is never more than one usable copyof any given file page. The use of RMA keeps I/O opera-tions in DAChe passive, but requires thatdache open anddache close be collective in order to set things up andbreak things down safely. Afterdache open completes,all communication is one-sided unless it is with a mutexserver described in more detail in the Mutual Exclusion 3.3subsection.

3.1. Remote Memory Access

Remote Memory Access (RMA) interfaces provide whatcan be thought of as shared-memory emulation for a dis-tributed memory environment. RMA in its truest form al-lows for the movement of data to or from a remote pro-cess without its active participation. The primary functionsof any RMA interface mirrors shared-memory should con-sist of a get, put, and some sort of atomic test & set or swap.DAChe uses RMA to remotely access cache metadata andcached data.

DAChe currently uses Portals [1] for RMA. While porta-bility is definitely a con, the Portals implementation is fairlymature and exists on specific large-scale platforms of inter-est. At the time of development, MPI-2’s one-sided commu-nication was not yet ready in MPICH2.

3.2. Cache Metadata

Cache metadata maintains basic state for each page inthe file. Most importantly, this metadata provides the where-abouts of any given file page. File page refers to the logicalpartitioning of the entire file into blocks of a size match-ing the page size of the cache. If a page is cached on any

process the metadata reflects the caching process as wellas an index location into that process’s cache. cache meta-data is remotely accessible through RMA and distributedacross the application nodes in a deterministic fashion; inthis case a basic striping algorithm. By striping the meta-data array, or table as it will be called, across nodes deter-ministically, not only is a potential bottleneck avoided, butthere is also no communication required to find the meta-data associated with any file page.

Creating metadata for each logical file page brings up theissue of metadata allocation. The allocation process requiresexplicit coordination amongst the application processes,and this is only available at the collectivedache open anddache close functions. Ideally, one would want the sizeof the metadata table to be directly related to the size ofthe file. What this basically entails is some level of coop-eration among processes for growing or shrinking the tablesize during run time. Since the write operations are inde-pendent, however, there is no opportunity to coordinate allthe processes in order to modify the size of the table. With-out this coordination, the last resort would be the ability toremotely allocate globally accessible memory. Needless tosay, remote memory allocation brings its own set of chal-lenges.

Given the distributed and passive nature of cache meta-data, one crucial element is enforcing mutually exclusiveaccess to it.

3.3. Mutual Exclusion

The purpose of the mutual exclusion subsystem is to en-sure safe access to read and modify cache metadata. Ide-ally, mutual exclusion is directly supported in the RMA in-terface. With proper RMA support for mutual exclusion, thelocking subsystem can be also distributed across the appli-cation nodes as passive remote accessible state.

MPI-2 explicitly provides theMPI Win Lock and cor-responding unlock functions. Since the current test platformfor DAChe is a threadless environment, however, this rulesout the use of the MPICH2 RMA interface. MPICH2 is fo-cused on portability, and threads are more commonly avail-able than hardware supported one-sided communication. Atthe same time, the atomic swap operation in the Portals li-brary has yet to be implemented.

In the absence of an RMA solution to mutual exclusion,several processes from the allocated user processes are si-phoned off from the main group to act as mutex servers.They are spun off duringdache open and returned duringdache close. During this time, the mutex servers cannotexecute any user application code. So while passive RMAmutual exclusion is preferred, DAChe can still be evaluatedusing dedicated mutex servers. Later, these mutex serverswill become passive elements on the application processes.

Since mutex responsibilities will eventually be movedto the client, the mutex servers are intentionally kept quitesimple. Lock responsibility for each file page is spreadacross the mutex servers in the same way cache metadatais spread across application processes. The mutex serversservice locks in the order they come, queueing requests tothe same cache metadata. Polling and simple queueing areboth implementable when the mutex servers become pas-sive state on remote processes. A process must block untilits lock request is fulfilled by the mutex server.

Typically, cache metadata is not “held” for extended pe-riods of time. It is locked only briefly for modification. It isnot held for the duration of access to the actual cached data.Minimizing the time that the metadata is locked, should re-duce the amount of simultaneous lock requests to the samemetadata. The exception to short lock times is when a par-ticular file page is being brought into the cache or a cachepage is being evicted. In either case, the cache metadatamust be locked for entire I/O phase in order to prevent earlyor late cache accesses, respectively.

3.4. Cache Management

Pages cached on one process are globally accessible toany other process through RMA. Although access to meta-data is carefully mediated, remote access to cache data isbasically a free-for-all. Since any file page can be cachedin at most one cache, all accesses to that page are coher-ent.

Cache management and eviction is handled locally withone exception to be discussed a little further along. Whatdata to cache is determined by the local procces’s I/O ac-cesses. If a process accesses a file page that is not yet cachedon any of the other processes, it caches it itself and updatesthe corresponding cache metadata to reflect this change.Should a process run out of cache space locally, it mustevict a page based on some local policy such as a least re-cently used (LRU) policy. Remember that during eviction, apage’s metadata cannot be accessed at all. Another precau-tion alluded to earlier is that processes accessing a remotelycached page must “pin” the page in cache so the page can-not be evicted while being accessed. This pin is a semaphorecontained in the cache metadata along with location infor-mation. While a page is pinned, the process on which thecache page resides cannot evict the page, and must eitherwait until the page is “un-pinned” or try to evict a differ-ent page. New data is not written to disk until it is eitherevicted or written out atdache close.

3.5. Initial Performance Evaluation

A preliminary analysis of DAChe is done using a syn-thetic I/O access pattern that should, by design, benefit from

...

� �� �� �� �� � �� � �

� �� �

� �� �� �� � � �� �

� �� �

� � �� � �

� �� �

� �� � � �� �

� �� �

� �� �� �� � � �� �

� �� �

� �� �� �� � � �� �

� �� �

i=0, j=0

i=1, j=0

i=2, j=0

i=3, j=0

i=0, j=1

i=0, j=2

Figure 2. Several iterations of the sliding win-dow I/O pattern for 4 processes.

cached data. It is roughly based on the regular noncontigu-ous access patterns often found in scientific applications andis meant to test the performance of DAChe. The benefits areal application may derive from DAChe are also clearly ofinterest. The basic labeling convention is as follows:

• dache-n where n is the number of mutex servers

• Nio is the number of clients actually caching data(non-mutex servers)

• 50:50 refers to an equal mix of clients and mutexservers

The sliding window application uses a repetitive I/O ac-cess pattern, and the underlying caching library uses thelock service to gain exclusive access to cache page meta-data. 2 describes the access pattern of the synthetic bench-mark. Each process accesses a contiguous chunk of data. Ineach subsequent iteration, processes circular shift theirac-cesses until all chunks are read before sliding to the nextset of four chunks. Since there is one lock per page, and thebenchmark accesses are page-aligned, the number of metadata locks is tied directly to the number I/O operations. Thecyclic access pattern makes the total amount of I/O done in-crease along a power function rather than linearly.

It is important to note that since the DAChe system is theprimary subject of evaluation, actual I/O is removed fromDAChe to prevent interference from any specific filesys-tem. The sliding window access pattern is such that once afile page is brought into cache, it remains there throughoutthe execution and is only remotely accessed. Though actualI/O would have been performed to bring in the file data, allsubsequent accesses are really accessing cached data, pos-sibly on remote nodes. Figure 3 illustrates the infeasibilityof running this benchmark with actual I/O and without anycaching. The resulting amount of aggregate I/O would be

Simulated I/O to File System

0

100000

200000

300000

400000

500000

600000

700000

0 16 32 48 64 80 96 112 128 144

DAChe Clients

DAChe Read Ops (32K each)

*nix Reads (64K each)

Figure 3. The number of I/O calls that wouldhave been made to the filesystem is signifi-cantly reduced by the presence of DAChe.

from 1 GB to 80 GB. There is little doubt this reduction inI/O calls would also reduce the power consumption of thedisks. Bandwidth is calculated based on the amount of I/Othe sliding window application thinks it has performed.

Since scalability is the primary goal of the DAChe, it wastested with various numbers of mutex servers running. In-tuitively, the heavier the lock system is taxed, the more mu-tex servers are needed to accomodate the increased load.This is illustrated clearly in figure 4 where a larger num-ber of mutex servers allows a larger number of clients with-out severely hindering performance when fewer than themaximum clients is used. From a cost efficiency perspec-tive, one would like to stay on the outside curve using theminimum number of processes at each point. This client tomutex server ratio is highly dependent on the properties ofthe specific machine. A 50:50 mix where there are an equalnumber of clients and mutex servers allows the mutex ser-vice to scale with the application size. As expected from thegrowth rate of I/O amount, this outer bandwidth curve isroughly an inverse power function. This 50:50 mix is in an-ticipation of each client doubling as a passive lock server.The most important point from Figure 4 is the number ofmutex servers determines at which number of clients per-formance will plateau.

4. Compiler Directed Disk PowerManagement

Our overall approach is depicted in Figure 5. A uniquecharacteristic of our approach is that itexposesthe disk pa-

Aggregate Bandwidths

0

200

400

600

800

1000

1200

0 32 64 96 128 160 192 224 256 288

DAChe Clients

Mtx-2

Mtx-4

Mtx-6

Mtx-8

Mtx-10

Mtx-12

Mtx-50:50

MB

/s

Figure 4. The number of clients at whichbandwidth knees over is dependent on howmany lock servers there are.

Figure 5. High-level view of our compilerdriven approach to power reduction.

rameters and the disk layout of array data to the compiler.The goal of doing so is to allow the compiler to determine(estimate) the disk active and idle times. The parametersused by the compiler in this approach include the numberof ids of the disks over which each array is striped (i.e., thestripe factor), the stripe size used, and the id of the disk thatcontains the first stripe of data. Using this information andthe data access pattern extracted by analyzing the applica-tion source code, the compiler determimnes what we referto as thedisk access pattern(DAP). Basically, a DAP indi-cates how the disks in the I/O subsystem are accessed and

Figure 6. An example of application of our ap-proach.

reveals information that is vital for disk power management:disk inter-access times, i.e., the gap between the two succes-sive accesses to a given disk. This information can be usedin two different ways to do proactive power management:(1) placing a disk into a suitable low-power mode after itscurrent access is complete, and (2) pre-activating a disk toeliminate the potential performance penalty due to powermanagement. Then, based on this information, the applica-tion code is modified to insert explicit power managementcalls. The nature of these calls depends on the underlyingmethod used (e.g., TPM versus DRPM), and will be ex-plained shortly. This compiler transformed code can executeon a system that uses TPM or DRPM disks to do proactivepower management. In the rest of this section, we explainthe three important components of our compiler-driven ap-proach: the compiler-analysis to identify disk accesses, theDAP, and the insertion of explicit power management callsin the code.

In order to determine the disk access pattern, we needtwo types of information: data access pattern and disk lay-out of array data. The data access pattern indicates the orderin which the different array elements are accessed, and is ex-tracted by the compiler by analyzing the source code of theapplication. To determine which particular disks are beingaccessed, the compiler also needs the layout of array data(i.e., the file that holds the array elements) on the disk sub-system. In this context, the disk layout of an array (which isstored in a file) is specified using a 3-tuple:

(startingdisk, stripefactor, stripesize).

The first element in this 3-tuple indicates the disk fromwhich the array is started to get striped. The second ele-ment gives the number of disks used to stripe the data, andthe third element gives the stripe (unit) size. As an exam-ple, in Figure 6(b), arrayU1 is striped over all four disksin the figure. Assuming that the stripe size isS and the to-tal array size is 4S (for illustrative purposes), the disk lay-out of this array can be expressed as (0, 4,S). Now, letus consider the other array (U2) in Figure 6(b), assuming

its size and stripe size are 4S and 2S, respectively. Con-sequently, its disk layout can be expressed as (2, 2, 2S).To illustrate the process of identifying the disk accesses,let us consider the code fragment in Figure 6(a). Duringthe execution of the first loop nest, this code fragment ac-cesses the array elementsU1[1], U1[2], . . . , U1[2S]andU2[1], U2[2], . . . ,U2[2S]. Consequently, for ar-rayU1, we access the first two disks (disk0 and disk1); andfor arrayU2, we access only the third disk (disk2). Notethat, the several current file systems and I/O libraries forhigh-performance computing support calls available to con-vey them the disk layout information when the file is cre-ated. For example, in PVFS [6], we can change the defaultstriping parameter by settingbase (the first I/O node tobe used),pcount (stripe factor), andssize (stripe size)fields of thepvfs filestat structure. Then, the stripinginformation defined by the user via thispvfs filestatstructure is passed to thepvfs open() call. When creat-ing a file from within the application, this layout informa-tion can be made available to the compiler as well, and, asexplained above, the compiler uses this information in con-junction with the data access pattern it extracts to determinethe disk access pattern. On the other hand, if the file is al-ready created on the disk subsystem, the layout informationcan be passed to the compiler as a command line parame-ter.

The DAP lists, for each disk, the idle and active times ina compact form. An entry for a given disk looks like:

< Nest 1, iteration 1, Idle>< Nest 2, iteration 50, Active>< Nest 2, iteration 100, Idle>

We see from this example DAP that, the disk in question re-mains in the idle state (not accessed) until the 50th iterationof the second nest. It is active (used) between the 50th it-eration and the 100th iteration of the second nest, follow-ing which it becomes idle again, and remains so for the restof execution. For the example code fragment in Figure 6(a)and the disk layouts illustrated in Figure 6(b), Figure 6(c)gives the DAPs for each of the four disks in the system. Thelast component of our compiler-driven strategy is responsi-ble from inserting explicit disk power management calls inthe code. Let us first focus on the TPM based disks. It is im-portant to note that a DAP is given in terms of loop iter-ations. In order to determine the appropriate places in thecode to insert explicit power management calls, we need tointerpret the loop iterations in terms of cycles, which can beachieved as follows. The cycle estimates for the loop itera-tions are obtained from the actual measurement of the pro-gram by using a high-quality timer calledgethrtime, whichis available on the UltraSPARC-based systems. Since themeasured execution time is given in nanoseconds scale and

we are given the machine’s clock rate, we can estimate thenumber of cycles per each loop iteration.

Once we determine that the estimated disk idleness (interms of cycles), if this idleness is larger than thebreakeventhreshold, i.e., the minimum amount of idle time requiredto compensate the cost of spinning down in a TPM disk, thecompiler inserts a spin down call in the code. The format ofthis call is as follows:

spin down(diski),

wherediski is the disk id. Since a DAP indicates not onlyidle times but also active times anticipated in the future, wecan use this information to preactivate disks that have beenspun down by aspin down call. To determine the appro-priate point in the code to spin up the disk, we take into ac-count the spin-up time (delay) of the disk. Specifically, thenumber of loop iterations before which we need to insert thespin-up (pre-activation) call can be calculated as:

d = dTsu

s + Tm

e (1)

whered is the pre-activation distance (in terms of loop it-erations),Tsu is the expected spin-up time,Tm is the over-head of aspin up call, ands is the number of cycles in theshortest path through the loop body. Note that,Tsu is typi-cally much larger thans. We also stripe-mine the loop, be-cause it is unreasonable to unroll the loop to make explicitthe point at which the spin-up call is to be inserted. The for-mat of the call that is used to pre-activate (spin up) a disk isas follows:

spin up(diski),

where as beforediski is the disk id. For our running exam-ple, Figure 6(d) shows the compiler-modified code with thespin down andspin up calls. In this transformed code,the calculated pre-activation distance,d, is assumed toS/2.Let us assume that, at the beginning of the first loop, disk0and disk2 are active, and disk1 and disk3 are in the low-power mode. Since the analyzed DAP indicates that disk1and disk3 will be accessed afterS iterations, we insert aspin up call to pre-activate disk1 and disk3 between thefirst and the second loop nests in Figure 6(d). After execut-ing S iterations, we insert aspin down call for disk0 anddisk2 because they are not accessed for the remaining exe-cution of the code. Note that, if we do not use pre-activation,the disk is automatically spun up when an access (request)comes; but, in this case, we incur the associated spin-up de-lay fully. The purpose of the disk pre-activation is to elimi-nate this performance penalty.

While our discussion so far has focused on the TPMdisks as the underlying mechanism to save power, thecompiler-driven approach can also be used with DRPM.The necessary compiler analysis and the DAP con-struction process in this case are the same as in the

TPM case. The main difference is how the informa-tion recorded in the DAP is used (by taking into ac-count the times to change RPM) and in the calls in-serted in the code. In this case, we employ the followingcall:

set RPM(rpm levelj,diski),

wherediski is the disk id, andrpm levelj is the jth

RPM level (i.e., disk speed) available. When executed, thiscall brings the disk in question to the speed specified. Theselection of the appropriate disk speed is made as follows.Since the transition time from one RPM step (level) to an-other is proportional to the difference between the two RPMsteps involved [4], we need to consider the detected idletime in order to determine the target RPM step. Conse-quently, we select an RPM level if and only if it is the slow-est RPM level whose transition latency can be captured bythe estimated idleness. To evaluate our proposed compiler-directed proactive approach to disk power management, wewrote a trace generator and a disk power simulator (see Fig-ure 5).

4.1. Brief Results

To compare different approaches to disk power manage-ment, we implemented and performed experimentswith dif-ferent schemes:

• Base: This is the base version that does not employany power management strategy. All the reported diskenergy and performance numbers are given as valuesnormalized with respect to this version.

• TPM: This is the traditional disk power managementstrategy used in studies such as [3] and [2], using twomodes, normal and low-power.

• Ideal TPM (ITPM): This is the ideal version of theTPM strategy. In this scheme, we assume the exis-tence of an oracle predictor for detecting idle peri-ods. Consequently, the spin-up and spin-down activi-ties are performed in an optimal manner.

• DRPM: This is the dynamic RPM strategy proposedin [4], which steps the RPM at several different incre-ments.

• Ideal DRPM (IDRPM): This is the ideal version of theprevious strategy. In this scheme, we assume the exis-tence of an oracle predictor for detecting idle periods,as in the ITPM case. Consequently, the disk speed tobe used is determined optimally.

• Compiler-Managed TPM (CMTPM): This corre-sponds to our compiler-driven approach when it isused with TPM. The compiler estimates idle pe-riods by analyzing code and considering disk lay-

Figure 7. An example of application of our ap-proach.

outs, and then makes spin-down/up decisions basedon this information.

• Compiler-Managed DRPM (CMDRPM): This corre-sponds to our compiler-driven approach when it isused with DRPM. The compiler estimates idle peri-ods by analyzing code and considering disk layouts,and then selects the best disk speed to be used basedon this information.

It must be emphasized that, the ITPM and IDRPMschemes are not implementable. The reason that we makeexperiments with them is that we want to see how closeour compiler-based schemes come close to the opti-mal. All necessary code modifications are automated usingthe SUIF infrastructure [5].

The graph in Figure 7 gives the energy consumption ofour benchmarks under the different schemes described ear-lier. One can make several observations from these results.First, the TPM version (ideal or otherwise) does not achieveany energy savings. Second, while the DRPM version gen-erates savings (26% on the average), the difference betweenit and the ideal DRPM (IDRPM) is very large; the latterreduces the energy consumption by 51% when averagedover all benchmarks in our suite. This shows that a reac-tive strategy is unable to extract the potential benefits fromthe DRPM scheme. Our next observation is that the CM-DRPM scheme brings significant benefits over the DRPMscheme, and improves the energy consumption of the basescheme by 46%. In other words, it achieves energy savingsthat are very close to those obtained by the IDRPM strat-egy. These results demonstrate the benefits of the compiler-directed proactive strategy.

It is to be noted, however, that the energy consump-tion is just one part of the big picture. In order to havea fair comparison between the different schemes that tar-get disk power reduction, we need to consider their perfor-mances (i.e., execution times/cycles) as well. The bar-chartin Figure 8 gives the normalized execution times (with re-

Figure 8. An example of application of our ap-proach.

spect to the base version) for the different schemes evalu-ated. The reason that the TPM-based schemes do not incurany performance penalty is because they are not applicable,given the short disk idle times discussed earlier. When welook at the DRPM-based schemes, we see that the conven-tional DRPM incurs a performance penalty of 15.9%, whenaveraged over six benchmarks. We also see that the CM-DRPM scheme incurs almost no performance penalty. Themain reason for this is that this scheme starts to bring thedisk to the desired RPM level before it is actually needed,and the disk becomes ready when the access takes place.This is achieved by accurate prediction of disk idle peri-ods. These results along with those presented in Figure 7indicate that the compiler-directed disk power managementcan be very useful in practice, in terms of both energy con-sumption and execution time penalty. Specifically, as com-pared to the reactive DRPM implementation, this schemereduces the disk power consumption and eliminates the per-formance penalty.

5. Conclusions and Future Directions

While preliminary results for DAChe are promising,there is still much to be investigated. In the current imple-mentation, the LRU queue is stored locally, and is only af-fected by local accesses to cache pages. Not only shouldthe basic eviction performance of DAChe be evaluated,but further research is planned into how taking remote ac-cesses into account in the eviction policy may affect overallcache performance. An extension of that would be migrat-ing cache pages frequently accessed by a particular processto that node.

Tighter integration with MPI is also planned. By movingaway from the Portals interface, DAChe will benefit fromthe portability of MPICH2.

Early performance results for DAChe suggest it scalesreasonably well. The extremely distributed characteristics

of DAChe get around a number of potential bottlenecks.While there were no explicit power studies done on the ef-fects of client-side caching, presumably DAChe can offera reduction in the number of disk accesses an applicationmay make. The two most interesting features of DAChe areits efficient use of the one-sided communication architec-ture and its scalable lock manager.

For array-intensive scientific applications, the compilercan extract a disk access pattern and use it for placing disksinto the most suitable low-power modes. In principle, thisapproach can be used with both TPM and DRPM disks.The compiler-directed proactive approach to disk power-managements is successful in imporoving the behavior ofthe DRPM based scheme. On average, it brings an addi-tional 18% energy savings over the hardware-based DRPM.

Acknowledgements

This work was supported in Part by NSF Grants#0444158, #0406340, #0093082, CNS-0406341, CCF-0444405, Sandia National Laboratory contract 28264, andDOE award DE-FC02-01ER25485.

References

[1] R. Brightwell, B. Lawry, A. Maccabe, and R. Riesen. Por-tals 3.0: Protocol building blocks for low overhead communi-cation.

[2] F. Douglis, P. Krishnan, and B. Bershad. Thwarting thepower-hungry disk. InProceedings of the USENIX WinterConference, pages 292–306, 1994.

[3] F. Douglis, P. Krishnan, and B. Bershad. Adaptive disk spin-down policies for mobile computers. InProceedings of the2nd Symposium on Mobile and Location-Independent Com-puting, pages 121–137, 1995.

[4] S. Gurumurthi, A. Sivasubramaniam, M. Kandemir, andH. Franke. DRPM: Dynamic speed control for power man-agement in server class disks. InProceedings of the Interna-tional Symposium on Computer Architecture, pages 169–179,June 2003.

[5] M. W. Hall, J. M. Anderson, S. P. Amarasinghe, B. R. Mur-phy, S.-W. Liao, E. Bugnion, and M. S. Lam. Maximizingmultiprocessor performance with the suif compiler.Com-puter, 29(12):84–89, 1996.

[6] R. Ross, P. Carns, W. L. III, and R. Latham. Using the parallelvirtual file system. July 2002.