Toward a Cost Model for System Administration · 2019-02-25 · Toward a Cost Model for System Administration Alva L. Couch, Ning Wu, and Hengky Susanto – Tufts University ABSTRACT

Toward a Cost Model forSystem Administration

Alva L. Couch, Ning Wu, and Hengky Susanto – Tufts University

ABSTRACT

The core of system administration is to utilize a set of ‘‘best practices’’ that minimize costand result in maximum value, but very little is known about the true cost of system administration.In this paper, we define the problem of determining the cost of system administration. For supportorganizations with fixed budgets, the dominant variant cost is the work and value lost due to timespent waiting for services. We study how to measure and analyze this cost through a variety ofmethods, including white-box and black-box analysis and discrete event simulation. Simplemodels of cost provide insight into why some practices cost more than expected, and whytransitioning from one kind of practice to another is costly.

Introduction

What is a set of ‘‘best practices’’? Ideally, it is aset of practices that cost the least while having themost value, i.e., a model of practice for which the ratiovalue/cost is maximal over the lifecycle of the equip-ment being managed. We have not succeeded in evalu-ating practices according to this yardstick, however,because there is no usable model of cost for any par-ticular set of practices. We would like a model thatpredicts, based upon particular management decisions,the total cost of operations that results from thosedecisions over the lifecycle of the network being man-aged. This is one goal of ‘‘analytical or theoretical sys-tem administration’’ [5, 6].

Many system administrators and managers con-sider a complete cost model to be an impossible goalfor several reasons. First, the actual cost of systemadministration is a relatively constant and monolithicline item in many IT budgets; it is commonly techni-cally infeasible to break the lump sum cost into com-ponent costs for the purpose of evaluating strategy.Mechanisms for recording day-to-day costs (e.g.,detailed time sheets) are often more expensive to man-age than the money they might potentially save. Andfor the organizations whose audit requirements forcethem to maintain detailed costing data, these recordsare usually confidential and unavailable to researchersoutside the organization. Thus any really usable costmodel has to be practical in not consuming resources,tunable for specific situations by the end-user, andmust allow that user to incorporate confidential datainto the model without divulging it to outsiders.

Currently, instead of considering costs, we jus-tify best practices by what might best be called a‘‘micro-economic’’ model. We say that certain prac-tices ‘‘make the job easier’’, or other weak justifica-tions. Is ‘‘simpler ’’ really ‘‘cheaper ’’? We have yet toprove this assertion and – in many cases – the micro-cosmic reasoning we use today seems to be incorrect

at a macrocosmic (lifecycle) scale. A case in point isthe use of automation, which is ‘‘better than manualchanges’’ except that – at a macrocosmic scale – scal-ing increases costs in ways that are inconceivablewhen working on a small number of machines. Thereasons for this apparent contradiction are deep andwill be discussed later in the paper.

Current Ideas About Cost

The first step toward a cost model was made byPatterson [18], who points out that while ‘‘administra-tive costs’’ may be fixed and non-varying, the cost ofdowntime varies with scale of outage and disruptsmore than computer use. Patterson’s formula for thecost of downtime is based upon calculation of two fac-tors we previously ignored as system administrators:revenue lost and work lost. Even if our system admin-istration group has a fixed size and operating budget,the cost of downtime varies with the severity andscope of outage, and lifecycle cost of operations thusvaries with risk of outage. Patterson also points outthat there are more subtle costs to downtime, includingmorale and staff attrition. But how do we quantifythese components in a cost model?

Cost modeling also depends upon risk and costof potential catastrophes. Apthorpe [1] describes themathematics of risk modeling for system administra-tors. By understanding risk, we can better make costdecisions; the lifecycle cost of an administrative strat-egy is the expected value of cost based upon a riskmodel, i.e., the sum of ‘‘cost of outcome’’ times‘‘probability of outcome’’ over all possible outcomes.

Cost modeling of risks is not simple; Cowan, etal. [8] point out that simply and blindly mitigatingrisks does not lead to a lowest-cost model. It is oftenbetter to wait to apply a security patch rather thanapplying it when it is posted, because of the likelihoodof downtime resulting from the patch itself. Thus theglobal minimum of lifecycle cost is not achieved by

19th Large Installation System Administration Conference (LISA ’05) 125

Toward a Cost Model for System Administration Couch, Wu, and Susanto

simply minimizing perceived risks; other risks enterthe system as a result of mitigating the perceived ones.

Further, Alva Couch made the bold claim at thelast LISA [7] that the essential barrier to deploymentof better configuration management is ‘‘cost of adop-tion’’. The reason that configuration managementstrategies such as automation are not applied morewidely is that it costs too much to change from unau-tomated to automated management. But he stoppedshort of truly quantifying cost of adoption in that pre-sentation, due to inadequate models. Meanwhile, manypeople pressured him in one way or another to formal-ize the model and demonstrate his claims rigorously.This paper is the first small result of that pressure.

administratorpopulation

queue

requests completed requestspopulation

client

internal requests

client

Figure 1: System administration as a queueing system.

queue

requests completed requests

m2

m1

administratorpopulation

clientpopulation

simpletasks

complextasks

junior staff

seniorstaff

internal requests

client

Figure 2: Multiple classes of requests and administrators.

In this paper, we make the first step toward a costmodel for system administration, based upon relatedwork in other disciplines. We begin by defining thecomponents of an overall lifecycle cost model. We lookat real data from a trouble-ticketing system to under-stand the qualities of load upon a support organization,and discuss the problems inherent in collecting datafrom real systems. We explore the relationship betweensystem administration and capacity planning, and showthat we must determine specific rates in order to deter-mine costs. We borrow mechanisms for determiningthose rates from white-box and black-box cost modelsin software engineering. Finally, we turn to discreteevent simulation in order to understand the relation-ships between rates and cost. As a result, we can beginto quantify the cost of some decisions about practice,including deployment schedules for new software.

A Simple Model Of System Administration

First, system administration can be modeled as aqueueing system (Figure 1) in which incomingrequests arrive, are queued for later processing, andeventually dequeued and acted upon, and completed.Each kind of request arrives at the queue with an‘‘arrival rate’’ and is completed in a length of timewhose reciprocal represents a ‘‘service rate.’’ Weembody all changes made to the network as requests; arequest may indicate a problem or ask for a change inthe nature of services offered. Requests arise frommany sources, including users, management, and eventhe system administrator herself may make a note tochange something. Likewise, requests are granted viamany mechanisms, including work by system admin-istrators and work by others.

Note that this is a more complex model than rep-resented by the typical helpdesk. In a typical ticketsystem, tickets represent external requests, whileinternal requests (e.g., actions generated by a securityincident report) are not given ticket numbers. In ourrequest queue, all change actions are entered into thequeue, serviced, and closed when done.

System administration has complex goals, so therequest queue has a complex structure; it is (in the lan-guage of capacity planning [17]) a multi-class queue-ing system consisting of a mixed set of several‘‘classes’’ of requests (Figure 2). Many kinds ofrequests, with different average arrival rates, are com-bined into one request stream. Each kind of request K

126 19th Large Installation System Administration Conference (LISA ’05)

Couch, Wu, and Susanto Toward a Cost Model for System Administration

has a distinct average service rate µK (and perhaps, adistinct statistical distribution of service times). Aswell, a realistic system administration organization is anon-product system: system administrators do notfunction independently like a set of cloned web-servers; they communicate and interact with oneanother, affecting throughput. A product system (as inCartesian product) consists of a number of compo-nents that function independently (so that the state-space of the whole system is a Cartesian product ofthe state-spaces of its parts).

Request ArrivalsWhile the overall structure of the request queue

is complex, we observe that the structure of someclasses of requests is easy to understand. Many classesof requests arrive with a ‘‘Poisson distribution’’ ofinter-arrival times. In a Poisson distribution witharrival rate of λ requests per unit time,

1. The mean inter-arrival time is 1/λ.2. The standard deviation of the inter-arrival time

is 1/λ.3. The arrival process is memoryless; the probabil-

ity that a request will arrive in the next t sec-onds is independent of whether one arrivedrecently.

Many kinds of requests naturally obey this distri-bution. For example, any kind of request in which alarge population operates independently of one anotherhas a Poisson distribution, e.g., forgotten passwords.

As well, many non-Poisson classes of requests(e.g., virus attacks) arrive with a Poisson distributionif viewed at the proper scale. While requests for viruscleaning of individual hosts arrive in bursts and arenot memoryless, the arrival of the virus at one’s site isan independent, memoryless event. If we treat thevirus arrival at one’s site as one event, rather than thethousands of requests it may generate, then newviruses arrive with a roughly Poisson distribution(because they originate from independent sources atrelatively constant rates). Likewise, while an outage ofa particularly busy server may generate thousands oftickets, the outage itself obeys a Poisson distributioneven though the tickets resulting from the outage donot. Many other kinds of requests have this character;although actual tickets arrive in bursts, the real prob-lem occurs in a memoryless way that is independent ofall other problem occurrences. Examples include hard-ware failures, power outages, denial-of-service attacks,spam, etc.

Request ProcessingThe second part of our model is how requests are

processed. Like request arrivals, request processing iscomplex but there are parts of it that are understand-able. For example, many kinds of requests are com-pleted via an ‘‘exponential distribution’’ of servicetime. The properties of an exponential service time aresimilar to those for a Poisson arrival; if a class of

requests is serviced with an exponential rate of µrequests per unit time, then:

1. The mean time for servicing a request is 1/µ.2. The standard deviation of service time is 1/µ.3. The service process is memoryless; the proba-

bility that a request will be finished in the next tseconds is independent of whether we know ithas been in progress for s seconds already.

The last assumption might be paraphrased ‘‘Awatched pot never boils.’’

Examples of requests that exhibit an exponentialservice time include password resets, routine accountproblems, server crashes, etc. For each of these, thereis a rate of response that is independent of the natureof the specific request (i.e., which user) and seldomvaries from a given average rate µ. Requests that can-not be serviced via an exponential distribution includecomplex troubleshooting tasks, and any request wherethe exact method of solution is unknown at the time ofrequest. In general, a request for which the answer iswell documented and scripted exhibits an exponentialdistribution of service times; requests with no docu-mented response do not.

Lessons From Capacity Planning

Real data discussed below shows that inter-arrival times may not exhibit a Poisson distribution,and that service times may not be exponentially dis-tributed. However, much is known about the perfor-mance of idealized queues governed by Poisson andexponential distributions, and there are many systemadministration tasks for which these performance esti-mates are reasonable.

A queue that exhibits Poisson arrivals with rate λand has c independent system administrators workingwith service rates µ is called an ‘‘M/M/c’’ queue. Thefirst M stands for ‘memoryless’ (Poisson) arrivals, thesecond M stands for ‘memoryless’ (exponential) ser-vice times, and c is a count of servers (administrators)all of whom complete requests with rate µ. The behav-ior of an M/M/c queue is well understood and is com-monly used for capacity planning of server farms andIT infrastructure.

For an M/M/c queue, whenever λ/cµ < 1, theprobability that the queue is empty is

S0 =1

c−1

n=0Σ (λ/µ)n

n!+

1

c!

(λ/µ)c

1 − λ/(cµ)

(1)

and the ‘‘mean time in system’’ (average wait) for arequest [15] is

W =S0(λ/µ)c

c!cµ(1 − λ/(cµ))2+

1

µ. (2)

The mean time spent waiting for n requests to beserviced is n times the mean wait for one. Moreimportant, this equation allows us to predict whetheradding more system administrators will not solve aresponse-time problem. As c grows, the first term of



the above equation goes to 0 and the response timeconverges toward the theoretical minimum 1/µ.

Many other equations and relationships exist formore general queues. In this paper, we will consideronly M/M/c models; for an excellent guide to othermodels and how to predict performance from them(including excel spreadsheets for decision support),see [17].

0

500

1000

1500

2000

0 50 100 150 200 250 300 350 400

Num

ber

of ti

cket

s

Days

Ticket arrival and completion

Figure 3: Ticket durations in ECE/CS from 7/2004 to 7/2005.

0

500

1000

1500

2000

0 50 100 150 200 250 300 350 400

Num

ber

of ti

cket

s

Days

Ticket arrival and completion

Figure 4: Ticket durations less than 30 days.

Learning From Real DataFrom above, it is easy to analyze systems that

behave according to Poisson arrivals and exponentialservice. How realistic are these assumptions about

system administration? To explore this, we examined aticket queue from a live site (Tufts ECE/CS). Note thatno one knew, until very recently, that anyone wasgoing to analyze this ticket data. It is thus free ofmany sampling biases. It is, however, difficult todetermine exactly when many tickets were closed.This is because there is no site requirement to closetickets promptly, and many tickets are closed by stu-dent staff who monitor the ticket queue, sometimeslong after the problem has been addressed.

Plotting ticket numbers (an increasing sequence)against time (Figure 3) shows little or no evocative



patterns. Each ticket is plotted as a horizontal line,with beginning and end representing creation andcompletion time. The Y axis represents ticket number;tickets due to spam have been deleted and the result-ing queue renumbered as increasing integers with nogaps. Note particularly that several tickets are stillopen after several months.

0

20

40

60

80

100

120

140

160

180

200

00:00 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 00:00

Num

ber

of ti

cket

s

Time

Number of tickets created

Figure 5: Ticket arrivals exhibit sinusoidal rate variation over 24 hours.

0

50

100

150

200

250

00:00 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 00:00

Num

ber

of ti

cket

s

Time

Number of tickets resolved

Figure 6: Ticket closures exhibit a sinusoidal time distribution with a hotspot at 15:00-16:00.

We discovered very quickly that there were twoclasses of service: one for short-duration requests andanother for long-duration requests. Viewed alone, therequests that took less than a month exhibit relativelyconsistent response times (Figure 4).

Request arrivals are not Poisson. For arrivals toexhibit a Poisson distribution, the mean of inter-arrivaltimes must be equal to their standard deviation. In thiscase, the overall standard deviation of inter-arrivaltimes (9580 seconds or ≈ 2.65 hours) is about 1.37

times the mean (6971 seconds or ≈ 1.94 hours), indi-cating that there are periods of inactivity. Lookingdeeper, Figure 5 shows one problem: arrival rates arenot constant, but instead sinusoidal over a 24-hourperiod. In this graph, ticket arrivals are shown by hour,summed over the lifetime of the Request Tracker (RT)database. The times are corrected for daylight savingstime, and show more intense traffic 9 am to 5 pm witha small dip in traffic at lunch. Ticket closures show adifferent pattern (Figure 6) with a hotspot at 3 pm thatdefies explanation, until one realizes that a studentadministrator charged with monitoring and closingtickets starts work at that time!

Measured ‘‘time in system’’ does not seem to beexponential, either. If, however, one omits requestswith time in system greater than one month, theremaining requests exhibit a distribution that looks



similar to exponential (Figure 7). The figure contains ahistogram of the number of requests processed in eachnumber of days. Note, however, that this figure repre-sents service time plus time spent waiting in queue, soit cannot be used to compute an accurate service rate.

0

200

400

600

800

1000

1200

0 5 10 15 20 25 30 35 40

Num

ber

of ti

cket

s

Days

Number of tickets resolved within period

Figure 7: A histogram of the frequency of tickets resolved in each number of days has a near-exponential shape.

From the data, we see that requests are multiclasswith at least two distinct classes of requests:

1. A vast majority of requests are resolved quickly(in less than one month, with a mean time insystem of about 3.6 days). Arrival times forthese requests seem to be governed by a sinu-soidal non-stationary Poisson process, i.e.,arrival rates seem to vary between a daily highand low on a sine-wave pattern.

2. A small number of requests have an indetermi-nate and long time in system. Arrival times forthese requests show no discernible structure(perhaps due to lack of enough examples).

3. The average rate of ticket arrival is graduallyincreasing over time. In our case, this seems tobe partly due to new faculty hires.

This data also exhibits, however, the main diffi-culties of extracting performance statistics from ticketqueues:

1. Service times are recorded inaccurately becausethere is no particular advantage to recordingthem accurately. Most tickets are closed late,because it is not the job of the administratoranswering the ticket to close it, but just to solvethe problem. In our case, many tickets areclosed by student staff some time after theproblem has been solved.

2. The class of a particular request is not alwayseasily discernible. It is easier to group requestsby time of service rather than class of problem.In our case, there is a clear distinction betweenrequests for which an appropriate response is

documented, and those for which an appropri-ate response is unknown. The former take onaverage the same time to resolve, while the lat-ter vary widely.

3. Emergent patterns in the data are only obviousif one is very careful about correcting for envi-ronmental issues. For example, data did notexhibit a sinusoidal arrival rate until it was cor-rected for daylight savings time (DST)!

4. Ticket data does not indicate the severity of aproblem. There are no discernible ‘‘flurries’’ or‘‘bursts’’ of data for severe problems; oftenonly one or two people bother to report a majoroutage.

Other practitioners have mentioned that there areseveral ways that request queue data can be biased byoperating policy.

1. If people are rewarded for closing ticketsquickly, they tend to close them early, before anactual resolution.

2. If people are rewarded for only the tickets theychoose to resolve, obviously difficult ticketswill be avoided.

The final issue plaguing the use of real requestqueue data is privacy. Real request data contains flawsin practice. For example, some requests for whichthere should be documented scripts remain undocu-mented, some requests are forgotten, and somerequests can take an embarrassing amount of time toresolve. For this reason, it is difficult for researchers toget real data on the nature of requests and their servicetimes, for sites other than their own.

One lesson learned from our data is the power ofgood documentation. If an appropriate response to aproblem is documented or otherwise well known,there seems to be no significant difference in responsetime invariant of the nature of the problem. It is



surprising that to a first approximation, differences inservice times for subclasses of requests do not emergefrom the data. One possible reason for this effect isthat in these cases, communication time with clientsmay dominate the time needed to solve the problemonce it is clearly defined.

Conversely, problems with no documentedresponse wait longer and may never be solved. At ourhelpdesk, student staff solve routine problems anddefer only problems with no documented solution tosecond-level triage staff. Since the second-level staffare often planning or deploying new architectures,requests without documented solutions await theirattention and compete with deployment tasks. Ofcourse, once solved and documented, such a requestbecomes quickly solvable.

In our data, a simple pattern emerges. Systemadministration is composed of a combination of rou-tine tasks and complex troubleshooting and discoverythat borders upon science. Our site’s practice is per-haps best described as a two-class queueing system,with a large number of routine requests with docu-mented and/or known solutions, and a smaller numberof requests requiring real research and perhaps devel-opment. For the most part, the routine requests areaccomplished by system administrators acting inde-pendently, while more complex requests may requirecollaboration between system administrators and takea longer, relatively unpredictable time to complete.

A Simple Model Of Cost

Given the above model of system administrationas a queueing system, we construct a coarse overallmodel of cost, based upon the work of Patterson [18]with some clarifications.

First, cost can be expressed as a sum of two com-ponents: the ‘‘cost of operations’’ and the ‘‘cost ofwaiting for changes.’’ The ‘‘cost of operations’’ con-tains all of the typical components of what we nor-mally consider to be cost: salaries, benefits, contracts,and capital expenditures such as equipment acquisi-tion. For most sites, this is a relatively predictable costthat remains constant over relatively long time peri-ods, e.g., a quarter or a year. The ‘‘cost of waiting’’ isa generalization of Patterson’s ‘‘cost of downtime’’,that includes the cost of waiting for changes as well asthe cost of waiting for outages to be corrected.

While the cost of downtime can be directly cal-culated in terms of work lost and revenue lost, the costof waiting for a change cannot be quantified so easily.First we assume that R represents the set of requests tobe satisfied. Each request r ∈ R has a cost Cr and thetotal cost of waiting is

Cw =r∈RΣ Cr . (3)

We assume that for a request r (corresponding to eitheran outage or a desired change in operations), there is acost function cr(t) that determines the instantaneous

cost of not accomplishing the change, and times tr1 andtr2 at which the change was requested and accom-plished. Then the tangible cost of waiting is the integral(running sum) of cr(t) over the waiting period:

Cr =

tr2

tr1

∫ cr(t)dt . (4)

If as well cr(t) is a constantCr = (tr2 − tr1)cr = trcr (5)

as in Patterson’s paper. In general, this may not betrue, e.g., if the change reflects a competitive advan-tage and the effects of competition become moresevere over time. For example, in the case of securityvulnerabilities, vulnerability is known to increase overtime as hackers gain access to exploits.

System administrators control very little of theprocess that leads to lifecycle cost, but the part theycontrol – how they work and accomplish tasks – canpartly determine the cost of waiting. In this paper, weconsider the effects of practice upon the cost of wait-ing in a situation in which the budget of operations isheld constant over some period, e.g., a quarter or ayear. Situations in which cost of operations can vary(e.g., by hiring, layoffs, or outsourcing) are left forlater work.

The cost function cr(t) must model both tangible(work lost) and intangible (contingency) factors. Forrequests concerning downtime, the cost of waitingmay be directly proportional to work and revenue lost,while for requests involving enhancements rather thandowntime, work lost and revenue lost can be more dif-ficult to quantify. Also, the costs of waiting forenhancements vary greatly from site to site. For busi-ness sites, delays often incur real revenue loss, whilefor academic sites, the effects of delays are moreintangible, resulting in missed grant deadlines, studentattrition, and other ‘‘opportunities lost’’. In the lattercase, it is better to model cost as risk of potential lossrather than as tangible loss.

We can best cope with uncertainty and risk bycomputing the expected value of each potential risk.Contingencies are like requests; they arrive during aperiod of vulnerability with a specific rate dependingupon the severity of the vulnerability; these arrivalsare often Poisson. The total expected value of an out-age or wait is the sum of expected incident costs,taken over the various kinds of incidents. If incidentsarrive with a constant Poisson rate λ, the expectedincident cost is the number of expected incidents timesthe cost of an incident. This is in turn a product of therate of arrival for the incident, the elapsed time, andthe average cost per incident. Note that the word‘‘incident’’ applies not only to security problems, butalso to lost opportunities such as students dropping outof school, employees quitting, etc.

Thus we can think of the cost function cr(t) for aparticular request r as



cr(t) = crm(t) + cri(t) (6)where crm(t) represents tangible losses and cri(t) repre-sents intangible losses. While crm(t) represents workand revenue losses and is proportional to the scale ofthe outage, cri represents possible losses due to ran-dom events. If contingencies are elements d of a set Drof all possible contingencies that can occur duringrequest r, and contingencies in Dr are statisticallyindependent, then the cost cri for all of them is the sumof their individual costs

cri(t) =d∈Dr

Σ crid(t) (7)

where crid is the contingency cost for d∈Dr whilewaiting for r. If contingencies d∈Dr have Poissoninter-arrival times λd, then

crid = λdCd (8)where Cd is the average cost per incident for d. Thus

cr(t) = crm(t) +d∈Dr

Σ λdCd. (9)

If crm, λd, and Cd are constants, thenαr = crm + (

d∈Dr

Σ λdCd) (10)

is also a constant, and

Cr =

tr2

tr1

∫ cr(t)dt = αrtr . (11)

Note that there are a lot of if’s in the above justi-fication and the reader should be warned that assump-tions abound here. The formula for cost of waitingsimplifies easily only if particular assumptions hold.As we make these assumptions, our model loses accu-racy and expressiveness. With all assumptions inplace, we have Patterson’s model; as he states, it is anoversimplification.

If requests can be divided into classes k∈K, eachwith a different proportionality constant αk, then thetotal cost of processing a set of requests is the totaltime spent waiting for each class, times the propor-tionality constant for that class. Thus, in the simplestcase, the total cost of waiting is

Cw =k∈KΣ

r∈KΣ αktr (12)

orCw =

k∈KΣ αk

r∈KΣ tr. (13)

Thus the contribution of each class k is proportional tothe total time spent waiting for events of that class.

In this approximation we make many simplifyingassumptions:

1. Contingencies arrive with Poisson rates.2. Contingencies are statistically independent of

one another.3. The effect of a contingency does not change

over time.

These are limits on how we formulate a problem;we must not allow dependencies to creep into our clas-sifications. Part of this formulation is to think of burstsof events as single events with longer service times.For example, it is incorrect to characterize ‘‘bursty’’

contingencies such as virus attacks as host events;these events are not independent of one another. How-ever, the event in which the virus entered the system isnot bursty, independent of all other like events, andthus can be treated as one contingency. Likewise, spamfrom a particular site is not an independent event foreach recipient, though spam from a particular source isoften independent of spam from other sources.

The main conclusion that we make from theseobservations is that

The intangible cost of waiting for a request is, toa first approximation, proportional to time spentwaiting (though the proportionality constant mayvary by request or request class).

While some constants remain unknown, the values forsome proportionality constants are relatively obvious.If n users are affected by an outage, then the tangiblecost of downtime is usually approximately propor-tional to n. Likewise the rate of incidents that involveone member of the population (such as attrition) isusually approximately proportional to the number ofpeople involved (due to independence of people asfree agents).

Estimating Service Rates

In the above sections, we show a linear relation-ship between the cost of waiting and amount of timespent waiting, and show that the amount of time spentwaiting depends upon arrival rate and service rate fortasks. In our observation of real systems, arrival ratewas relatively easy to determine. To determine thecost, however, we must also know the service ratewith which requests are completed. We cannot mea-sure this parameter directly; we can only measure thewaiting time that results from it. How do we estimatethe service rate itself? To answer this question, weborrow from a broad body of work on complexity esti-mation in software engineering [19].

Cost modeling in software engineering concernsthe cost of maintaining a large piece of software (suchas a program or configuration script). The basic strat-egy is to measure the complexity of the final product insome way, and then predict from that complexity howmuch it will cost to craft and maintain the program.

Complexity metrics that can aid in determiningcost of a software engineering project include both‘‘white-box’’ and ‘‘black-box’’ methods. A ‘‘black-box’’ method looks at the complexity of requirements,while a ‘‘white-box’’ method looks at the complexityof a potential finished product. The goal of either kindof analysis is to produce a service rate that can be uti-lized for later analysis. To map this to system adminis-tration, a ‘‘white box’’ method would base cost esti-mates on the structure of practice, while a ‘‘blackbox’’ approach would base cost estimates upon thestructure of the problem.



White-box MethodsIn software engineering, white-box software

metrics include:1. Lines of code (LOC): the complexity of a soft-

ware product is proportional to its length inlines of code.

2. cyclomatic complexity [16]: the complexity ofa piece of software is proportional to the num-ber of ‘‘if ’’ statements in the code.

Procedure A

Procedure C

Procedure B

decision

decision

Procedure D

Procedure E

Procedure F

start

finish

Figure 8: An example troubleshooting flowchart.

A

B C

D E

FFigure 9: The flow graph corresponding to Figure 8.

It is generally agreed that cyclomatic complexityis a much better measure of complexity than LOC, fora variety of reasons, including variations in the expres-siveness of programming languages; long programs inone language can be short in another. The key is tofind something about the program that is more closelyrelated to its cost than its length. For programs, thenumber of branches contributes to the difficulty ofdebugging or maintaining the program. The key towhite-box analysis of system administration is to findan analogue to the branches for programs.

Whitebox analysis of programs inspires a similarform of analysis for system administration procedures.While white-box analysis of programs starts withpseudo-code, white-box analysis of practice starts witha recipe or instructions to service a particular kind ofrequest. If we treat each recipe as a ‘‘program’’, with‘‘branches’’ at particular steps, then we can computethe average time taken for the recipe by keeping statis-tics on the number of times that individual steps areutilized in practice. This provides a way to come upwith estimated rates for a procedure, given estimatesfor subparts of the procedure.

Note that white-box analysis of a recipe for sys-tem administration is quite different than white-boxanalysis of a program. In the case of the program, thewhite-box measurement of complexity does notdepend upon the input to the program. In systemadministration, the performance of a proceduredepends upon the nature of the environment. A white-box estimate of the time needed to service a request is



a measure of both the complexity of the procedure andthe complexity of the environment.

A

B C

D D

F F F

E

Figure 10: The flow tree corresponding to Figure 9.

A

B C

D D

F F F

E

mF1 mF2 mF3

mD1,1.0 mD2,1.0 mE,1.0

mB,1.0 mC1,pC1 mC2,pC2

mA1,pA1 mA2,pA2

Figure 11: An annotated flow tree tracks statisticsthat can be used to compute average completionrate.

One way of performing white-box analysis startswith an (acyclic) troubleshooting chart for a proce-dure. We start with a a troubleshooting chart (Figure8) that describes procedures to perform and branchesto take after each procedure. We convert this to a flowgraph (Figure 9) by representing only decision nodes.Since a typical troubleshooting chart has no loops, weconvert this graph into a flow tree by duplicatingnodes with two or more incoming edges (Figure 10).We then annotate branches in that tree with statisticsto be collected or estimated about the branch (Figure11). These statistics allow us to compute the mean ser-vice rate for the whole tree.

The key to the computation is that given that weknow a service rate for the subtrees of a node in thetree, we can compute the service rate for the nodeitself. The nature of the computation is illustrated in

Figure 12. Suppose we know the service rate mB forsubtree B and mC for subtree C. Suppose that we wantto compute the service rate mA for A, and know foreach branch out of A, the service rate for A given thatit takes the branch (mA1,mA2) and the probabilitywith which that branch is taken (pA1,pA2). If we takethe branch from A to B, and A has service rate mA1,then the average service time for the branch is1/mA1 + 1/mB. If we take the branch from A to C, theaverage service time for the branch is 1/mA2 + 1/mC.If we take the branch to B with probability pA1, andthe branch to C with probability pA2, then the averageservice time for both is the expected value

pA1(1

mA1+

1

mB) + pA2(

1

mA2+

1

mC) . (14)

Thus the average rate is the reciprocal of this.

mA1,pA1 mA2,pA2

C:mCB:mB

A:mA

Figure 12: Computing average completion rate for aflow tree.

To enable this computation, each edge in the pro-gram graph is labeled with two quantities: the meanservice rate for the predecessor of the edge, given thatthis branch is chosen, as well as the probability thatthis branch is chosen. We can either measure thesedirectly or estimate them by some method. One obvi-ous way to measure both rates and probabilities is toperform the procedure many times and record thenumber of times each node is visited, the average ser-vice time before taking each branch, and the numberof times each branch is taken. Then the ratio of thetimes the branch is taken, divided by the times its par-ent is visited, is the sample probability that the branchwill be taken.

In this abstraction there hides an astonishing fact:the order in which decisions are made strongly affectsthe time-in-system for such a graph. While the ratesare properties of the administrative process, the proba-bilities of branching are properties of the environment.Further, these probabilities are not conditional in theBayesian sense; they are temporo-conditional in thatthey depend upon the previous occurrence of a spe-cific procedure. In Figure 11, the probability of goingto B from A is not the conditional probability P(B|A),but the probability of B after A: the probability that we



choose B given that A has already been completed.Bayesian identities do not hold; any change in a proce-dure affects the sample probabilities of all branchesafter it in the script.

One way to estimate branch probabilities in thismodel is that certain subtasks depend upon hetero-geneity that is routinely tracked. For example, onestep might be to determine whether the affected hostruns Solaris or Linux. In this case, the sample proba-bilities for the branches are both known in advancefrom inventory data. In the same way, one can esti-mate some branch probabilities from overall statisticson the sources of trouble within the network.

Black-box Methods

White-box methods depend upon the fact that thenature of practice is already known, i.e., we know thesteps that people will take to accomplish tasks. In sys-tem administration, as in software, we are often askedto estimate the cost of a process without knowing thesteps in advance. To do this, we must use a methodthat estimates cost based upon the complexity of theoutcome rather than the complexity of the process.

Black-box methods for measuring software com-plexity include COCOMO [2, 3, 4]: the complexity ofsoftware depends upon an overall characterization ofthe software’s character and mission. COCOMOdepends upon use of one of two estimations of codecomplexity:

1. ‘‘object points’’ [3, 4]: the complexity of apiece of software is proportional to the com-plexity of the objects it must manipulate.

2. ‘‘function points’’: the complexity of a piece ofsoftware is proportional to the number of func-tions that it must perform.

The key idea in COCOMO is that there is a rela-tionship between the cost of maintaining a programand the complexity of its interactions with the outsideworld, though we may not know the exact nature ofthat relationship in advance. COCOMO is ‘‘tuned’’for a specific site by correlating object or functionpoints with actual costs of prior projects. COCOMO issite-specific; the relationship between complexity andcost varies from site to site. By computing a ratio esti-mating the relationship between requirements andcapabilities, one estimates the time that will be takento complete requirements.

We can apply the idea of COCOMO to systemadministration in a very direct way. While the softwaremodel for function points considers open files, the anal-ogous concept of function points for a network servicewould describe that service’s dependencies and interre-lationships with others. We know that the number ofdependencies within a service influences the cost ofmaintaining it; we do not know the exact relationship.

For example, we might compute the functionpoints for an apache web server by assigning a number

of points to its relationship with each of the followingsubsystems: the network (DHCP, DNS, Routing), thefilesystem (protections, mounts), and the operating sys-tem (users and groups). In a function point model, eachof these attributes is assigned a ‘‘weight’’ estimatinghow difficult it is for a system administrator to dealwith that aspect of configuration and management. Thesum of the weights is then an estimator of ‘‘how com-plex’’ the service will be to manage.

The main difficulty with this approach is the num-ber of potential weights one must estimate; virtuallyevery subsystem mentioned in the system administra-tor ’s book of knowledge [11, 13] has to be assigned aweight. Further, these weights are not universal; theyvary from site to site, though it is possible that similarsites can use similar weights. For example, weightsassigned to subsystems vary greatly with the level ofautomation with which the subsystem is managed.

The cost of providing any service depends notonly upon the complexity of the service, but also uponthe capabilities of the staff. Our next step in defining afunction point estimate of the complexity of systemadministration is to derive a capability summary of theadministrative staff and site in terms of service rate.Obviously, a more experienced staff deals with tasksmore effectively than a less experienced one. Capabili-ties in the model might include end-user support, ser-vice support, architecture, etc. If each staff member isassessed and the appropriate attributes checked, and asum is made of the results, one has a (rough) estimateof capabilities of one’s staff. This has similarities tothe SAGE levels of system administrator expertisedefined in the SAGE booklet on job descriptions [9].

The last step in defining a function point esti-mate of the complexity of system administration is toassess the capabilities maturity of the site itself. Onemight categorize the site into one of the followingmaturity categories [14]:

1. ad-hoc: everything is done by hand.2. documented: everything is documented, no au-

tomation.3. automated: one can rebuild clients instantly.4. federated: optimal use is made of network ser-

vices.

Again, each one of these has a weight in deter-mining the overall capabilities. The sum of admin-istrator capabilities and site capabilities is an estimateof overall ‘‘capability points’’ for the site.

It can then be argued that the complexity ofadministering a specific subsystem can be estimatedby a fraction

service rate =estimated service points

estimated capability points(15)

where service points and capability points are sums ofweighted data as described above. If the weights forcapability points are rates in terms of (e.g.) service-points per hour, then the complexity is the averageresponse time in hours to a request.



The overwhelming problem in tuning COCOMOfor system administration is that tuning the modelrequires detailed data on actual measured rates. The tun-ing process requires regression to determine weights foreach of the complexity and quality factors. This is accom-plished by studying a training set of projects with knownoutcomes and properties. To use COCOMO-like systems,we must be able to gather more accurate data on the rela-tive weights of subsystems than is available at present.

0

100

200

300

400

500

600

700

800

900

0 100 200 300 400 500 600 700 800

Cum

ulat

ive

time

spen

t wai

ting

Elapsed time

two administratorsthree administratorsfour administrators

Figure 13: Diminishing returns when adding administrators to a queue.

Some Experiments

So far, we have seen that we can estimate thecost of system administration via one of two models.‘‘Black box’’ methods require that we assess the timeimpact of the complexities of the problem beingsolved, while ‘‘white box’’ methods require that weestimate the time taken for a detailed set of tasks. Ofthese methods, ‘‘black box’’ methods give us informa-tion more quickly, but these methods require that we‘‘score’’ facets of the problem being solved as harderor easier than others. These scores must be developedvia practice, but we can learn something about the rel-ative complexity of black-box attributes via simula-tion. By simulating practice, we can account for realis-tic human behaviors that cannot be analyzed viaknown queueing theory. We can also observe how realsystems can potentially react to changes in a less idealway than ideal queueing models suggest. Particularly,we can study behavior of queueing systems ‘‘on theedge’’; almost out of control but still achieving asteady state. In our view, this situation describes moreIT organizations than it should.The Simulator

The simulator, written in C++, is basically anM/M/c queue simulator with the ability to simulatenon-ideal (‘‘non-product’’) behaviors. It assumes that

we have c identical system administrators working24x7 and generates a variety of requests to whichthese ideal administrators must respond. One can varythe behavior of the system administrator and therequest queue and measure the results. The input tothe simulator is a set of classes of tasks, along witharrival and service rates for each task. The output isthe time spent waiting for requests (by all users), bothper time-unit of the simulation and overall. We assumefor this simulator that the cost of waiting is a constant;a unit wait by a single individual results in some con-stant intangible cost. These simulator assumptions arevery much less complex than real practice, but one canmake some interesting conclusions from even so sim-ple a model.

Diminishing ReturnsOur first simulation exercise is to study the

effects of adding system administrators to a group ser-vicing simple requests. We set up a simple multi-classsystem with a varying number of system administra-tors all of whom have identical average response rates.There are four request classes, corresponding torequests whose service time is an average of 1, 3, 8,and 24 hours, respectively. The service rate of eachrequest class is twice its arrival rate, creating a balancebetween arrivals and service. We ran the exact samesimulation for two, three, and four system administra-tors. The cumulative time spent waiting for service isshown in Figure 13. There is clearly a law of dimin-ishing returns; the change in wait time from three tofour system administrators does not significantlychange the time spent waiting for service.

SaturationRealistic system administration organizations can

be faced with request volume that is impossible to



resolve in the time available. We know from classicalqueueing theory that an M/M/c queuing systemexhibits predictable response time only if λ/cµ < 1,where λ is the arrival rate, c is the number of adminis-trators, and µ is the service rate per administrator. Inother words, there has to be enough labor to goaround; otherwise tickets come in faster than they canbe resolved, the queue grows, and delays becomelonger and longer as time passes.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

0 100 200 300 400 500 600 700 800

Cum

ulat

ive

time

spen

t wai

ting

Elapsed time

one administratortwo administrators

three administratorsfour administrators

Figure 14: One administrator performs very poorly compared to two, three, and four.

0

50

100

150

200

250

0 100 200 300 400 500 600 700 800

Incr

emen

tal t

ime

spen

t wai

ting

Elapsed time



Figure 15: Incremental data for Figure 14 shows that utilizing one administrator leads to chaotic wait times.

Figure 14 shows the same simulation as before,but adds the case of one administrator. This seems likean unbalanced situation in which request rate isgreater than service rate, but looking at waiting timeper unit time (Figure 15) we see that waiting time isnot always increasing with time. So although oneadministrator is very much slower than two or three,the situation is not completely out of control. Note,

however, that the situation of the single administratoris very sensitive to load; he is ‘‘on the brink ofdestruction.’’ Small changes in load can cause largevariations in response time, and the cost of administra-tion due to waiting is ‘‘chaotic’’, especially when arequest when a long service time enters the queue.Nevertheless, on average, the time spent waitingvaries directly with elapsed time of the simulation.

Figure 16 shows incremental waiting time for atruly ‘‘saturated’’ system in which there is no way foradministrators to ever catch up. We saturate the queuein the previous example by multiplying the arrivalrates for all requests by four. In this case, one and twoadministrators are in trouble; queue length is growinglinearly with time along with wait time. Figure 17shows the cumulative time for this example. In a satu-rated queueing system, since time spent waiting


To w a r d a Cost Model for System Administration Couch, Wu, and Susanto

increases linearly with elapsed time, the cumulativetime spent waiting varies as the square of elapsed time.

BrinksmanshipWe consider it a fair statement that many IT

organizations run with λ/cµ quite close to 1. It is thusno surprise that it is very difficult for these organiza-tions to cope with changes that might increase loadupon system administrators, even for a short time.

0

1000

2000

3000

4000

5000

6000

0 100 200 300 400 500 600 700 800

Incr

emen

tal t

ime

spen

t wai

ting

Elapsed time



Figure 16: Multiplying the arrival rate by four overloads one or two system administrators.

0

50000

100000

150000

200000

250000

300000

0 100 200 300 400 500 600 700 800

Cum

ulat

ive

time

spen

t wai

ting

Elapsed time



Figure 17: Cumulative wait time for overloaded system administrators varies as the square of elapsed time.

There is a solution, but it is counter-intuitive. Figure18 shows the effect of a ‘‘catastrophic’’ flurry ofrequests arriving in a near-saturated system. For ashort while, wait times go way up, because the systemis already almost saturated and the new requests pushit over the limit. The key is to distribute the samerequests over a long time period (Figure 19), to avoidpushing the system over the limit and save waitingtime. Note that in both figures, one administrator

alone simply cannot handle the load and chaotic waitsoccur in both cases.

Lessons Learned

Human systems self-organize around maximumefficiency for the task at hand, but not necessarily forfuture tasks. As such, we as system administrators areoften near the ‘‘saturation point’’ in our practice. As inFigure 18, small increases in load can lead to cata-strophic delays. But the strategies in Figure 19 can help.

One part of coping with being understaffed is toutilize automation to lessen workload, but that can leadto queue saturation in an unexpected way. The quandaryof automation is that when something goes wrong, it isnot one host that is affected, but potentially hundreds. Ifan automation mistake affects hundreds of nodes, weoften have the situation in Figure 18; there are hundreds


Couch, Wu, and Susanto To w a r d a Cost Model for System Administration

of questions to answer and the queue is already satu-rated. These questions can be as simple as educatingusers about a different command for invoking software;it takes little perturbation to saturate a queue that isalready nearly saturated. The worst possible case is thatautomation uncovers latent pre-existing conditions thatcause failures. In this case, troubleshooting may take alarge amount of time while the queue fills.

0

100

200

300

400

500

600

700

0 100 200 300 400 500 600 700 800

Incr

emen

tal t

ime

spen

t wai

ting

Elapsed time



Figure 18: A flurry of 100 requests causes a

0

100

200

300

400

500

600

700

0 100 200 300 400 500 600 700 800

Incr

emen

tal t

ime

spen

t wai

ting

Elapsed time



Figure 19: Distributing the 100 requests in Figure 18 over a longer time interval improves wait times except for anadministrator working alone.

The main lesson of this paper is that stageddeployment is often better than large-scale automateddeployment, when system administrators are near satu-ration. It is often far better to control the request queueby upgrading a small number of users at a time, ratherthan risk a flood of potentially unmanageable requestsdue to a massive upgrade. If a massive upgrade isrequired, extra staff are needed to handle the loadthrough the upgrade period. It is no shame to ask for

help when the alternative is that your organization losesmuch more money than it would spend as a result ofhiring help.

Open Questions

Obviously, this paper is a very small step towardunderstanding the effects of practice upon cost. Simu-lations are no replacement for real measurements, andreal measurements remain impractical. We end thisstudy with more questions than when we started.

First, there are at least a hundred factors affectingthe practice that we are not simulating. Some haveperipheral effects, such as human learning; we were sur-prised at how little an effect it has when running simplesimulations. Others have major effects, such as peermentoring, user conditioning, and error propagation.Models of user behavior (such as those described in


To w a r d a Cost Model for System Administration Couch, Wu, and Susanto

[12]) have not been incorporated here, nor have weincorporated behavioral aspects such as conditioning topre-existing circumstances. For example, it is common,in the presence of poor response time, for users to stopmaking requests and go elsewhere for help. Likewise,events and incidents are often not independent of oneanother; requests can be bursty or sparse. Like Patter-son’s paper, this one also oversimplifies a complexproblem, giving slightly more sophisticated methodsthan the ‘‘back of an envelope’’ to estimate the results ofvery complex processes.

Second, we should not forget the wealth of workon queueing systems upon which we can draw. Refer-ence [10] analyzes the properties of sinusoidal arrivals– like the ones we observed – and gives a method forcomputing the number of servers that are needed toachieve best-case performance. Can these methods beused to analyze system administration? The burningquestion is what can we afford to idealize and for whatmust we account by simulating realistic practice. Sim-ulations are particularly difficult to use for asking‘‘what-if ’’ questions because realistic answers requirethe results of thousands of runs to be averaged.

Integrating Measurement with PracticeOne of the largest blockades against understand-

ing the cost of practice is that the activity of cost mea-surement is separate from that of practice. Can weintegrate this with practice? Could we develop toolsthat – by their use – provided input data to a costmodel? This approach seems to have some promise.

Imagine a tool that – when you utilize it – keepsrecords on how long the task takes. Imagine this data-base being used to populate a function point model, sothat the complexity of specific tasks can be accuratelypredicted. If done correctly, this would make the costanalysis intrinsic, transparent, and completely invisi-ble to the practitioner. It should neither limit nor delaythe practitioner, but should keep track of realistic timeestimates for specific tasks.

One idea is that of a ‘‘smart troubleshootingguide’’ that keeps records on how long was spent oneach procedure. While the administrator was follow-ing a procedure, this guide would record time spent oneach page and in each procedure, for the purpose ofrecording how long, on average, each procedure takes.

Of course, the large question here is not that ofefficiency or transparency but that of privacy. Anymechanism that measures our practice also keeps datathat we might not want to be stored, such as individualperformance records. As well, the potential exists forthis data to be misused in ways that damage the pro-fession; e.g., punishing administrators who are‘‘slower ’’ but consistently make fewer errors.

Conclusions

No simulator or model is a perfect substitute forreality. In this paper, we have studied some of the

easiest ways to obtain predictions about the cost ofsystem administration, when it is considered to be asum of infrastructure cost and an indirect cost propor-tional to time spent waiting. They are of course notparticularly accurate, but are they accurate enough touse in making intelligent decisions?

One lesson to take from software engineering isthat often an educated guess is better than no informa-tion at all. Even if we get within an order of magni-tude of estimating the cost of managing a particularservice, we know more than when we started, and cantune that figure by observing practice. The first step isto get real data.

And this process cannot wait. At this time, theprofession is ‘‘under siege’’ from those who wouldeliminate system administration as a profession. Thegrand promise of autonomic computing, however, isnot easy to obtain, and the cost justifications of thetechnology often do not include an analysis of the costof troubleshooting when things go wrong. By under-standing the cost of our practice, we can betterrespond to arguments claiming superiority of auto-nomic computing methods, and be able to realisticallycompare human-centered and machine-centered stylesof system administration.

This is a very small step in a new direction. If ithas sensitized practitioners to the idea that waitingtime matters, it has accomplished its goal. The bestmodels and measurements for analyzing indirect costsare yet to be discovered. But if the reader is – likemany of us – near saturation, one can modify one’spractice to ease the pain, not by applying automationblindly, but by strategically planning changes so thatrequests do not become overwhelming. This is the firststep toward a practice in which queues never saturateand IT managers understand the difference betweenunder-utilization and required capacity in systemadministration organizations. System administratorsare like insurance; a properly functioning organizationdoes not have them all busy all of the time, except dur-ing contingency periods.

Acknowledgements

As usual, this work was a product of collabora-tion among a broader community of learning than theauthors. Paul Anderson, Mark Burgess, and many oth-ers nudged the authors toward developing their modelmore carefully. The systems staff at Tufts UniversityECE/CS graciously donated their time and ticketqueue for study; John Orthoefer was particularly help-ful in helping us obtain data. Marc Chiarini workedlong hours to offload the authors from other tasks dur-ing writing and to proofread the document.

Author Biographies

Alva L. Couch was born in Winston-Salem,North Carolina where he attended the North Carolina



School of the Arts as a high school major in bassoonand contrabassoon performance. He received an S.B.in Architecture from M. I. T. in 1978, after which heworked for four years as a systems analyst and admin-istrator at Harvard Medical School. Returning toschool, he received an M.S. in Mathematics fromTufts in 1987, and a Ph.D. in Mathematics from Tuftsin 1988. He became a member of the faculty of TuftsDepartment of Computer Science in the fall of 1988,and is currently an Associate Professor of ComputerScience at Tufts. Prof. Couch is the author of severalsoftware systems for visualization and system admin-istration, including Seecube (1987), Seeplex (1990),Slink (1996), Distr (1997), and Babble (2000). He canbe reached by surface mail at the Department of Com-puter Science, 161 College Avenue, Tufts University,Medford, MA 02155. He can be reached via electronicmail as [email protected].

Ning Wu is pursuing his Ph.D. at Tufts Univer-sity. His research interests are in system management,wireless ad-hoc networking, security, and P2P sys-tems. Before studying at Tufts, he had worked as anengineer for Genuity and Level 3 CommunicationsInc. He received an M.S. from State University ofNew York at Albany, an M.E. from East China Insti-tute of Computer Technology, and a B.S. from South-east University in China. Ning can be reached viaemail at [email protected] .

Hengky Susanto is a computer science Ph.D. stu-dent at Tufts University. His research interests are inAutonomic computing, System Management, and Net-working Area. He also worked as a software engineerat EMC and StorageNetworks Inc prior to returning toschool. He received a B.S from University of Mas-sachusetts at Amherst and a M.S from University ofMassachusetts at Lowell, both in computer science.Hengky can be reached at [email protected] .

References

[1] Apthorpe, R., ‘‘A Probabilistic Approach to Esti-mating Computer System Reliability,’’ Proc.LISA 2001, USENIX Assoc., 2001.

[2] Boehm, Barry, ‘‘Anchoring the SoftwareProcess,’’ Barry Boehm, IEEE Software, July,1996.

[3] Boehm, Barry, Bradford Clark, Ellis Horowitz,Ray Madachy, Richard Shelby, and Chris West-land, ‘‘Cost Models for Future Software LifeCycle Processes: COCOMO 2.0,’’ Annals ofSoftware Engineering, 1995.

[4] Boehm, Barry, Bradford Clark, Ellis Horowitz,Ray Madachy, Richard Shelby, and Chris West-land, ‘‘COCOMO 2.0 Software Cost EstimationModel,’’ International Society of ParametricAnalysts, May, 1995.

[5] Burgess, Mark, ‘‘Theoretical System Administra-tion,’’ Proc. LISA 2000, USENIX Assoc., 2000.

[6] Burgess, Mark, Analytical Network and SystemAdministration: Managing Human-ComputerSystems, Wiley and Sons, 2004.

[7] Couch, Alva and Paul Anderson, ‘‘What is thisthing called system configuration,’’ an invitedtalk to LISA-2004, USENIX Assoc., 2004.

[8] Cowan, et al., ‘‘Timing the application of secu-rity patches for optimal uptime,’’ Proc. LISA2002, USENIX Assoc., 2002.

[9] Darmohray, T., Ed, Job Descriptions for SystemAdministrators, Revised Edition, SAGE ShortTopics in System Administration, USENIXAssoc.

[10] Eick, S. G., W. Massey, and W. Whitt, ‘‘Mt/G/∞queues with sinusoidal arrival rate,’’ Manage-ment Science 39, Num. 2, 1993.

[11] Halprin, G., et al., ‘‘SA-BOK (The SystemsAdministration Body of Knowledge),’’ http://www.sysadmin.com.au/sa-bok.html .

[12] Haugerud, Harek and Sigmund Straumsnes,‘‘Simulation of User-Driven Computer Behav-iour,’’ Proc. LISA 2001, USENIX Assoc., 2001.

[13] Kolstad, R. et al., ‘‘The Sysadmin Book of Knowl-edge Gateway,’’ http://ace.delos.com/taxongate .

[14] Kubicki, C., ‘‘The System Administration Matu-rity Model – SAMM,’’ Proc. LISA 1993, UsenixAssoc., 1993.

[15] Maekawa, M., A. Oldehoeft, and R. Oldehoeft,Operating Systems: Advanced Concepts, Ben-jamin/Cummings, 1987.

[16] McCabe, T. J., and C. W. Butler, ‘‘Design Com-plexity Measurement and Testing.’’ Communica-tions of the ACM, Vol. 32, Num. 12, pp.1415-1425, December, 1989.

[17] Menasce, David, Performance by Design: Com-puter Capacity Planning by Example, Prentice-Hall, 2004.

[18] Patterson, David, ‘‘A simple model of the cost ofdowntime,’’ Proc. LISA 2002, USENIX Assoc.,2002.

[19] Pressman, Roger S., Software Engineering: APractitioners’ Approach, Fifth Edition, McGraw-Hill, 2001.


Documents

Toward a Cost Model for System Administration · 2019-02-25 · Toward a Cost Model for System Administration Alva L. Couch, Ning Wu, and Hengky Susanto – Tufts University ABSTRACT