THE SCHEDULES, COSTS, AND VALUEOF SOFTWARE PROCESS IMPROVEMENTS

Version 6 – August 6, 2005

Abstract

The topic of software process improvement is now very popular in the United States, Europe, and the Pacific Rim. There are many local chapters of the well-known software process improvement network or SPIN. Unfortunately the popularity of a topic is not commensurate with the quantitative data that is available about a topic. In the case of process improvement, there is a severe shortage of information on the schedules, costs, and results of moving from marginal to superior performance in software development.

This report shows a 36 month case study derived from several of SPR’s clients to illustrate four tangible aspects of software process improvement: 1) What it costs to achieve software excellence; 2) How long it will take to achieve excellence; 3) What kind of value will result from achieving software excellence; 4) What kinds of quality, schedule, and productivity levels can be achieved?

Capers Jones, Chief Scientist EmeritusSoftware Productivity Research LLC

Email: cjonesiii@cs.comWeb http://www.spr.com

Copyright 1998 - 2005 by Capers Jones.All Rights Reserved.

INTRODUCTION

Thanks to the pioneering work of Watts Humphrey (Humphrey 1989) and the Software Engineering Institute (SEI) the topic of software process improvement has become one of the major themes of the software engineering community.

The topic of software process improvement is featured in many international conferences. It has also been the impetus for the creation of a non-profit software process improvement network or SPIN, with chapters in many major cities.

Although the qualitative aspects of software process improvement have been published in scores of journals and books, there has been comparatively little empirical data published on the investments, schedules, costs, and benefits of moving from “marginal” to “superior” in terms of software development practices, tools, and infrastructure. For an interesting view of the status of software assessments and improvements, the annual process maturity profiles published by the Software Engineering Institute summarize trends from 1995 through 1999 (SEI Process Maturity Profile, March 2000.)

This report derived from several clients of Software Productivity Research (SPR) discusses the observed rates at which companies have been able to make tangible improvements in their software processes. Because most of SPR’s clients are major corporations in the Fortune 500 class, this report concentrates on the results of process improvement in large organizations that have at least 1,000 software professionals employed.

While many of the same principles are valid for small companies or even for government agencies, the timing and investments required can be significantly different from those discussed in this report. In general, smaller organizations can move more rapidly and do not require the same level of investment as major corporations. See the same author’s report “Assessments Benchmarks and Best Practices” for additional information on small company improvement costs (Jones 2000).

The phenomenon of higher costs for larger companies is true for many other topics beside process improvement. For example finding office space for a company with only 10 employees is far easier and usually much cheaper than finding office space for 1000 employees.

In general, technology transfer is slower in large corporations than in small companies. Also, large organizations tend to build up a bureaucratic structure with many levels of approval needed before doing anything new and different. All of these factors tend to slow down the progress of wide-spread undertakings such as software process improvement programs.

On the other hand, large corporations often have a process improvement infrastructure and can afford to create special software “centers of excellence” or research laboratories, which smaller organizations cannot afford.

2

A PROCESS IMPROVEMENT CASE STUDY

In order to give a context to the speed and value of software process improvements, it is convenient to think in terms of real organizations. In this report we will synthesize the results from several of our clients, and set up a generic but typical organization that is interested in pursuing a course of software process improvement.

Let us assume that the organization illustrated in this case study is one of a score of software development laboratories for a major telecommunications manufacturer. The total software employment totals to 1000 personnel, and includes software engineers, project managers, quality assurance, technical writers, testing specialists, and perhaps 25 other specialized and general occupations.

To simplify the case study example, we can assume that half of the software population is associated with new development work, and the other half is devoted to maintenance and enhancement of legacy applications: 500 personnel in each wing. This is a simplifying assumption and the ratio of development to maintenance personnel can vary widely from company to company and lab to lab.

Assume that in 2000 the lab director was charged by corporate management to improve the lab’s on-time performance of software schedules, and to improve software quality and customer support as well.

The Assessment and Baseline Analysis

In order to fulfill the corporate mandate of improving software performance, the lab director commissioned an external process assessment and a quantitative benchmark study by an outside consulting organization. As part of the final report, the consulting group presented the quantitative results shown in table 1, which is based on the kinds of data produced by Software Productivity Research (SPR). However, in order to understand table 1, several terms need to be introduced and defined.

A “software process assessment” is a qualitative study of development and maintenance practices. Several varieties of assessment are performed, although they share a nucleus of common features.

Assessments performed using the approach of Software Productivity Research (SPR) compare organizations against development and maintenance practices noted by other companies within the same industry, and attempt to determine whether the client’s practices are equivalent, better, or worse than normal industry practices. Results are statistically aggregated and the client is evaluated using a five-point scale running from “excellent” to “poor.”

Assessments performed using the approach of the Software Engineering Institute (SEI) compare organizations against a set of specific criteria and are aimed at determining the

3

“capability maturity level” of the organization on the well-known five-point maturity scale.

Both SPR and SEI express their results using five-point scales but the significance runs in the opposite direction. The SPR scale was first published in 1986 in Capers Jones’ book Programming Productivity (Jones 1986) and hence is somewhat years older than the SEI scale:

SPR Excellence Scale Meaning Frequency of Occurrence

1 = Excellent State of the art 2.0%2 = Good Superior to most companies 18.0%3 = Average Normal in most factors 56.0%4 = Poor Deficient in some factors 20.0%5 = Very Poor Deficient in most factors 4.0%

The SEI maturity level scale was first published by Watts Humphrey in 1989 in his well-known book Managing the Software Process (Humphrey 1989):

SEI Maturity Level Meaning Frequency of Occurrence

1 = Initial Chaotic 75.0%2 = Repeatable Marginal 15.0%3 = Defined Adequate 8.0%4 = Managed Good to excellent 1.5%5 = Optimizing State of the art 0.5%

Simply inverting the SPR excellence scale or the SEI maturity scale is not sufficient to convert the scores from one to another. This is because the SEI scale expresses its results in absolute form, while the SPR scale expresses its results in relative form. Large collections of SPR data from an industry typically approximate a bell-shaped curve. A collection of SEI capability maturity data is skewed toward the Initial or chaotic end of the spectrum. However, it is possible to convert data from the SPR scale to the equivalent SEI scale by a combination of inversion and compression of the SPR results.

The term “software baseline” refers to a collection of quantitative data on productivity, quality, schedules, costs, or other tangible information gathered during a specific time period. The purpose of the software baseline is to provide a tangible starting point for software process improvement work.

A “software benchmark” refers to a collection of quantitative data on productivity, quality, schedules, costs or other tangible information which is then compared against similar data points from competitors or from other companies within the same industry.

There is little or no difference in the actual data collected for either a baseline or a benchmark analysis. The only difference is that the baseline is used for comparing rates

4

of progress against your own initial conditions, while the benchmark is used for comparing your performance against similar organizations. Indeed, the same data can be used for both baseline and benchmark purposes.

The phrase “defect potential” refers to the sum total of bugs or errors that are likely to be found in five major artifacts: requirements, design, source code, user documentation, and a special category called “bad fixes” or the number of defect repairs which themselves contain an error.

The phrase “defect removal efficiency” refers to the percentage of defects actually eliminated prior to delivering the software to clients. Current U.S. averages are about 85% so in this case the client is approximately average.

Measurement of defect removal efficiency is a characteristic of every “best in class” company. Defect removal efficiency is one of the most powerful software metrics, since high values of > 95% are critical for achieving other key attributes such as schedule reductions, effective software reuse programs, and excellence in terms of customer satisfaction.

The phrase “delivered defects” refers to the numbers of latent defects still present in a software application when it is delivered to clients. The value for this topic is derived by subtracting the defect removal efficiency level from the defect potential.

The phrase “development productivity” refers to the software staff effort expended during a full development cycle which starts with requirements and commences with delivery to clients. All major activities are included; i.e. project management, software engineering, technical writing, testing, reviews and inspections, quality assurance, data base administration, etc.

Development productivity is expressed using the metric “function points per staff month” which is abbreviated in the table to “FP/Month.” Function point metrics have substantially replaced the older lines of code metric in software benchmark and baseline studies. The specific form of function point used in this report is version 4 of the counting rules published by the International Function Point Users Group (IFPUG 1995).

Note that the reciprocal metric “work hours per function point” is also used in baseline and benchmark studies, but this form is not shown in the current report simply to conserve space. The two formats are mathematically equivalent and which method is used in a specific report is a matter of preference.

The phrase “maintenance productivity” refers to the effort expended after the initial release of a software application. The work includes both enhancements (adding new features) and maintenance (defect repairs).

Maintenance productivity is expressed using the metric “function points per staff month” which is abbreviated in the table to “FP/Month.” Another interesting maintenance metric

5

using function points is that of the “maintenance assignment scope” or the amount of software one person can support in the course of a year. This metric is not illustrated in the current report but averages about 750 function points per maintenance programmer in the telecommunications industry. (A full baseline report can exceed 100 pages of information expressed using quite a few metrics. This overview report only discusses a few highlights to explain the essential features of a baseline study.)

The term “schedule” refers to the elapsed time in calendar months from the start of requirements until delivery to customers. Because schedules vary with the size of the application, the table illustrates the typical average schedule for applications which are nominally 1000 function points in size.

Note that within the telecommunications software community, the term “interval” is often used as a substitute for “schedule.” The reason for this is that AT&T used the word “interval” to mean development schedule prior to its break-up into many other companies, so the word continues to be used very widely among companies that were once part of the AT&T system such as Lucent or Bellcore.

Quantitative Baseline and Benchmark Results

The client baseline results are shown in the top row of table 1. The two lower rows illustrate industry benchmarks from within the telecommunications industry: one for industry averages, and one for “best in class” results which are derived from the top 5% of projects within other telecommunications manufacturing enterprises.

The comparison of a client’s data against both industry averages and “best in class” results is a common practice for baseline and benchmark studies carried out by consulting groups such as Software Productivity Research (SPR) or some of the other specialized benchmark and baseline organizations such Gartner Group, Meta Group (now merged with Gartner), Giga Group, the Davids Group, Quantitative Software Management, the International Function Point Users Group (IFPUG), or the Australian Software Metrics Association.

Incidentally, all of the benchmark groups cited now use function point metrics as their primary reporting tool. It is possible to express productivity and quality data using the older “lines of code” metric but this metric is not as reliable for large-scale studies which include projects created in multiple programming languages such as COBOL, FORTRAN, C, SMALLTALK, C++, Eiffel, Visual Basic, etc.

Table 1 illustrates a sample of the basic quantitative results in high-level form:

Table 1: 2000 Client Baseline Quality and Productivity Compared to Similar Groups

Baseline Defect Defect Delivered Develop. Maint. SchedulePotential Removal Defects Productivity Productivity Months

per FP Efficiency per FP (FP/Month) (FP/Month) (1000 FP)

6

Client 5.00 85.00% 0.750 7.00 9.00 18.00Results

Industry 5.25 83.00% 0.893 6.50 9.00 21.00Average

Best in 2.75 97.00% 0.083 10.50 13.00 15.00Class

For table 1, make the assumption that a total of 25 software projects from the client organization were measured, and then compared against similar projects from other companies within the telecommunications industry. A set of 20 to 50 projects is the normal sample size for Software Productivity Research baseline studies although for very large multi-national corporations a suitable sample may exceed 100 software projects.

(Note that since the lab in this case study is stated to be one of 20 labs in the same corporation, it often happens that internal benchmark data is collected from all locations so that each lab can be compared against the others. In this situation, then as many as four or five hundred projects can be collected for the same benchmark study.)

As can be seen from table 1, the client organization is close to the industry average in overall results, and indeed slightly better than industry norms in terms of quality control and development productivity. However, the “best in class” results are significantly better than the client’s results. The significant gaps between a client’s results and “best in class” records within the same industry is the most common impetus for starting a process improvement program.

Qualitative Assessment Data

In the course of gathering the quantitative data, the consulting organization also gathered a significant volume of qualitative data. The qualitative data is often aimed at ascertaining the “maturity level” of the organization using the well-known capability maturity model pioneered by Watts Humphrey and published more recently by the Software Engineering Institute (Paulk et al, 1995).

In this case study assume that the client is currently a fairly sophisticated Level 1 group under the SEI capability maturity model (CMM) concept. That is the organization is still in Level 1 but on the cusp of Level 2, rather than being well back in the set of Level 1 organizations.

Incidentally, the quality levels shown in the case study correlate exactly with the Level 1 quality data shown in a prior report on Becoming Best in Class (Jones 1998). Table 2 shows the current quality results associated with the five levels of the CMM:

Table 2: Software Defect Potentials and Defect Removal Efficiency Targets Associated With Each Level of the SEI CMM

7

(Data Expressed in Terms of Defects per Function Point)

SEI CMM Levels Defect Removal Delivered Potentials Efficiency Defects

SEI CMM 1 5.00 85% 0.75SEI CMM 2 4.00 90% 0.40SEI CMM 3 3.00 95% 0.15SEI CMM 4 2.00 97% 0.06SEI CMM 5 1.00 99% 0.01

These results are somewhat hypothetical at the highest levels, but from observations of organizations at various CMM levels seem to be within the range of current technologies from levels 2 through 4.

The Level 5 defect potential target, or lowering defect potentials down to 1 per function point, is the hardest target of the set, and would require a significant volume of certified reusable material of approximate zero-defect quality levels.

Moving up the SEI CMM scale brings to mind learning to play golf. For a beginning player, it can take several years before they are able to break 100, or shoot a round of golf in less than 100 strokes. In fact, most golfers never do break 100. This is somewhat equivalent to the observation that many companies who are initially assessed as being at Level 1 tend to stay at the level for a very long time. Indeed, some may never advance to the higher levels.

However, golfers that do manage to break 100 can usually break 90 in less than a year. This parallels the observation that moving up the SEI CMM ladder is quicker once the initial hurdle of moving from Level 1 to Level 2 is accomplished.

In general, the rate of progress is relative to an organization’s status on their first assessment. Organizations that are well back in the “level 1” CMM zone have trouble getting started. Organizations that are fairly close to “level 2” status usually are more flexible and can accelerate their process improvement work.

Strength and Weakness Analysis

The qualitative assessment results produced by Software Productivity Research include “strength” and “weakness” reports which discuss specific methodologies, development processes, tools, etc. in the context of whether the client organization is better or worse than typical patterns noted within the same industry. In the case study, the following pattern of strengths and weaknesses were presented to the client:

Clients Strengths (Better than Average Performance)

8

Staff experience with application types Staff experience with in-house development processes Staff experience with development tools Staff experience with programming language(s) Staff specialization: testing Staff specialization: quality assurance Staff specialization: maintenance Requirements analysis (Quality Function Deployment) Change control methods Design methods Development process rigor Customer support tools Customer support methods Maintenance release testing Development testing

Since both the client company and the telecommunications industry have been building software applications for more than 35 years, they know a quite lot about standard development practices. For example the client had long recognized that change control was a key technology, and had stepped up to fully automated change management tools augmented by a change control board for major projects.

However, many companies tend toward weakness in the project management domain. Indeed, project management failures are often much more common and also more serious than development technology failures as causes of missed schedules or outright cancellations of software projects.

Another very common weakness is failure to use formal design and code inspections prior to commencing testing. This is considered to be a weakness because inspections are about twice as efficient as most forms of testing in finding errors or bugs. Formal design and code inspections can each average more than 60% in terms of defect removal efficiency, while most forms of testing are less than 30% efficient. Further, not only do inspections have an excellent record in terms of defect removal, but they also are synergistic with testing and raise the efficiency of standard test stages such as new function test, regression test, and system test.

Another common weakness is in the software maintenance domain. Although software development is often funded and supplied with state of the art tools, maintenance is not as “glamorous” as development and hence may lag. In the case of the client, none of the more powerful maintenance support tools were deployed; i.e. complexity analysis tools, code restructuring tools, reverse engineering tools, reengineering tools, etc.

Yet another common weakness or gap in software development is failure to move toward a full software reusability program. An effective software reuse program entails much more than just source code reuse, and will include many other artifacts such as design specifications, test materials, and also user documents and even project plans.

9

Although the client had a good track record for standard development practices and methods, there were some significant weaknesses visible in terms of project management, pre-test defect removal, and software reuse:

Client Weaknesses (Worse than Average Performance)

Project management: annual training in state of the art methods Project management: cost estimating Project management: quality estimating Project management: risk analysis Project management: schedule planning Project management: lack of productivity measurements Project management: partial quality metrics Project management: lack of productivity metrics Project management: incomplete milestone tracking Quality control: no use of formal design inspections Quality control: no use of formal code inspections Maintenance: no use of complexity analysis Maintenance: no use of code restructuring tools Maintenance: inconsistent use of defect tracking tools Maintenance: no use of inspections on enhancements No reuse program: requirements No reuse program: design No reuse program: source code No reuse program: test materials No reuse program: documentation No reuse program: project plans

As can be seen from the patterns of strengths and weaknesses, the client organization was pretty solid in basic development practices but somewhat behind the state of the art in terms of project management and advanced quality control methods, although the use of quality function deployment is certainly innovative. The management and quality problems, in turn, made software reuse questionable because reuse is only cost effective for artifacts which approach zero-defect levels.

A final weakness is failure to provide enough training for managers and technical personnel. Usually in the range of 10 to 15 days per year is an optimum amount for teaching new skills and for refreshing current skills. See the interesting work by Bill Curtis et al on the People Capability Model (Curtis et al 1995) for additional insights.

The combination of the assessment and baseline study took about two months to perform from the day that the contract was signed until the final report was presented to the lab director and his principal management team.

10

Incidentally, once the data has been presented to client executives and management, the next step is to present the findings to the entire development and maintenance community. The reason for doing this is to pave the way for a subsequent process improvement program. Unless the specific strength and weakness patterns are clearly stated and understood by all concerned, it is hard to gain support for process improvement activities.

It is always best to explain both the assessment, baseline, and benchmark results to the entire software community. Unless this is done, then social resistance to proposed software process improvements are likely. In general, people are not likely to respond well to change unless they know the reason for it. If a company is well behind the state of the art and competitors are visibly better, that is usually a strong motivation to experiment with software process improvements.

11

BEGINNING A SOFTWARE PROCESS IMPROVEMENT PROGRAM

Assume that the assessment and baseline study began in September and the final report was delivered in October of 1997. After that the client took two month in conjunction with the consulting group to develop a three-year process improvement program. The process improvement program was scheduled to kick off in January of 1998 and had the following targets:

Three-Year Software Process Improvement Targets

Set aside 12 days a year for training in software management topics Set aside 10 days a year for training in software process improvement topics Establish a local software “center for excellence” to pursue continuous improvements Budget $10,000 per capita for improved tools and training Achieve Level 3 status on the SEI CMM maturity scale No more than 5% difference between estimated schedules and real delivery dates No more than 5% difference between estimated costs and actual costs Raise defect removal efficiency above 97% as the corporate average Reduce defect potentials below 3.0 per function point as the corporate average Reduce development schedules or intervals by 50% from requirements until delivery Raise development productivity rates by more than 50% Reduce development costs by more than 40% Reduce maintenance costs by 50% for first two years of deployment Achieve more than 50% reusability by volume for design, code, and test artifacts Establish an in-house measurement department Publish monthly reports on software quality and defect removal Publish overall results in an annual “state of the art” report for group executives

Although these are fairly aggressive improvement targets, they are all achievable within about a 36 months time span for companies that are not so bureaucratic that change is essentially impossible.

However, making significant process improvements does not come for free. It is necessary to set aside time for training, and also to budget fairly large amounts for improved training, improved tools, and other attributes.

Removing the Mystery of Software Process Improvement

Although there are thousands of tools and hundreds of vendors making claims about improving software productivity by vast amounts, most of these claims have no solid empirical data behind them.

The basic approach to improving software productivity is actually fairly simple: Find out what your major cost and schedule obstacles are and then strive to eliminate them. For most large software companies and large software projects, the rank order of software schedule and cost drivers is the same:

12

1. Defect removal is the most expensive and time consuming activity2. Producing documents is the second most expensive and time consuming activity3. Meetings and communications is the third most expensive consuming activity4. Coding is the fourth most expensive and time consuming activity5. Project management is the fifth most expensive and time consuming activity

Thus it is obvious that if you want to improve productivity and cut your development schedules down to something shorter than today, you have to concentrate your energies on the top-ranked cost and schedule drivers; i.e. 1) Improve quality; 2) Control paperwork; 3) Improve communications.

For example, the distribution of software development expenses in the case study enterprise would approximate the following pattern shown in table 3:

Table 3: Development Expense Pattern for the Client Case Study

Activity Percent of Development Cost

Defect removal 30%Paperwork 22%Meetings 19%Coding 17%Project management 12%

The only thing unusual about this pattern is the fact that older measurements based on the “lines of code” metric rather than function points would not usually focus on the high costs associated with quality control, paperwork, meetings, and project management. Since these activities cannot be easily be measured using lines of code metrics there is no clear understanding of the relative percentages associated with non-coding work. Thus for many years the costs associated with non-coding work have been understated in the software literature.

This gap in the metrics literature caused many companies to focus only on coding productivity, and to essentially ignore the higher costs associated with quality control, paperwork, project management and other non-coding activities.

For a company with this kind of expense pattern, it is obvious that software defects have to be both prevented and removed somewhat better than today. It is also obvious that paperwork costs have to be controlled, and this leads directly to the concept of reusing portions of specifications, plans, user manuals, and other paper-based documents.

However, reusing artifacts with a lot of bugs or defects is counter productive. Therefore the sequence of effective process improvement requires that quality control come before reusability. The normal sequence of process improvement is more or less the same for all companies:

13

1. Begin with an assessment and baseline2. Start by upgrading management skills in planning and estimating3. Start a measurement program to track progress4. Improve software defect removal via formal inspections5. Improve defect prevention via better requirements and design6. Improve the maintenance process via complexity analysis and restructuring7. Improve the conventional development process8. Stock a library with high-quality reusable artifacts9. Expand the volume of reusable materials

There are minor variations in this sequence, but management improvements and quality improvements are definitely the top priorities for the first year. Incidentally from analyses of many organizations, the general rates of software process improvement are now starting to be known:

Software quality can improve at 15% to as much as 40% per year. Software development productivity can improve at rates of 5% to 20% per year. Software maintenance productivity can improve at rates of 10% to 40% per year Schedules can be shortened at rates of 5% to 15% per year Reuse volumes can grow from < 20% to > 75% over about a five-year period

These rates of improvement can occur for four or five years in a row before they reach “terminal velocities” at which point improvements taper down and the results become stable again.

Because of the need to climb a fairly steep learning curve, the first year results are usually at the low end of the improvement range while the second and third years are usually nearer to the peak of the improvement range.

Incidentally, continuous improvement is not likely to occur unless there is constant pressure from top management and continuous focus on software process and tool issues. Since software personnel who are fully occupied on normal projects lack the free time for this purpose, the normal situation is to establish a “center for excellence” whose main task is software process improvement.

For the case study organization, whose total software employment is nominally about 1000, the size a local center for excellence would be about 10 people. However at a corporate level for organizations with perhaps 20,000 software personnel the overall number of personnel devoted to software process improvement would be in the range of 200 and they would probably be concentrated in one or two larger facilities rather than being distributed in small groups among every location.

For a discussion of the creation and evolution of the well-known ITT Programming Technology Center which served as the model for many software centers for excellence,

14

refer to the author’s “A Ten Year Retrospective of the ITT Programming Technology Center” (Jones 1988).

The First Year Goals and Accomplishments: 2001

In the first year of the process improvement program, the basic goals are to attack the most significant problems identified by the assessment and baseline studies. For this case study there were two pressing needs identified: 1) The need for better quality by means of formal inspections; 2) The need for better project management tools and methods, and especially for better planning and estimating methods.

It is also obvious that if you want to prove to higher management that your process is actually improving, you need to measure your quality and productivity results. The reason for this is to make a convincing case that things are getting better and your are not just spending money without achieving any results. Thus measurements must start at the beginning of the process improvement program. Indeed, you need an initial baseline of your pre-improvement status in order to validate that you are truly improving.

Table 4 shows the pattern of results that might be experienced in the first year of a successful process improvement program. Table 4 expresses the current situation in terms of percentages so all of the columns begin at 100%, except for the last column. The last column reflects the average volume of reusable material and at the time of the baseline analysis, only 10% of any given artifact, on average, was reused.

In the course of the first year, the specific targets set by the lab manager were to improve defect removal efficiency by 40% compared to the baseline, reduce defect potentials by 10%, increase development productivity by 10%, improve maintenance productivity by 25%, and increase the volume of reusable material up to 25% by volume.

Note that because of the need to take time out from work for classes, and to learn to feel comfortable with some of the new approaches such as design and code inspections, productivity will decline for about the first four to six months when first starting out on a process improvement program.

The initial reduction in productivity due to the steep learning curve associated with most kinds of process improvement strategies is often ignored by vendors and is not well covered in the process improvement literature. In any human activity, time must be set aside for training and preparation before tangible benefits start to accrue.

As a rule vendor advertisements and even journal articles on software process improvement typically ignore start-up problems and initial set-backs. A thoughtful book which discusses these problems has already been cited, and is Karl Wiegers’ excellent treatment of how Kodak carried out their software process improvement program (Wiegers 1996).

15

Table 4 reflects the short-term loss of productivity, which is then followed by productivity gains in the last half of the year.

Table 4: First Year Rates of Improvement in Software Attributes

Calendar Defect Defect Develop. Maint. Schedules SoftwareMonths Removal Potentials Productivity Productivity Reuse

0 100.00% 100.00% 100.00% 100.00% 100.00% 10.00%1 105.00% 100.00% 90.00% 95.00% 105.00% 10.00%2 107.00% 100.00% 92.00% 97.00% 104.00% 10.00%3 110.00% 100.00% 95.00% 100.00% 102.00% 10.00%4 112.00% 98.00% 96.00% 105.00% 100.00% 10.00%5 115.00% 97.00% 98.00% 107.00% 98.00% 10.00%6 117.00% 96.00% 100.00% 110.00% 97.00% 12.00%7 120.00% 95.00% 102.00% 112.00% 96.00% 15.00%8 122.00% 94.00% 103.00% 114.00% 95.00% 18.00%9 125.00% 93.00% 105.00% 116.00% 94.00% 20.00%10 130.00% 92.00% 106.00% 116.00% 93.00% 22.00%11 135.00% 91.00% 108.00% 120.00% 92.00% 24.00%12 140.00% 90.00% 110.00% 125.00% 90.00% 25.00%

Average 119.83% 95.50% 100.42% 109.75% 97.17% 15.50%

Because of the need to move up the learning curve in the new quality control approaches and to learn to use the new project management tools, it is interesting that none of the first year improvement targets were actually met when measured on a full calendar year basis. This is because productivity declined for the first half of the year when managers and developers were in courses, or were climbing the inspection and project management learning curves.

However, smaller projects commenced and finished in the second half of the year did achieve the first year goals, even though the entire year did not reflect but about half of the desired accomplishments.

This kind of analysis of how a company performed versus its annual goals and targets is exactly the kind of information that would be published in an annual report, which is a major attribute of “Best in Class” software groups.

The first year is the most difficult period for a software process improvement program. Resistance to change will be at its peak, the costs are likely to be higher than in the following years, and due to the steep learning curve, the first year goals and targets can easily be missed.

For some interesting discussions of software process improvements in real companies, refer to Karl Wiegers’s multi-year history of process improvements within Kodak

16

(Wiegers 1996) and Robert Grady’s account of software process improvements within Hewlett Packard (Grady 1996).

Both of these excellent studies are based on long-range analysis of improvements over a number of years and both are written by authors who were active participants in the process improvement programs.

The Second Year Goals and Accomplishments: 2002

In any organization, process improvements do not happen overnight. Work is pressing and overtime is a way of life, so not everyone can learn new methods at the same time. In the first year usually less than half of the personnel and managers can be brought up to speed. By the middle of the second year a “critical mass” point will be reached where more than 50% of the entire software community is now fully up to speed in some of the new process improvement activities. As a result, second year accomplishments are usually more visible than first year accomplishments.

Incidentally, one of the hidden problems with making process improvements visible is the large component of unpaid overtime that is common in software organizations. As software development practices improve, unpaid overtime tends to drop down from about 15% to less than 5% in a typical business week. Since unpaid overtime is seldom tracked or included in measurements, the reduction in unpaid overtime makes it harder to demonstrate improvements during the first year of a process improvement program.

For the second year, formal inspections are now a standard practice and this has improved the quality of software artifacts to such a point that it is now possible to make rapid progress in a software reusability program.

When expressing the goals and targets for a multi-year improvement program there are a number of possible ways that the goals can be expressed:

They can be compared to the original baseline They can be compared to the year just finished They can be expressed in terms of percentage changes They can be expressed in absolute terms such as function points per staff month

All of these alternatives have potential value and all have been noted among client organizations. However to simplify the presentation of results in this short case study, the results will be compared as percentage changes to the original baseline.

Incidentally, any full software assessment or benchmark study will include dozens of charts, graphs, and illustrations in order to get the points across. Somewhat surprisingly, companies vary in their preferences for how information should be presented. Therefore consulting groups such as Software Productivity Research usually select the metrics and presentation methods that suit the desires of the client organizations.

17

Table 5 highlights the quantitative results from the second year of the process improvement program:

Table 5: Second Year Rates of Improvement in Software Attributes

Calendar Defect Defect Develop. Maint. Schedules SoftwareMonths Removal Potentials Productivity Productivity Reuse

13 145.00% 88.00% 112.00% 127.00% 90.00% 27.00%14 150.00% 86.00% 112.00% 130.00% 87.00% 30.00%15 150.00% 85.00% 113.00% 132.00% 85.00% 33.00%16 150.00% 84.00% 114.00% 134.00% 83.00% 35.00%17 150.00% 82.00% 115.00% 136.00% 81.00% 38.00%18 150.00% 81.00% 116.00% 138.00% 80.00% 40.00%19 155.00% 80.00% 117.00% 140.00% 79.00% 44.00%20 155.00% 79.00% 118.00% 142.00% 77.00% 47.00%21 155.00% 78.00% 120.00% 144.00% 75.00% 50.00%22 155.00% 77.00% 122.00% 146.00% 73.00% 52.00%23 160.00% 76.00% 124.00% 148.00% 72.00% 54.00%24 160.00% 75.00% 125.00% 150.00% 70.00% 55.00%

Average 152.92% 80.92% 117.33% 138.92% 79.33% 42.08%

This fashion of expressing results by comparing them to the staring condition is similar to they way many of us would go about expressing the results of dieting. If when we start a diet we weigh 200 pounds and we want to lose 10% of our weight then we would end up at 180 pounds. In the course of our diet we would obviously pass through other weights such as 195, 190, etc. but the real target of the diet is the difference between where we start and our desired destination.

In the course of the second year, the organization should be able to achieve the criteria for successfully entering the Level 2 plateau of the SEI capability maturity model concept. By the end of the second year the group should be approaching the Level 3 criteria although it will probably not be quite time to seek Level 3 certification.

By the end of the second year, measurements of quality, productivity, schedules, and customer satisfaction should show some interesting and significant gains compared to the original baseline.

18

The Third Year Goals and Accomplishments: 2003

By the start of the third year of software process improvement the lab’s software performance is visibly improved:

Defect removal efficiency levels are at state-of-the art levels Defect potentials have been significantly reduced Software reuse is reaching a critical mass (> 50% by volume) Development productivity is significantly better than the baseline Maintenance productivity is significantly better than the baseline Schedules are significantly shorter than the baseline Customer satisfaction is significantly better than the baseline

Table 6 illustrates the results of the third year of continuous process improvement:

Table 6: Third Year Rates of Improvement in Software Attributes

Calendar Defect Defect Develop. Maint. Schedules SoftwareMonths Removal Potentials Productivity Productivity Reuse

25 160.00% 75.00% 125.00% 150.00% 70.00% 55.00%26 160.00% 73.00% 128.00% 155.00% 68.00% 57.00%27 160.00% 70.00% 132.00% 160.00% 66.00% 60.00%28 160.00% 68.00% 135.00% 165.00% 64.00% 62.00%29 161.00% 64.00% 140.00% 170.00% 62.00% 64.00%30 162.00% 62.00% 143.00% 175.00% 60.00% 65.00%31 163.00% 60.00% 148.00% 180.00% 58.00% 66.00%32 164.00% 58.00% 150.00% 185.00% 57.00% 68.00%33 165.00% 56.00% 155.00% 190.00% 55.00% 70.00%34 167.00% 55.00% 160.00% 193.00% 54.00% 72.00%35 168.00% 53.00% 165.00% 197.00% 52.00% 73.00%36 170.00% 50.00% 175.00% 200.00% 50.00% 75.00%

Average 163.33% 62.00% 146.33% 176.67% 59.67% 65.58%

By the third year, notable sociological changes should occur too. Organizations and personnel that are doing a very good job have a pride of workmanship that is quite visible to clients and company executives alike.

It is very satisfying and enjoyable to pay repeat visits to software organizations which have been pursuing software process improvement programs for three years are approaching or achieving state of the art results. Indeed, it is strongly recommended that organizations which are just starting out on process improvement work pay a visit to such groups so they can see the levels of energy and competence which are visible in groups that are “top guns” in terms of software excellence.

19

In addition to relative performance against the baseline, it is also interesting to examine the actual quality and productivity results at various points during the process improvement program. Usually data of this kind would be assembled and produced in an overall annual report.

Table 7 shows the results of the original baseline, and then the results at month 12, month 24, and month 36:

Table 7: Baseline Quality and Productivity and 36 Months of Improvement

Baseline Defect Defect Delivered Develop. Maint. SchedulePotential Removal Defects Productivity Productivity Months

per FP Efficiency per FP (FP/Month) (FP/Month) (1000 FP)

Month 0 5.00 85.00% 0.750 7.00 9.00 18.00

Month 12 4.50 92.00% 0.360 7.70 11.25 16.20

Month 24 3.75 96.00% 0.150 8.75 13.50 12.60

Month 36 2.50 99.00% 0.025 12.25 18.00 9.00

By the end of 36 months of continuous process improvement work, the quality data is now better than the “best in class” results and this in turn has created significant improvements in productivity and schedules.

By the end of the 36 month period, the company should not only be able to pass the criteria for SEI CMM Level 3, but could also make a strong run for a Baldrige Award and should easily be able to pass the ISO 9000-9004 quality criteria.

Another key result by the third year is the ability to make use of substantial volumes of reusable artifacts in every major project. By the third year of a software process improvement program, the following levels of reuse are possible:

Reusable design material should approach 50% by volume for new projects. Reusable source code should approach 75% by volume for new projects Reusable test materials should approach 75% by volume for new projects Reusable plans, manuals, and paper materials should approach 50% by volume

The topic of software reuse is beginning to get an extensive literature of its own, and is a key factor in every major process improvement program. Ivar Jacobsen and colleagues have assembled some interesting data on the technologies of reuse within corporations (Jacobsen et al 1997).

Incidentally, there is a situation that companies should be alert to in this era of almost universal shortages of software personnel. Both the software technical workers and the

20

managers of companies with successful software practices have highly marketable skills on the software job market.

Although the topic is not discussed in depth in this report, if you plan to achieve “best in class” status in terms of software engineering and software project management capabilities, you should also plan to achieve “best in class” status in terms of your compensation and benefit packages. Otherwise you will be creating a highly skilled set of employees who may find better opportunities elsewhere.

For an excellent overview of the implications of the software personnel shortage, Dr. Howard Rubin and a number of industry experts were part of a national task force which explored the shortage in considerable depth. The task force report is still evolving, but already contains a wealth of useful information and insights, much of which is unavailable from other sources and is quite new (Rubin 1998).

Software Costs and the Software Labor Shortage

Software productivity in terms of “function points per staff month” or “work hours per function point” can be improved at a rate of perhaps 10% per year for several years. However, productivity in terms of “cost per function point” is not as easy to improve, and may even decline due to the impact of the current software labor shortage.

The result of the software personnel shortage is that for about 10 years in a row software salaries and benefits have been rising faster than labor productivity has been improving. Thus if a company undergoing an effective process improvement program is increasing labor productivity at a rate of 10% per year, but personnel costs are going up at a rate of 15% per year, then the net cost per function point will be getting higher at roughly 5% per year.

Due to the growth in the software personnel shortage, it will be very hard to reduce costs per function point while salary inflation is increasing, and even “signing bonuses” are starting to be offered to software personnel with key skills.

Suffice it to say in the context of immediate process improvement work over the next three to five years, it is questionable that it will be possible to make substantial reductions in “cost per function point” because the inflation in software salaries is greater than typical productivity improvement rates on an annual basis.

The Value and Return on Investment in Software Process Improvement

As pointed out in the introduction, software process improvement is not inexpensive. In this case study a total of $10,000 per capita was set aside for process improvement expenses such as training, tool acquisition, consulting fees, etc. Since the software personnel in the case study were stated to comprise 1000 workers and managers, this implies that the total expenses for software process improvement activities amounted to $10,000,000 over a three-year period.

21

If the average annual burdened compensation rate for the staff of 1000 personnel is $120,000 per year, then the organization has an approximate annual budget of $120,000,000.

Investments of the $10,000,000 magnitude require funding approval, and this means that the investment must meet normal corporate criteria for returns on investment (ROI). Let us examine some of the significant value aspects of software process improvement that can justify a $10,000,000 investment.

A software group with a total employment of 1000 personnel will normally be able to develop about 42,000 function points per year and maintain about 1,250,000 function points per year. Although the situation is more complex in real life, let us assume that 50% of personnel are on the development side and 50% of the personnel are on the maintenance side. To further simplify, let us assume that half of the $10,000,000 for process improvements will go to the development side and half will go to the maintenance side.

Cost Recovery on the Development Side. In the original baseline average productivity was stated to be 7 function points per staff month for development, which amounts to 84 function points per year. Assuming 500 development personnel and 84 function points per year, the annual rate of new work is 42,000 function points.

Since 500 personnel with fully burdened compensation rates of $120,000 per year have annual expenses of $60,000,000 it can be seen that the development cost per function point for the organization amounts to roughly $1429 per function point for new function points added to the corporate inventory.

Without the process improvement program, the development group would have created 126,000 new function points over the three-year period shown in the case study. However, as a result of the improvement program, the average rate went from 7 to about 10 function points per month over the three-year improvement program.

Thus instead of creating 126,000 function points in three years the same group of 500 development personnel would be able to create 60,000 function per year or 180,000 function points in the same three year period.

Using only the principles of cost recovery, the 54,000 additional function points at a value of $1429 per function point means additional corporate software assets worth $77,166,000 were created as a byproduct of an investment of $5,000,000. This is an ROI of about $15.43 for every $1.00 invested, with a 36-month recovery period. Although this calculation makes some simplifying assumptions, an ROI of $15.43 per $1.00 expended in three years is more than sufficient to be funded by most corporations.

Cost Recovery on the Maintenance Side. An even more significant aspect of cost recovery can be achieved on the maintenance side of the case study. Of the 500

22

personnel on the maintenance side about 250 will be working on enhancements. Obviously these personnel will be more productive too, so value will be associated with their higher output.

The other 250 maintenance personnel spend much of their time on software defect repairs. Now fixing bugs has been a necessary and expensive activity for software groups ever since the industry began. Although bug repairs are needed to stay in business, every dollar spent on fixing bugs after a product is released is really a subtraction from the bottom line and should be viewed as a liability.

The greatly improved quality levels associated with the process improvement program will probably cut down the number of full-time staff working on bug repairs from 250 down to about 150 and hence free up 100 personnel for other assignments.

This means that the software process improvement program will unexpectedly free up about 10% of the total software employment, or a set of 100 experienced personnel, who are no longer going to be locked into heavy-duty bug repairs and customer support tasks.

Assuming the $120,000 a year for burdened compensation that was stated, this means a possible savings of $12,000,000 a year. There are several options, which the lab director can consider as to how to best utilize the windfall of 100 spare personnel:

They can be reassigned to development and thereby raise software production by about 12,000 function points per year. At the assumed cost of $1429 per function point this would supply added value at a rate of $17,148,000 per year.

They could be assigned to maintenance projects not discussed in this report such as Year 2000 work or Euro-currency updates. Since telecommunications companies are heavily impacted by Year 2000 work, and international companies by the Euro, this would be an unexpected windfall from the process improvement program.

They can be “downsized” or transferred to other labs and locations within the company and removed from local payrolls. However since most locations have a shortage of software personnel in the late 1990’s, this is unlikely to occur.

In any case, the significant increase in software quality and resulting decrease in software defect repairs is one of the most valuable aspects of the process improvement program. Here too, returns in the vicinity of $15.00 can be projected for every $1.00 expended, although the exact ROI must be calculated for each company and each specific situation.

Asset Value of a Library of Reusable Artifacts. A typical Fortune 500 corporation owns a software portfolio or library that totals somewhere between 250,000 function points to more than 5,000,000 function points of software applications. Prior to the commencement of the formal process improvement program, about 15% of this volume was derived from reusable materials. However much of the normal day to day reuse is in the form of “private” reuse by individual technical personnel. Private reuse may be

23

valuable to individual programmers, but much of the value is invisible in the sense of having any tangible value on corporate books.

As a result of the emphasis on formal reuse as part of the planned improvement program, the total volume of reusable artifacts owned by the enterprise may total to 50,000 function points after three years. A library of reusable artifacts is a valuable corporate asset. However, the value of a library of reusable software assets may have tax consequences in the United States, so companies are advised to seek legal counsel about the implications of Internal Revenue Service Code Rule 482.

The essence of Rule 482 is that when one division or affiliate in a controlled group of related corporations such as a conglomerate or multinational company provides goods or services to another division, these goods or services may be treated by the IRS as though they were taxable income to the receiving division. This is potentially a “stake through the heart” of a formal corporate reuse program in multi-national corporations.

For example, if Division A of Mega Company in San Jose supplies 1000 function points of reusable artifacts to Division B in Boston, then the transfer may be treated as a taxable transaction by the IRS.

Of course, if the enterprise is not a U.S. corporation or if the transfers are made abroad, such as transferring reusable assets between London and Paris, then Rule 482 may not apply.

In any case, a formal library of reusable artifacts is a valuable corporate asset and that raises questions of how to determine the value. Since reusable artifacts are more difficult to develop and than normal software, their costs are often about 30% to 50% higher than “normal” artifacts of the same nature. Let us assume that the case study company has developed a library of 10,000 function points of reusable materials at an average cost of $2,000 per function point. Thus the replacement value of these artifacts would amount to $20,000,000.

However, if each of these reusable artifacts is reused an average of 10 times, and each instance of reuse saves 70% of normal development costs, then the effective value of the library of reusable artifacts would amount to about $98,000,000. This value assumes a “normal” development cost of about $1,400 per function point where each reused function point saves 70% or $980.

As can be seen, calculating the real value of a library of reusable artifacts is a fairly complex situation. The replacement costs of the reusable assets themselves can be calculated fairly easily. However the true business value of the library of reusable artifacts is more complex, and requires analysis of normal development costs, the effectiveness of the reusable materials in reducing those costs, and the number of times each artifact is reused during its life expectancy.

24

These calculations assume reusable artifacts that are of zero-defect status. If any of the reusable artifacts contain serious flaws or bugs (say a year 2000 bug embedded in a reusable module) then the value of the reusable artifact will be degraded by the recall and repair costs.

Adding Value through Shorter Development Schedules. One of the primary benefits of a software process improvement program is the ability to shorten typical software development schedules by 50% to 70% compared to the initial baseline. The main technology for achieving shorter schedules is of course having significant volumes of reusable artifacts available.

The direct beneficiaries of the shorter development schedules are the clients and users of the software applications that result. There may well be substantial financial benefits accruing from these shorter schedules, but the quantification of such values must be derived from the nature of the business activities that can commence earlier, or from the business improvements that result from more rapid turnaround times.

For example, if a company can bring out a new product 50% faster than their main competitors as a result of their software process improvement activities, then no doubt significant business value will result. However, quantifying this value requires specific knowledge about the revenue stream of the product in question, and does not lend itself to abstract generalization.

Adding Value through Higher Revenues. Thus far we have discussed value only in the form of cost recovery to recoup the $10,000,000 investment in process improvement by lowering in-house development costs and raising internal productivity rates.

However if some of the software is part of marketed products, or is itself commercially marketed, then the software process improvement program can also generate value by raising revenues.

Two aspects of the process improvement program can benefit software-related revenue streams:

1. The shorter development schedules will get products to the market faster.

2. The higher quality levels will increase market shares.

These two phenomena are both known to occur when software processes are improved, but their value is too specific to individual products and to focused markets to allow general rules of thumb to be developed.

Adding Value from Disaster Avoidance. Because many large software projects are cancelled or fail to perform when installed, an additional value of process improvement is that of avoiding the consequences of software disasters. Unfortunately, this kind of

25

value is hard to quantify. When disasters do occur their costs are highly visible. But if no disasters occur, then there is no way of being sure how many might have occurred under less stringent development practices.

Adding Value from Reduced Litigation Risk. The author often works as an expert witness in software lawsuits where applications either did not get finished, or worked so poorly when deployed as to be unusable. Litigation costs can be enormous, and litigation can absorb hundreds of staff and executive hours for several years. Here too the value of litigation avoidance is hard to quantify.

Adding Value from Improved Staff and Management Morale. No one likes to work on poorly planned and carelessly executed projects. Everyone likes to work on projects that are successful and yield good customer satisfaction. Therefore another form of value from process excellence is very high morale levels on the part of both managers and staff. This in turn yields very low voluntary attrition rates, and a very good “team spirit” among the software community. If voluntary employment declines from 3% per year to less than 1% per year, it is possible to quantify the savings in recruitment and training costs. However, collecting the data for this kind of value analysis is outside the scope of normal quality and productivity measurements.

Adding Value from Recruiting Top-Ranked Personnel. One interesting phenomenon was first noted in the 1970’s and remains true in 2005. Technical personnel who quit a company voluntarily are often the best qualified and have the highest appraisal scores. There are two reasons for this: 1) Top technical personnel are the most frustrated with poor performance and bad management; 2) Top technical personnel are the ones most likely to be sought by other companies and professional recruiters. Therefore once a company gets a reputation in the industry as being a high-performance organization, they can attract the best people in the industry. Of course this means that companies with excellent technological prowess must also have good compensation and benefits plans.

Adding Value from Customer Loyalty. Customers quickly learn to avoid vendors with shoddy goods. Conversely, customers tend to buy multiple products from vendors with excellent manufacturing and customer support quality levels. Therefore an investment in achieving software excellence will yield returns through customer loyalty. Here too the data for this kind of value analysis is outside the scope of normal quality and productivity measurements.

26

SUMMARY AND CONCLUSIONS

The phrase “software process improvement” covers a very wide spectrum of methodologies, tools, quality control methods, project management functions, and enhanced volumes of reusable artifacts.

Although the literature on software process improvement is quite large and growing larger, much of the information is subjective and deals with qualitative factors. If the concept of software process improvement is going to become a permanent fixture of the software engineering world, then it must begin to augment subjective observations with quantitative information based on empirical baseline and benchmark studies.

This report covers some of the highlights of the economics of software process improvement using a case study approach. But this report is intended simply to illustrate the basic structure of software process improvement economics. There is a continuing need for more extensive coverage in the software literature using both case studies and statistical analysis of software process improvement results. Qualitative information is useful but not sufficient. Software process improvements need to be firmly based on solid empirical findings.

REFERENCES AND READINGS

Abran, A. and Robillard, P.N.; “Function Point Analysis, An Empirical Study of its Measurement Processes”; IEEE Transactions on Software Engineering, Vol 22, No. 12; Dec. 1996; pp. 895-909.

Austin, Robert d:; Measuring and Managing Performance in Organizations; Dorset House Press, New York, NY; 1996; ISBN 0-932633-36-6; 216 pages.

Bogan, Christopher E. and English, Michael J.; Benchmarking for Best Practices; McGraw Hill, New York, NY; ISBN 0-07-006375-3; 1994; 312 pages.

Brown, Norm (Editor); The Program Manager’s Guide to Software Acquisition Best Practices; Version 1.0; July 1995; U.S. Department of Defense, Washington, DC; 142 pages.

Curtis, Bill, Hefley, William E., and Miller, Sally; People Capability Maturity Model; Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA; 1995.

Department of the Air Force; Guidelines for Successful Acquisition and Management of Software Intensive Systems; Volumes 1 and 2; Software Technology Support Center, Hill Air Force Base, UT; 1994.

27

Dreger, Brian; Function Point Analysis; Prentice Hall, Englewood Cliffs, NJ; 1989; ISBN 0-13-332321-8; 185 pages.

Grady, Robert B.; Practical Software Metrics for Project Management and Process Improvement; Prentice Hall, Englewood Cliffs, NJ; ISBN 0-13-720384-5; 1992; 270 pages.

Grady, Robert B. & Caswell, Deborah L.; Software Metrics: Establishing a Company-Wide Program; Prentice Hall, Englewood Cliffs, NJ; ISBN 0-13-821844-7; 1987; 288 pages.

Grady, Robert B.; Successful Process Improvement; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-626623-1; 1997; 314 pages.

Humphrey, Watts S.; Managing the Software Process; Addison Wesley Longman, Reading, MA; 1989.

IFPUG Counting Practices Manual, Release 4, International Function Point Users Group, Westerville, OH; April 1995; 83 pages.

Jacobsen, Ivar, Griss, Martin, and Jonsson, Patrick; Software Reuse - Architecture, Process, and Organization for Business Success; Addison Wesley Longman, Reading, MA; ISBN 0-201-92476-5; 1997; 500 pages.

Jones, Capers; “A Ten-Year Retrospective of the ITT Programming Technology Center”; Software Productivity Research, Burlington, MA; 1988.

Jones, Capers; Applied Software Measurement; McGraw Hill, 2nd edition 1996; ISBN 0-07-032826-9; 618 pages.

Jones, Capers; Assessment and Control of Software Risks; Prentice Hall, 1994; ISBN 0-13-741406-4; 711 pages.

Jones, Capers; Patterns of Software System Failure and Success; International Thomson Computer Press, Boston, MA; December 1995; 250 pages; ISBN 1-850-32804-8; 292 pages.

Jones, Capers; Software Assessments, Benchmarks, and Best Practices; Addison Wesley Longman, Boston, MA; 2000 (due in May of 2000); 600 pages.

Jones, Capers; Software Quality – Analysis and Guidelines for Success; International Thomson Computer Press, Boston, MA; ISBN 1-85032-876-6; 1997; 492 pages.

Jones, Capers; The Economics of Object-Oriented Software; Software Productivity Research, Burlington, MA; April 1997; 22 pages.

28

Jones, Capers; Becoming Best in Class; Software Productivity Research, Burlington, MA; January 1998; 40 pages.

Kan, Stephen H.; Metrics and Models in Software Quality Engineering; 2nd edition; Addison Wesley Longman, Boston, MA; ISBN 0-201-72915-6; 2003; 528 pages.

Keys, Jessica; Software Engineering Productivity Handbook; McGraw Hill, New York, NY; ISBN 0-07-911366-4; 1993; 651 pages.

Love, Tom; Object Lessons; SIGS Books, New York; ISBN 0-9627477 3-4; 1993; 266 pages.

McCabe, Thomas J.; “A Complexity Measure”; IEEE Transactions on Software Engineering; December 1976; pp. 308-320.

Melton, Austin; Software Measurement; International Thomson Press, London, UK; ISBN 1-85032-7178-7; 1995.

Multiple authors; Rethinking the Software Process; (CD-ROM); Miller Freeman, Lawrence, KS; 1996. (This is a new CD ROM book collection jointly produced by the book publisher, Prentice Hall, and the journal publisher, Miller Freeman. This CD ROM disk contains the full text and illustrations of five Prentice Hall books: Assessment and Control of Software Risks by Capers Jones; Controlling Software Projects by Tom DeMarco; Function Point Analysis by Brian Dreger; Measures for Excellence by Larry Putnam and Ware Myers; and Object-Oriented Software Metrics by Mark Lorenz and Jeff Kidd.)

Paulk Mark et al; The Capability Maturity Model; Guidelines for Improving the Software Process; Addison Wesley, Reading, MA; ISBN 0-201-54664-7; 1995; 439 pages.

Perry, William E.; Data Processing Budgets - How to Develop and Use Budgets Effectively; Prentice Hall, Englewood Cliffs, NJ; ISBN 0-13-196874-2; 1985; 224 pages.

Perry, William E.; Handbook of Diagnosing and Solving Computer Problems; TAB Books, Inc.; Blue Ridge Summit, PA; 1989; ISBN 0-8306-9233-9; 255 pages.

Putnam, Lawrence H.; Measures for Excellence -- Reliable Software On Time, Within Budget; Yourdon Press - Prentice Hall, Englewood Cliffs, NJ; ISBN 0-13-567694-0; 1992; 336 pages.

Putnam, Lawrence H and Myers, Ware.; Industrial Strength Software - Effective Management Using Measurement; IEEE Press, Los Alamitos, CA; ISBN 0-8186-7532-2; 1997; 320 pages.

29

Rubin, Howard; Software Benchmark Studies For 1997; Howard Rubin Associates, Pound Ridge, NY; 1997.

Rubin, Howard (Editor); The Software Personnel Shortage; Rubin Systems, Inc.; Pound Ridge, NY; 1998.

Shepperd, M.: “A Critique of Cyclomatic Complexity as a Software Metric”; Software Engineering Journal, Vol. 3, 1988; pp. 30-36.

Strassmann, Paul; The Squandered Computer; The Information Economics Press, New Canaan, CT; ISBN 0-9620413-1-9; 1997; 426 pages.

Stukes, Sherry, Deshoretz, Jason, Apgar, Henry and Macias, Ilona; Air Force Cost Analysis Agency Software Estimating Model Analysis ; TR-9545/008-2; Contract F04701-95-D-0003, Task 008; Management Consulting & Research, Inc.; Thousand Oaks, CA 91362; September 30 1996.

Symons, Charles R.; Software Sizing and Estimating – Mk II FPA (Function Point Analysis); John Wiley & Sons, Chichester; ISBN 0 471-92985-9; 1991; 200 pages.

Thayer, Richard H. (editor); Software Engineering and Project Management; IEEE Press, Los Alamitos, CA; ISBN 0 8186-075107; 1988; 512 pages.

Umbaugh, Robert E. (Editor); Handbook of IS Management; (Fourth Edition); Auerbach Publications, Boston, MA; ISBN 0-7913-2159-2; 1995; 703 pages.

Weinberg, Dr. Gerald; Quality Software Management - Volume 2 First-Order Measurement; Dorset House Press, New York, NY; ISBN 0-932633-24-2; 1993; 360 pages.

Wiegers, Karl A; Creating a Software Engineering Culture; Dorset House Press, New York, NY; 1996; ISBN 0-932633-33-1; 358 pages.

Yourdon, Ed; Death March - The Complete Software Developer’s Guide to Surviving “Mission Impossible” Projects; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-748310-4; 1997; 218 pages.

Zells, Lois; Managing Software Projects - Selecting and Using PC-Based Project Management Systems; QED Information Sciences, Wellesley, MA; ISBN 0-89435-275-X; 1990; 487 pages.

Zvegintzov, Nicholas; Software Management Technology Reference Guide; Dorset House Press, New York, NY; 1994; ISBN 1-884521-01-0; 240 pages.

30