CPSC 547 – Software Measurement

Software Quality Metrics

Professor: Dr. Bing Cong

By:

Aakash Juneja

Jaskaran S. Gandhi

Phong Nguyen

Saba Yaghoobi Saray

Department of Computer Science

California State University Fullerton

Fall 2014

Table of Contents Abstract ................................................................................................................................................. 4

1. What is Software Quality? ............................................................................................................ 5

1.1 Why Measure Software? ................................................................................................................. 6

1.2. Software metrics and its objectives ................................................................................................ 7

1.3. The Need for Metrics ...................................................................................................................... 8

1.4 Types of Metrics- ............................................................................................................................ 10

2. Background ................................................................................................................................... 11

2.1 Waterfall model .............................................................................................................................. 12

2.2. Spiral model ................................................................................................................................... 13

3. Software Quality Metrics Classification ...................................................................................... 14

Figure 1. Classification of Software Metrics ........................................................................................ 16

3.1 Product Quality Metric .................................................................................................................. 17

3.1.1 Scenario ........................................................................................................................................ 18

Table 1: Requirements Coverage Table ............................................................................................. 20

3.1.2 Goal ............................................................................................................................................. 20

3.1.3 Product Risk Metrics ................................................................................................................... 21

Figure 2: Quality Risk Status (Early Test Execution) ........................................................................ 23

Figure 3: Quality Risk Status (Middle of Test Execution) ................................................................. 24

Figure 4: Quality Risk Status (Late Test Execution) .......................................................................... 25

3.2 In-Process Quality Metric ............................................................................................................. 26

Figure 5: Two contrasting Defect Arrival patterns during Testing ................................................... 28

Figure 6: Defect Removal by Phase for two products ........................................................................ 30

Figure 7: Phase Effectiveness of a Software Project .......................................................................... 32

3.3 Resources ....................................................................................................................................... 32

Table 2 : Comparison between resources ........................................................................................... 33

Table 3: Capacity/Load ....................................................................................................................... 33

Figure 8: Resource Distribution .......................................................................................................... 34

5. Real World Examples of Software Metrics ................................................................................. 34

a. Motorola ....................................................................................................................................... 34

b. Hewlett - Packard ........................................................................................................................ 40

Figure 9: Testing Metrics .................................................................................................................... 43

Figure 10: Defect Summary ................................................................................................................ 45

Figure 11: Time Data and Return ....................................................................................................... 45

5. What are the advantages of using software metrics? .................................................................. 46

5.1 Limitations of Software Metrics .................................................................................................... 48

6. References – ................................................................................................................................. 49

Abstract

Software development is the active conduct of program design systems as they are

retained and improved over their periods. Software systems need to repeatedly change during

their life cycle for numerous causes; adding new features to satisfy user requirements, changing

business needs, introducing novel technologies, correcting faults, improving quality, etc. The

increase of changes, along the development of a software system, can lead to a bad quality. It is,

thus vital to observe how software quality changes so that quality assurance activities can be

correctly scheduled. Software quality is basically a field of study and practice that describes the

necessary qualities of software products.

Software metrics can be used to examine the improvement of the software systems

quality. Empirical sign occurs presenting that there exists a connection among metrics and

software quality. Software metric is a tool for accepting changing phases of the code base, and

the development of the plan. The goal is gaining impartial, reproducible, and computable

measurements, which may have plentiful valued applications in plan and budget planning, cost

estimation, quality assurance testing, software debugging, software performance optimization,

and optimal personnel task assignments.

What seems to be challenging on the way of quality engineering and management is that

the word quality per se is kind of unclear and this problem can consequences to mistake in

delicate processes. There are numerous details behind this mistake. First of all, each person can

have different knowledge about the idea of quality based on their perspectives. Secondly, there

are different levels of concept for each term; while you are debating about quality, we may refer

to the inclusive meaning of quality, or you may refer to obvious sense of it. Moreover, since we

use the term quality in our day to day language it causes difficulties on the technique of using

this term in specialized positions, as these two are so different.

We are planning to discourse the goal of software quality metrics on this paper, while we

explain about metrics classification in software quality namely product metrics, project metrics,

and in-process quality metrics. Meanwhile, we are going to discuss the importance of collecting

software engineering data. In other words, the accumulated data will be the results of well-

defined metrics and may help to improve our system quality.

1. What is Software Quality?

Our world uses software everywhere. Every profession depends on it, every cell-phone

needs software, and even each car depend on it. Without software, our world would fall apart.

With this in mind, the quality of that software is important. Because it is widely used, low

quality software is not satisfactory. But what is software quality? High quality software meets

the needs of customers while being reliable, well supported, maintainable, portable, and easily

integrated with other tools. The most basic search in a software quality would be lack of bugs. It

is important that our requirements have as minimum bugs as possible because it will cost a lot of

fix them later on in the software cycle; if the software contains too many defects, the requirement

will be unfulfilled. We express these two ways: defect rate which is no. of defects/million lines

of code and reliability which is no. of failures per n hours of action and mean time failure.

Customer approval is usually measured by taking the percent between (neutral and non-

happy) by the customer satisfaction surveys. To decrease unfairness, methods such as blind

surveys are frequently used. In addition to complete client contentment with the software

product, satisfaction toward specific qualities is also estimated. For example, IBM estimates

satisfaction with their software products in terms of capability, functionality, usability,

performance, reliability, install ability, maintainability, documentation, service, and overall.

Hewlett-Packard focuses on functionality, usability, reliability, performance, and service ability.

Other corporations use related measurements of software customer satisfaction.

We must take these quality attributes into account in the planning and design phases of the

software to increase overall customer satisfaction. However, these quality attributes are not

always mix well with each other. For example, it is hard to attain maintainability when the

complexity of the product is high. Depending on the type of product and customers, different

factors are mandatory for different feature attributes example real-time processing, performance

and reliability might be the most significant attributes. For customers with individual systems

and simple operations, documentation may be more important.

1.1 Why Measure Software?

• Determine the quality of the software

• Calculate qualities of a software

• Improve quality of a software

• Estimation and cost of software

• Estimate the efficiency effects of new tools and techniques

• Establish efficiency developments over time

• Improve software quality

• Decrease future maintenance needs

1.2. Software metrics and its objectives

Software metrics is used to define the wide-ranging of activities used with measurement. The

practice of metrics involves measures and metrics which leads to long term process

improvement. It assesses the state of the project, tracks potential risks, uncover problem areas,

adjust workflow or tasks, and evaluate team’s ability to control quality. Software product metrics

focuses on measuring attributes of the software product, which is any object or document

subsequent from the software development method. Software product objects descent into one of

the four categories: specifications, designs, source code, and test records.

Software metric is simply measurement of software and primary purpose is to help plan and

foresee software growth. Any software metric is an effort to measure or foresee some attribute of

software, process, or resource. Attributes can be internal or external. Internal attributes are those

of the system representations, such as design diagrams, coding or documentation which can be

controlled and measured directly, but their measurement slight off unless there is indication that

the specific metric is related to an external attribute. Examples of internal attributes of source

code are:

Size

Complexity

Coupling

Modularity and

Reuse.

External attributes are usually those influenced by the system in implementation and cannot be

measured directly. They are for example:

Functionality

Reliability

Usability

Efficiency

Maintainability and

Portability

External attributes are important because knowing or foreseeing them plays a vital role in quality

assurance. Some internal attributes are connected with external attributes therefore their metrics

can be used as secondary measures for external attributes.

1.3. The Need for Metrics

Software testing is the process of recognizing the errors or defects in the system and make sure it

is resistant with the customer requirements before releasing it to the market. But testing can

never guarantee 100% bug free software. This might show the unpredictability and

ineffectiveness of testing process or testing procedures followed, it could also account the human

mistakes in case of manual testing or script errors in case of automated testing. This indirectly

influences the quality of the software. Hence, in order to increase the software quality, certain

procedures and guidelines need to be placed down and match the efficiency of the software.

Every software experiences the risk phase. Hence risk management is a very significant factor

which needs to be taken care in order to recover the software quality, consistency and strength of

the software. There should be high amount of clearness as to where the software views in terms

of quality, quantity, development, customer efficiency, obedience to requirements etc. This raises

a distress to have an efficient metric and report based procedure, thus giving growth to test

metrics and reports.

Following are the arguments that support why software testing metrics are necessary.

• Metrics helps in tracking of the project status and supports in presenting the statistics to

the senior management in a structured way.

• Back tracking can be very stress-free if every action is followed correctly.

• Metrics and reports help in accumulating data using which further processes in testing

can be completed more effectively.

• Software Metrics and reports help in both project management and process management

• Metrics can directly impact both the effectiveness and productivity of a software

• Helps in early defect detection and defect removal, thus reducing the cost of defects

• Assists the managers in effective decision making

• Metrics also act as a benchmark for estimations and provide bottleneck to the testers

• Manages risk at ease

Software metrics are used to obtain objective measurements that can be useful for quality

assurance, performance, correcting, managing, controlling and estimating costs. Finding defects

in code (post release and prior to release), foreseeing faulty code, calculating project success, and

predicting project risk. The utility of metrics is limited to quantifying one of the following goals:

Schedule of a software project, Size/complexity of development involved, cost of project, and

quality of software

To summarize, the goal of software metric is to understand the state of the software product.

Hence, it is very much vital to select the correct set of metrics for our product during the testing

process.

1.4 Types of Metrics-

Requirements metrics

Size of requirements

Traceability

Completeness

Volatility

Product Metrics

Code metrics

Lines of code LOC

Design metrics – work out from requirements or design documents before the system has

been implemented

Object oriented metrics- helps in finding faults, and allow team to see directly how to

make their classes and objects simpler.

Test metrics

Communication metrics – looking at artifacts i.e. email, and meetings.

Process metrics

Measure the process of software development

Frequently used by management to check the budget and processes for efficiency.

Evaluate and track aspects of the software design process like: human resources, time,

and schedule

2. Background

First thing we need to consider when we are talking about the background of software

measurement is the potential impact of the different software development models on quality.

What seems impossible is discussing software metrics and models without considering the effect

of software development process type on them. Depending on our project’s goals and objectives

we designate the preferred software development model among the variety of different processes

and methodologies. Based on which objectives we are pointing at, there are several development

life cycle models to adopt. Each of the known models defines different stages of the process and

also dissimilar set of orders in which the stages should be performed.

Here, we are going to discuss two of the best or the most popular life cycles. Meanwhile, we are

going to explain why it is needed to merge the metrics.

2.1 Waterfall model

During 60s and 70s, software developers were more concerned about controlling and planning

difficulties. At that era, almost all of the software development projects were facing gigantic cost

escalations on one hand and schedule deferrals issues on the other hand. It was exactly when

waterfall process aroused to put an end on the increasing complication of development projects

and its consequences. The waterfall process model raises the development team’s sprits to clarify

what are the expectations of this projects to be met before beginning development of the system.

It is simply known as collection and description of the system requirements. The next motivating

asset of waterfall model is its inspiration to break the whole development process into some

reasonable phases such as design, code, test, and etc. which are supposed to deliver some

halfway products leading to the final product. It seemed crucial to make sure of the appropriate

execution and also delivery of good-quality products, hence, each step was required to pass some

criteria such as validation, entry, and exit. This is well known as Entry-Task-Validation-Exit

(ETVX) paradigm which is one of the major specifications of waterfall process.

Waterfall process has various advantages because of this divide and conquer approach. First,

precise tracking of the project evolution is more possible while early recognition of errors is

more likely. Second, by this approach, the software system development organization is obliged

to be more arranged and controllable.

For outsized companies, having huge and complicated development projects, structural tactic

plays a significant role. Using this approach, during the process, bunch of documents must be

created for future use in testing and maintenance arenas. The most important aspect of this

method is making large projects easier to manage and possible for on time delivery without cost

escalations.

2.2. Spiral model

The spiral model which has been developed by Boehm is one of the models of software

development, being fallowed in some organizations. It has been formed after lots of

improvements on the waterfall model while it applied to variety of enormous government

software projects.

The spiral model mostly relies on creating prototypes and using risk management methods.

These features of spiral model increase the spiral model’s level of flexibility comparing to

waterfall model. As Boehm has explained, the most inclusive application of this model is related

to the expansion of the TRW Software Productivity System, which is known as TRW-SPS.

During these last few years spiral model has been positively recognized between engineers and

project managers due to its high risk management ability.

The need for software metrics had been more palpable as introducing increasing amount of

object oriented projects were occurring. In other words, we can say the available software

development models were effective for the variety of software versions but there were some

restrictions when it came to object oriented software’s. That was when engineers were convinced

the combination of software metrics and software development models is the best solution to the

problem.

3. Software Quality Metrics Classification

Software Quality Metrics are a subclass of software metrics that emphasis on the quality

features of the product, process, and project. In most cases, software quality metrics are more

thoroughly related with process and product metrics rather than with project metrics. Software

metrics can be categorized into 3 classifications:

Product Metrics

Process Metrics

Project Metrics

Product metrics help software engineers to better appreciate the qualities of models and

measure the quality of software by describing the characteristics of the product such as size,

difficulty, design features, presentation, etc. A vision into the design and structure of the

software is provided, and also they help in measuring quality based on a set of distinct rules, that

helps the software engineers. Process metrics are composed over a long period of time through

all projects, and they can be used to help improve software development & maintenance, and aid

in making strategic decisions. The goal is long-term process improvement that could be attained

by providing a set of process indicators. Project metrics are used by project managers and a team

of software designers to adjust project workflow and practical activities that allows them to

define project features and implementation designs. At times, metrics belong to multiple

categories: in-process quality metrics of a project both involve process metrics as well as project

metrics.

Having said above that, software quality metrics are closely associated to process and

product metrics rather than with project metrics; the parameters related to project development

such as the number of developers, their individual skill levels, the size of the project, etc.

certainly affects the quality of the product. The intention of Software Quality Engineering is to

look in to the relationships that exist amongst in-process metrics, project characteristics, and end-

product quality. The findings of this research helps in engineering improvements in both process

and product quality.

In the discussions below, we shall discuss several kinds of metrics in each of the three

classifications of software quality metrics as described briefly above.

Figure 1. Classification of Software Metrics

Software Metrics

Product Metrics

Static Metrics

Size Metrics

LOC Token Count Function

Count

Design Metrics

Control Flow Metrics

Function Count

Information Metrics

Henry & Kafura

Weighed Metrics

Data Structure Metrics

Software Science Metrics

Dynamic Metrics

Testability

Process Metrics

3.1 Product Quality Metric

Product Quality Metrics help us to better understand the attributes of models and assess

the quality of software under test as well as help us gain insight into the design and construction

of the software. A good test allows us to measure the quality and risk in a system, however

proper product metrics are needed to capture the test measures. Product Quality Metrics provide

an on-the-spot rather than after-the-fact insight into the software development of a product, and

at the same time provide insights to guide where product improvements should occur.

Further discussing the 3 sub-groups of Product Quality Metrics:

Efficiency Product Metrics – The degree of measure to which a product can attain that

anticipated level of quality carefully.

Effectiveness Product Metrics – The degree of measure to which the product is attaining

anticipated levels of quality.

Elegance Product Metrics – The extent of measure to which a product effectively and

efficiently achieves the results in an elegant, and well-executed way.

A Quality Metric is a predictor for Product Quality, and both are used to degree the

properties of software and at the same time help in refining a system module by matching them

with current better-quality systems.

The steps to develop a good quality Product Metrics [2]:

Define test coverage and quality-related objectives for the product.

The effectiveness, efficiency, and elegance with which the product is achieving those

objectives should be taken into consideration.

Devise measurable metrics, either direct or surrogate, for each effectiveness, efficiency,

and elegance question.

Determine realistic goals for each metric, such that we can have a high level of

confidence in the quality and test coverage for the product prior to release.

Monitor progress towards those goals, determining product status, and making test &

project control decisions as needed to optimize product quality and test-coverage

outcomes.

3.1.1 Scenario

We have the following information on a project:

95% of the tests have run

90% of the tests have passed

5% of the tests have failed

4% of the tests are ready to run

1% of the tests are blocked.

It has been assumed that we are on schedule as per the text execution. However, this

information is not accurate enough to confirm that this product will be of standard quality by the

last phase. Since, this information is not accurate, we would need product metrics to assist us in

determining the quality throughout test execution, and at the same time making sure that quality

is on track for successful delivery of the product. Testing Product Metrics are focused on the

quality of the system under test, and it is important to have this information on testing

dashboards so that the stakeholders, participants, and other involved members are on the same

track.

In addition to making sure the quality of the product is at par, Product Metrics also reflect

the entire team’s efforts towards quality. Stakeholders, and the management are key as they’re

the ones who determine the software process and its quality capabilities, and the only role of the

testing team is to measure quality, and they cannot control the behaviors of stakeholders or

management with respect to quality.

The objectives for test coverage and quality for every product vary, but often includes

ensuring complete coverage of requirements. A requirements coverage table would help in

monitoring progress towards the goals of complete testing and fulfillment of the requirements, as

shown below [3]:

Table 1: Requirements Coverage Table

3.1.2 Goal

An analytical requirements based test strategy is used, and one or more tests should be

created for every requirement during the test design & implementation phase. If bi-directional

traceability is maintained, complete coverage can then be assured. During test execution, we run

these tests, and we can report the results in terms of requirements fulfilled, and unfilled using the

traceability.

The progress can be monitored towards these goals of complete testing and fulfillment of

requirements by using the metrics analysis as shown in the table above. At the same time, the

table helps in understanding the quality of the product and it eases the decisions to make test and

project control decisions that help us to achieve the best possible test coverage.

The table above depicts quality stands on the major requirements groups for an e-

commerce site, and the status of each requirement in each group. Based on the analysis, the

status of each requirement may be determined as follows:

If all the tests related to requirement have been run, then it is said that the requirement

is tested.

If all the tests related to requirement have been run and have passed, then it is said

that the requirement is also classified as passed.

If all the tests relayed to requirement have been run, but one or more have failed, then

it is said that the requirement is classified as failed.

If any of the tests related to requirements are blocked, then it is said that the

requirement is classified as blocked.

If the metrics in a table highlight problems, then it is very helpful for the engineers. This helps to

increase the level of confidence in terms of positive measurements of requirements. Practically,

it is difficult to develop metrics that will predict the quality for software, but with multi-

dimensional coverage metrics, we can have a higher level of confidence when it comes to

satisfaction.

3.1.3 Product Risk Metrics

If we consider product metrics for testing strategies other than requirements based

testing, but now we shall consider risk based testing in which the objective is to typically reduce

product quality risk to an acceptable level, and these two questions can be taken into

consideration:

How effectively are we reducing quality risk overall?

For each quality risk category, how effectively are we reducing quality risk?

Answering the first question, when using a risk based strategy, each quality risk item

deemed sufficiently important for testing should have one or more tests create for it during test

design and implementation. With the help of bi-directional traceability, the coverage can be

assured and measured between the tests and the risk items.

The tests are run and defects are reported during test execution. Bi-directional traceability

is needed between the defects and risk items as well in addition to the test results and risk items.

This enables us to report which risks are fully mitigated, which risks are partially mitigated, and

which risks are unmitigated.

Figure 2: Quality Risk Status (Early Test Execution)

The above figure graphically represents all information across all risk items. The region

in green represents risks for which all tests were run and passed. The region in orange represents

risks for which at least one tests has failed or at least one must-fix bug is known. The region in

blue represents other risks, which have no known must-fix bugs but still have tests pending to

run.

Quality Risk Status

Other Risks

Risks for which at leastone test has failed ormust-fix bug known

Risks for which all testswere run & passed

Figure 3: Quality Risk Status (Middle of Test Execution)

During the second half of the test execution, the green region starts to grow very quickly.

The tests those are most likely to find out bugs in the early half of the test executions have

already been carried out. Testers focus on running confidence-building tests that lower risk

(turning blue region to green), while developers fix the bugs that were found.

Quality Risk Status

Other Risks

Risks for which at leastone test has failed ormust-fix bug known

Risks for which all testswere run & passed

Figure 4: Quality Risk Status (Late Test Execution)

As mentioned above, the green region completely takes over all the regions during the

end of the testing phase, and the blue region would slowly be faded out that is a good sign since

it represents unmitigated risks.

When a risk-based test strategy is followed, the project management team decides

adequate test coverage. The risks are said to be balance, and quality risk mitigation has been

optimized only if the project management team believes that the quality risks posed by known

defects, test failures or yet un-run tests are acceptable, compared to the schedule and budget risks

associated to continue the testing.

Quality Risk Status

Other Risks

Risks for which at leastone test has failed ormust-fix bug known

Risks for which all testswere run & passed

3.2 In-Process Quality Metric

Process metrics are collected over long periods of time and they pretty much cover all

projects involved, and they are used for making strategic decisions. A set of indicators is

provided by process metrics that leads to software process improvement. Organizational

sensitivity is to be kept in mind when interpreting metrics data, and at the same time it is

required to provide regular feedback to engineers who are directly involved with the collection of

measures and metrics.

The basis for in-process quality management is formed by the following metrics:

1. Defect density during Machine Testing: The defect rate is correlated to the defect rate

during formal machine testing. A higher defect rate is an indicator that during the development

phase, the software was injected with a lot of errors. In some cases, the extra-ordinary testing

effort could also be the reason for a high defect rate. A uniform distribution is never followed by

software defect density. If a piece of code has higher testing defects, it is a result of more

effective testing or it could be because of higher latent defects in the code. The team of engineers

can use the following scenarios to judge the release quality [1]:

If the defect rate during testing is the same or lower than that of the previous

release, then ask: Does the testing for the current release deteriorate?

If the answer is no, the quality perspective is positive.

If the answer is yes, extra testing needs to be done.

If the defect rate during testing is substantially higher than that of the previous

release, then ask: Did we plan for and actually improve testing effectiveness?

If the answer is no, the quality perspective is negative. The only

remedial approach in this case is to do more testing, which yields

higher defect rates.

If the answer is yes, then the quality perspective is the same or

positive.

2. Defect arrival pattern during Machine Testing: The defect density during testing is a

summary indicator. More information is given by the pattern of defect arrivals. Even with the

same overall defect rate during testing, different patterns of defect arrivals indicate different

quality levels in the field. Both the defect arrival rate and the cumulative defect rate are shown

by the figure below. The objective of this pattern is always to look for defect arrivals that

stabilize at a very low level, or times between failures that are far apart, before ending the testing

effort and releasing the software to the field. These declining patterns of defect arrival during

testing are indeed the basic assumption of many software reliability models, and the time unit for

observing these patterns is usually weeks and months. [1]

Figure 5: Two contrasting Defect Arrival patterns during Testing

There are three slightly different metrics when we talk about defect arrival patterns

during testing:

The defects reported during the testing phase by time interval. These are just the

raw number of arrivals, not all of which are valid defects.

The true defect pattern is the pattern of valid defect arrivals – when problem

determination is done on the reported problems.

The pattern of defect backlog overtime, this is needed because development

organizations cannot investigate and fix all reported problems immediately. A

large defect backlog at the end of development cycle and a lot of fixes has yet to

be integrated into the system, the stability shall be affected. A regression test is

needed to ensure that targeted product quality levels are reached.

3. Phase-based defect removal pattern: The test defect density metric extends to the

phase-based defect removal pattern. It requires the tracking of defects at all phases of the

development cycle (including design reviews, code inspections, and formal verifications) in

addition to testing. Conducting formal reviews or functional verifications to enhance the defect

removal capability of the process reduces error injection, and the overall defect removal ability

of the development process is reflected by the pattern of phase-based defect removal.

Many development organizations use metrics like inspection coverage and inspection

effort for in-process quality management. The figure below shows the patterns of defect removal

of two development projects: Project A was front-end and Project B was heavily testing-

dependent for removing defects. The different phases of defect removal are high-level design

review (I0), low-level design review (I1), code inspection (I2), unit test (UT), component test

(CT), and system test (ST).

Figure 6: Defect Removal by Phase for two products

4. Defect Removal Effectiveness: Also referred as efficiency,

DRE= (Defects removed during development phase/Defects latent in the product) x 100%

The total number of latent defects in the product at any given phase is not known, the

denominator of the metric can only be approximated. It is usually estimated by:

Defects removed during the phase + defects found later

For each phase, and for the front end, the metric can be calculated. It is known as early

defect removal and phase effectiveness when used for the front end, and other specific phases.

The higher the value of the metric, the more effective the development process and hence, the

fewer the defects are likely to escape to the next phase. The figure below shows DRE by phase

for a software project, the weakest phases were unit test (UT), code inspections (I2), and

component testing (CT). The action plans to improve the effectiveness of these phases were then

established and deployed.

Figure 7: Phase Effectiveness of a Software Project

3.3 Resources

There are certain metrics related to resource distribution measurement such as: utilization

percentage, effort distribution, capacity/load, and response time.

Response Time:

Using response time metric, we will be able to recognize what is the state of our

desired resource. By picking a baseline, we will have a scale to use later on. The next

step is comparing the response times to the baseline whenever we are trying to access

a resource. It is so simple:

Comparison Resource status

Response time < baseline Resource is available

Respond time > baseline Resource is overloaded

Table 2: Comparison between resources

Utilization Percentage

(Total Effort spent by resource) / (Total Budgeted Effort for the resource)

Utilization Percentage simply provides the utilization related to a specific resource in

order to avoid having under-utilized or over-utilized resources. Here, the point to

keep in mind is that the utilization stages should be kept optimal.

Capacity/Load:

It is the best utensil to calculate the maximum load that your project can tolerate and

by using this you can easily figure out if your project is kind of overloaded.

Ratio amount Project status

Ratio =1 Ideal

Ratio <1 Project can take more load

Ratio >1 Project is overloaded

Table 3: Capacity/Load

Effort Distribution:

It is supposed to show the share of particular resources among a number of

tasks/components. We use pie charts to show the percentage of resources assigned to

each task or component more clearly. For sure, one of the ways to ensure about the

appropriate distribution of resources is using effort distribution measure.

Figure 8: Resource Distribution

5. Real World Examples of Software Metrics

a. Motorola

Many companies attempt to use software metrics to better their position, but they often find it too

complex and unnecessary to truly follow-through. On one industry survey that was taken, less

than 10% of the companies surveyed considered the use of metrics as “positive” and

0.2

0.3

0.27

0.2

0.05 0.05

Resource Destribution

Requirement Analysis

Design

Implementation

Testing

Documentation

Rework

“enthusiastic” [100]. One of the few companies that was able to positively turn metrics into

improvement was Motorola.

The managers and engineers at Motorola were yearning for more understanding and insight into

the software development process of their company. They wanted to see if improvement could

be done to quality, productivity, and cycle time.

The first step Motorola did was to define a Metrics Working Group (MWG) in their

organization. This group would be in charge of establishing and defining a company-wide

adopted set of software metrics to be used for improving the quality of the software. This group

was and is very successful in their work, they have been responsible for numerous processes and

metrics in the company. One of these being the Goal/Question/Metric approach.

“Measurement is not the goal. The goal is improvement through measurement, analysis, and

feedback.” That was the philosophy of the Motorola software metrics initiative. This philosophy

was put in practice when senior management made it a requirement to software development

process (in their Quality Policy for Software Development (QPSD)). The target areas in this

requirement were:

Delivered defects and delivered defects per size;

Total effectiveness throughout the process;

Adherence to schedule;

Estimation accuracy;

Number of open customer problems;

Time that problems remain open;

Cost of nonconformance;

Software reliability.

There actually were some conflicts by all the different business units on deciding the proper

definition on terms like “software problem”, “fault”, and many more. These were to be

eventually settled though for the progress of the company’s software metrics goals. After the

definitions and other small problems were settled, a Goal/Question/Metric structure was decided,

and this structure had 7 goals:

1. Improve project planning,

2. increase defect containment,

3. increase software reliability,

4. decrease software defect density,

5. improve customer service,

6. reduce the cost of nonconformance

7. increase software productivity

Using these goals, they went on to crease metrics based on the scope.

Below, I’ve listed some of the metrics that were borne to address these goals. These metrics are

by no means set in stone and can always change over time due to feedback and effectiveness:

Metric 1.1: Schedule Estimation Accuracy (SEA) = Actual Project Duration / Estimated

Projection Duration (goal 1)

Metric 1.2: Effort Estimation Accuracy (EEA) = Actual Project effort / Estimated Project Effort

(goal 1)

Metric 2.1: Total Defect Containment Effectiveness (TDCE) = Number of pre-release defects /

(number of pre-release defects + number of post-release defects) (goal 2)

Metric 2.2: Phase Containment Effectiveness for phase I (PCEi) = number of phase I errors /

(number of phase I errors + number of phase I defects) (goal 2)

Metric 3.1: Failure Rate (FR) = number of failures / execution time (goal 3)

Metric 4.1a: In-Process Faults (IPF) = in-process faults caused by incremental software

development / assembly-equivalent delta source size (goal 4)

Metric 4.1b: In-Process Defects (IPD) = in-process defects caused by incremental software

development / assembly-equivalent delta source size (goal 4)

Metric 4.2a: Total Released Defects total (TRD total) = number of released defects / assembly-

equivalent total source size (goal 4)

Metric 4.2b: Total Released Defects delta (TRD delta) = number of released defects caused by

incremental software development / assembly-equivalent total source size (goal 4)

Metric 4.3a: Customer-Found Defects total (CFD total) = number of customer-found defects /

assembly-equivalent total source size (goal 4)

Metric 4.3b: Customer-Found Defects delta (CFD delta) = number of customer-found defects

found by incremental software development / assembly-equivalent total source size (goal 4)

Metric 5.1: New Open Problems (NOP) = total new post-release problems opened during the

month (goal 5)

Metric 5.2: Total Open Problems (TOP) = total number of post-release problems that remain

open at the end of the month (goal 5)

Metric 5.3: (mean) Age of Open Problems (AOP) = (total time post-release problems remaining

open at the end of the month have been open) / (number of open post-release problems remaining

open at the end of the month) (goal 5)

Metric 5.4: (mean) Age of Closed Problems (ACP) = total time post-release problem closed

within the month were open / number of post-release problems closed within the month (goal 5)

Metric 6.1: Cost of Fixing Problems (CFP) = dollar cost associated with fixing post-release

problems within the month (goal 6)

Metric 7.1a: Software Productivity total (SP total) = assembly-equivalent total source size /

software development effort (goal 7)

Metric 7.1b: Software Productivity delta (SP delta) = assembly-equivalent total source size /

software development effort (goal 7)

Alongside the goals and metrics, Motorola encouraged each software project/business unit to

create their own goals using the agreed-upon metrics based on how they were faring (baseline).

Not to say that one area would be more important than the other, but there were some that stood

out. One of the most important areas was the defect data area. Because if properly analyzed and

addressed, the information gathered would lead to significant improvement. Other metric areas

such as estimation accuracy and software problem-related really helped the units/projects get a

better grip on things.

To complete their engraining of software metrics into their company and its culture, Motorola

wanted to build a software metrics infrastructure, not only would this include the MWG, it would

go to include the Metrics User Group (MUG). This group was established as a forum to share

experiences of software metrics implementation. This group would meet every quarter to share

their experiences with things like tools to automate metrics. Also, there were many additions

and outputs created by this infrastructure. Bullet-pointed below are some of these changes

established by the MWG:

The clarification of metric definition and interpretation.

Two-day training workshops set up for training and consulting support of metric

implementation

A standardized requirements for the automation of data collection and analysis to give to

the tools group making them.

A criteria for the evaluation of metric tracking systems to choose one to buy if necessary.

Also, there’s a current list of all the metric tools available if anyone wants to use it.

Support for analysis of the metric data collected and some generic defect classification

scheme and their respective examples for using them to recommend improvements.

Guidelines to whichever unit is interested in creating a function responsible for software

metrics implementation.

A method for software measurement technology assessment and it provides feedback on

priority items that would help the implementation cause further.

Surveys involving customer satisfaction (from the software perspective).

One of MWG’s great contribution is their metric and process definition for software review and

test processes. A survey of the software engineers and managers showed that 67% of them use a

software review package made by MWG. The MWG was also faced with requests to centralize

all of the metric data but they decided that it was best to keep the data localized to which ever

entity needed it.

So how did software metrics turn out for Motorola? We’d have to say “pretty well!” Motorola’s

focus on improving their software quality through metrics turned in an astounding 50 time

reduction in software defect density within 3.5 years. That’s amazing. This reduction led

straight to cost reduction and cycle time reduction too! Not to boot that in 1988, the Motorola

Company was awarded the First Malcolm Baldrige national Quality award.

Example 2-

b. Hewlett - Packard

Another well-known successful metric-user is Hewlett-Packard (HP). One of HP’s great story is

how the use of software inspection metrics helped them achieve different levels of success in

finding defects early in the software development life cycle. It’s not surprising to say that by

finding those early defects, HP went on to save countless effort and money. This successful

project which they embarked on was called the sales and inventory tracking (SIT) project.

Though there were many various goals for this project, it all involved the same type of data:

computer dealer sales and inventory levels of HP products. Because that was the case, HP

decided to centralize all of the data for the SIT project to make it easy access.

HP followed their normal software life cycle of investigation, design, construction and testing,

implementation and release, and post-implementation review. But in this case, the metrics were

only being used for inspection and testing in the design, and construction and testing phase. In

the inspection process, HP broke it down into an 8 step process:

Step 0: Planning

Step 1: Kickoff

Step 2: Preparation

Step 3: Issue and Question Logging

Step 4: Cause Brainstorming

Step 5: Question and Answer

Step 6: Rework

Step 7: Follow-up

On a funny side note, HP found it much more effective to change the term “defect” to “issue and

question logging” because they realized calling something a defect makes the author/engineer

less receptive to the process of improvement.

So which metrics did HP decide to collect for this SIT project? There were three forms used to

collect the inspection metrics. First one was the inspection issue log. The second was the

inspection summary form. And the last was the inspection data summary and analysis form.

From these, HP’s team selected for key metrics and those were:

1. Number of critical defects found and fixed

2. Number of noncritical defects found and fixed

3. Total time used by inspections

4. Total time saved by inspections

Their testing process was broken down into test planning, unit testing, module testing, and

system testing. Of course not all of these are created equal and there’s only a finite amount of

resources so HP created their Master Test Plan after their assessment to know where to spend the

time and effort at. The system that was to be inspected would be broken down into logical

modules. Each module performed a different specific function. The Master Test Plan told of

how the primary and secondary features were to be tested against their respective product

specifications and how accurate the output of the data was, not to mention the given goals of

optimum speed and efficiency, ease of user use, and system supportability.

It was the risk assessment’s turn in their test planning phase. They took the modules and their

sub modules and assigned a risk number to it. That number involved their operational

importance to overall system functionality, complexity, and technical difficulty.

The unit testing was done on all the jobs and program streams. Each program and job stream

were counted as an individual entity. HP had the author of those entities to do their own testing

respectively to save money and time. Those authors then filled out the forms for further analysis.

The module testing was more of a double-checking type test. Since after the unit tests were

done, the module testing was there just to ensure that all of them worked together in a nice

cohesive manner. After that came the system testing. HP decided to conduct the system test

along with a pilot test at one of their HP dealers. The values gathered there was then inputted

into their central SIT database. The testing done had their own metrics, and there were two kids

of those used to measure their testing effort:

1) Total number of critical defects and

2) Total testing time for each phase

Which looked like this:

Figure 9: Testing Metrics

So how did HP decide if all of this was worth it or not? HP used an ROI model to measure the

value of inspections and testing in terms of how much time it saved and how quick it got it to the

market. Their reasoning, which is a sound one, is that there should be a value put on how much

time is saved if a defect was found in the inspection/testing phase than if it was found after at the

system test phase.

The model had these metrics to be gathered and analyzed:

Total Time Saved = Total Time Saved by inspection + Total Time Saved by Unit and Module

Testing

Total Time Used = Total Time used by inspection + OTtal Time used by Unit and Module

Testing

Prerelease ROI = Total Time Saved / Total Time Used

Inspection ROI = Total Time Saved by Inspection / Total Time used by inspection

Testing ROI = Total Time Saved by Unit and Module Testing / Total Time used by unit and

module testing

Total Time Saved by Inspection = time saved on critical defects + time saved on noncritical

defects

Total Time used by inspection = Inspection Time + Time to Fix and follow up for defect

resolution

Time Saved on Critical Defects = Black Box Testing Time x Number of Critical Defects – Total

Time Used

Mean Total Time to Rework (MTTR) = Time to Find defect + time to Fix defect + Time to

release to Production

Time Saved on Noncritical Defects = MTTR x Number of Noncritical defects

Total Time Saved = Time Saved on Critical defects

Total Time Used = Time to design and build a test + time to execute + time to find and fix a

defect

With those metrics, and the chart below:

Figure 10: Defect Summary

HP was able to conclude that their return on investment in the prerelease was a whopping 355%:

Figure 11: Time Data and Return

Thus another case of software metrics gone right. It is good to have the numbers back up the

practice.

Conclusion

We have just shown two examples of real world applications of software metrics in action. In

both cases we saw how successful they were and if done right, using software metrics can benefit

the project/company in quite a big way. We also see, though, that it takes a lot of effort and

commitment to make software measurement work. These two companies are actually in the

minority of real-life examples gone right. Today, there is still resistance to the implementation

and follow through of solid software metrics. It takes time, effort, cooperation, and a lot of

communication to get it done, but it is well worth it.

5. What are the advantages of using software metrics?

We have gathered a list of all of the possible advantages to software metrics which are as

following:

The basic advantage of software metrics become apparent during the process of

providing feedback to managers about the evolution and quality of the software

during several phases of software development life cycle.

In order to analyze and compare different programming languages, considering their

differences in characteristics, using software metrics is one of the best options.

One of the possible usages of software metrics is in the defining of software quality

specifications.

Software metrics can simply be used to enhance the comparison process between the

varieties of design approaches for software systems.

When it comes to checking software systems requirements in accordance with

specifications, one may benefit from software metrics during the verification process.

Software metrics are beneficial when you need a prediction about the amount of

effort needed to be put in the design and development of the software systems. It may

be possible by analyzing the old data related to some similar standard software

processes.

Software metrics can be used to measure the complexity of the software under

development.

When it comes to comparing and assessing the proficiency of individuals in software

development process, one can definitely take advantage of software metrics

approaches.

In some circumstances you need to divide the complicated modules in your system to

smaller pieces in order to reduce the percentage of software’s complexity. Using

software metrics can help you to decide when the correct time is to stop the division

process.

For appropriate utilization decisions resource managers need to use software metrics

to provide them a correct view.

In situations that you need to make important design decisions for software

developments issues related to maintenance costs you can basically take advantage of

software metrics to make design tradeoffs.

When you are trying to find the best solution for allocating your available resources

in order to test the developed software’s ode, software metrics seem such a useful

tool.

5.1 Limitations of Software Metrics

Software metrics are difficult to apply in some cases, and that increases the cost of

application.

Since software metrics are based on data that has been recorded in the past, its

authenticity is difficult to verify.

Software products can be managed, and their quality could be verified but Software

Metrics are not useful to evaluate the performance of the engineers.

Most of the applications of Software Metrics are based on assumptions, whose definitions

and derivations are not verified.

As you can see above, all the methods and tools of discussed are very old, and empirical.

Estimation is key in Software Metrics, and verifying the authenticity of these estimations

is very hard.

6. References –

1. Mishra, Suchitra. "Five Project Management Performance Metrics Key to Successful

Project Execution – Operational Excellence." Suchitra Mishra. N.p., 30 Sept. 2013. Web.

03 Dec. 2014.

2. Subramaniam, Anand. "Project Metrics & Measures." Project Metrics & Measures.

Slideshare.net, 28 July 2009. Web. 03 Dec. 2014.

3. "Earned Value Management." Wikipedia. Wikimedia Foundation, 29 Nov. 2014. Web.

02 Dec. 2014.

4. Jonathan, Louis. "Estimating the Value of Inspections and Early Testing for Software

Projects." Estimating the Value of Inspections and Early Testing for Software Projects

(1994): n. pag. Www.hp.com. Hp, 12 Dec. 1994. Web. 29 Nov. 2014.

5. M, Daskalantonakis. "A Practical View of Software Measurement and Implementation

Experiences within Motorola." IEEE Xplore. Software Engineering, IEEE Transactions

on (Volume: 18, 06 Aug. 2012. Web. 30 Nov. 2014.

6. Kan, Stephen. "Metrics and Models in Software Quality Engineering." Metrics and

Models in Software Quality Engineering. Addison-Wesley Longman Publishing, 12 May

2003. Web. 03 Dec. 2014.

7. Rex Black.” Metrics for Software Testing: Managing with Facts Part 4: Product Metrics”

http://www.rbcs-us.com/images/documents/Metrics-Article-4.pdf” Web. 03 Dec. 2014.