Test coverage

Tor Stålhane

What is test coverage

Let c denote the unit type that is considered – e.g. requirements or statements. We then have

Cc = (unitsc tested) / (number of unitsc)

Coverage categories

Broadly speaking, there are two categories of test coverage:

• Program based coverage. This category is concerned with coverage related to the software code.

• Specification based coverage. This category is concerned with coverage related to specification or requirements

Test coverage

For software code, we have three basic types of test coverage:

• Statement coverage – percentage of statements tested.

• Branch coverage – percentage of branches tested.

• Basic path coverage – percentage of basic paths tested.

Finite applicability – 1

That a test criterion has finite applicability means that it can be satisfied by a finite test set.

In the general case, the test criteria that we will discuss are not finitely applicable. The main reason for this is the possibility of “dead code” – e.g infeasible branches.

Finite applicability – 2

We will make the test coverage criteria that we use finitely applicable by relating them to only feasible code.

Thus, when we later speak of all branches or all code statements, we will tacitly interpret this as all “feasible branches” or all “feasible code”.

Statement coverage

This is the simplest coverage measure:

Cstat = percentage of statements tested

Path diagram

P2

P3

P1

S4

S3

S2

S1

<empty>

<empty>

<empty>Predicates

PathsP1 P2 P30 0 0 S40 0 1 S3, S40 1 0 S2, S40 1 1 S2, S3, S41 0 0 S1, S41 0 1 S1, S3, S41 1 0 S1, S2, S41 1 1 S1, S2, S3, S4

Branch coverage

Branch coverage tells us how many of the possible paths that has been tested.

Cbranch = percentage of branches tested

Basis path coverage

The basis set of paths is the smallest set of paths that can be combined to create every other path through the code.

The size of this set is equal to v(G) – McCabe’s cyclomatic number.

Cbasis = percentage of basis paths tested

Use of test coverage

There are several ways to use the coverage values. We will look at two of them coverage used

• As a test acceptance criteria

• For estimation of one or more quality factors, e.g. reliability

Test acceptance criteria

At a high level this is a simple acceptance criterion:

• Run a test suite.• Have we reached our acceptance criteria

– e.g. 95% branch coverage?– Yes – stop testing– No – write more tests. If we have tool that

shows us what has not been tested, this will help us in selecting the new test cases.

Avoid redundancy

If we use a test coverage measure as an acceptance criterion, we will only get credit for tests that exercise new parts of the code.

In this way, a test coverage measure will help us to

• Directly identify untested code

• Indirectly help us to identify new test cases

Fault seeding – 1

The concept “fault seeding” is used as follows:

• Insert a set of faults into the code

• Run the current test set

• One out of two things can happen:– All seeded faults are discovered, causing

observable errors– One or more seeded faults are not discovered

Fault seeding – 2

The fact that one or more seeded errors are not found by the current test set tells us which parts of the code that have not yet been tested – e.g. which component, code chunk or domain.

This info will help us to define the new test cases.

Fault seeding – 3

Fault seeding has one problem – where and how to seed the faults.

There are at least two solutions to this:

• Save and seed faults identified during earlier project activities

• Draw faults to seed from an experience database containing typical faults and their position in the code.

Fault seeding and estimation – 1

X

X

X

X

XX

X X

Realfault

Seededfault

Inputdomain

Test domain

Fault seeding and estimation – 2

We will use the following notation:

• N0: number of faults in the code

• N: number of faults found using a specified test set

• S0: number of seeded faults

• S: number of seeded faults found using a specified test set

Fault seeding and estimation – 3

X

X

X

XX

X

X X

Realfault

Seededfault

Inputdomain

Test domain

N0 / N = S0 / Sand thusN0 = N * S0 / SorN0 = N * S0 / max{S, 0.5}

Capture – recapture One way to get around the problem of fault

seeding is to use whatever errors are found in a capture – recapture model.

This model requires that we use two test groups.

• The first group finds M errors

• The second group finds n errors

• m defects are in both groups

m / n = M / N => N = Mn / m

Capture – recapture

No Customer 1 Customer 2 Common N

1 25 36 17 52

2 29 30 11 79

3 23 21 13 37

4 0 - 1 0 - 2 0 0 - 4

Output coverage – 1 All the coverage types that we have looked

at so far have been related to input data. It is also possible to define coverage based

on output data. The idea is as follows:• Identify all output specifications• Run the current test set• One out of two things can happen:

– All types of output has been generated– One or more types of output have not been

generated

Output coverage – 2

The fact that one or more types of output has not been generated by the current test set tells us which parts of the code that have not yet been tested – e.g. which component, code chunk or domain.

This info will help us to define the new test cases.

Output coverage – 3

The main challenge with using this type of coverage measure is that output can be defined at several levels of details, e.g.:

• An account summary

• An account summary for a special type of customer

• An account summary for a special event – e.g. overdraft

Specification based coverage – 1

Specification based test coverage is in most cases requirements based test coverage.

We face the same type of problem here as we do for output coverage – the level of details considered in the requirements.

In many cases, we do not even have a detailed list of requirements. This is for instance the case for user stories frequently used in agile development.

Specification based coverage – 2

The situation where this is most easy is for systems where there exist a detailed specification, e.g. as a set of textual use cases.

Use case name (Re-)Schedule train

Use case actor Control central operator

User action System action

1. Request to enter schedule info

3. Enter the schedule (train-ID, start and stop place and time, as well as timing for intermediate points)

5. Confirm schedule

2. Show the scheduling screen

4 Check that the schedule does not conflict with other existing schedules; display entered schedule for confirmation

Quality factor estimation

The value of the coverage achieved can be used to estimate important quality characteristics like

• Number of remaining fault

• Extra test time needed to achieve a certain number of remaining faults

• System reliability

Basic assumptions

In order to use a test coverage value to estimate the number of remaining faults, we need to assume that:

• All faults are counted only once.

• Each fault will only give rise to one error

• All test case have the same size

Choice of models – errors We will use the notation

• N(n): number of errors reported after n executions

• N0: initial number of faults

There exists more than a dozen models for

N(n) = f(N0, n, ). It can be shown that when we have N(n) -> N0, we have

N(n) = N0(1 – exp(-n)]

Choice of models – coverage (1)

We will use the notation

• C(n): the coverage achieved after n tests

• C0: final coverage. We will assume this to be 1 – no “dead” code.

Further more, we will assume that

C(n) = 1 / [1 + A exp( – an)]

Choice of models – coverage (2)

n

1 / (1 + A)

C(n)

1

Parameters We need the following parameters:

• For the N(n) model we need– N0: total number of defects

: mean number of tests to find a defect

• For the C(n) model we need– A: first test coverage– a: coverage growth parameter

All four parameters can be estimated from observations using the Log Likelihood estimator.

Final modelWe can use the C(n) expression to get an

expression for n as a function of C(n).

By substituting this into the N(n) expression we get an estimate for the number of remaining fault as a function of the coverage:

a

nAC

nC

N

nNN

)(

)(1)(

0

0