THE BUILDING-BLOCK ALGORITHM: A NEW METHOD FOR CONSTRUCTING

COVERING ARRAYS

by

Andreas Hundseid Ronneseth

A Thesis Presented in Partial Fulfillmentof the Requirements for the Degree

Master of Science

ARIZONA STATE UNIVERSITY

May 2006

THE BUILDING-BLOCK ALGORITHM: A NEW METHOD FOR CONSTRUCTING

COVERING ARRAYS

by

Andreas Hundseid Ronneseth

has been approved

April 2006

APPROVED:

, Chair

Supervisory Committee

ACCEPTED:

Department Chair

Dean, Division of Graduate Studies

ABSTRACT

Software systems consist of multiple interchangeable elements that interact with

each other. As the number of components increases, the size of a test suite that covers all

interactions grows exponentially. Test suites that cover all pairs of component interactions

have been proposed to mitigate this cost and still provide adequate fault detection. This

thesis proposes a new greedy test-generation method, called the Building-Block Algorithm

(BBA), that allows system testers to test a subset of the components in unit test and then

later incorporate the same set of test configurations in integration and system tests. It

uses smaller test suites to build larger test suites and often yields the smallest test suite

when compared to other greedy approaches. In one instance, 1124, a new lower upper

bound of 229 is reported. The results show that combining smaller test suites as building

blocks generated test suites faster than similar methods, thus reducing the cost of test suite

development.

iii

To my parents

iv

ACKNOWLEDGMENTS

First, I would like to thank Dr. Charles J. Colbourn for his excellent advice and

direction through this past year. His knowledge in this area is breathtaking and his insight

helped shape this thesis. With his schedule, I am amazed how he takes the time and care

to help each student, including myself. He is truly unique.

Dr. David C. Pheanis guided me through my undergraduate years, and taught

me computer engineering, personal finance, English grammar, and helped me obtain both

scholarships and my first job. I owe him my deep gratitude.

Also, I would like to thank Dr. James S. Collofello and Dr. Goran Konjevod for

serving on my supervisory committee. Dr. Renee C. Turban introduced me to this topic

during the course of my honors thesis and also provided the implementation of the Deter-

ministic Density Algorithm (DDA). Dr. Myra B. Cohen at University of Nebraska–Lincoln

shared her simulated annealing results. Thank you!

I would like to thank my family and especially my parents. They pushed me to come

to the United States in the first place, and I have not regretted that for even a second. Along

the way they have offered their love and advice from thousands of miles away. Without my

grandparents here in Arizona, I probably would have starved to death. I love you all.

Finally, my soulmate Melissa makes every day a little brighter!

v

TABLE OF CONTENTS

Page

LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

CHAPTER 1 INTRODUCTION AND MOTIVATION . . . . . . . . . . . . . 1

CHAPTER 2 TECHNICAL BACKGROUND . . . . . . . . . . . . . . . . . 5

1. Combinatorial Models . . . . . . . . . . . . . . . . . . . . . . . 5

2. One-Row-at-the-Time Greedy Constructions . . . . . . . . . . . . . . 9

2.1. Automatic Efficient Test Generator . . . . . . . . . . . . . . . 9

2.2. Test Case Generator . . . . . . . . . . . . . . . . . . . . . . 10

2.3. Deterministic Density Algorithm . . . . . . . . . . . . . . . . . 12

3. IPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4. Combinatorial Constructions . . . . . . . . . . . . . . . . . . . . . 16

4.1. Orthogonal-Array Construction . . . . . . . . . . . . . . . . . 16

4.2. TConfig . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

5. Heuristic Search . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5.1. Simulated Annealing . . . . . . . . . . . . . . . . . . . . . . 20

5.2. Tabu Search . . . . . . . . . . . . . . . . . . . . . . . . . 20

6. Criteria for Comparing Algorithms . . . . . . . . . . . . . . . . . . 21

CHAPTER 3 THE BUILDING-BLOCK ALGORITHM AND DENSITY IMPLE-MENTATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1. An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2. The Building-Block Algorithm . . . . . . . . . . . . . . . . . . . . 27

3. Density Implementation . . . . . . . . . . . . . . . . . . . . . . . 29

vi

Page

3.1. Building-Block Density . . . . . . . . . . . . . . . . . . . . . 31

3.2. Building Blocks with Different Sizes . . . . . . . . . . . . . . . 33

3.3. Don’t-Care Positions . . . . . . . . . . . . . . . . . . . . . . 33

3.4. Implementation Summary . . . . . . . . . . . . . . . . . . . 34

4. Comparison of the Building-Block Algorithm and Other Methods . . . . 34

CHAPTER 4 COMPUTATIONAL RESULTS . . . . . . . . . . . . . . . . . 36

1. Different Building Blocks . . . . . . . . . . . . . . . . . . . . . . 37

1.1. Building Blocks from Bose’s Algorithm . . . . . . . . . . . . . . 37

1.2. Building Blocks from Heuristic Search . . . . . . . . . . . . . . 39

1.3. Building Blocks from DDA . . . . . . . . . . . . . . . . . . . 40

2. Comparison with Published Results . . . . . . . . . . . . . . . . . . 41

3. Execution Time . . . . . . . . . . . . . . . . . . . . . . . . . . 43

CHAPTER 5 FUTURE WORK AND CONCLUSION . . . . . . . . . . . . . 45

1. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

2. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

vii

LIST OF TABLES

Table Page

1. A modular ticket reservation system . . . . . . . . . . . . . . . . . 2

2. Test suite for ticket reservation system . . . . . . . . . . . . . . . . 2

3. Orthogonal array OA(4; 3, 2, 2) . . . . . . . . . . . . . . . . . . . . 6

4. Covering array CA(6; 2, 4, 2) . . . . . . . . . . . . . . . . . . . . . 7

5. Orthogonal Latin squares of order three . . . . . . . . . . . . . . . . 16

6. Construction of OA(9; 4, 3, 2) using Bose’s algorithm . . . . . . . . . . 18

7. Covering arrays A and B, both MCA(6; 2, 2, (2131)) . . . . . . . . . . 25

8. Partial covering array PMCA(6; 2, 4, (2232)) . . . . . . . . . . . . . . 26

9. Covering array MCA(N ; 2, 4, (2232)) . . . . . . . . . . . . . . . . . 26

10. BBA (using Bose’s algorithm) versus DDA . . . . . . . . . . . . . . 38

11. Mixed-level BBA (using Bose’s algorithm) versus DDA . . . . . . . . . 39

12. BBA (using tabu search) versus DDA . . . . . . . . . . . . . . . . . 40

13. BBA (using simulated annealing) versus DDA . . . . . . . . . . . . . 40

14. BBA (using DDA) versus DDA . . . . . . . . . . . . . . . . . . . 41

15. Comparison with Published Results . . . . . . . . . . . . . . . . . . 42

16. Building blocks for BBA for published results . . . . . . . . . . . . . 43

17. Execution time (BBA versus DDA) in seconds . . . . . . . . . . . . . 44

viii

LIST OF FIGURES

Figure Page

1. Concept figure of IPO growing the test suite horizontally and then vertically 15

2. Pseudo-code for the Building-Block Algorithm . . . . . . . . . . . . . 30

ix

CHAPTER 1

INTRODUCTION AND MOTIVATION

Software testing is an important part of the software development process. Software

testing is costly and time-consuming and therefore much research has gone into selecting the

appropriate test cases. As software is generally becoming more complex, it is also becoming

more modular in an attempt to reduce the complexity. A significant number of faults can

be attributed to unexpected interactions between software modules. Often, these faults are

caused by bad interactions between a relatively low number of modules. Kuhn and Reilly

studied 171 bugs for the Apache webserver and found that more than 70 % of the bugs

could be attributed to a bad interaction between only two modules. Interactions between

four or fewer modules caused 96 % of all the bugs in the study. Thus, we focus on creating

test sets that cover at least all t-way interactions where t is a small number, generally in

the range 2 ≤ t ≤ 6 [1]. Pairwise testing means t = 2, i.e., we test every pair of possible

interactions in a system.

A small example illustrates this approach. A software company is building a modular

software system for a travel agency built with four components. The software system has

to interact with three major airlines’ booking systems. A customer can book their flight

through the web, through an automated phone system, or by contacting an agent that uses

a specialized agent system. The travel agency issues the tickets through a web system,

through an e-ticket system, or by sending the customer a paper ticket. Finally, the travel

2

Table 1. A modular ticket reservation system

Airline Database Interface Issue Method

US Air MySQL Web Web TicketSouthwest Oracle Phone E-Ticket

KLM Postgre SQL Agent System Paper Ticket

Table 2. Test suite for ticket reservation system

Test No. Airline Database Interface Issue Method

1 US Air MySQL Web Web Ticket2 US Air Oracle Phone E-Ticket3 US Air Postgre SQL Agent System Paper Ticket4 Southwest MySQL Agent System E-Ticket5 Southwest Oracle Web Paper Ticket6 Southwest Postgre SQL Phone Web Ticket7 KLM MySQL Phone Paper Ticket8 KLM Oracle Agent System Web Ticket9 KLM Postgre SQL Web E-Ticket

agency uses three different database systems. Thus, we have a system of four components,

each with three different configuration possibilities as shown in Table 1.

If the software company wanted to exhaustively test this system, they would have to

test 34 = 81 different configurations. Instead we can employ pairwise testing and cover all

pairwise interactions in nine test cases as shown in Table 2. This offers substantial savings,

and for systems with more components and configurations, the savings become even more

apparent. Exhaustive testing for a system with 20 components each with three options

would require 320 or over three billion test cases. Pairwise testing on the same system

would require at most 15 test cases [2].

An interaction test can be modeled as a combinatorial object called a covering ar-

ray, which we define more formally later. Many methods have already been proposed to

3

construct these covering arrays. Several methods are direct constructions, such as TConfig

and Bose’s algorithm [3, 4]. These types of techniques do well on a limited set of inputs.

Heuristic-search methods, such as tabu search and simulated annealing have been used to

find some of the smallest known covering arrays [2, 5, 6]. Although the heuristic-search

methods can handle a wide array of inputs, this approach often take months of computa-

tion time. Greedy algorithms such as AETG, TCG, IPO, and DDA can construct a covering

array quickly, but with some loss of accuracy versus heuristic-search methods [7, 8, 9, 10].

All these methods are introduced in more detail later.

We present a new greedy method for constructing covering arrays called the Building-

Block Algorithm (BBA). Because this method uses smaller covering arrays to construct

larger covering arrays, it enables testers to reuse unit tests in integration or system tests.

An example illustrates the advantages of this attribute next.

Software systems are often built in subsystems and then integrated later in the

development cycle. For example, subsystem one has five components and subsystem two

has four components. The software tester tests subsystem one using automated pairwise

interaction testing. Subsystem two is tested the same way, although with a different set

of interaction tests. In integration testing, components in subsystem one interact with

individual components in subsystem two so the tester needs a test set that covers all pairwise

interactions between the nine components. Instead of starting over in integration testing,

BBA enables the tester to reuse all the automated tests from the unit test. The tester

also has to write a few more automated scripts to cover the additional interactions between

individual components in subsystem one and two. However, this is far less effort than

rewriting all the test scripts.

In this thesis we first introduce some combinatorial definitions in Chapter 2. Using

4

these definitions, we discuss some existing methods for constructing covering arrays and

criteria for evaluating different methods. Chapter 3 builds arguments that lead to the

development of the Building-Block Algorithm. An implementation of the algorithm is also

discussed in this chapter. The results in Chapter 4 show that BBA is generally as accurate or

more accurate and faster than existing greedy techniques. Finally, the last chapter discusses

future endeavors and draws some final conclusions.

CHAPTER 2

TECHNICAL BACKGROUND

The introduction of AETG in the mid-90s [7] sparked an increased interest in interac-

tion testing. The combinatorial, testing, and algorithm communities have all contributed to

the area. This chapter summarizes some of the researchers’ advances, especially in the last

decade. First, we introduce some combinatorial models, which help us describe the problems

in interaction testing. We then cover relevant algorithms for constructing test suites. The

algorithms fall into three main groups: Heuristic search, combinatorial constructions, and

greedy algorithms. The chapter ends with a discussion of criteria for comparing methods

of test suite generation.

For additional background information, we refer the reader to [11], which gives a

thorough review of research related to interaction testing and contains 140 references.

1. Combinatorial Models

An orthogonal array OAλ(N ; k, v, t) is an N × k array on v symbols such that every

N × t subarray contains all ordered subsets of size t from v symbols exactly λ times [12].

In other words, each tuple (or pair for t = 2) is covered exactly λ times. Orthogonal arrays

have the property that λ = Nvt . Table 3 is an orthogonal array OA(4; 3, 2, 2).

Mandl used orthogonal arrays for interaction testing of an Ada compiler [13].

Williams and Probert also did pair-wise testing using orthogonal arrays on a network in-

6

Table 3. Orthogonal array OA(4; 3, 2, 2)

1 a x1 b y2 a y2 b x

terface [3]. Orthogonal arrays provide the optimal test suites for certain values of v and

k when λ equals 1. However, for other values of v and k, orthogonal arrays only exist for

λ greater than 1. These orthogonal arrays may create unacceptably large test suites. The

literature has therefore largely moved away from orthogonal arrays for interaction testing

and instead concentrated on the less restrictive covering arrays, which require that every

tuples are covered at least once.

A covering array, CAλ(N ; t, k, v), is an N × k array with the exact same definition

as an orthogonal array, except it contains all ordered subsets from v symbols of size t at

least λ times. When λ equals 1, we omit λ and use the shorthand notation CAλ(N ; t, k, v).

In such a covering array we call t the strength, k the degree, and v the order [14]. The

covering array number, CAN(t, k, v) is the number of rows in an optimal covering array,

i.e., the minimum number of rows. If an orthogonal array exists for a set of parameters with

λ = 1 , it always yields CAN(t, k, v) since an orthogonal array is always a covering array

(but not vice versa). For example, an orthogonal array of size 9 exists for OA(9; 3, 2, 2),

thus CAN(2, 3, 3) equals 9. It also follows that the covering array number is always larger

or equal to vt (or λ × vt if λ is greater than 1).

Covering arrays as a model naturally map to the test suites. Each row of the covering

array represents a test case. The k columns represent the components or factors in the

system under test (SUT). The value in a column of a row represents the configuration or

7

Table 4. Covering array CA(6; 2, 4, 2)

Covering Array

Test f1 f2 f3 f4

1 a 1 x 32 b 2 y 33 a 2 x 44 b 1 y 45 a ⋆ y ⋆6 b ⋆ x ⋆

level for the factor for this test case. As shown in Table 4, we label the columns or factors

f1, f2, . . . , fk and each row 1, 2, . . . ,N .

The definition of a covering array does not require each column in each row to

contain a value. For example, say we have constructed the first five rows of covering array

CA(6; 2, 4, 2) in Table 4. At this point only one pair is uncovered overall, between factors

f1 and f3. The pair is {b, x}, so set column one in test case six to ’b’ and column three to

’x’. The covering array is now complete, and we can arbitrarily fix column two to ’1’ or

’2’ and column four to ’3’ or ’4’. Columns two and four of row six are don’t-care positions,

denoted by a ⋆. These positions can be fixed to any level of the particular factor without

changing the covering array’s property of covering every tuple at least once.

Although covering arrays give a natural mapping to test suites, covering arrays lack

the properties necessary to handle every testing situation. Specifically, covering arrays can

only handle systems where all the factors have the same number of levels. For example,

a system may have four factors with three levels and two factors with five levels, and we

should assume this is the more frequent case. Several authors have suggested that a mixed-

level covering array can handle this situation (see [15, 5, 16]). Cohen et al. [14] defined

good notation for these types of arrays and we use that notation in this thesis.

8

A mixed-level covering array, MCAλ(N ; t, k, (v1, v2, . . . , vk)), is an N × k array on

v symbols, where v =∑k

i=1vi, with the following properties:

1. Each column i (1 ≤ i ≤ k) contains only elements from set Si with |Si| = vi.

2. The rows of each N × t sub-array cover all t-tuples of values from the t columns at

least λ times.

We omit λ from the notation when λ equals 1, which is the normal occurrence.

In addition, the notation combines factors with an equal number of levels. For

example, ninety factors with ten levels can be combined to 1090. Thus, a multi-level cov-

ering array with ten factors, three with two levels and seven with four levels, can be writ-

ten as MCA(N ; t, 10, 2347). More formally, we can write a multi-level covering array as

MCA(N ; t, (wr1

1wr1

1. . . wrs

s )), where k =∑s

i=1ri and (wj : 1 ≤ j ≤ k) ⊆ {v1, v2, . . . , vk}

[10].

Finally, we introduce a combinatorial object called the partial mixed-level covering

array, PMCA(n/N ; t, k, (v1, v2, . . . , vk)), an n×k array on a maximum of v symbols, where

v =∑k

i=1vi, and the rows of each n×t subarray overall does not cover all t-tuples, although

some subarrays may. In other words, a partial mixed-level covering array (PMCA) is a set of

test rows that does not cover all the t-tuples yet. If you add N−n rows, the PMCA becomes

a complete mixed-level covering array. We generally do not know the value of N up front.

Analogously, a partial covering array, PCA(n/N ; t, k, v), shares the same definition as the

PCMA for a covering array. We use PMCAs in intermediate steps of the Building-Block

Algorithm (BBA) that we develop.

9

2. One-Row-at-the-Time Greedy Constructions

There are many greedy algorithms to construct covering arrays, and all of them

favor execution speed over accuracy. Three of the algorithms, the Automatic Efficient Test

Generator system (AETG) [7], Test Case Generator (TCG) [8], and Deterministic Density

Algorithm (DDA) [10], all build the covering arrays one row at the time. The general

approach is as follows:

1. Start with an empty test row, i.e., all of the factors are unfixed.

2. Select a factor to treat next.

3. Fix the factor you selected in step two to a level within the factor.

4. Repeat steps two and three until all the factors are fixed.

5. Add the complete test row to the test suite.

These steps are repeated N times until all the t-tuples are covered. Note that these algo-

rithms are strictly greedy; none of the algorithms undo a decision later.

This section gives a brief overview of AETG and TCG, and a more thorough overview

of DDA as it plays a larger role in the Building-Block Algorithm (BBA).

2.1. Automatic Efficient Test Generator. AETG is a random greedy algorithm

[7] that constructs multi-level covering arrays, MCA(N ; t, k, (v1, v2, . . . , vk)), one row at the

time. It generates M test-case candidates and then includes the test case that covers the

most new pairs in the test suite. If AETG has selected r test cases, next test case candidate,

r + 1 or T , is selected as follows:

1. Select a factor s and a value ν that appears in the most number of uncovered pairs.

Fix the level in the test case, i.e., Ts = fs,ν.

10

2. Let o2, o3, . . . , ok be a random order of the remaining k − 1 factors. The first factor

in the order is o1 = fs.

3. Say we have fixed a level for the first j ordered columns of T . Let fi be the factor and

fi,w be the level for the factor selected, for 1 ≤ i ≤ j and 1 ≤ w ≤ vi. For each level in

fj+1, count the number of uncovered pairs that appear with fj+1,w and the value at

Ti, for 1 ≤ i ≤ j. Fix column j +1 to level ν that appears with the maximum number

of uncovered pairs., i.e., Tj+1 = fj+1,ν. Each level of each factor is only considered

once per candidate test case. Also, any level is only compared to the previous j fixed

columns, o1, o2, . . . , oj , without considering what pairs may be covered ahead [7].

Although there is a large number of permutations of ordering the factors (k! to be

exact), AETG attempts to find at least one order that yields a higher number of newly

covered pairs than an average test row. The number of test cases were observed to grow

logarithmically in k for M = 50. The authors proved that a greedy algorithm could guar-

antee logarithmic growth if each new test case covered at least N/v2 new pairs, where N is

the remaining uncovered pairs and v is the number of levels in each factor. The algorithm

does not provide a guarantee that it will cover the maximum possible number of new pairs.

In fact, Colbourn et al. later proved that selecting a test case covering at least p pairs is

NP-complete [10]. We discuss this result in more detail later as we discuss DDA. The point

here is that AETG cannot guarantee logarithmic growth, only observe it.

2.2. Test Case Generator. Tung and Aldiwan introduced the TCG algorithm in

2000 [8], not long after the introduction of AETG. TCG’s main advantage over AETG is

that it is deterministic. The main steps of the algorithm are as follows:

1. The algorithm orders the factors decreasingly according to the of number levels for

11

each factor. Assign i to vi, where 1 ≤ i ≤ k such that v1 ≥ v2 ≥ · · · ≥ vk. Let M be

the maximum number of test-case candidates for each row, and set M = v1.

2. Generate as many as M test-case candidates. If the first j factors have been selected,

where 1 ≤ j ≤ k − 1, we select the (j + 1)th factor as follows: For each level in vj+1,

count the number of new uncovered pairs the level appear in with the previously j

selected levels. Select the level with the largest number of newly covered pairs. If

multiple levels cover the same number of pairs, add both partial arrays to the pool of

test-case candidates.

3. Select the test-case candidate that covers the most new pairs from the M candidates,

up to M candidates.

4. Repeat step two and three until all the pairs are covered.

The main difference between AETG and TCG is the factor ordering; AETG ran-

domly orders the factors (after fixing the very first factor) M times, where M is usually

fifty, and TCG deterministically orders the factors and then saves up to M ties, where M is

equal to the largest number of levels in a factor. However, as the two algorithms construct

each test row, they use a different approach when fixing a the factor involved in the most

number of uncovered pairs before fixing the factors involved in fewer uncovered pairs. This

heuristic merits some further discussion.

AETG handles this by first selecting a level for the factor with the most uncovered

pairs and then randomly ordering the rest of the factors a number of times, hoping to find a

test-case candidate that covers as close to the maximum number of pairs as possible. Note

that the first factor changes as test cases are added to the test suite since the factors are

covered unevenly.

12

TCG deals with this problem by initially defining an order that treats the factors

with the higher number of uncovered pairs first. However, this order does not change

during the course of the algorithm and thus ignoring the possible gain one would have if

the treatment of the factors were reordered after each test case.

Perhaps recognizing these heuristics, Cohen et al. [10] introduced the Deterministic

Density Algorithm (DDA) in 2004. This algorithm dynamically reorders the treatment of

factors as it adds new test rows.

2.3. Deterministic Density Algorithm. DDA constructs covering arrays one

row at the time, so the construction is similar to that of AETG and TCG. However, it adds

two more heuristics we call lookahead and dynamic factor ordering [10].

AETG and TCG selects a level based on the number of new pairs it would cover.

Specifically, when selecting a level for the ith factor, they only consider the interactions with

i−1 other factors. One could ask whether the interaction with the k−i other factors matter?

For example, if there are 100 factors, and we are currently selecting a level for the tenth

factor, the interaction with the last 90 is arguably more important than the interaction with

the first nine. DDA develops a heuristic that lets each greedy step look ahead and consider

the later interactions when selecting a level.

DDA also dynamically selects which factor to treat next. The factor is selected

based on the number of pairs it may cover with the treated factors and the number of

potential pairs it may cover with untreated factors. AETG calculates a similar heuristic for

the treatment of the first factor of each row (but uses a random order after that). TCG

also does this for the first few test rows, since it treats the factors with the most levels first,

but it never changes the order. This especially poses a problem for a covering array with

a constant number of levels. The pairs between the first factors get priority over the pairs

13

between the latter factors in the ordering. DDA consider the interactions with the later

factors when it selects which factor to treat next.

Let us develop the notion of density. Consider a covering array with k factors and

a factor i has vi number of levels. A local density between factors i and j is defined as

δi,j =ri,j

vivj, where ri,j is the number of uncovered pairs between the two factors. In plain

terms, the local density is the ratio of uncovered pairs versus total pairs between two factors.

Let the factor density for factor s be defined as

δs =∑

1≤i≤ki6=s

δi,s

At every step we treat the factor s with the highest density next.

To treat a specific factor we must select a level as the configuration for this compo-

nent in the test case. Let ρi,s,σ be the uncovered pairs between level σ in factor s and all

the levels in factor i times 1

vi. In other words, ρi,s,σ is the ratio of uncovered pairs between

a level and all levels in another factor and the initial number of pairs between these. Thus,

level density, ρs,σ, can be written as

ρs,σ =∑

1≤i≤ki6=s

ρi,s,σ

We use the level density as a heuristic to select a level when we treat a factor. The level

with the highest level density is added to the test case. Note that ρi,s,σ is equal to 1 if factor

i already has a selected level.

The densities as defined have some problems in practice, especially when they are

calculated for mixed covering arrays. Each pair between two factors, each with three levels,

contribute 1

9to the local density. A pair between two ten-level factors contribute 1

100to

the local density. The pairs in the first example are easier to cover than the pairs in the

14

second example, so we must adjust the density so each pair contribute an equal amount

to the local density. Let v2max be the largest number of levels for any factor. When we

calculate the local density for two factors, each with more than two levels, we replace the

denominator by vmax. When one of the factors have only one level or is already treated, we

use the denominator vmax. If both of the factors have only one level or are already treated,

we use 1 as the denominator.

The major steps of DDA then become:

1. Create a test row using one level from each of the k factors.

(a) Select factor s with the highest factor density δs.

(b) Select a level σ from fs with the highest level density ρs,σ and assign it to column

s of the current test row.

(c) Repeat steps (a) and (b) k times until there are no untreated factors.

2. Add the test row from step 1 to the test suite.

3. Repeat steps 1 and 2 until there are no uncovered pairs.

Unlike AETG and TCG, DDA does not create multiple test-case candidates per test

row in the final test suite. As the authors suggest, the time/accuracy tradeoff of keeping

several candidates merits further exploration [10].

3. IPO

Unlike the previous three greedy algorithms, IPO does not generate a test suite one

row at the time [9]. Instead, it first grows the partial test suite horizontally by adding one

factor at a time. In each step the test suite may also grow vertically if needed as shown in

Figure 1.

15

Figure 1. Concept figure of IPO growing the test suite horizontally and then vertically

IPO initially starts out by generating a pairwise test suite for the first two factors.

This test suite has size v1 × v2. Then it adds the third factor as a third column of the test

suite (see 3a in Figure 1), and selects levels for the third factor for each of the test rows

from the previous step. If there are still uncovered pairs between the first two factors and

the third factor, IPO vertically extends the test suite with additional rows to cover these

(see 3b in Figure 1). The horizontal and vertical growth continues until the algorithm has

included all the k factors, and the result is a complete covering array. The factors are also

added in a decreasing order of the number of levels, just like TCG, and IPO appears to have

some trouble with covering arrays where all the factors have the same number of levels.

The authors of IPO point out several advantages of IPO’s approach. Consider a

system with k factors, and the tester has developed a pairwise test suite for the system.

Later, the system grows, and the tester must add five factors. IPO can inherently extend the

current test suite to accommodate for these additional factors by using horizontal growth

and thus reuse the original test suite In another scenario, a test also has a complete test

16

Table 5. Orthogonal Latin squares of order three

0 1 21 2 02 0 1

0 2 11 0 22 1 0

(0,0) (1,2) (2,1)(1,1) (2,0) (0,2)(2,2) (0,1) (1,0)

suite. Later, the tester adds additional levels to some of the factors. IPO’s vertical growth

can also handle this situation.

As we show later, the Building-Block Algorithm has similar abilities to add both

factors and levels.

4. Combinatorial Constructions

Mathematicians have worked on different forms of combinatorial constructions

for centuries. Bose introduced a method for constructing orthogonal arrays of the size

OA(n2;n + 1, n, 2) [4]. More recently, Williams presented a more general algorithm called

TConfig that constructs covering arrays mainly from orthogonal arrays [3].

4.1. Orthogonal-Array Construction. A Latin Square is an n × n matrix on n

symbols such that each symbol appears exactly once in each row and each column. Two

Latin squares L1 = |aij | and L2 = |bij | of order n on n symbols 0, 1, . . . , n−1 are orthogonal if

every ordered pair occurs exactly once among the n2 pairs (aij , bij), where i = 0, 1, . . . , n−1

and j = 0, 1, . . . , n− 1. Table 5 shows two such Latin squares of order 3 and the nine pairs

these squares cover.

We can now start to see how Latin squares relate to orthogonal and covering arrays.

Each n2 ordered pair on the n symbols appear exactly once, which coincides with the exact

definition of an orthogonal array where every N × t subarray contains all the ordered pairs.

The example in Table 5 cover only two factors. We can assign row one through nine to each

17

of the elements by traversing from right to left and top to bottom over the elements in the

Latin square. The result would be orthogonal array OA(9; 2, 3, 2).

A set of mutually orthogonal Latin squares (MOLS) is a set of Latin squares where

every pair of squares are orthogonal. If we can find larger set of MOLS, we also can find

larger orthogonal arrays. A complete set of MOLS has n−1 Latin squares; in fact, no larger

set of MOLS exists. This paper uses one method that can construct OA(n2;n + 1, n, 2) if

n is a prime or a power of a prime. However, this would require a set of MOLS larger

than n − 1. Bose found he could add two additional squares that are mutually orthogonal

with all the other squares, but not Latin squares themselves. The technique uses properties

of Galois fields and calculates the orthogonal array directly using Galois-field arithmetic.

We simply present the algorithm here, but refer the reader to [4] for more details on this

construction.

Consider constructing an orthogonal array OA(v2; v + 1, v, 2), where v is a prime

power. The comma-separated columns of row (i, j), where 0 ≤ i < v and 0 ≤ j < v, may

be calculated using the following equations

(i, j) : {i, j, i + j, i + 2 · j, i + 3 · j, . . . , i + (v − 1) · j}

and each calculation is modulo v. We get N equals v2 as we loop over all values for i and

j, as shown in Table 6 for orthogonal array OA(9; 4, 3, 2). The last two columns of the

orthogonal array are Latin squares (as in Table 5) and form a complete set of MOLS. The

first two columns are mutually orthogonal with each other and the two other columns, but

they are not Latin squares.

If an orthogonal array exists with λ = 1, it always gives the lower bound for the

covering-array number, CAN(t, k, v), and provides a lower bound for other, seemingly

smaller, covering arrays. Consider an orthogonal array for k factors and v levels. You

18

Table 6. Construction of OA(9; 4, 3, 2) using Bose’s algorithm

(i, j) i j i + j i + 2 · j OrthogonalRow 0 ≤ i, j < 3 (mod 3) (mod 3) (mod 3) (mod 3) Array

1 (0, 0) 0 0 0 + 0 0 + 2 · 0 0 0 0 02 (0, 1) 0 1 0 + 1 0 + 2 · 1 0 1 1 23 (0, 2) 0 2 0 + 2 0 + 2 · 2 0 2 2 14 (1, 0) 1 0 1 + 0 1 + 2 · 0 1 0 1 15 (1, 1) 1 1 1 + 1 1 + 2 · 1 1 1 2 06 (1, 2) 1 2 1 + 2 1 + 2 · 2 1 2 0 27 (2, 0) 2 0 2 + 0 2 + 2 · 0 2 0 2 28 (2, 1) 2 1 2 + 1 2 + 2 · 1 2 1 0 19 (2, 2) 2 2 2 + 2 2 + 2 · 2 2 2 1 0

can remove up to k − 1 columns from the orthogonal array and still have a covering array

with 2 to k columns with a minimum number of rows. Similarly, you can remove levels from

k−2 factors s.t. at least two factors still have v levels, and still have a CA with the minimum

number of rows. The levels you remove are replaced by don’t-care positions. We call the

process of removing factors and levels a reduction of the orthogonal (or covering) array. For

example, OA(9; 4, 3, 2) in Table 6 can reduce to MCA(9; 2, 4, 2232) or CA(9; 2, 3, 3). We

use this in several results in Chapter 4 as building blocks for the building-block algorithm.

Further reductions, removing more than k − 2 factors or levels from more than k − 2 fac-

tors, result in a covering array where certain rows contain k − 1 or k don’t-care positions.

These rows can be removed, and thus we no longer have the same covering-array-number

guarantee, and a smaller covering array must exist.

4.2. TConfig. Williams and Probert use building blocks to construct covering ar-

rays in an algorithm they label TConfig [16]. The building blocks are orthogonal arrays and

reduced orthogonal arrays. In addition, TConfig utilizes some building blocks not based

19

on orthogonal arrays. These building blocks act as glue to cover the pairs not covered by

combining the orthogonal arrays.

Their basic idea is as follows: Place two identical orthogonal arrays side by side. If

the orthogonal array has k factors, this partial covering array has 2 · k factors. Each pair

within each orthogonal array is covered. Let i and j be indices into the columns of each

of the two orthogonal arrays. All pairs between columns i and j, where i 6= j, are covered

by the same argument as each of the pairs in the original array is covered. When i = j,

however, only pairs of the form (x, x) are covered, where 1 ≤ x ≤ v). There are are now

v2 − v uncovered pairs between the two columns, meaning at least v2 − v rows are required

to complete the covering array. TConfig can indeed complete the covering array in 2 ·v2 −v

rows total using a ”reduced array” in the last v2 − v rows.

TConfig has other generalizations that make it possible for the algorithm to handle

any covering array, mixed or fixed level. Our purpose is not to explain all these details, but

rather illustrate the use of building blocks to build larger covering arrays. The Building-

Block Algorithm uses building blocks similarly, but allows reordering of the rows within a

building block.

5. Heuristic Search

Heuristic-search techniques, also known as local-search techniques, have successfully

been applied to many combinatorial optimization problems, including the covering-array

problem. If you give these approaches enough run-time they usually find very small covering

arrays. This is an important criteria if each test case is expensive to run and time and

computation power is plentiful.

20

In this thesis we use both tabu-search and simulated-annealing results in the

Building-Block Algorithm.

5.1. Simulated Annealing. Stevens [6] used simulated annealing in his thesis to

construct smaller covering arrays. The algorithm starts out with a k×n matrix, where the

rows are test cases and the values in each column represent a level for a factor. Each test

row is filled with random configurations. This matrix will have an initial cost according to

the cost function; Stevens used the number of uncovered pairs as the cost. At each step,

the algorithm randomly selects a row and a column and change the value to a different

value within the levels of the corresponding factor. The proposed move is accepted with a

probability that decreases exponentially with time, i.e., the matrix cools until the probability

is zero. If the cost after a move is zero, the algorithm has found a covering array.

Since the algorithm cannot calculate the size of the resulting covering array (this is

NP-hard, see [10]), the algorithm has to guess an initial size. The size of the matrix stays

constant during one run. If the algorithm finds a covering array, the size is decreased by one

row and a new run is started. If a covering array cannot be found in a reasonable time, the

algorithm may have cooled too quickly or selected an initial size that was too optimistic.

5.2. Tabu Search. Tabu search also starts with an k×n matrix, just like simulated

annealing [2]. The matrix is initially filled with random values. The number of uncovered

tuples is the cost of the matrix at each step.

Nurmela uses a special heuristic neighborhood to propose moves [2]. First, randomly

select an uncovered tuple. Iterate over the n rows and find which rows only require changing

a single element to cover the tuple. These changes form the moves in the current neighbor-

hood. The algorithm tries each move and calculates the new cost after the proposed move.

21

The move with the highest reduction in cost is selected, unless the move is tabu. If several

moves has the same cost reduction, the algorithm selects one nontabu move at random.

A move is tabu if the element has changed in the last T moves. This prevents the

algorithm from looping and helps diversify the search. In his paper, Nurmela found that

relatively small values of T were sufficient, usually ten or fewer.

6. Criteria for Comparing Algorithms

As we have seen, many algorithms exist for creating covering arrays, and all are

designed with different goals in mind. Some algorithms are fast, others have more accuracy

(produce smaller covering arrays), and yet others offer more flexibility (seeding, uneven

strength coverage, avoids). This section defines these criteria and discusses some of the

algorithms in terms of these criteria.

A primary requirement, rather than criteria, is completeness of coverage [10], i.e.,

every pair (or tuple) must be covered. In most cases, we achieve this by creating a covering

array of strength two. A tester rarely have no additional constraints, though. Avoids are

tuples that are not necessarily needed in a test suite. For example, there may be certain

configurations that are possible, but that a company does not want to officially support.

Disallowed tuples are tuples that cannot appear in a test suite, e.g., a test configuration

may be physically impossible to run. If a tester has too many constraints, there may

be a problem with how the system is modeled, and the tester may want to go back and

reconsider the levels and factors. Another important criteria for an algorithm is the ability

to seed an algorithm with some initial test cases. The tester may have run these test cases

earlier, e.g., during requirements testing or as part of a regression test. AETG developed

two different input languages for constraints and discussed some possible implementation

22

approaches [7]. However, they failed to mention that some constraints make it impossible to

even complete one test case. This is an area that merits additional research in the future.

The combinatorial algorithms cannot handle constraints and seeds very well, since these

techniques rely on a certain internal structure of the tuples. Heuristic-search and greedy

algorithms’ approaches more easily lend themselves to handling constraints.

All the algorithms must make a classical tradeoff between efficient construction and

accuracy. For example, a heuristic-search algorithm may run for weeks to find a very

accurate test suite (i.e., with very few test cases). A greedy algorithm may find a test

suite in seconds, but the test suite may be significantly larger. Combinatorial constructions

are usually extremely fast and are very accurate in some cases. However, combinatorial

constructions often has constraints on the input, e.g., Bose’s algorithm requires v to be a

prime power. TConfig is more general, but often create overly large test suites for mixed-

level covering arrays, since it was designed with fixed-level covering arrays in mind. A tester

also has to make a similar tradeoff. If test execution is extremely expensive, the tester may

be willing to wait for weeks for a heuristic-search algorithm to find an accurate test suite.

However, if the tester has automated scripts that can quickly run on a software system, the

size of the test suite is of little importance.

Another desirable criteria is a worst-case guarantee on the size of the final test suite.

TConfig and DDA offer such a guarantee. Also, an algorithm’s average test-suite size should

be as low as possible. Finally, predictability may be an important criteria in some cases.

Results from heuristic-search or AETG may be impossible to reproduce and confirm later.

There are other, more complex, criteria that are still subject to additional research.

For example, an algorithm could create a covering array with uneven strength. Some factors

could be covered at strength two, others at strength three. One could take this a step further

23

and cover certain levels of a factor at a higher strength. A default value for a factor could

be covered at a higher strength if it is unlikely that a user will ever change this value.

From this section we can deduce that no algorithm that constructs covering arrays

will meet all testers’ requirements. A tester has to analyze the circumstances of the testing

environment and decide which algorithm handles the circumstances the best. Some of these

criteria may aid the tester in this decision.

CHAPTER 3

THE BUILDING-BLOCK ALGORITHM AND DENSITY

IMPLEMENTATION

In this chapter we introduce a new algorithm for constructing covering arrays, the

Building-Block Algorithm (BBA). A simple example first illustrates the approach. Next,

we define the algorithm formally and describe an implementation. Finally, we compare the

BBA approach to other algorithms for constructing covering arrays according to the criteria

discussed in Chapter 2.

1. An Example

Let us consider a small multi-level covering array such as MCA(N ; 2, 4, (2232)). To

construct this covering array, we break the problem into two covering arrays, A and B, of

the form MCA(6; 2, 2, (2131)) as shown in Table 7. By combining the two covering arrays

we can create a partial solution to MCA(N ; 2, 4, (2232)) with six rows. A complete covering

array, however, must have at least nine rows since there are nine pairs between factors f2

and f4. Thus, after combining the first six rows, we add extra rows at the end to cover the

remaining pairs.

What are the remaining pairs? There are two types of pairs to consider: The inner

pairs and the cross pairs. Covering arrays A and B cover the inner pairs, i.e., A covers

the pairs formed by f1 and f2 and B covers the pairs formed by f3 and f4. The remaining

pairs are cross pairs, i.e., the first symbol in a cross pair appear in f1 or f2 and the second

25

Table 7. Covering arrays A and B, both MCA(6; 2, 2, (2131))

A

Row f1 f2

1 0 a2 0 b3 0 c4 1 a5 1 b6 1 c

B

Row f1 f2

1 2 x2 2 y3 2 z4 3 x5 3 y6 3 z

symbol appear in f3 or f4. Unless A and B change, the inner pairs are guaranteed to be

covered in this construction and thus we must concentrate on covering the cross pairs.

How A and B are combined affects which cross pairs and how many cross pairs are

uncovered after the first six rows. First, consider simply combining row one in A with row

one in B, row two in A with row two in B, and so on until each row in A is combined with

the corresponding row in B. This operation preserves covering arrays A and B, so the inner

pairs are still covered in the first six rows as shown in Table 8. However, this partial covering

array leaves a very unsatisfactory set of uncovered cross pairs. In particular, between f2

and f4, only cross pairs (a, x), (b, y), and (c, z) are covered, leaving six uncovered cross

pairs between these two crucial factors. This approach forces a complete covering array to

twelve rows, a dismal result considering the problem. A second and better approach will

be discussed next.

The first approach did not consider covering the most important or hardest-to-cover

cross pairs first as it combined covering arrays A and B. If the first six rows cover six cross

pairs between factors two and four there would be only three pairs left to cover in this

factor pair. We achieve this by reordering the rows in A and B, such that any row in A can

combine with any row in B. Each row in A and each row in B appears exactly once, so the

26

Table 8. Partial covering array PMCA(6; 2, 4, (2232))

Row f1 f2 f3 f4

1 0 a 2 x2 0 b 2 y3 0 c 2 z4 1 a 3 x5 1 b 3 y6 1 c 3 z

Table 9. Covering array MCA(N ; 2, 4, (2232))

Covering Array

Row rA rB f1 f2 f3 f4

1 1 1 0 a 2 x2 5 2 1 b 2 y3 3 5 0 c 3 y4 4 6 1 a 3 z5 2 3 0 b 2 z6 6 4 1 c 3 x

7 - - ⋆ b 3 x8 - - ⋆ c 2 z9 - - ⋆ a ⋆ y

resulting partial covering array still has six rows and still covers the inner pairs in A and

B. We complete the covering array by adding three additional rows to cover the remaining

cross pairs as shown in Table 9. Columns rA and rB refer to the row numbers in covering

arrays A and B in Table 7.

By allowing reordering on the rows within A and B, the number of rows in the final

covering array is dramatically reduced. The Building-Block Algorithm’s fundamental idea

is to combine smaller covering arrays by reordering the rows and then append additional

rows to construct a complete covering array. The next section generalizes this idea more

formally.

27

2. The Building-Block Algorithm

The Building-Block Algorithm (BBA) consists of four major steps:

Step 1 Divide the set of k factors {f1, f2, . . . , fk} into ǫ factor groups

{e1, e2, . . . , eǫ}. Each factor is assigned exactly one factor group, such that

|e1|+ |e2|+ · · ·+ |eǫ| equals k and the set e1 ∪ e2 ∪ · · · ∪ eǫ equals {f1, f2, . . . , fk}.

Step 2 Construct a building block for each factor group ei. A building block

BB(ni; t, |ei|, ei) is a covering array with ni rows, strength t, with |ei| factors

specified by the factor group ei. All building blocks have the same strength as

the original covering array. A shorthand notation for a building block for a

factor group ei is bi.

Step 3 Construct a partial covering array, PMCA(η; t, k, (v1v2. . .vk)), by

combining the building blocks b1, b2, . . . , bǫ from step 2.

Step 4 Complete the covering array, MCA(N ; t, k, (v1v2. . .vk)), by adding

additional rows to the partial covering array in step 3.

In the example in section 1 we performed step one by dividing the factors

{f1, f2, f3, f4} into two factor groups, namely e1 = {f1, f2} and e2 = {f3, f4}. Next, we con-

structed the two building blocks, b1 = BB(6; 2, 2, {f1, f2}) and b2 = BB(6; 2, 2, {f3, f4}).

In step three we combined row one of A with row one of B, row two of A with row five of

B, and so on until each of the six rows were used once. In the last step we added test cases

(⋆, b, 3, x), (⋆, c, 2, z), and (⋆, a, ⋆, y) to complete the nine-row covering array.

At every step there are several decisions that must be made. The algorithm poses

these decisions; an implementation of the algorithm must answer how to make these deci-

sions.

28

In step one, an algorithm must decide ǫ and how to assign the k factors to the

ǫ factor groups. Some choices for ǫ appear undesirable, for example ǫ = k (one factor

per factor group), ǫ = 1 (all factors in one factor group), or any other ǫ that causes any

|ei| < t (a factor group has fewer than t factors, thus the building block will not cover any

inner tuples). The assignment of factors to factor groups greatly affect the size of each

building block in step two. For example, a good combinatorial construction or a previously

computed heuristic-search result may be available for step two if a factor group has certain

characteristics. Furthermore, depending on the implementation of step three, it may be

desirable that the size of the building blocks have approximately the same size.

The next step constructs building blocks b1 through bǫ. An implementation of the

BBA may freely select any method to construct these covering arrays. They may be con-

structed with accuracy as the main goal, or perhaps with a certain internal structure. One

could even construct different building blocks with different methods. In our implementa-

tion, smaller building blocks appear to have a positive effect on the overall size of the final

covering array.

The third step reorders the rows in each building block and combines them to create

a partial covering array. The most important decision here is how the rows are reordered

and combined so that the best mixture of cross tuples get covered. The building blocks

are rarely the same size, so the algorithm must decide, for building blocks bi and bj and

ni > nj , how to combine rows in bi with nonexisting rows in bj after nj rows have been

fixed. Also, there may be don’t-care positions in the building blocks; how and when will an

implementation fix these positions? They could be fixed in step three or they could be left

open and fixed before step four by going back through the rows.

29

Several challenges similar to those AETG [7], TCG [8], and DDA [10] face present

themselves here. First, the algorithm must determine in what order to consider the factor

groups when constructing a row. Next, the algorithm must select which test row within the

building block should be fixed. Other approaches could be considered; however, this paper

only considers this general greedy approach.

Finally, additional rows are appended in the fourth step to cover any uncovered

cross tuples not covered in the partial covering array from step three. Although the fourth

step should attempt to optimize the number of added test rows, it ultimately hinges on

the uncovered tuples left in step three. Any algorithm that does not expect a certain order

in the uncovered pairs is suitable. For example, TConfig may have problems efficiently

completing this step. AETG, DDA, TCG are greedy algorithms that would work, and

even heuristic-search approaches such as simulated annealing [6], tabu search [2], and hill

climbing [14] could complete the covering array.

All the steps of the algorithm is summarized in pseudo code in Figure 2.

3. Density Implementation

This section explains the details of one implementation of BBA. We discuss step

three in great detail, including how to reorder the rows in the building blocks, how to

handle don’t-care positions, and how to handle building blocks of different sizes. This

implementation does not discuss step one, but acknowledges that this is an area that merits

further study.

Both AETG and DDA naturally maps to BBA’s approach to constructing covering

arrays, especially in step three. A factor and a building block are similar, since the rows in

a building block can be viewed as the levels in a factor. Density seems like a natural choice

30

1 BBA(k, t, (f1, f2, . . . , fk))2

3 Declare MCA(N ; k, t, (v1v2. . .vk))4

5 Divide f1, f2, . . . , fk into ǫ factor groups e1, e2, . . . , ek

6

7 for i from 1 through ǫ8 Compute bi

9 endfor10

11 while (rows are unfixed)12 O = Select order to treat factor groups13 for j from 1 through ǫ14 Select row to fix in bOj

15 endfor16 Add newly created row to MCA17 endwhile18

19 Cover remaining uncovered tuples in MCA by adding additional test rows20 Return MCA

Figure 2. Pseudo-code for the Building-Block Algorithm

31

for several reasons: Density provides a logarithmic bound on the number of test rows [10].

Also, density is deterministic, thus giving it a slight edge when reproducibility is important.

Finally, DDA appears to have a slight advantage on execution time. The two algorithms

are essentially equal for other criteria such as accuracy, ability to find mixed-level covering

arrays, and allowing seeding.

This section therefore discusses how to implement the algorithm using density. In

this chapter we assume that existing algorithms construct the building blocks. The next

chapter presents results where DDA, combinatorial construction, and tabu search perform

step two. Throughout this paper we let DDA complete the covering array in step four.

3.1. Building-Block Density. Let us explore how we can select a heuristic based

on density for ordering the rows in the building blocks. We then use this heuristic to greedily

find and fix rows from each building block, constructing one row of the partial covering array

at the time. We can quickly identify that an exhaustive search over all the orders are not

feasible. There are∏ǫ

i=1ni! different combinations!

DDA [10] defines a local density between factors; see Section 2.3. We now define a

similar density for factor groups. Let wi be the sum of the levels in a factor group ei. The

number of cross pairs involving factor groups i and j is wjwj . For factor groups i and j, we

define the factor-group pair density to be Γi,j =Ri,j

wiwj, where Ri,j is the number of uncovered

pairs involving a value of a factor in factor group i and a value of a factor in factor group j.

In other words, the factor-group pair density is the fraction of cross pairs already covered

between the two factor groups. From this we can define a factor-group density for factor

group s to be

Γs =∑

1≤i≤ǫi6=s

Γi,s.

32

We fix the factor groups in descending factor-group density order, since a higher density

indicates that a factor group (and the factors within the factor group) is likely harder to

cover.

After selecting a factor group to fix we must select a row within the building block

for that factor group. Let Ψi,r,φ be 1

witimes the number of uncovered cross pairs involving

row φ in building block r and factor group i. The row density for row φ in building block

r is defined to be

Ψr,φ =∑

1≤i≤ǫi6=r

Ψi,r,φ

We select φ to maximize Ψr,φ and fix that building-block row in the appropriate row and

columns in the partial covering array before we select the next factor group to fix.

Colbourn et al. point out some concerns with their initial definition for density in

[10], and we have the same concerns here. The factor-group pair density is scaled by the

initial number of cross tuples between the two factor groups. If a pair of factor groups i

and j has wi = wj = 4, the scale factor is 16. Each cross pair covered reduces the density

by 1

16. A different pair of factor groups has a scale factor of 100. Each cross pair covered

between these two factor groups reduces the density by 1

100, a smaller amount per pair than

the first two factor groups. This runs counter to the notion that the first pair of factor

groups are easier to cover than the second pair. We address this by redefining the scaling

factor. Let wmax be the largest sum of the levels of any factor group. When calculating

the factor-group pair density for two unfixed factor groups, let the denominator be w2max.

When one factor group has already been fixed, let the denominator be wmax (and treat each

fixed factor as if it only had one level). Lastly, when both factor groups have already been

fixed, let the denominator be 1. We now correctly treat each pair equally from a density

standpoint and focus on the factor groups with a higher density of uncovered cross pairs.

33

3.2. Building Blocks with Different Sizes. Our implementation adds don’t-care

rows to smaller building blocks to make all the building blocks the same size. A don’t-care

row is a row with all don’t-care positions. First, find building-block r with the largest

number of rows and set nmax = nr. For each building block i, add nmax − ni don’t-care

rows. As the algorithm builds the partial array, fixing these positions covers more cross

pairs. Since all the inner pairs are already covered, these don’t-care positions simply cover

some of the inner pairs more than once, which still maintains the covering array property.

3.3. Don’t-Care Positions. Any implementation of the BBA should fix the don’t-

care positions in the building blocks. If these positions are not fixed, the size of the final

covering array is unnecessarily inflated. Our implementation fixes the don’t-cares after all

the rows of the building blocks have been combined, after step three but before step four.

The algorithm uses density (as DDA defines it) to fix these positions. After all the rows

are combined, there is a smaller chance that the new pairs that a don’t-care position covers

now, also are covered in later rows. Consider an algorithm that fixes the don’t-cares as the

building-block rows are combined. A certain level for a factor may appear attractive at the

time, so the algorithm fixes the position to this level. However, other rows remaining in the

building block cover these exact same cross pairs later, thus the earlier don’t-care position

was fixed without actually covering any pairs that would not be covered otherwise.

A problem with not fixing the don’t-care position is that we cannot calculate the row

and factor-group densities. The algorithm solves this problem by using density as defined

by DDA to fix the don’t-care positions in a temporary row and then calculates the row

densities. Before the algorithm adds the combined temporary row to the partial covering

array it “unfixes” the original don’t-care positions.

There is a caveat with this approach that is worth mentioning [17]. An unfixed

34

don’t-care position affects the densities later. For example, a factor-group density may be

higher because a don’t-care position was unfixed when it could otherwise have been fixed

and thus covered additional cross pairs. The algorithm may end up picking an undesirable

building-block row without taking into account that the don’t-care position earlier covers

some of the same cross pairs with a high probability when we later fix the position. This

area merits further exploration and is not treated here.

3.4. Implementation Summary. This section summarizes how the density im-

plementation performs each step of the algorithm. The user supplies step one in the argu-

ments. For example, the user may provide MCA(N ; t, k, (3446)), MCA(N ; t, k, (255366)),

and MCA(N ; t, k, (2632)). The algorithm treats these as building blocks and combines them

to create MCA(N ; t, k, (21136465366)). The building blocks are either supplied by the user

as seeds or the algorithm uses DDA1 to construct the building blocks. Before the third step

we add don’t-care rows to smaller building blocks until they all have a size of nmax. Next,

the algorithm iteratively selects a factor group and a row within the corresponding building

block and builds the partial covering array one row a the time. The next step fixes the

don’t-care positions in the partial covering array. Finally, DDA appends additional rows as

needed to cover the remaining cross pairs.

4. Comparison of the Building-Block Algorithm and Other Methods

The Building-Block Algorithm draws many ideas from existing algorithms in the

literature. In some ways, one can view the BBA as a natural evolution of these ideas. This

section discusses how these other algorithms are similar to the BBA and how they differ.

Williams et al. introduced TConfig [16], which uses building blocks to construct

covering arrays. As opposed to BBA, TConfig does not allow reordering of the rows in the

1Renee C. Turban at Arizona State University wrote the DDA implementation

35

building blocks. This inflexibility sometimes forces TConfig to create larger covering arrays.

Also, because the building blocks are specific combinatorial constructions, TConfig cannot

take advantage of previously computed building blocks. Overall, BBA and TConfig share

the idea of using building blocks, but the similarities end there.

IPO builds a covering array adding factors one at the time [9] . BBA could implement

a form of IPO by iteratively use one factor as a building block. However, this approach

would not cover any pairs in one of the building blocks since it only has one column. Both

IPO and BBA can easily add new factors to an existing covering array and both add new

rows to accomplish this. Since IPO handles the factors in a decreasing order of the number

of factors, IPO runs into trouble if the new factors have many levels compared to the factors

in the original covering array. BBA, on the other hand, handles the factor groups according

to density, so the order of the factors has less of an impact.

Finally, BBA draws many ideas from purely greedy algorithms such as AETG, TCG,

and DDA. All of these algorithms treat factors (or factor groups) in some order and then

fix a level for that factor. The covering array is built one row at the time, just like BBA.

All of them can take advantage of previously computed results. However, none of these

algorithms can take advantage of several previously computed results the way BBA does.

CHAPTER 4

COMPUTATIONAL RESULTS

In this chapter we present the computational results for BBA. There are three areas

of interest: the accuracy of BBA compared to IPO, AETG, TCG, TConfig, and especially

DDA; the execution speed of BBA compared to DDA; and a comparison of different invoca-

tions of the BBA using building blocks created by simulated annealing, tabu search, Bose’s

algorithm, and DDA. BBA uses DDA for step four to cover the remaining cross pairs in all

the results. Also, we only consider strength two throughout the chapter.

In order to save space and avoid redundant information, the tables use a shorthand

notation, where multiple building blocks represent a single, larger covering array. Let

β = (N, (v1v2 . . . vk)) be a single building block with k factors, N rows, strength two, and

vi levels for factor i, 1 ≤ i ≤ k. Note that β is also a covering array with N rows. BBA can

combine multiple βs together, i.e., β1, β2, . . . , βǫ, and collectively these form the covering

array β1β2 · · · βǫ. If several building blocks are the same, we use an exponent to signify a

multiple of a building block, e.g., β5 is a covering array where building block β is combined

five times. Thus, β = (9, 34) equals CA(9; 2, 4, 3) and β4 equals CA(N ; 2, 16, 3), where N

is the number of rows in the final covering array after all four building blocks are combined

and the uncovered cross pairs are covered using additional rows. The number of rows in the

covering array β is given by S(β) and the running time to construct β is given by T (β).

All the calculations were performed on a Pentium D 2.8 GHz processor, a dual-core

37

64-bit processor, with 2 GB of physical memory running Linux. Although the processor is

dual-core, only one core was used to perform the calculations.

1. Different Building Blocks

As we mentioned earlier, BBA can use covering-array results from other algorithms

as building blocks to build an even larger covering array. This section focuses on results

using building blocks from Bose’s algorithm, simulated annealing, tabu search, and DDA.

We compare all the computational results against results using DDA from [10], using

an implementation by Renee C. Turban at Arizona State University. Her implementation

contains several levels of tie breaking for both the factor density and level density. We

did not attempt to optimize the reported DDA results by trying different tie-breaking

techniques, but left all the tie-breaking settings at reasonable defaults. The tie-breaker for

factor density was to select the factor with the most pairs left, and if this was equal, to select

the first factor. The same tie-breakers were used for level density. Since our implementation

of BBA does not address tie-breaking, this appears to be a fair comparison.

1.1. Building Blocks from Bose’s Algorithm. Bose’s algorithm constructs very

small covering arrays (or orthogonal arrays) for a restricted set of inputs. Table 10 gives a

number of results using the direct construction and reduced orthogonal arrays (see Section

4.1) as building blocks. The top half of the table uses orthogonal arrays and the bottom

half uses reduced orthogonal arrays by inserting don’t-care positions.

The results in Table 10 indicate that BBA produces accurate results using good

building blocks compared to DDA. The first six results, which use orthogonal arrays as

building blocks, have an improvement of about 10 % or more over DDA. The last five BBA

results, which use reduced orthogonal arrays, have a smaller improvement over DDA than

38

Table 10. BBA (using Bose’s algorithm) versus DDA

Bose’s BBA using Bose’s DDA (combined)Row β S(β2) S(β3) S(β4) S(β2) S(β3) S(β4)

1 (4, 23) 6 7 8 7 8 92 (9, 34) 15 15 18 17 19 213 (25, 56) 45 48 51 49 55 594 (49, 78) 91 99 104 97 111 1195 (121, 1112) 229 253 266 256 287 3116 (169, 1314) 325 350 370 367 412 446

7 (48, 68) 71 81 87 73 84 908 (49, 77) 91 98 103 94 105 1179 (120, 1010) 191 222 243 203 228 24910 (121, 119) 218 243 253 234 266 28711 (121, 118) 212 238 265 225 256 27912 (119, 99) 157 180 197 162 183 198

the first six results. These results suggest that accurate building blocks lead to smaller

covering arrays overall. The building block of size 119 for 99 in row ten of Table 10 does

not give a tight bound for this problem, and this fact carries over to the final covering-array

size for BBA. In this case, BBA constructs covering arrays comparable to that of DDA.

TConfig and BBA construct covering arrays with the same size for rows one through

four and row six for β2. BBA leaves v2−v pairs to cover between at least one pair of factors

after step three, just like TConfig, and the final size is thus 2 · v2 − v. Surprisingly, BBA

improved its solution for 1124 (in row five) by two rows over TConfig, from 231 to 229.

This improvement can be attributed to BBA’s step that allows reordering of the rows in

the building blocks. Colbourn reports a higher upper bound for 1124 [18], so we can report

that CAN(t, k, v) ≤ 229.

We see further evidence of these improvements in Table 11. In the first two rows of

this table, the building blocks for 68 and 77 are both reduced from the orthogonal array 78 of

size 49. TConfig would construct covering arrays with 91 rows for 7768 and 7868 (essentially

39

Table 11. Mixed-level BBA (using Bose’s algorithm) versus DDA

Building Blocksa BBA DDAb

Row Parameters β1 β2 β3 S(β1β2β3) S(β1β2β3)

1 7768 (48, 68) (49, 77) - 82 832 7868 (48, 68) (49, 78) - 79 863 101078 (49, 78) (120, 1010) - 139 1644 1112101078 (49, 78) (120, 1010) (121, 1112) 212 2325 11121020 (120, 1010) (120, 1010) (121, 1112) 240 2496 563423 (4, 23) (9, 34) (25, 56) 31 39

aThe first three rows use only two building blocksbInvoked directly with the parameters as the input

a reduced array from 716), but BBA improves these results to 82 and 79, respectively. Similar

improvements can be seen in Table 10, row seven, for inputs derived from the same source.

Table 11 also illustrates that BBA performs well with uneven building blocks.

1.2. Building Blocks from Heuristic Search. Heuristic-search algorithms gen-

erally produce covering arrays close to the upper bound when given enough execution time.

We obtained results from two heuristic-search algorithms, tabu search and simulated anneal-

ing, and used these results as building blocks for BBA. Overall, BBA’s results are slightly

better than DDA’s results.

BBA performs well in the first three rows of Table 12 against DDA, and slightly

better in the last two rows. Nurmela spent several CPU months calculating these building

blocks, and many give new best upper bounds. For example, in row five, Nurmela found a

covering array for 415317229 with 29 rows [2]. AETG found a covering array with 41 rows

using the same input. The tabu-search results for v = 3 are particularly good, and BBA

appears to use these good results to construct competitive covering arrays.

BBA continues to have slight edge over DDA using results from simulated annealing

as building blocks in Table 13. In rows one through four, simulated annealing found orthog-

onal arrays. Indeed, row three of Table 10 and row four of Table 13 are identical. Although

40

Table 12. BBA (using tabu search) versus DDA

Tabu BBAa DDA (combined)Row β S(β2) S(β3) S(β4) S(β2) S(β3) S(β4)

1 (18, 330) 25 28 30 29 32 332 (20, 343) 28 30 32 31 34 343 (23, 383) 32 34 36 35 37 394 (21, 41339235) 30 34 36 32 35 375 (29, 415317229) 41 46 49 44 48 50

aThe building blocks originate from http://www.tcs.hut.fi/~kjnu/covarr.html (accessed 2/25/2006),a website mentioned in [2]

Table 13. BBA (using simulated annealing) versus DDA

SA BBAa DDA (combined)Row β S(β2) S(β3) S(β4) S(β2) S(β3) S(β4)

1 (25, 53) 36 43 48 36 44 492 (25, 54) 40 48 52 42 49 533 (25, 55) 43 49 53 43 51 574 (25, 56) 45 48 51 49 55 595 (29, 57) 48 52 60 50 57 616 (36, 59) 52 60 65 55 61 677 (37, 510) 53 62 67 57 64 688 (40, 512) 57 65 70 59 67 719 (41, 513) 58 65 71 62 68 7310 (42, 514) 59 67 72 61 68 75

aMyra B. Cohen at University of Nebraska–Lincoln wrote the simulated annealing algorithm that pro-duced the covering arrays used as input in this table.

the building block for 56 have the same number of rows in each table, the actual order of

the rows is different. BBA still maintains the overall size of the final covering array. We

also combined a 56 building block from the simulated-annealing results and one 56 building

block from Bose’s algorithm and BBA still produced a 512 covering array with 45 rows.

1.3. Building Blocks from DDA. Finally, in Table 14 we used building blocks

constructed by DDA. A greedy algorithm such as DDA typically does not achieve the

same accuracy as Bose’s algorithm or heuristic search, so the building blocks in this table

are larger. Table 14 is divided in three sections, and each section corresponds to some of

41

Table 14. BBA (using DDA) versus DDA

DDA BBA DDA (combined)Row β S(β2) S(β3) S(β4) S(β2) S(β3) S(β4)

1 (9, 34) 15 15 18 17 19 212 (71, 77) 95 108 118 94 105 1173 (75, 78) 100 112 121 97 111 1194 (159, 1010) 211 237 258 203 228 2495 (127, 99) 166 189 204 162 183 198

6 (25, 330) 30 32 33 29 32 337 (28, 343) 32 34 36 31 34 348 (32, 383) 37 38 40 35 37 399 (28, 41339235) 34 38 41 32 35 3710 (35, 415317229) 45 50 52 44 48 50

11 (36, 56) 49 55 62 49 55 5912 (42, 58) 54 60 65 53 59 6413 (43, 510) 56 63 68 57 64 6814 (46, 511) 59 67 71 58 65 7015 (50, 513) 61 70 73 62 68 73

the same inputs as Bose’s algorithm, tabu search, and simulated annealing respectively.

Overall, BBA constructs covering arrays that are slightly larger than the covering arrays

DDA constructs. The only real exception is the first row, where DDA constructs the

orthogonal array for 34.

2. Comparison with Published Results

A common set of inputs for constructing covering arrays has emerged, and many

different methods have published results for at least some of these common inputs. Colbourn

et. al. published a table that summarized and compared results from AETG, TConfig, IPO,

TCG, and DDA [10]. We reproduced [10, Table 3] and added a column for BBA as shown

in Table 15.

BBA appears to be even with DDA and AETG and slightly better than IPO and

TConfig. TCG is also about even with BBA, although there are very few samples available

for comparison. IPO can easily reuse old test cases and add new factors to an existing

42

Table 15. Comparison with Published Results

Minimum Size of a Test Suite

Row Parameters BBA DDA AETG TCG IPO TConfig

1 513822 21 21 19 20

2 716151453823 45 43 45 45

3 514431125 28 27 30 30

4 6151463823 35 34 34 33

5 415317229 33 35 41 34 40

6 41339235 28 27 28 26 30

7 313 19 18 15 19 15

8 2100 14 15 10 15 14

9 440 40 43 49 40

10 4100 47 51 52 43

11 1020 191 201 180a 212 231

aor 198, see discussion in the text.

covering array. Since DDA can also perform the same operations, DDA appear to be a

better choice if these are important criteria for a specific application.

AETG delivers impressive results for 2100 and 1020. Other one-row-at-the-time

greedy algorithms such as DDA, TCG, and BBA construct dramatically larger covering

arrays for these inputs. We must differentiate between the commercially available (and

patented) AETG system and the greedy AETG algorithm described in [7]. Cohen et. al.

implemented AETG as described in [7] and they reported that their AETG implementation

constructed 1020 in 198 test rows [14]. The commercial product may include some simple

construction techniques, look-up tables of previously constructed covering arrays, and post

processing, none of which are included in the other greedy implementations. This must be

kept in mind when comparing the various greedy algorithms.

The composition of the factor groups in Table 15 affects the size of the final covering

array BBA constructs. Table 16 breaks down the input parameters into specific factor

groups. Some of the factor groups are repeated to match the factor groups to the input

43

Table 16. Building blocks for BBA for published results

Row Parameters β1 β2 β3 β4 β5

1 513822 (10, 5122) (9, 34) - - -

2 716151453823 (14, 7121) (12, 6121) (10, 5121) (16, 45) (9, 34)

3 514431125 (10, 5123) (16, 4421) (9, 34) (9, 3321)

4 6151463823 (16, 45) (12, 6121) (10, 5121) (9, 34) (8, 4121)

5 415317229 (16, 45) (9, 34) (9, 33) (4, 23) (4, 22)

6 41339235 (8, 4122) (9, 34) (9, 33) (4, 23) -

7 313 (9, 33) (9, 34) - - -

8 2100 (4, 23) (4, 22) - - -

9 440 (16, 45) - - - -

10 4100 (16, 45) - - - -

11 1020 (120, 1010) - - - -

parameters. For example, in row nine we repeat the factor group for 45 eight times to get

440. Most of the building blocks are orthogonal arrays or reduced orthogonal arrays. In

cases where there is not an easy mapping to an orthogonal array, we used DDA to create

the building block. These cases also illustrate that BBA can handle building blocks from

different sources, such as DDA and Bose’s algorithm.

3. Execution Time

Fast test-suite generation is one criteria we can use to evaluate a technique that

constructs covering arrays. All greedy methods are intended for speed, so TCG, IPO, and

DDA all are in the same range of timing, and AETG is slower due to the fact that it

generates M candidates per test row[10]. TConfig is faster since its method is almost a

direct construction. This section reports running times for BBA.

Table 17 compares BBA and DDA running times on the same computer system.

T (βx) means the time it took to construct covering array βx. It appears that BBA executes

faster, sometimes running five times faster than DDA. However, BBA is using precom-

44

Table 17. Execution time (BBA versus DDA) in seconds

BBA (s) DDA (s)β T (β2) T (β3) T (β2) T (β3)

(18, 330) 1.2 6.6 3.6 19(22, 368) 24 98 110 571

(169, 1314) 5.1 23 5.9 29(107, 815) 1.0 3.8 1.9 8.9

puted covering arrays as building blocks. For example, tabu search may have spent days

computing 815 with 107 test rows, and this time is not reflected in Table 17. BBA also

appear to have an advantage when the building blocks have fewer rows. DDA can construct

CA(295; 2, 14, 13), but in the table Bose’s algorithm constructed the same building block

with 169 rows. BBA spent 9.4 seconds constructing CA(N ; 2, 28, 13) using DDA’s building

block, almost doubling the comparable execution time in Table 17.

Although we cannot draw definitive conclusions from the Table 17, we can conclude

that BBA appears to have a reasonable execution time, especially when building blocks

near the upper bound is available.

CHAPTER 5

FUTURE WORK AND CONCLUSION

Although we have presented the Building-Block Algorithm (BBA) and our imple-

mentation as a fully functioning system, there are still many unanswered questions. This

chapter summarizes some of these questions and proposes possible new directions one could

explore. Finally, we draw some conclusions from our results and highlight some of the

advantages BBA has over other similar methods.

1. Future Work

As we mention in Section 3.4, our implementation does not select how to divide

the factors into factor groups. In some cases the user may have a specific set of factor

groups in mind, but as a general algorithm, this step should be performed automatically.

If this was done automatically, what should be the general goal? Should each building

block have approximately the same number of rows, and if so, how could one determine (or

approximate) that a specific division would reach that goal? If a few good building blocks

are available, one approach could divide the factor groups so they fit with the available

building blocks. This could have adverse effects on the remaining factor groups (if we

cannot find a perfect match). Further investigation is necessary to answer these questions.

BBA constructs competitive covering arrays when using building blocks constructed

by Bose’s algorithm and heuristic-search algorithms. BBA would be slow if it had to run

46

the heuristic-search methods in step two to get accurate building blocks, so a database of

the most accurate covering arrays available would benefit BBA. Each time BBA needs a

building block, it could first query a database before creating its own building block, thus

minimizing the time spent re-creating known results.

Our implementation of BBA did not include any tie-breakers for selecting factor

groups nor building-block rows. Tie breakers for factor groups may include random, first,

last, most uncovered pairs, etc. Ties between building-block rows could be broken using

several methods such as least-frequently used levels, most uncovered pairs between level and

single other factor, first, last, random, least-recently used levels. Some of these tie-breakers

would also require more calculations, so one must make a trade-off between execution time

and accuracy.

The definition of density in Section 3.1 may average over too many factors and

miss important information about cross pairs left between two factors. The local density

between two factors can still be high, even as the factor-group pair density is low compared

to other factor group pairs. This could lead to an effect similar to TConfig, where BBA

leaves v2 − v cross pairs between two factors after step three, thus forcing step four to add

v2 − v additional rows (see Section 1.1). We could either change the definition of density

or find a similar heuristic we could use as a tie breaker for building-block rows. Further

investigation should reveal how large of a problem this is, and from there one could make

a decision whether to change the definition of density or simply make it a tie breaker.

A larger question remains: Is density even the best way to match up building blocks?

We tried many different heuristics, mainly greedy strategies, but we found density to be the

most accurate out of these heuristics. However, other as yet unknown heuristics may prove

more efficient and accurate in the future. Heuristic-search techniques that move blocks of

47

configurations in a test row (instead of individual configurations) within the BBA framework

may be an interesting path to explore.

Although our current implementation for handling don’t-care positions appears rea-

sonable, we still have some concerns (see Section 3.3). We know that don’t-care positions

are important. For example, the result for β2 in row eleven of Table 15 increases from 191

to 206 if we do not fix the don’t-care positions at all. The impact of different methods

for handling don’t-care positions on the final covering-array size is largely unknown. If the

impact is rather small using other methods, we should prefer the fastest method.

DDA calculates step four in our implementation of BBA. We could use a heuristic-

search technique on these additional rows to improve the overall size of the covering array

without altering the rows fixed in step three. Depending on the user’s preference, the

heuristic search could be given a fixed time to improve the result in a post-processing step.

There is a limit to how many rows the post processing could optimize. Let Υ be the

maximum number of uncovered cross pairs between any two factor pairs after step three.

The post-processing step could not improve the result by more than N − (nmax + Υ) rows,

and in some cases even fewer rows.

Finally, the problem with avoids and disallowed pairs remains an issue for BBA (and

most other algorithms that construct covering arrays). This is largely an unsolved problem,

and we expect that extensive research will go into this area in the future.

2. Conclusion

The Building-Block Algorithm (BBA) is a general technique for constructing cover-

ing arrays one row at a time and is similar to techniques such as AETG, TCG, and DDA.

However BBA differs from these methods in that it combines several building blocks which

48

speed up the construction and potentially create more accurate covering arrays. TConfig

also uses building blocks, although in a stricter sense since TConfig does not reorder any

rows in its building blocks. The results show that BBA can take advantage of reordering

and thus constructs smaller covering arrays in many cases.

Furthermore, the results show that our implementation using density generally cre-

ates smaller covering arrays if the building blocks are more accurate. In a sense, BBA can

leverage the accuracy of its building blocks to create a smaller final test suite. This notion

should not necessarily be carried over to other possible future implementations since these

implementations may prefer other heuristics in their building blocks.

Finally, we can conclude that a test engineer can use BBA to leverage test cases from

part of a system to later test the complete system. IPO can grow a test suite by adding

factors, but it cannot take test suites from n subsystems and use those test suites in the

final test suite for the entire system. This attribute of BBA can save the test engineer time

and money, which are often scarce in software testing.

REFERENCES

[1] D. R. Kuhn and M. J. Reilly, “An investigation of the applicability of design of ex-periments to software testing,” in SEW ’02: Proceedings of the 27th Annual NASAGoddard Software Engineering Workshop (SEW-27’02). Washington, DC, USA: IEEEComputer Society, 2002, p. 91.

[2] K. J. Nurmela, “Upper bounds for covering arrays by tabu search,” Discrete AppliedMath., vol. 138, no. 1-2, pp. 143–152, 2004.

[3] A. W. Williams and R. L. Probert, “A practical strategy for testing pair-wise coverageof network interfaces,” in Proc. Seventh Intl. Symp. on Software Reliability Engineering,1996, pp. 246–254.

[4] R. C. Bose, “On the application of the properties of Galois fields to the constructionof hyper Graeco-Latin squares,” Sankhya, no. 3, pp. 323–338, 1938.

[5] J. Stardom, “Metaheuristics and the search for covering and packing arrays,” Master’sthesis, Simon Fraser University, 2001.

[6] B. Stevens, “Transversal covers and packings,” Ph.D. dissertation, University ofToronto, Toronto, 1998.

[7] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG system: Anapproach to testing based on combinatorial design,” IEEE Transactions on SoftwareEngineering, vol. 23, no. 7, pp. 437–444, 1997.

[8] Y. Tung and W. S. Aldiwan, “Automating test case generation for the new generationmission software system,” in IEEE Aerospace Conference, 2000, pp. 431–437.

[9] K. C. Tai and Y. Lei, “A test generation strategy for pairwise testing,” IEEE Trans-actions on Software Engineering, vol. 28, no. 1, pp. 109–111, 2002.

[10] C. J. Colbourn, M. B. Cohen, and R. C. Turban, “A deterministic density algorithmfor pairwise interaction coverage,” in Proceedings of the International Conference onSoftware Engineering (SE 2004), February 2004, pp. 345–352.

[11] C. J. Colbourn, “Combinatorial aspects of covering arrays,” Le Matematiche (Catania),no. 58, pp. 121–167, 2004.

[12] A. Hedayat, N. Sloane, and J. Stufken, Orthogonal Arrays. New York: Springer-Verlag,1999.

50

[13] R. Mandl, “Orthogonal latin squares: an application of experiment design to compilertesting,” Commun. ACM, vol. 28, no. 10, pp. 1054–1058, 1985.

[14] M. B. Cohen, P. B. Gibbons, W. B. Mugridge, and C. J. Colbourn, “Constructingtest suites for interaction testing,” in ICSE ’03: Proceedings of the 25th InternationalConference on Software Engineering. Washington, DC, USA: IEEE Computer Society,2003, pp. 38–48.

[15] M. Chateauneuf and D. Kreher, “On the state of strength-three covering arrays,”Journal of Combinatorial Designs, vol. 10, no. 4, pp. 217–238, 2002.

[16] A. W. Williams and R. L. Probert, “A measure for component interaction test cover-age,” in Proc. of the ACS/IEEE Intl. Conf. on Computer Systems and Applications.Washington, DC, USA: IEEE Computer Society, 2001, p. 304.

[17] C. J. Colbourn, Arizona State University, 2006, private communication.

[18] ——, “Strength two covering arrays: existence tables and projection,” Discrete Math-ematics, to appear (accepted December 2005).