JOURNAL OF ELECTRONIC TESTING: Theory and Applications 18, 583594, 2002c 2002 Kluwer Academic Publishers. Manufactured in The Netherlands.

Fast Anti-Random (FAR) Test Generation to Improve the Qualityof Behavioral Model Verification

TOM CHEN, ANDRE BAI AND AMJAD HAJJARDepartment of Electrical and Computer Engineering, Colorado State University, Fort Collins, CO 80523, USA

chen@engr.colostate.eduba206534@engr.colostate.edu

amjad@engr.colostate.edu

ANNELIESE K. AMSCHLER ANDREWSSchool of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 99164-2752

aandrews@eecs.wsu.edu

C. ANDERSONDepartment of Computer Science, Colorado State University, Fort Collins, CO 80523, USA

anderson@cs.colostate.edu

Received November 19, 1999; Revised May 2, 2002

Editor: B.T. Murray

Abstract. Anti-random testing has proved useful in a series of empirical evaluations. The basic premise ofanti-random testing is to chose new test vectors that are as far away from existing test inputs as possible. Thedistance measure is either Hamming distance or Cartesian distance. Unfortunately, this method essentially requiresenumeration of the input space and computation of each input vector when used on an arbitrary set of existing testdata. This prevents scale-up to large test sets and/or long input vectors.

We present and empirically evaluate a technique to generate anti-random vectors that is computationally feasiblefor large input vectors and long sequences of tests. We also show how this fast anti-random test generation (FAR)can consider retained state (i.e. effects of subsequent inputs on each other). We evaluate effectiveness of applyinganti-random vectors for behavioral model verification using branch coverage as the testing criterion.

Keywords: test data generation, anti-random testing, code coverage improvement

1. Introduction

Testing techniques employ a variety of mechanisms,automated, tool assisted, and manual, for test genera-tion. One of the techniques that has gained support andhas shown to be useful in a series of empirical evalu-ation is anti-random testing [7, 11]. The basic premiseof anti-random testing is that in order to achieve higher

coverage (of whatever type) one should, after havingexercised a set of tests, now choose tests that are asdifferent as possible from the tests previously used.Such a strategy will ensure that tests are not repeated,and therefore, uncover design errors or faults sooner.The dissimilarity of two given vectors can be measuredusing the concept of distance between them. Thegreater the distance, the less similar they are. The

584 Chen et al.

distance between two vectors can be measured usingeither the Hamming distance or the Cartesian distance.New test patterns are chosen that maximize thedistance.

In previous analyses, this approach has improvedcode coverage for boundary conditions, and has shownto be more efficient than random testing [5, 7].

The basic anti-random pattern generation methodproposed in [7, 11] has the following two shortcomings:

1. The method essentially requires enumeration of theinput space and computation of distance for each po-tential input vector. This prevents scale-up to largetest sets and/or long input vectors, in which casecomputations become too expensive.

2. The input vectors on which the anti-random vec-tors are computed have to be binary. The currentway around this problem is to use checkpoint en-coding [11]. Non-binary inputs are grouped intopartitions which are then given a binary encoding.This binary encoding is used for anti-random testgeneration. The anti-random vectors computed aremapped back into the actual input space by select-ing from each of the partitions identified by the bi-nary encoding. Unless the partitions are very small,this approach has low confidence in the programthrough successful tests and would also have littlevalue over the random sampling approach [3]. Onthe other hand, when we have many small partitions,the size of the input vector grows and computationbecomes expensive, and quickly impossible.

Our objective was to find a more efficient method togenerate anti-random test patterns that would be com-putationally feasible for large input vectors and longsequences of tests. This would enable a promising tech-nique to be applied to larger problems. Although theproposed method can be applied to hardware ATPG toidentify defective parts, our primary target applicationsare in the area of hardware behavioral model verifica-tion where a vast amount of test cases and test vectorsare applied to a given behavioral model to ensure error-free designs. Therefore, the terms used in this paper,such as testing, test vectors, and coverage, should betreated in the context of hardware behavioral model ver-ification where test vectors are applied to a behavioralmodel under test to verify its error-free operations.

Section 2 illustrates the approach to developing FastAnti-Random Pattern Generation (FAR). We start withan informal example to explain the basic approach,i.e., the concept of balancing points using the Multi-

dimensional Sphere Model [1]. This is followed by theexplanation of the algorithm and an example of its ap-plications. Section 3 analyzes the complexity of FARcompared to the enumerative algorithm and evaluateshow close the FAR generated inputs are compared withthose by the enumerative method in [7, 11]. Section 4presents a derivative of FAR to further speed-up thepattern generation process. Section 5 reports on apply-ing FAR to seven different applications. We concludein Section 6 by summarizing our results and pointingout open questions with regard to anti-random testingin general and the FAR method in particular.

2. New Approach for Anti-RandomPattern Generation

2.1. Method

The anti-random test vector is defined as a vector thathas the maximum distance from all previous vectorswhich were applied during a test [7, 11]. For binaryvectors, there are two ways to calculate the distance,Hamming Distance (HD) and the Cartesian Distance(CD). Hamming Distance is defined as the numberof corresponding bits that are different between twobinary vectors. The Cartesian Distance between twovectors AM and BM is defined as:

CD(AM , BM ) = M

i=1(Ai Bi )2 (1)

For example, for two binary vectors A = [1, 1, 0] andB = [0, 1, 1], the Hamming Distance between the vec-tors is 2 (the first and third bits differ). Their CartesianDistance is

(0 1)2 + (1 1)2 + (1 0)2 = 2.

Exhaustive anti-random test generation calculatesthe Hamming Distance (HD) and Cartesian Distance(CD) for all potential input test vectors. The Fast Anti-Random (FAR) approach generates new test sequencesby centralizing all existing input test vectors into onetest vector. A centroid vector of a set of vectors is theiraverage. Next, FAR finds all potential input test vectorsthat are orthogonal to the centralized vector. Vectors areorthogonal if their dot product is zero. Finally, FARfinds an anti-random vector with maximum distancefrom the centroid vector.

Let M be the number of bits in each input vector. M isthe dimension of the input space. Let N be the numberof such vectors (i.e., the length of the test sequence).The set of all M-bit binary vectors defines a EuclideanM-space, and can be denoted by 2M . The elements of

Fast Anti-Random (FAR) Test Generation 585

Fig. 1. Three dimensional binary vector space.

2M , points in the M-space, are called M-vectors. TheFAR algorithm finds a new point in M-space, providedN points, to make the N +1 points distributed as evenlyas possible in the M-space. It is interpreted as findinga point with maximum distance from the existing Npoints in M-space [7].

A test sequence of N vectors is a sampling of the M-dimensional space with 2M points in this space [1]. FARconstructs an Anti-random test sequence that balancesthe points in this space, given the known test sequenceof N vectors.

To illustrate the concept of balanced space, Fig. 1shows a three dimensional binary space with a total ofeight possible points in the space. Vectors A, B and Drepresent three given points in the space. Vector C isthe centroid vector of A, B and D. If the centroid vectoris rounded to binary integers, then vector C is the sameas vector A. The property of the centroid vector is thatit has the minimum distance to all the existing vectors.Therefore, one of the orthogonal vectors to the centroidvector will have the maximum distance to all existingvectors. In Fig. 1, vector F is the orthogonal vector ofthe centroid vector and has the maximum distance tovectors A, B and D. Vector F is the anti-random vector.The vectors A, F are symmetrical to each other. Soare B and D. This makes the three dimensional spacebalanced. Table 1 shows the sum of Cartesian distancebetween all vectors in the space to vectors A, B and D.Vector F has indeed the largest distance to the existingvectors.

More formally, the FAR method consists of the fol-lowing three steps:

Table 1. The Cartesian distance between vectors.

Vectors Vector A Vector B Vector D CD

(0, 0, 0) 2 1 2 1 + 2 + 2(0, 1, 0) 1 2 1 1 + 1 + 2(1, 0, 1) 2 1 2 1 + 2 + 2(1, 1, 1) 1 2 1 1 + 2 + 2F(0, 0, 1) 3 2 1 1 + 2 + 3

1. Centralization: This step determines the centroidvector of the random test sequence by using theMultidimensional Sphere Model [1]. This involvescalculating the average of the N vectors, then round-ing the resulting values to 0 or 1. Given N binarytest vectors, first sum the N vectors, then divide thevector sum by N . The result is the centroid vector, avector whose elements are between 0 and 1. Becausewe need a binary vector, we project these floating-point numbers into binary numbers by rounding ei-ther down to 0 or up to 1. This projection betweena floating point number and a binary one will intro-duce truncation error. This is shown in Fig. 1 as thechange of from the centroid vector C to the binarycentroid vector A.

FAR rounds vector elements to 0 for values lessthan 0.5. If they are greater it rounds them to 1. Ifthe value equals 0.5, FAR randomly selects either 0or 1. This creates a binary centroid vector.

2. Orthogonal vector calculation: This step finds allorthogonal vectors to the centroid vector. The vec-tor farthest from the centroid vector will be an or-thogonal vector, making orthogonality a necessarycondition to being farthest away. Two vectors areorthogonal, if and only if their dot product is zero[6]. Orthogonal vectors include those with maxi-mum distance provided all vectors are nonnegative.Not all orthogonal vectors will have maximum dis-tance to the centroid vector, but the vector with themaximum distance is orthogonal to the centroid.

3. Maximum distance calculation: Since all new testvectors generated are orthogonal to the given cen-troid vector, the orthogonal vector with the maxi-mum vector length will have the maximum distance.For binary vectors this is the vector containing themost 1s.

For implementation, steps 2 and 3 can be combined,as inverting the binary centroid will result in an orthog-onal vector with maximum distance.

586 Chen et al.

During the process of centralization, a centroid vec-tor bit might be 0.5. This means that the space isbalanced with respect to that particular vector element.When a vector element in the centroid vector becomes0.5, FAR randomly selects it to be 0 or 1. When generat-ing an anti-random test sequence based on the centroid,FAR iterates until all vector elements in the centroidvector become 0.5, implying the M-space is balanced.While the FAR algorithm is an approximation, it isfairly effective until the centroid vector has too manyentries of 0.5 (the space is balanced). Thus we wouldexpect to use the FAR algorithm in cases where enu-meration is not possible and the input space is large.The concept of a balanced space provides a naturalstopping criterion for this technique.

2.2. Example

Let us now illustrate with an example how to generatea six bit anti-random vector when given five input testvectors. Assume a six dimensional input space (M = 6)with five input test vectors (N = 5). The existing testvectors are given in Table 2. We now use FAR to gen-erate anti-random vectors to these test vectors.

Step 1: Centralization: Find the centroid vector X ofthe five given test vectors. Vector X is the average ofvectors A through E. Vector X is the rounded binarycentroid vector. The computation of vectors X andX is shown in Table 3.

Table 2. The exist-ing test sequence.

The test vectors

A = [1, 1, 1, 0, 1, 0]B = [0, 1, 0, 1, 1, 0]C = [1, 0, 0, 0, 1, 1]D = [0, 1, 1, 0, 0, 0]E = [1, 0, 0, 0, 0, 1]

Table 3. The centralization pro-cedure and the binary centroid X.

Centroid vector X calculation

X = (A + B + C + D + E)/5= [3, 3, 2, 1, 3, 2]/5= [0.6, 0.6, 0.4, 0.2, 0.6, 0.4]

Corresponding binary centroid X

X = [1, 1, 0, 0, 1, 0]

Table 4. All vectorsthat are orthogonal toX.

Orthogonal vectors

X1 = [0, 0, 1, 1, 0, 1]X2 = [0, 0, 0, 1, 0, 1]X3 = [0, 0, 1, 0, 0, 1]X4 = [0, 0, 1, 1, 0, 0]X5 = [0, 0, 0, 0, 0, 1]X6 = [0, 0, 1, 0, 0, 0]X7 = [0, 0, 0, 1, 0, 0]X8 = [0, 0, 0, 0, 0, 0]

Step 2: Orthogonal vector calculation: Find all vectorsin M-space that are orthogonal to the centroid vectorX. The list of all the orthogonal vectors is shown inTable 4.

Step 3: Maximum distance calculation: Next, FARfinds the orthogonal vector that has maximum vec-tor length, i.e., the binary vector containing the most1s. For this example, X1 = [0, 0, 1, 1, 0, 1] con-tains the most 1s, therefore, it is the anti-randomvector for X. The original vector set upon whichanti-random vectors are generated using FAR usu-ally include some previously generated anti-randomvectors. In such a scheme, even if the original vectorset were clustered to a small input space, the anti-random vectors will not be clustered to the oppositesmall input space. Rather, the anti-random vectorstend to be spread to cover the entire input space.

3. Evaluation of FAR

3.1. Computational Complexity of FAR

We compare the computational complexity between theexhaustive search algorithm and the search algorithmproposed here in FAR. The exhaustive anti-random testgeneration is completed by first obtaining a binary testsequence. Next, find a new test vector for which the sumof Hamming distance and Cartesian distance from allprevious vectors is greatest. However, calculating thetotal Hamming distance and total Cartesian distancegenerally requires enumeration of the input space anddistance computation for each potential input vector.Since there are 2M N potential input vectors, (Mbeing the number of bits in a binary vector, and N beingthe length of the test sequence given), the complexity

Fast Anti-Random (FAR) Test Generation 587

of exhaustive calculation then becomes of the order(2M N ) N .

Conversely, the FAR algorithm is based on the cal-culation of the centroid vector. Given N test inputs withM bits, the FAR algorithm treats the given test sequenceas a two dimensional array of size N M . It first sumseach column of the two dimensional array and thendivides it by the number of rows to obtain a centroidvector. It then rounds each bit of the centroid vectorto 0 or 1. A new anti-random vector is constructed byinverting each bit of the centroid vector. Therefore, theFAR algorithm only requires on the order of N M cal-culations. Thirty sets of anti-random generations wererun to illustrate the complexity of the FAR algorithm,as compared to an exhaustive search algorithm. In eachrun we generated 200 binary vectors, each with a bitlength of 15. We used the FAR algorithm to find oneanti-random vector based on 200 test vectors and thencompared the result with an anti-random vector gener-ated by the exhaustive search algorithm. In the exper-iment, M is 15, and N is 200, then the complexity ofthe exhaustive search algorithm becomes on the orderof (215 200) 200, or 6.5 106. By comparison, thecomplexity of FAR is only on the order of 3000. There-fore, we would expect that in this study, the cost forgenerating one anti-random vector using the exhaustivesearch algorithm will be 3 degrees of magnitude higherthan the anti-random vector generated by FAR. Fig. 2shows the cost of finding one anti-random test vectorusing the FAR algorithm versus the exhaustive searchalgorithm, as measured in CPU time. The X axis is thenumber of experiments and the Y axis is the CPU time.

Fig. 2. The CPU time of FAR algorithm vs. exhaustive algorithm.

As expected, the results show that the cost for gen-erating one anti-random vector using the exhaustivesearch algorithm is indeed 3 orders of magnitude higherthan the FAR algorithm.

3.2. Quality of FAR

Because FAR uses an approximation of the centroidvector by rounding bits to 0 or 1, a rounding error willoccur when given an even number of input test vec-tors. On the other hand, the cost for generating theanti-random test sequence using the FAR algorithm islow compared to the exhaustive search algorithm. Thuswe are interested in examining the quality of the anti-random vectors generated by the FAR algorithm. Thequality of the anti-random vector generated by FAR ismeasured by the distance between the vector generatedby FAR and that generated by the exhaustive search al-gorithm from the same set of existing test inputs. Thesame data described in Section 3.1 was used to mea-sure the quality of the anti-random vectors generatedby FAR. Fig. 3 shows the differences in the anti-randomvectors generated by FAR versus using the exhaustivesearch algorithm. Distance is measured in terms of theHamming distance of the two anti-random vectors. TheX axis identifies each run in the sequence, while the Yaxis shows the Hamming distance between the anti-random vector generated by the FAR algorithm versusthe exhaustive search algorithm for a given run.

The results show that for the majority of experiments,the Hamming distance between the anti-random vectorgenerated by the FAR algorithm and the exhaustive

588 Chen et al.

Fig. 3. Quality of FAR algorithm vs. exhaustive algorithm.

search algorithm is zero. There are only two caseswhere Hamming distance between the anti-randomvector generated by FAR versus the exhaustive searchalgorithm is more than 1. Given that the FAR algorithmis three degrees of magnitude faster than exhaustivesearch, we conclude that the overall performance ofthe FAR algorithm by far exceeds that of the exhaus-tive search algorithm.

4. Input Packing

In behavioral model verification we often have to dealwith the issue of retained state [4], that is, the same testinput pattern can generate different execution behav-ior depending on its order in the sequence. This canmake a huge difference in the quality of the verifica-tion process. In this case we are interested in generatingsequences of tests, rather than single, independent in-put patterns. Therefore, the key to successful testingis generating appropriate sequences. Usually, even afinite window of sequences is better than a single, in-dependent generation. Because of the speed of the FARgeneration method, we can generate test sequences ofdesired length through packing. Packing of size k is de-fined as concatenating k input patterns of size M intoone new input of size k M . When packing, we are mod-eling sequences of test inputs, of length k. A packed in-put with packing size k is called a scenario segment. Forpacked input, FAR generates new scenario segmentsthat are as far away from the current usage pattern aspossible.

Packing can be expected to be beneficial in situationswhere code execution depends heavily on the order inwhich a series of inputs is applied in addition to thevalue of the individual input. Optimal packing size isproblem dependent and currently must be determinedby the tester.

Most behavioral models will have state dependentaspects. If the input space is small, this can lead toexhaustive testing of all inputs without fully verifyingthe behavioral model. Applying packing increases theinput space of low dimensional input vectors and alle-viates this situation.

5. Applications of FAR

We have applied the FAR technique to test code writ-ten in VHDL, a hardware design language. Each of theVHDL programs represents a model of a hardware de-sign. We selected this language for our empirical inves-tigation, because its input variables (signals) are binaryvectors. Using the FAR technique we generated anti-random test sequences with vectors of varying size (Mvaried from 2 to 62). For the larger dimensions, enu-merative computation is clearly out of the question.The FAR algorithm quickly handled even the largerdimensions.

Similar to fault coverage in hardware testing, behav-ioral model verification has its own metrics. The met-rics are established based on a set of test criteria forbehavioral model verification. The most widely usedmetrics include statement coverage, toggle coverage,

Fast Anti-Random (FAR) Test Generation 589

and branch coverage. Other metrics include theobservability-based metrics [2] and validation vectorgrade [8]. A previous analysis [10] have shown that thebranch coverage is one of the best metrics to be used tomeasure the effectiveness of the verification process.It was shown [10] that branch coverage contains othercoverage metrics such as statement coverage and ismuch simpler to handle than some of the more complexcoverage metrics such as path-based coverage metrics.The previous results illustrate that if a higher branchcoverage can be achieved for a given set of vectors, ahigher coverage is also likely for the behavioral modelusing other coverage metrics such as the statement cov-erage. Although there is no theoretical proof to showthat such a relationship always exists, and the strengthof such a relationship may be HDL-model-dependent,our study presented in this paper adopts the use ofbranch coverage for its effectiveness and simplicity.

Using seven different VHDL models, we performedan empirical study which generated a series of randomtest cases first, then created anti-random test sequencesfor these tests. Table 5 lists for each model its name,

Table 5. Benchmark characteristics.

Model name LOC Branches Control Data

b01 115 28 1 2b03 141 27 1 4b04 80 17 3 9b08 130 33 2 8b10 167 44 3 8b13 296 79 3 8sys7 3785 591 7 62

Table 6. Random test followed by random test.

Model name b01 b03 b04 b08 b10 b13 sys7

First phase (Random test) 89.29 77.78 70.59 72.73 70.45 69.62 57.53Second phase (Random test) 89.29 77.78 70.59 72.73 72.73 69.62 70.05Branch coverage increase 0 0 0 0 2.28 0 12.52

Table 7. Random test followed by anti-random test.

Model name b01 b03 b04 b08 b10 b13 sys7

First phase (Random test) 89.29 77.78 70.59 72.73 70.45 69.62 57.53Second phase (Anti-random test) 92.86 77.78 76.47 72.73 70.45 69.62 69.71Branch coverage increase 3.57 0 5.88 0 0 0 12.18

the number of lines of VHDL code, the number ofbranches, the number of control bits and the number ofdata bits.

For each model, we compared anti-random testingversus random testing. First, we applied random tests toeach model followed by an anti-random test. We com-pared the branch coverage increase of the anti-randomtest against cases where a random test was followed byanother random test with the same number of vectorsas for the anti-random strategy, so that the total num-ber of patterns for both setups are the same. Table 6shows the branch coverage after the first phase (ran-dom test), the branch coverage after the second phase(random test), and the percentage increase after the sec-ond phase. Table 7 shows the branch coverage after thefirst phase (random test), the branch coverage after thesecond phase (anti-random test), and the percentageincrease after the second phase.

Fig. 4(a) shows the graph for a branch coverage in-crease from applying a set of random test sequencesfollowed by another set of random test sequences.Fig. 4(b) shows the branch coverage increase when ap-plying the same set of random test sequences followedby a set of anti-random test sequences. The X axis ofFig. 4 is branch coverage percent for seven differentmodels at the end of phase 1. The Y axis of Fig. 4 isthe percentage increase after the second phase of thetest.

Fig. 5 is the curve fitted graph of the comparison ofFig. 4(a) and (b). The solid line is the case where arandom test sequence was followed by more randomtest inputs. The dashed line represents the case where arandom test sequence was followed by anti-random testinputs. The X axis of Fig. 5 is the branch coverage inpercent for the seven models after executing the random

590 Chen et al.

Fig. 4. The graph of branch coverage increase.

Fig. 5. The curve fitted graph of Fig. 4.

patterns of phase 1. The Y axis is the percent increase inbranch coverage after the second phase of the test. Weused this format to visualize the effect of the coveragelevel at the beginning of phase 2 on possible further

coverage. Usually, it is harder to achieve additionalcoverage once a certain coverage level is reached.

The results show that using anti-random tests aftera set of random test sequences tended to improve

Fast Anti-Random (FAR) Test Generation 591

Table 8. Average branch coverage for each packing size.

Packing size 225 random 1 5 10 15 20 25 30

Average branch coverage 67.78 67.97 68.52 67.85 67.86 67.86 68.06 68.19

branch coverage, as opposed to using further randomtest sequences. It is not, however, a strong result infavor of anti-random testing. The amount of cover-age increase for anti-random versus random testingin the second phase was more for two models, lessfor two models, and was the same for three models.This could be due to the issue of retained state, or or-dering of inputs, particularly since the VHDL modelsthat show the least improvement have state-dependentcode.

To investigate this further, we used the sys7 model toevaluate the effectiveness of packing. The sys7 modelis a highly state sensitive model where the order of thetest patterns executed will have a significant impact onbranch coverage increase.

The FAR algorithm generates an anti-random testsequence very quickly even with a big M-space. Thisallowed us to use packing to improve the branch cov-erage of the sys7 model. In the first phase of the ex-periment we treated the model with 200 random testsequences, and then generated 25 anti-random test

Fig. 6. Effectiveness of packing size on sys7 model.

sequences using packing sizes k of 1, 5, 10, 15, 20,25 and 30 of the original input vectors.

In the second phase of the study, we treated the sys7model with 25 anti-random test sequences that weregenerated and measured cumulative branch coveragefor each packing size. We repeated this 19 times withdifferent seeds for generating the random test vectors.In the same set up, we also tested the case where 225random tests were applied to the model, and measuredthe branch coverage for each test. In this way we wantedto average the effect of the random seed used for therandom test generation. The average branch coverageof the 19 runs for each packing size compared to the225 random tests is shown in Table 8. Anti-randomtesting improved branch coverage in all cases over purerandom testing. The amount of improvement variedwith the packing size.

Fig. 6 graphs the average branch coverage in percentfor each packing size for the sys7 model. The X axisis the packing size. The Y axis is the branch coveragein percent. From the figure, we can clearly tell that the

592 Chen et al.

order of the test inputs in the sequence of execution hadan impact on branch coverage. The two best packingsizes for the sys7 program are either relatively small(k = 5) or large (k = 30). We believe packing size isproblem and code structure dependent.

Another experiment setup was also conducted onthe sys7 model to show more effectiveness of apply-ing random/anti-random pairs versus pure random. Thetesting procedure consisted of the following:

(a) Generate r (r = 65, 91) random input patterns.(b) Generate the maximum number of possible anti-

random patterns (until space is balanced).

Repeat steps (a) and (b) 10 times for each r . Comparethe resulting coverage against the same number ofrandom tests. Table 9 shows an example of such resultsfor each step when r = 91. Column 2 lists coverageafter applying the random inputs in step (a). Column 3reports coverage after applying the anti-random inputs(step (b)). Column 3 states branch coverage in percentwhen the same number of pure random inputs areapplied.

Testing with anti-random vectors is clearly the bet-ter approach to increase branch coverage. After 10 it-erations of random and anti-random, branch coverageis 84.43% compared to 76.48% (row 10, columns 3and 4). This represents a 7.95% increase in branch cov-erage or 47 branches. Note also, that the sequences ofrandom followed by anti-random seem to get eithertesting technique unstuck. For example, in iteration4, random patterns cannot improve upon the coverageof anti-random from iteration 3, but random showsslight improvement in iteration 5 again. Similarly,anti-random cannot improve upon random in iteration

Table 9. Branch coverage increase with anti-random testing.

Trial 91 random Anti-random Pure random

1 50.93 56.51 61.592 73.77 76.48 74.453 80.71 81.39 74.624 81.39 81.90 74.795 82.23 82.74 75.306 82.91 83.42 75.637 83.76 83.76 75.638 84.09 84.26 76.489 84.26 84.26 76.48

10 84.26 84.43 76.48

7, but after applying random again in iteration 8, anti-random is able to contribute to increased branch cov-erage for iteration 8. By contrast, pure random staysstuck at a given coverage level longer. It appears thusthat the combination of random with anti-random datais helpful.

However, whenever random data are generated,choice of seed influences what data get generated.When a good portion of the input space is covered thismay not make as much difference, but in our case (inputspace is of dimension 62), we should evaluate the effectof the seed used. To investigate this and determine av-erage performance of the anti-random strategy, we per-formed the above comparison using 10 seeds for eachstrategy, anti-random and pure random. Table 10 showsthe results for the 10 anti-random (column labeledAR) and 10 pure random trials (column labeled R).

Looking at the data, the first observation is thatchoice of seed and thus the random inputs upon whichthe anti-random generation is based influences whetheror not anti-random improves coverage over randomtesting. The other factor at work is the number of ran-dom inputs used in step (a). r = 65 does not perform aswell on average as r = 91. This may be because it is asparse sampling of the M-space. r = 65 appears to betoo small for the M-space of sys7 which is 262.

r = 91 works much better overall. The average im-provement in branch coverage percent is 2.23% or 13branches versus 0.62% or 4 branches in the case ofr = 65. But even for r = 91 not all trials showed im-provement, thus it is by no means guaranteed that anti-random assisted testing will always improve coverage,although on average using anti-random rather than purerandom is better for increased branch coverage in either

Table 10. Branch coverage for r = 65, r = 91.r = 65 r = 91

Trial AR R dcov AR R dcov

1 83.25 80.2 3.05 84.6 82.74 1.862 80.20 81.22 1.02 85.28 80.20 5.083 80.37 78.34 2.03 81.39 80.88 0.514 83.42 82.06 1.36 82.91 82.91 05 74.49 80.37 5.88 84.60 81.56 3.046 82.59 80.03 2.56 81.56 83.76 2.027 76.82 78.68 1.86 84.43 76.48 7.958 79.19 81.22 2.03 83.59 82.06 1.539 81.22 75.80 5.42 86.29 83.76 2.53

10 82.23 79.7 2.53 83.93 81.9 2.03

Fast Anti-Random (FAR) Test Generation 593

Fig. 7. Branch coverage comparison.

case. Fig. 7 visually compares branch coverage im-provement for r = 91 over pure random for all ten trials.

Our next question was whether it is important to gen-erate and use the maximum possible anti-random num-bers, as they represent a balanced space. This was oneof the assumptions in the prior experiment. Thus wedecided to violate this assumption and generate a fixednumber of anti-random inputs in step (b), but generateda comparable number of total test cases. Results of 10trials based on different seeds showed that anti-randomvectors performed much worse than before. Pure ran-dom outperformed anti-random in 5 of the 10 trials.Moreover, pure random performed on average 0.84%better than anti-randomit covered 5 more brancheson average. Thus we recommend using the maximumpossible number of anti-random inputs when generat-ing anti-random vectors.

6. Conclusion

We demonstrated a Fast Anti-Random Test generationalgorithm that is able to improve branch coverage inbehavioral model verification. It can be used to gener-ate single, independent input patterns, or sequences ofrelated inputs via packing. When applying it to behav-ioral models with more sophisticated data types andinputs, a mapping or encoding can be used to map the

binary vectors into the representation required for theactual input to the program. This mapping could po-tentially be checkpoint encoding [11].

While our small study was not able to achieve largeimprovements in branch coverage, as for example re-ported in [7, 11], several things must be considered. Based on work by Hamlet and Taylor [3], and

Tsoukalas, et al. [9], we cannot expect any singletesting technique to be consistently superior. Thusanti-random testing will not always be the bettertechnique. When it is advisable, it is of course im-portant to have a fast implementation like FAR.

Packing improved FARs performance in uncoveringnew branches. The best packing size depends on thecode under test (its state transition behavior).

Even if only moderate increases in branch cover-age are possible, this should be compared to howmany more random tests one would have to executeto achieve the same coverage as via anti-random test-ing. Such a comparison would indicate the amountsavings in test execution and validation.

The nature of the inputs upon which anti-randominputs are based makes a difference in how muchbranch coverage increases.

When using anti-random inputs it is beneficial to usethe maximum number of such inputs (representing abalanced space) rather than a fixed set.

594 Chen et al.

Ultimately, one would want to combine a testingtechnique like FAR with a stopping rule (e.g., [4]), sothat it will be used only as long as it yields results. Sec-ondly, each testing technique should come with rules ofthumb when their use is likely to be advantageous. Theuse of the proposed anti-random method will increasethe likelihood of finding more design errors in behav-ioral models earlier in the verification process thanother random test methods if the initial vector set doesnot cover the input space very well. This is typicallythe situation at the early stage of the verification pro-cess where a limited amount of vectors have been ap-plied and maximally effective vectors are needed sub-sequently to potentially shorten the verification cycle.However, the benefit of using the anti-random methoddecreases when a huge amount of vectors that coverthe input space very well have already been appliedto behavioral models. This is typically at the very latestage of the verification cycle.

Acknowledgments

This work was supported by the National Science Foun-dation through grant MIP-9628770. We are gratful toYashwant Malaiya of Colorado State University for in-sightful discussions about anti-random test-generationmethods.

References

1. T.J. Bai, T. Cottrell, D.-Y. Hao, T. Te, and R.J. Brozka, Multi-Dimensional Sphere Model and Instantaneous VegetationTrend Analysis, Ecological Modeling, vol. 97, no. 1/2, pp. 7586.

2. F. Fallah, S. Devadas, and K. Keutzer, OCCOM: Efficient Com-putation of Observability-based Metrics for Functional Veri-fication, in Proc. 35th Design Automation Conference, 1998,pp. 152157.

3. R. Hamlet and R. Taylor, Partition Testing does not In-spire Confidence, IEEE Transactions on Software EngineeringSE-16, vol. 12, pp. 14021411, December 1990.

4. W. Howden, Systems Testing and Statistical Test Data Cover-age, in Proc. COMPSAC 97, Washington, DC, 1997, pp. 500504.

5. R. Kapur, S. Patil, T. Snethen, and T.W. Williams, A WeightedRandom Pattern Test Generation System, IEEE Transactionson Computer-Aided Design of Integrated Circuits and Systems,vol. 15, no. 8, pp. 10201025, August 1996.

6. S.J. Leon, Linear Algebra with Applications, 4th ed., New Jersey:Prentice Hall, Inc., 1994.

7. Y.K. Malaiya, Anti-random Testing: Getting the Most out ofBlack-Box Testing, in Proc. ISSRE95, Toulouse, 1995, pp. 8695.

8. P.A. Thaker, V.D. Agrawal, and M.E. Zaghloul, Validation Vec-tor Grade (VVG): A New Coverage Metric for Validation andTest, in Proc. 17th IEEE VLSI Test Symp., 1999, pp. 182188.

9. M.Z. Tsoulakos, J.W. Duran, and S.-C. Ntafos, On Some Re-liability Estimation Problems in Random and Partition Test-ing, IEEE Transactions on Software Engineering SE-19, vol. 7,pp. 687697, July 1993.

10. A. von Mayrhauser, T. Chen, et al., On Choosing Test Criteriafor Behavioral Level Hardware Design Verification, in Proc.HLDVT2000, Bekerley, CA, November 2000.

11. H. Yin, Z. Lebnedengel, and Y.K. Malaiya, Automatic Test Gen-eration using Checkpoint Encoding and Anti-random Testing,in Proc. ISSRE97, Albuquerque, NM, 1997, pp. 8495.

Tom Chen obtained his Ph.D. from the University of Edinburgh,U.K. He spent 3 years with Philips Semiconductors in Europe from1987 to 1999 before joining New Jersey Institute of Technology asan assistant professor. Since 1990, he has been with the Departmentof Electrical and Computer Engineering, Colorado State University.He is currently an associate professor. Dr. Chens research interestsare in the areas of VLSI design and test, CAD techniques for VLSIdesign, and high level design verification.

Andre Bai received his B.S. degree from the Department of Electricaland Computer Engineering at Colorado State University in 2000. Heis currently at IBM engaging in the design and analysis of mainframecomputers. His research interests are in the areas of hardware andsoftware testing methods.

Amjad Hajjar received his M.S. and Ph.D. degrees from the De-partment of Electrical and Computer Engineering at Colorado StateUniversity in 1998 and 2001. He is currently holding a post-doc po-sition at the department of Electrical and Computer Engineering atColorado State University. His research interests are in the areas ofhigh level verification techniques for behavioral hardware modelsand statistical methods for improving verification efficiency.

Anneliese Amschler Andrews is the Huie Rogers Endowed Chair inSoftware Engineering at Washington State University. Dr. Andrewsis the author of a text book and over 130 articles in the area of Soft-ware Engineering, particularly software testing and maintenance.Dr. Andrews holds an MS and Ph.D. from Duke University and aDipl.-Inf. from the Technical University of Karlsruhe. She served asEditor in Chief of the IEEE Transactions on Software Engineering.She has also served on several other editorial boards including theIEEE Transactions on Reliability, the Empirical Software Engineer-ing Journal, the Software Quality Journal, the Journal of InformationScience and Technology, and the Journal of Software Maintenance.She contributes to many conferences in various capacities includinggeneral or program chair, or as a member of the steering or programcommittee.

Charles Anderson is an associate professor of Computer Scienceat Colorado State University. He graduated with a Ph.D. in com-puter science from the University of Massachusetts, Amherst, in1986, and worked at GTE Laboratories in Waltham, MA, until 1991.His research interests include neural networks, reinforcement learn-ing, EEG pattern recognition, neural modeling, HVAC control, adap-tive tutoring, computer graphics, computer vision, and software andhardware testing.