Verification and Validation(Part 2)

John MorrisComputer Science/Electrical and Computer Engineering

19 Apr 2023 1

A hard day’s work ensuring that some Japanese colleagues understand why Auckland is The City of Sails!

Credit:Several slides in this presentation were adapted from originals written by

Stephen M. Thebaut, University of Florida

Validation or Verification?

ValidationBuilding the right softwareMake sure it’s what the user wants

VerificationBuilding the software rightMake sure it works

Accurate, complete specification essential!

19 Apr 2023 2

Testing

• Aim• Locate and repair defects

• Axiom

• No matter how much testing you do, you can’t be sure that there isn’t an error waiting to bite you!

Testing only reveals the presence of defects,it never proves their absence!!

19 Apr 2023 3

Testing

The alternative?• Formal verification

• Uses formal logic to prove that software is correct• Currently:

• Prohibitively expensive• Little automated support• Mainly manual techniques• Error prone

• Only feasible when cost of failure is extreme• Usually when failure leads to loss of life

• Air and space craft control• Medical systems• Nuclear plants

19 Apr 2023 4

Testing - Motivation

Definitely the least glamorous part of software development

• Possibly the most expensive!• If not carried out thoroughly!• Estimates of the economic cost of software failure

produce astronomic numbers• US: $59.5 billion in 2002

• http://www.nist.gov/public_affairs/releases/n02-10.htm

• ~10% of projects are abandoned entirely• Including some very large ones

19 Apr 2023 5

Exhaustive Testing

• Inefficient, naive?• Never forget the KISS principle

• An automated test system can do a very large numbers of tests in a reasonable time

and do them while you’re designing the next test!• Analysis needed is trivial

whereas

• Analysis for a more efficient test regime may be quite complex and error-prone

• It’s easy to convince someone that an exhaustively tested system is reliable

19 Apr 2023 6

Equivalence classes – formal definition

• A set of equivalence classes is a partition of a set such that• Each element of the set is a member of exactly

one equivalence class• For a set, S, and a set of equivalence classes, Ci

• U Ci = S

• Ci Cj = (null set) unless I = j

19 Apr 2023 7

Equivalence classes – formal definition

• A set of equivalence classes is a partition of a set such that• The elements of an equivalence class, C, are

classified by an equivalence relation, ~• If a C and b C , then a ~ b

• The equivalence relation is • Reflexive a ~ a• Transitive if a ~ b and b ~ c, then a ~ c• Symmetric if a ~ b then b ~ a

• A Representative of each class is an arbitrary member of the class• They’re all ‘equivalent’ – so choose any one!

19 Apr 2023 8

Equivalence classes – verification

• Equivalence relation• In the verification context, the elements of the set

are the sets of input values for a function under test

eg we are verifying a function

f( int a, int b )• The 2-tuples (1,1), (1,2), (1,3) .. (and many more!)

are the elements of the set of all possible inputs for f

• The equivalence relation is “behaves the same way under testing”

• One common interpretation of this is:• “follows the same path through the code”

19 Apr 2023 9

Black Box and White Box Verification

• Black Box • Specification is available but no code

• White Box• Code is available and can be analyzed• Equivalence classes are derived from rules in the

specification and the code

19 Apr 2023 10

Generating the Equivalence Classes

Outputs• Find equivalence classes that cover outputs also!• Same general rules apply as for inputs• One representative of each class plus• Boundaries• Null output

eg No items in a report – does the user want a confirming report anyway?

• Just one outputeg Reports often have header and trailer sections - are

these correctly generated for a short (<1 page) report?

19 Apr 2023 11

Never neglect the null case!•It’s very easy to neglect at specification stage•Required behaviour may be ‘obvious’• No need to write it down!•It will require coding•Experienced programmers know that it’s a very common source of error!

Coverage in White Box Testing

• Black Box testing will not usually cover all the special cases required to test data structures• Often, the functional goals of the specification

could be met by one of several data structures• Specification may deliberately not prescribe the

data structure used• Allows developers to choose one meeting performance goals• Permits substitution of an alternative with better

performance (vs non functional specifications)

• Coverage defines the degree to which white box testing covers the code• Measurement of completeness of testing

19 Apr 2023 12

Coverage in White Box Testing

• Usually, at least some white box coverage goals will have been met by executing test cases designed using black-box strategies • How would you know if this were the case or

not?• In simple modules, which don’t use internal data

structures, black box classes may be adequate• This is not the general case though!

• Various coverage criteria existEvery statement at least once

Every branch taken in true and false directions

Every path through the code

19 Apr 2023 13

Coverage in White Box Testing

• Coverage criteria• Logic coverage

• Statement: each statement executed at least once• Branch: each branch traversed (and every entry point

taken) at least once• Condition: each condition True at least once and False at

least once• Branch/Condition: both Branch and Condition coverage• Compound Condition: all combinations of condition values

at every branch statement covered (and every entry point taken)

• Path: all program paths traversed at least once

19 Apr 2023 14

Pseudocode and Control Flow Graphs

input(Y)

if (Y<=0) then

Y = −Y

end

while (Y>0) do

input(X)

Y = Y-1

end

19 Apr 2023 15

“nodes”

“edges”

Statement Coverage

• Statement Coverage requires that each statement is executed at least once

• Simplest form of logic coverage• Also known as Node Coverage

• What is the minimum number of test cases required to achieve statement coverage for the program segment given next?

19 Apr 2023 16

Pseudocode and Control Flow Graphs

input(Y)

if (Y<=0) then

Y = −Y

end

while (Y>0) do

input(X)

Y = Y-1

end

19 Apr 2023 17

“nodes”

“edges”

Branch coverage

• Branch Coverage requires that each branch will have been traversed, and that every program entry point will have been taken, at least once

• Also known as Edge Coverage

19 Apr 2023 18

Branch Coverage – Entry points

• Why include “…and that every program entry point will have been taken, at least once.”

• Not commonin HLLs (eg Java) now

• Common in scripting languages• Any language that

allows a goto and a statement label!

19 Apr 2023 19

Branch Coverage (cont’d)

• Relationship between Statement and Branch Coverage?Possibilities:

1. None.2. Statement Coverage subsumes Branch

Coverage (“statement => branch”).3. Branch Coverage subsumes Statement

Coverage (“branch => statement”).

“statement => branch” ?

Minimum input sets required for Statement Coverage?

Minimum input sets required for Branch Coverage?

“branch => statement” ???

“branch => statement” ???

• Normally, YES.• But not in the presence of Dead Code

Dead Code is not reachable via any executable program path.

Condition Coverage

• A branch predicate may have more than one condition

input(x,y);if ((y<=0) || (x=0) ){

y = -y;}

while ( (y>0) && (! EOF) )do {input(x);y = y-1;

}

Condition Coverage

• Condition Coverage requires that each condition will have been True at least once and False at least once

• What is the relationship between Branch and Condition Coverage?

Condition Coverage (cont’d)

if (A || B) s1;else s2;

A B Branch

test 1 T F true

test 2 F F false

A B Branch

test 3 T F true

test 4 F T true

BranchCoverage

+

ConditionCoverage

Checkeachcondition

Branch/Condition Coverage

• Branch/Condition Coverage requires that both Branch AND Condition Coverage be achieved

Branch/Condition Coverage subsumes both Branch Coverage and Condition Coverage

Compound Condition Coverage

• What if the compiler generates code that masks the evaluation of conditions?

• That is, supposeif ((A) || (y/x=5) ) { … }

is compiled so that if A is true, y/x=5 will not be evaluated

Compound Condition Coverage

• Compound Condition Coverage requires that all combinations of condition values at every branch statement be covered and that every entry point be taken at least once

• Also know as Multiple Condition Coverage• Subsumes Branch/Condition Coverage,

regardless of the order in which conditions are evaluated

Compound Condition Coverage (cont’d)

input(X,Y)if ((Y<=0) || (X==0) )

Y = -Y

Possible condition values: TT, TF, FT, FF

In general, how many different combinations of condition values must be considered when a

branch predicate has n conditions?

Path Coverage

• Path Coverage requires that all program paths be traversed at least once

• Often described as the “strongest” form of logic coverage? • Is it stronger than Compound Condition Coverage?

• Path Coverage is usually impossible when loops are present • How many test cases would be required to cover

all paths in the example below?)

Path Coverage

for (i=0;i<30;i++) {input(X,Y);if (Y<=0) { if (X<=0)

Y = -X; else

Y = -Y; }else Y = X+Y;}

repeat 30 times

Path Coverage

3 paths

3 paths

3 X 3 = 9 paths

Path Coverage (cont’d)

repeat 30 times

3 X 3 X…X 3

= 3 paths30

Path Coverage

• Various strategies have been developed for identifying useful subsets of paths for testing when Path Coverage is impractical: • Loop Coverage• Basis Paths Coverage• Dataflow Coverage

Loop Coverage

• Loop Coverage requires that the body of loops be executed • 0, 1, 2, t, max and max+1 times

• Rationale• 0: Is some action taken in the body that must also

be taken when the body is not executed?• 1: Lower bound on number of times body may be

executed• 2: Check loop re-initialization.• t: Check typical number of iterations.• max: Check upper (valid) bound on number of

times body may be executed.• max+1: If the maximum can be exceeded, what

behavior results?

Coverage tools

• Simple statement based coverage metrics can be checked by coverage tools

• Sometimes called profilers• Tool augments the code by adding additional code to

collect statistics – mainly counts of individual statement executions

• At the end of a run of tests, lists of unexecuted statements are readily produced

19 Apr 2023 37

input(a);if ( a < 0 ) {

print(“error”);}

elsex = f(a);

input(a); inc_count( 1002 );if ( a < 0 ) {

print(“error”); inc_count( 1004 );}

else {x = f(a); inc_count( 1006 ); }

Statement Coverage tools• Mechanical confirmation that every statement has been

executed once is clearly a valuable first stepto our goal of demonstrating that a system has been thoroughly tested

If a statement hasn’t been executed, it’s a potential timebomb!

• Can be augmented for compound condition coverage by parsing code and identifying multiple conditions

• However, complex or subtle interactions between input parameter values will not necessarily be caught if relying on statement coverage alone!

• Path coverage is needed

• Correct equivalence class analysis should produce this manually (and therefore at considerable cost!)

19 Apr 2023 38

Dataflow Coverage

19 Apr 2023 39

Procedure – Module Verification

Steps• Obtain precise specification

• Should include definitions of exception or illegal input behaviour

• For each input of module• Determine equivalence classes (inc special cases)• Choose representatives• Determine expected outputs

• Repeat for outputs• Many output equivalence classes are probably covered

by input equivalence classes

• Build test table

19 Apr 2023 40

Procedure – Module Verification

Steps• Write test program

• Tests in tables is usually the best approach• Easily maintained• Test programs are part of the deliverables for a system

• They need to be run when any change is made• To ensure that code that worked isn’t broken by a fix

of something else!!

or worse

• a new feature added!• Tables are easily augmented

• When you discover the case that you didn’t test for!

19 Apr 2023 41