Summary ofSummary ofTest Case Design Techniques

Brian Nielsen,Arne Skou{bnielsen | ask}@cs.auc.dk

Development of Test Cases

Complete testing is impossible

olog

i Testing cannot guarantee the absence of faults

tekn

o

How to select subset of test cases from all possible test cases

tions

t How to select subset of test cases from all possible test cases

with a high chance of detecting most faults ?

orm

at Test Case Design Strategies

Info

CISS

Whitebox Testing Overview

White box testing

olog

i White box testing Flowgraphs Test criteria/coverage

St t t / b h / d i i / diti / th

tekn

o Statement / branch / decision / condition / path coverage MC/DC coverage

Looptesting

tions

t

Def-use pairs Efficiency of different criteria

orm

at

BEWARE: Expected outcome cannot be determined from

Info code!

CISS

White-Box : Statement Testing Execute every statement of a program

Relatively weak criterion

olog

i Relatively weak criterion

Weakest white-box criterion

tekn

otio

nst

orm

atIn

fo

CISS

Example : Statement Testing((result = 0+1+…+|value|, if this <= maxint, error otherwise)

olog

i 1 PROGRAM maxsum ( maxint, value : INT )2 INT result := 0 ; i := 0 ;

tekn

o 3 IF value < 04 THEN value := - value ;5 WHILE ( i < value ) AND ( result < maxint )

tions

t 5 WHILE ( i < value ) AND ( result <= maxint )6 DO i := i + 1 ;7 result := result + i ;

orm

at 7 result := result + i ;8 OD;9 IF result <= maxint

Info 9 IF result maxint

10 THEN OUTPUT ( result )11 ELSE OUTPUT ( “too large” )

CISS12 END.

Flowgrah1

2Node = statement (blocks)Edges = possible successor (control flow)

olog

i

1 PROGRAM maxsum ( maxint, value : INT )2 INT result := 0 ; i := 0 ;3 IF l 0

3 4

tekn

o 3 IF value < 04 THEN value := - value ;5 WHILE ( i < value ) AND ( result <= maxint )6 DO i i + 1

5 6-7

tions

t 6 DO i := i + 1 ;7 result := result + i ;8 OD;9 IF result <= maxint

orm

at 9 IF result <= maxint10 THEN OUTPUT ( result )11 ELSE OUTPUT ( “too large” )12 END

9

Info 12 END.

1011

CISS12

Flow graph: Cyclomatic complexity #edges - #nodes + 2 Defines the maximal number of test cases needed to

provide statement coverage

olog

i provide statement coverage Mostly applicable for Unit testing Strategy for statement coverage:

tekn

o gy g1. Derive flow graph2. Find cyclomatic complexity #c3 Determine at most #c independent paths through the

tions

t 3. Determine at most #c independent paths through the program (add one new edge for each test case)

4. Prepare test cases covering the edges for each path( ibl f th # )

orm

at (possibly fewer than #c cases)

Info

CISS

Cyclomatic complexity?1

y p y

1 PROGRAM maxsum ( maxint, value : INT )2 INT lt 0 i 0

2

olog

i 2 INT result := 0 ; i := 0 ;3 IF value < 04 THEN value := - value ;5 WHILE ( i < l ) AND ( lt < i t )

3 4

tekn

o 5 WHILE ( i < value ) AND ( result <= maxint )6 DO i := i + 1 ;7 result := result + i ;8 OD;

5 6-7

tions

t 8 OD;9 IF result <= maxint10 THEN OUTPUT ( result )11 ELSE OUTPUT ( “too large” )

orm

at 11 ELSE OUTPUT ( too large )12 END.

9

Info 1011

CISS12

Example : Statement Testingstart

yes

olog

i value:= -value;value < 0 yes

no

tekn

o

Tests for completestatement coverage:

i:=i+1;result:=result+i;

tions

t g

maxint value(i<value) and

(result<=maxint)yes

orm

at 10 -1

0 -1

no

yes no

Info result<=maxint

output(result); output(“too large”);

yes no

CISS

output(result); output( too large );

exit

White-Box : Path Testing Execute every possible path of a program,

i e every possible sequence of statements

olog

i i.e., every possible sequence of statements

Strongest white-box criterion

tekn

o

Usually impossible: infinitely many paths ( in case of loops )

So: not a realistic option

tions

t

But note : enormous reduction w.r.t. all possible test cases

( each sequence of statements executed for only one value )

orm

at ( eac seque ce o s a e e s e ecu ed o o y o e a ue )

Info

CISS

White-Box : Branch Testing

Branch testing == decision testing

olog

i Execute every branch of a program :

each possible outcome of each decision occurs at least once

tekn

o

Example:

IF b THEN s1 ELSE s2

tions

t

IF b THEN s1; s2

CASE x OF

orm

at 1 : ….2 : ….3

Info 3 : ….

CISS

Example : Branch Testing

Tests for complete

start

yes

olog

i statement coverage:maxint value

10 1

value < 0 value:= -value;yes

no

value < 0

tekn

o 10 -1

0 -1

is not sufficient fori:=i+1;

result:=result+i;

tions

t is not sufficient forbranch coverage;(i<value) and

(result<=maxint)yes

T k

orm

at no

yes no

Take:

maxint value

10 3

Info result<=maxint

output(result); output(“too large”);

yes no 10 3

0 -1

for complete

CISS

output(result); output( too large );

exit

branch coverage

Example : Branch Testing

l ll < 0

start

yes maxint value

olog

i value:= -value;value < 0 yes

no

maxint value

-1 -1-1 -1

10 3

-1 -1

tekn

o

i:=i+1;result:=result+i;

10 3

But:

10 3

tions

t

(i<value) and(result<=maxint)

no

yesBut:No green path !

orm

at

lt< i t

no

yes no Needed :

Info result<=maxint

output(result); output(“too large”);

yes oCombination of decisions

10 -3

CISS

p ( ); p ( g );

exit

White-Box : Condition Testing

Design test cases such that each possible outcome

f

olog

i of each condition in each decision occurs at least once

Example:

tekn

o

decision ( i < value ) AND (result <= maxint )consists of two conditions : ( i < value ) AND (result <= maxint )t t h ld b d i d h th t h t l

tions

t test cases should be designed such that each gets valuetrue and false at least once

orm

atIn

fo

CISS

Example : Condition Testingstart

yes

olog

i ( i = result = 0 ) :

maxint value i<value result<=maxint

value:= -value;value < 0 yes

no

tekn

o

-1 1 true false

1 0 false truei diti

i:=i+1;result:=result+i;

tions

t gives condition coveragefor all conditions(i<value) and

(result<=maxint)yes

But it does not preserve

orm

at no

yes no

But it does not preservedecision coverage

Info result<=maxint

output(result); output(“too large”);

yes no always take care thatcondition coveragepreserves decision coverage :

CISS

output(result); output( too large );

exit

preserves decision coverage :decision / condition coverage

White-Box : Multiple Condition Testing Design test cases for each combination of conditions

Testingol

ogi

Example: ( i < value ) (result <= maxint )

tekn

o ( i value ) (result maxint )false falsefalse truet f l

tions

t true falsetrue true

Implies decision- condition- decision/condition coverage

orm

at Implies decision-, condition-, decision/condition coverage

But : exponential blow-up

Again : some combinations may be infeasible

Info Again : some combinations may be infeasible

CISS

White-box: loop testing Statement and branch coverage are not sufficient Single loop strategy:

Zero iterations

olog

i Zero iterations One iteration Two iterations

l b f

tekn

o Typical number of iterations n-1, n, and n+1 iterations (n maximum number of

allowable iterations)d l

tions

t

Nested loop strategy: Single loop strategy often intractable Select minimum values for outer loop(s)

orm

at Treat inner loop as a single loop Work ‘outwards’ and choose typical values for inner loops

Concatenated loops:

Info Concatenated loops: Treat as single, if independent Treat as nested, if dependent

CISS

Example : Loop testingstart

yes

Tests for complete loop coverage:

olog

i value < 0 value:= -value;yes

no maxint value

15 0

tekn

o

i:=i+1;result:=result+i;

15 0

15 1

15 2

tions

t

(i<value) and(result<=maxint)

yes 15 3

6 4

orm

at no

yes no

15 5

Info result<=maxint

output(result); output(“too large”);

yes no

CISS

output(result); output( too large );

exit

White-box testing: Data Flow Criteria

Basic idea: For each variable definition (assignment), find a path (and a corresponding test case), to its use(s). A

olog

i a path (and a corresponding test case), to its use(s). A pair (definition,use) is often called a DU pair.

Three dominant strategies:ll d f ( ) f ll l h f h

tekn

o All-defs (AD) strategy: follow at least one path from eachdefinition to some use of it

All-uses (AU) strategy: follow at least one path for each DU

tions

t

pair All-du-uses strategy (ADUP): follow all paths between a DU

pair

orm

at Complements the testing power of decision coverage

Info

CISS

Example: All-uses coverage 1 Dm,v

1 PROGRAM maxsum ( maxint, value : INT )2 INT result := 0 ; i := 0 ;3 IF value < 0 2 Dr,i

olog

i4 THEN value := - value ;5 WHILE ( i < value ) AND ( result <= maxint )6 DO i := i + 1 ; 3 4

UvUv;Dv

tekn

o7 result := result + i ;8 OD;9 IF result <= maxint10 THEN OUTPUT ( lt ) 5 6-7Ui v r m

Ur,i;Dr,i

tions

t10 THEN OUTPUT ( result )11 ELSE OUTPUT ( “too large” )12 END.

5 6i,v,r,m

orm

at

9Ur,m

Def-use pairs:

1 3 1 5 1 9 1 4Tests for complete all-uses coverage:

i t l

Info

1011Ur

1-3,1-5,1-9,1-4

2-5,2-9,2-6

4-5

maxint value

0 0

0 -1

CISS12

6-5,6-9,6-11

6-5-610 1

10 2

White-Box : Overviewstatementcoverage

olog

i conditioncoverage

decision(branch)coverage

tekn

o

decision/conditioncoverage

tions

t

multiple-condition

orm

at conditioncoverage

Info path

coverage

CISS

White-Box : Overviewstatementcoverage

f decision

olog

i all defscoverage

decision(branch)coverage

tekn

o

all usescoverage

tions

t

all du pathscoverage

orm

at

path

coverage

Info path

coverage

CISS

Additional techniques: mutation and random testingand random testing Mutation testing:

Intended for evaluating the test cases

olog

i Intended for evaluating the test cases Create at set of slightly modified mutants of the original

program containing errorsR th t t i t th t t

tekn

o Run the test cases against the mutants Criteria

All mutants must fail (strong)

tions

t

All mutants will eventually fail (weak) Random testing:

Basic idea: run the program with arbitrary inputs

orm

at Basic idea: run the program with arbitrary inputs Inherent problems: How to define the oracle for

arbitrary inputs and how to decide to stop?

Info Advantage: The program structure can be ignored

CISS

Efficiency of white box techniques:Efficiency of white-box techniques: two studies

olog

i

Strategy #test cases %bugs foundRandom 35 93 7

tekn

o Random 35 93.7Branch 3.8 91.6All 11 3 96 3

tions

t All-uses 11.3 96.3

orm

at

Random 100 79.5

Info Branch 34 85.5All-uses 84 90 0

CISS

All-uses 84 90.0

What is MC/DC?

MC/DC stands for Modified ol

ogi Condition / Decision Coverage

Main idea: Each condition must be h t i d d tl ff t th

tekn

o shown to independently affect the outcome of a decision, i.e. the outcome of a decision changes as a

tions

t outcome of a decision changes as a result of changing a single condition.

orm

atIn

fo

CISS

1. MC/DC Requirement The decision has taken all possible

outcomes at least once.ol

ogi

If ((a) & (b) | (c)) then

tekn

o If ((a) & (b) | (c)) then…

tions

t

true false

orm

at

We could also say we cover both the true d h f l b h (lik d i

Info and the false branch (like we do in Branch Testing).

CISS

2. MC/DC Requirement Every condition in the decision has taken

all possible outcomes at least once.

olog

i p

If ((a) & (b) | (c)) then…

tekn

o If ((a) & (b) | (c)) then…

true false true false true false

tions

t

Aims to cover compound conditions (like

orm

at

p (Condition/Decision Coverage)

Info

CISS

3. MC/DC Requirement

Every condition in the decision i d d tl ff t th d i i ’

olog

i independently affects the decision’s outcome.

tekn

o

If ((a) & (b) | (c)) then…

tions

t (( ) ( ) | ( ))

true false true false

orm

at Change the value of each condition individually while keeping all other

conditions constant.

Info

CISS

Creating MC/DC test cases

If (A and B) then…ol

ogi

(1) create truth table for conditions

tekn

o (1) create truth table for conditions. (2) Extend truth table so that it

indicated which test cases can be used

tions

t indicated which test cases can be used to show the independence of each condition.

orm

atIn

fo

CISS

Creating MC/DC test cases cont’dol

ogi Show independence of A:

Take 1 + 3

tekn

o Take 1 + 3

Show independence of B:

tions

t B: Take 1 + 2

Resulting test cases

orm

at

gare 1 + 2 + 3

(T T) (T F) (F T)

Info (T , T) + (T , F) + (F , T)

CISS

More advanced example

If (A and (B or C)) then…ol

ogi

tekn

o

Note: We want to determine the MINIMAL set of test casesH

tions

t Here:{2,3,4,6}{2,3,4,7}

orm

at

Non-minimal set is:{1,2,3,4,5}

Info

CISS

Argument in the software industry Federal Aviation Administration’s

requirement that test suites be MC/DC

olog

i q /adequate (DO-178B).

tekn

o

In the avionics domain, complex Boolean expressions are much more common

tions

t expressions are much more common.

orm

atIn

fo

CISS

Argument in the software industry Federal Aviation Administration’s requirement

that test suites be MC/DC adequate. Argument – Too expensive

olog

i Argument – Too expensive “For example, one of our industrial partners reports

that for one of its products of about 20,000 lines of code the MC/DC-adequate test suite requires seven

tekn

o code, the MC/DC-adequate test suite requires seven weeks to run.”

Counter Argument – Significantly more effectiveA t i i l t d f d d i th l

tions

t

A recent empirical study performed during the real testing of the attitude-control software for the HETE-2 (High Energy Transient Explorer) found that the test cases generated to satisfy the MC/DC coverage

orm

at cases generated to satisfy the MC/DC coverage requirement detected important errors not detectable by functional testing.

Info

CISS

Blackbox Testing Overview

Black-box testing (or functional testing):

olog

i Black box testing (or functional testing): Equivalence partitioning

Boundary value analysis Domain analysis

tekn

o y y

Cause-effect graphing

Behavioural testing / state transition testing

y

tions

t

Random testing, Error guessing etc…

Others: Classification Tree Method, Pairwise testing, Use Case

orm

at Testing, Syntax testing, ..

Basics : heuristics and experience

Info How to use black-box and white-box testing in combination

CISS

Black box testing

requirements

olog

i

output

tekn

o

SUTinput

tions

tput

events y domain testing

orm

atIn

fo

x

CISS

x

Black-box: Equivalence Partitioningol

ogi Strategy :

Identify input equivalence classes

tekn

o y p q

Based on conditions on inputs / outputs in specification / description

Both valid and invalid input equivalence classes

tions

t Both valid and invalid input equivalence classes

Based on heuristics and experience “input x in [1..10]” classes : x 1, 1 x 10, x 10

orm

at “enumeration A, B, C classes : A, B, C, not{A,B,C,} ……..

Define one / couple of test cases for each class

Info Define one / couple of test cases for each class

Test cases that cover valid eq. classes

Test cases that cover at most one invalid eq class

CISS

Test cases that cover at most one invalid eq. class

Example: Equivalence Partitioningol

ogi

Test a function for calculation of absolute value of an integer

Equivalence classes :

tekn

o Equivalence classes :

Condition Valid eq. classes Invalid eq. Classes

nr of inputs 1 0 > 1

tions

t nr of inputs 1 0, > 1

Input type integer non-integer

particular abs < 0, >= 0

orm

at

Test inputs :

Info • x = -10, x = 100

• x = “XYZ”, x = - x = 10 20

CISS

A Self-Assessment Test [Myers] “A program reads three integer values. The three values are

interpreted as representing the lengths of the sides of a triangle.

olog

i The program prints a message that states whether the triangle is scalene (uligesidet), isosceles (ligebenet) , or equilateral (ligesidet) ”

tekn

o (ligesidet).

tions

tor

mat

Write a set of test cases to test this program

Info Write a set of test cases to test this program.

CISS

A Self-Assessment Test [Myers]Test cases for:

olog

i 1. valid scalene triangle ?2. valid equilateral triangle ?3 lid i l t i l ?

9. one side larger than sum ofothers ?

10 3 permutations of previous ?

tekn

o 3. valid isosceles triangle ?4. 3 permutations of previous ?5 side = 0 ?

10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?

tions

t 5. side = 0 ?6. negative side ?7. one side is sum of others ?

12. non integer input ?13. wrong number of values ?14. for each test case: is

orm

at 7. one side is sum of others ?8. 3 permutations of previous ? expected output specified ?

15. check behaviour aftert t d d ?

Info output was produced ?

CISS

Example: Equivalence Partitioning

Test a program that computes the sum of the first value integers

l thi i l th i t Oth i

olog

i as long as this sum is less than maxint. Otherwise an error

should be reported. If value is negative, then it takes the

tekn

o

absolute value

Formally:

tions

t y

Given integer inputs maxint and value compute result :

orm

at

result = if this <= maxint, error otherwise|value|

k

Info

K=0

CISS

Example: Equivalence Partitioning Equivalence classes :

Condition Valid eq. classes Invalid eq. classesN f i t 2 2 2

olog

i Nr of inputs 2 < 2, > 2Type of input int int int no-int, no-int intAbs(value) value 0, value 0

tekn

o maxint k maxint, k > maxint

Test Cases : maxint value result

tions

t Test Cases : maxint value result

Valid 100 10 55100 -10 5510 10

orm

at 10 10 errorInvalid 10 - error

10 20 30 error

Info “XYZ” 10 error

100 9.1E4 error

CISS

Black-box: Boundary Value Analysisol

ogi

Based on experience / heuristics :

Testing boundary conditions of eq. classes is more effective

tekn

o

i.e. values directly on, above, and beneath edges of eq. classes

Choose input boundary values as tests in input eq. classes

tions

t

instead of, or additional to arbitrary values

Choose also inputs that invoke output boundary values( values on the boundary of output classes )

orm

at ( values on the boundary of output classes )

Example strategy as extension of equivalence partitioning: choose one (n) arbitrary value in each eq class

Info choose one (n) arbitrary value in each eq. class

choose values exactly on lower and upper boundaries of eq. class choose values immediately below and above each boundary

( if applicable )

CISS

( if applicable )

Example: Boundary Value Analysisol

ogi

Test a function for calculation of absolute value of an integer

Valid equivalence classes :

tekn

o Valid equivalence classes :

Condition Valid eq. classes Invalid eq. Classes

particular abs < 0, >= 0

tions

t particular abs 0, 0

• Test cases :

orm

at class x < 0, arbitrary value: x = -10class x >= 0, arbitrary value x = 100

Info classes x < 0, x >= 0, on boundary : x = 0

classes x < 0, x >= 0, below and above: x = -1, x = 1

CISS

Example: Boundary Value Analysis Given integer inputs maxint and value compute result :

|value|

olog

i

result = if this <= maxint, error otherwise|value|

k

tekn

o K=0

• Valid equivalence classes :

tions

t

Condition Valid eq. Classes

Abs(value) value 0, value 0

orm

at maxint k maxint, k > maxint

Sh ld l di ti i h b t i t 0 d i t 0 ?

Info • Should we also distinguish between maxint < 0 and maxint >= 0 ?

maxint maxint < 0, 0 maxint < k, maxint k

CISS

Example: Boundary Value Analysisol

ogi Valid equivalence classes :

Abs(value) value 0, value 0

tekn

o maxint maxint < 0, 0 maxint < k, maxint k

Test Cases :

tions

t C

maxint value result maxint value result

55 10 55 100 0 0

orm

at 55 10 55 100 0 054 10 error 100 -1 156 10 55 100 1 10 0 0

Info 0 0 0 …. …. …

-10 1 error

How to combine the boundary conditions of different inputs ?

CISSTake all possible boundary combinations ? This may blow-up.

Black-box: Cause Effect Graphingol

ogi

Black-box testing technique to analyse combinationsof input conditions

tekn

o of input conditions

Identify causes and effects in specification

tions

t inputs outputs

current state new state

orm

at

Make Boolean Graph linking causes and effects

Info Annotate impossible combinations of causes and effects

Develop decision table from graph with in each columna particular combination of inputs and outputs

CISS

a particular combination of inputs and outputs

Transform each column into test case

Black-Box: Cause Effect Graphing k maxint

k i t kd

xor

olog

i k maxint

value 0

error

and

xor and

tekn

o

C

value 0 xor

tions

t Causes k maxint 1 1 0 0

inputs k maxint 0 0 1 1

value 0 1 0 1 0

orm

at value 0 1 0 1 0

value 0 0 1 0 1

Info

Effects k 1 1 0 0

outputs error 0 0 1 1

CISSTestcase

Black-box: Cause Effect Graphing

Systematic method for generating test cases representing

combinations of conditions

olog

i combinations of conditions

Combinatorial explosion of number of possible combinations

S h i ti t d thi bi t i l l i

tekn

o Some heuristics to reduce this combinatorial explosion

Starting point is effects (outputs) then working ‘backwards’

tions

t

‘light-weight’ formal methods:

transformation into semi-formal Boolean graph

orm

at

A technique : to be combined with others

Info

CISS

Black-box: Error Guessing Just ‘guess’ where the errors are ……

Intuition and experience of tester

olog

i Intuition and experience of tester

Ad hoc, not really a technique

tekn

o

Strategy:

Make a list of possible errors or error-prone situations

tions

t

( often related to boundary conditions )

Write test cases based on this list

orm

atIn

fo

CISS

Black-box : Error Guessing More sophisticated ‘error guessing’ : Risk Analysis

Try to identify critical parts of program (high risk code sections):

olog

i Try to identify critical parts of program (high risk code sections):

parts with unclear specifications

tekn

o

developed by junior programmer while his wife was pregnant ……

complex code :

tions

t

measure code complexity - tools available (McGabe, Logiscope,…)

High-risk code will be more thoroughly tested

orm

at ( or be rewritten immediately ….)

Info

CISS

Black-Box Testing: Which One ? Black-box testing techniques :

Equivalence partitioning

olog

i

Boundary value analysis

Cause-effect graphing

tekn

o

Error guessing

………

Whi h t ?

tions

t

Which one to use ? None is complete

All are based on some kind of heuristics

orm

at All are based on some kind of heuristics

They are complementary

Info

CISS

Black-Box Testing: Which One ? Always use a combination of techniques

olog

i

When a formal specification is available try to use it

Identify valid and invalid input equivalence classes

tekn

o

Identify output equivalence classes

Apply boundary value analysis on valid equivalence classes

tions

t

Guess about possible errors

Cause-effect graphing for linking inputs and outputs

orm

atIn

fo

CISS

White-Box testing : How to Apply ?

Don’t start with designing white-box test cases !

olog

i Start with black-box test cases (because requirements must be tested

anyway)

tekn

o

equivalence partitioning, boundary value analysis,

cause effect graphing, test derivation with formal methods, …..

Check white box coverage

tions

t Check white-box coverage

( statement-, branch-, condition-, ….. coverage )

U t l b bi d ith U it f k

orm

at Use a coverage tool – maybe combined with a Unit framework

Design additional white-box test cases for not covered code

Info

CISS

Blackbox vs. Whiteboxol

ogi

I II III

tekn

otio

nst

orm

at

Specified Behavior Implemented Behavior

Info (Blackbox) (Whitebox)

I: Specified, not implementedII: Implemented and specified

CISS

II: Implemented and specifiedIII:Implemented, but not specified

A Coverage Tool : gcov

Standard Gnu tool gcov

O l t t t b t b li d ll f

olog

i Only statement coverage, but may be applied manually for

multiple condition coverage

tekn

o

Compile your program under test with a special option

Run a number of test cases

tions

t

A listing indicates how often each statement was executed

and percentage of statements executed

orm

at and percentage of statements executed

Info

CISS

A Coverage Tool : BullseyeCoverage

Supports decision/condition

olog

i

Compile your program under test with a special

option

tekn

o option

Run a number of test cases

tions

t

A listing indicates the coverage

orm

atIn

fo

CISS

END