1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13

1

Unit Testing

CS 4311

Hans Van Vliet, Software Engineering, Principles and Practice, 3rd edition, John Wiley & Sons, 2008. Chapter 13.

2

Hierarchy of TestingTesting

Program Testing

Top Down

Bottom Up

Integration TestingUnit Testing

System Testing

Big Bang

Sandwich

Black Box

White Box

Function

Performance

Reliability

Availability

AcceptanceTesting

Properties

Security

Equivalence

Boundary

Decision Table

State Transition

Use Case

Domain Analysis

Control Flow Data Flow

Usability

Documentation

Portability

Capacity

Ad Hoc

Benchmark

Pilot

Alpha

Beta

Black Box

Equivalence

Black Box

Boundary

Equivalence

Boundary

Black Box

Equivalence

Boundary

Black Box

Equivalence

Decision Table

Boundary

Black Box

Equivalence

State Transition

Use Case

State Transition

Domain Analysis

Use Case

State Transition

White Box

Domain Analysis

Use Case

State Transition

3

Outline

Basic concepts Black box testing White box testing

4

Basic Concepts

What is unit testing? Test focusing on the smallest units of code, such as

Functions, procedures, subroutines, subprograms Methods, classes

Component tested in isolation from the rest of the system and in a controlled environment:

Uses appropriately chosen input data Uses component-level design description as guide

Often the target of testing frameworks such as JUnit

5

Basics

Why unit testing? Foundation for other testing like integration and

system testing What to test?

Data transformations across the unit are tested Data structures are tested to ensure data integrity

When and who? Frequently done (at least partially) during code

development by code developers

6

Unit Test Procedures

Driver

Module to Be Tested

Stub Stub

Results

Testcases

7

Two Different Techniques

Black box Based on specification Inner structure of test object is not considered

White box Based on source code Inner structure of test object is the basis of test case selection

Often complementary Effectiveness of black box is similar to white box, but the

mistakes found are different (Hetzel 1976, Myers 1978) Use in combinations

8

Outline

Basic concepts Black box testing

Basic conceptsEquivalence classes…

White box testing

9

What Is Black Box Testing?

Unit (code, module) seen as a black box No access to the internal or logical structure Determine if given input produces expected

output

Input Output

10

Black Box Testing Test set is derived from specifications or requirements Goal is to cover the input space Lots of approaches to describing input space:

Equivalence classes Boundary value analysis Decision tables State transitions Use cases . . .

11

Advantage and Disadvantage

(Dis)Advantages It does not require access to the internal logic of a

component However, in most real-world applications, impossible

to test all possible inputs

Need to define an efficient strategy to limit number of test cases

12

General Process

Analyze specifications or requirements Select valid and invalid inputs (i.e., positive and

negative tests) Determine expected outputs Construct tests Run tests Compare actual outputs to expected outputs

13

Outline


Basic conceptsEquivalence classesBoundary values…

White box testing

14

Motivation Assume you test a sign function, int sign(int x),

whose result is defined as:

1 if x > 0

0 if x = 0

-1 otherwise (i.e., x < 0) One way to reduce the number of test cases is:

to partition the input into three subsets

{ x | x > 0}, { x | x = 0}, { x | x < 0} to pick one from each subset, e.g., 10, 0, and -10.

15

Basic Strategy of Equivalence Classes

Partition the input into equivalence classes This is the tricky part. It’s an equivalence class if:

Every test using one element of the class tests the same thing that any other element of the class tests

If an error is found with one element, it should be found with any element

If an error is not found with some element, it is not found by any element

Test a subset from each class

16

Exercise

Consider a factorial function, fact(n): if n < 0 or n >= 200, errorif 0 n 20, exact valueif 20 < n < 200, approximate value within 0.1%.

Q: What equivalence classes can you see?

17

Simple Example

Suppose you are building an airline reservation system. A traveler can be a child, an adult, or a senior. The price depends on the type of traveler. The seat reservation does not depend on the type of traveler.

Q: How many test cases can you identify for the reservation component and the billing component?

18

Heuristics for Finding Equivalence Classes

Identify restrictions for inputs and outputs in the specification.

19



If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).

20




If a number of values is required, create one valid and one or more invalid classes.

21





If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set.

22





If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set.

If there is a condition, create two classes, one satisfying and one not satisfying the condition.

23

Exercise

Define test cases for a program that reserves a golf tee time. The standard green fee is $65 on weekdays (Monday-Friday) and

$80 on weekend (Saturday and Sunday). However, an El Paso resident pays a reduced green fee of $45 and

$60 on weekdays and weekend, respectively. A senior (of age 60+) pays only $40 and $50 on weekdays and

weekend, respectively. A junior (of age <17) pays only $20 and $30 on weekdays and

weekend, respectively.

Q: How many equivalence classes? Define them.Q: Define one test case per equivalence class.

24

Outline


Basic concepts Equivalence classes Boundary values Decision tables …

White box testing

25

Boundary Values

Observations Programs that fail at interior elements of an

equivalence class usually fail at the boundaries too. Programmers often make errors on boundary

conditions (e.g., branch/loop condition x <= 10 instead of x < 10).

Test the boundaries if it should work for 1-99, test 0, 1, 99, 100. if it works for A-Z, try @, A, Z, [, a, and z

The hard part is identifying boundaries.

26

Hints

If a domain is a restricted set, check the boundaries. e.g., D=[1,10], test 0, 1, 10, 11 It may be possible to test the boundaries of outputs, also.

For ordered sets, check the first and last elements. For complex data structures, the empty list, full lists, the

zero array, and the null pointer should be tested. Extremely large data sets should be tested. Check for off-by-one errors.

27

More Hints

Some boundaries are not obvious and may depend on the implementation (use gray box testing if needed) Numeric limits (e.g., test 255 and 256 for 8-bit values) Implementation limits (e.g., max array size)

28

Example

Consider a simple voltage stabilizer with an input range of 190-240 volts 50-60 hertz.

29

Example


Voltage boundaries 189 volts (below the lowest boundary) 190 volts (lowest boundary) 240 volts (highest boundary) 241 volts (above the highest boundary)

Cycles (hertz) boundaries 49 hertz (below) 50 hertz (lowest) 60 hertz (highest) 61 hertz (above)

30

Example


Voltage boundaries 189 volts (below the lowest boundary) 190 volts (lowest boundary) 240 volts (highest boundary) 241 volts (above the highest boundary)

Cycles (hertz) boundaries 49 hertz (below) 50 hertz (lowest) 60 hertz (highest) 61 hertz (above)

How to combine?Exhaustive,

Pairwise (later),One extremes with

others fixed.

31

Exercise

Determine the boundary values for US Postal Service ZIP codes (5 digits such as 79912 or 5 digits hyphen 4 digits such as 79912-1818).

Determine the boundary values for a 15-character last name entry.

Hints: Two kinds of boundaries---size boundary and value boundary.

32

Outline


Basic concepts Equivalence classes Boundary values Decision tables Pairwise testing State transitions Use cases

White box testing

33

Decision Tables

Construct a table (to help organize the testing) Identify each rule or condition in the system that

depends on some input For each input to one of these rules, list the

combinations of inputs and the expected results

34

Example

Test Case C1

Student?

C2

Senior?

Result

Discount?

1 T T T

2 T F T

3 F T T

4 F F F

Theater ticket prices are discounted for senior citizens and students.

35

Exercise

Construct a decision table for a program that reserves a golf tee time. The standard green fee is $65 on weekdays (Monday-Friday) and

$80 on weekend (Saturday and Sunday). However, an El Paso resident pays a reduced green fee of $45 and

$60 on weekdays and weekend, respectively. A senior (of age 60+) pays only $40 and $50 on weekdays and

weekend, respectively. A junior (of age <17) pays only $20 and $30 on weekdays and

weekend, respectively.

36

Consider a website that must: operate correctly with different browsers: IE 5, IE 6, and IE 7;

Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome. work using RealPlayer, MediaPlayer, or no plug-in. run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0 accept pages from IIS, Apache and WebLogic running on

Windows NT, 2000, and Linux servers.

How many different configurations are there?

Pairwise Testing: Motivation

37

Consider a website that must: operate correctly with different browsers: IE 5, IE 6, and IE 7;

Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome. work using RealPlayer, MediaPlayer, or no plug-in. run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0 accept pages from IIS, Apache and WebLogic running on

Windows NT, 2000, and Linux servers.

How many different configurations are there?A: 9 browsers x 3 plug-ins x 7 OS’s x 3 web servers x 3

server OS’s = 1701 combinations

Motivation

38

Many such examples (i.e., finite sets of discrete values).

A bank is ready to test a data processing system Customer types: Gold, Platinum, Business, Non profits Account types: Checking, Savings, Mortgages, Consumer loans,

Commercial loans States (with different rules): CA, NV, UT, ID, AZ, NM

How many different configurations are there?

Motivation

39

Approaches?

When given a large number of combinations: Give up and don’t test Test all combinations: miss targets, delay product

launch, and go out of business Choose one or two cases Choose a few that the programmers already ran Choose the tests that are easy to create List all combinations and choose first few List all combinations and randomly choose some “Magically” choose a small subset with high probability of

revealing defects

40

Pairwise Testing

Combinatorial testing technique in which every pair of input parameters of software is tested.

Reasonable cost-benefit compromise Much faster than exhaustive testing More effective than less exhaustive methods that fail to exercise

all possible pairs of input parameters Why? Majority of software failures are caused by a single input

parameter or a combination of two input parameters. Each pair of input parameter values should be captured

at least by one test case. However, finding the least number of test cases is an

NP-complete problem.

41

Example

Consider software that takes three input parameters, say x, y, and z.

If each parameter can have three different values, then there will be 27 different pairs: (x1, y1), (x1, y2), …, (y3, z3).

A test case (x1, y3, z2), for example, captures three of these 27 pairs: (x1, y3), (x1, z2), and (y3, z3).

By selecting test cases judiciously, all pairs of input parameters can be exercised with a minimum number of test cases; e.g., a set of 9 test cases can capture all 27 pairs of three parameters, each with three different values.

42

State Transitions

Use state transitions (e.g., UML state machine diagrams) to derive test inputs

Cover a state transition diagram, e.g., Visit each state at least once Trigger each event at least once Exercise each transition at least once Exercise each path at least once*

* Might not be possible.

43

Example and Exercise

Reserved

Paid

Ticketed

Used

paidprinted

deliveredcanceled

Customer makes a reservation and has a limited time to pay. May cancel at any time.

Groups: How many tests to visit each state, trigger each event, exercise each transition, and exercise each path?

Cancelled

Unpaid

ordered/start timer

canceled/refund

timerexpired

canceled/refund

44

Use Cases

Use the use cases to specify test cases Use case specifies both normal, alternate, and

exceptional operations However, use cases may not have sufficient

details, esp. for unit testing

45

Outline


Control flow graph (CFG)Statement coverage…

46

White Box Testing

Test set is derived from structure of (source) code Also known as:

Glass box testing Structural testing

Often use a graph called a control flow graph (CFG) To represent code structure To cover it (CFG), e.g., all statements

Input Output

47

Control Flow Graphs Programs are made of three kinds of statements:

Sequence (i.e., series of statements) Condition (i.e., if statement) Iteration (i.e., while, for, repeat statements)

48

Control Flow Graphs Programs are made of three kinds of statements:

Sequence (i.e., series of statements) Condition (i.e., if statement) Iteration (i.e., while, for, repeat statements)

CFG: visual representation of flow of control Node represents a sequence of statements with single entry

and single exit Edge represents transfer of control from one node to another

49

Control Flow Graph (CFG)

Sequence If-then-else If-then Iterative

50

Control Flow Graph (CFG)

Join Join

Sequence If-then-else If-then Iterative

When drawing CFG, ensure that there is one exit: include the join node if

needed

51

Example

1. read (result);2. read (x,k)3. while result < 0 {4. ptr false5. if x > k then6. ptr true7. x x + 18. result result + 1 }9. print result

3

5

6

7

9

8

2

1

4

52

Example

1. read (result);2. read (x,k)3. while result < 0 {4. ptr false5. if x > k then6. ptr true7. x x + 18. result result + 1 }9. print result

3

5

6

7

9

8

2

1

4

3

4,5

6

Join

9

7,8

1,2

53

Exercise: Draw CFGs

Example 1

1. if (a < b) then 2. while (a < n)3. a a + 1;4. else 5. while (b < n)6. b b + 1;

Example 2

1. read (a, b); 2. if (a 0 && b 0) then {3. c a + b;4. if c > 10 then 5. c max;6. else c min; }7. else print ‘Done’;

Example 3

1. read (a, b); 2. if (a 0 || b 0) then {3. c a + b;4. while( c < 100)5. c a + b; }6. c a * b;

54

Outline


Control flow graph (CFG) Statement coverage Branch coverage Condition coverage Path coverage …

55

Coverage

Coverage-based testingAdequacy of testing in terms of the coverage of the product to be

tested, e.g., the percentage of statements executed or the percentage of functional requirements tested.

Control-flow coverage Statement Branch Condition Path

Data-flow coverage Def-use

56

Statement Coverage

Every statement gets executed at least once Every node in the CFG gets visited at least

once, also known as all-nodes coverage

57

Example

Q: Number of paths needed for statement coverage?

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

58

Branch Coverage

Every decision is made true and false. Every edge in a CFG of the program gets traversed at

least once. Also known as

Decision coverage All edges coverage Basis path coverage

Branch is finer than statement. C1 is finer than C2 if

T1 C1 (T2 C2 T2 T1)

true false

59

Example

Q: Number of paths needed for branch coverage?

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

60

Condition Coverage

Make each Boolean sub-expression (called a condition) of a decision evaluated both to true and false

Example:

if (x > 0 || y > 0) { … } else { … } C1: x > 0, x 0 C2: y > 0, y 0

Q: How many tests?

61

Many Variances of Condition Coverage

Condition coverage is not finer than branch coverage. There are pathological cases where you can achieve

condition and not branch. E.g., if (x > 0 || y > 0) { … } (x = 10, y = 0), (x = 0, y = 10): condition but not branch

Under most circumstances, achieving condition achieves branch

Many variances, e.g., Multiple condition Condition/decision Modified condition/decision (MC/DC)

62

CFG for Condition Coverage One way to determine the number of paths is to break the

compound conditional into atomic conditionals Suppose you were writing the CFG for the assembly language

implementation of the control construct if (A AND B) then

Cendif

(short circuit eval) (no short circuit eval) LD A LD A ; in general, lots of BZ :endif LAND B ; code for A and B LD B BZ :endif BZ :endif JSR C JSR C :endif nop

:endif nop

63

AND Condition

1. read (a, b); 2. if (a == 0 && b == 0) then {3. c a + b; }4. else c a * b;

paths:1, 2A,2B,3, J1, 2A, 2B, 4, J1, 2A, 4, J

2A

2B

Join

3

4

1

true false

true false

64

OR Condition

1. read (a,b); 2. if (a == 0 || b == 0) then {3. c a + b;4. while( c < 100)5. c a + b;}

Paths:1, 2A, 3, 4, 5, 4 … J1, 2A, 3, 4, J1, 2A, 2B, 3, 4, 5, 4, … J1, 2A, 2B, J

2A

3

Join

4

2B

5

1

true false

false

true

65

Outline


Control flow graph (CFG) Statement coverage Branch coverage Condition coverage Path coverage Def-use coverage

66

Path Coverage

Every distinct path through code is executed at least once.

All-paths coverage similar to exhaustive testing A path is a sequence of:

Statements Branches Edges

67

Example

1. read (x)2. read (z)3. if x 0 then {4. y x * z;5. x z; }6. else print ‘Invalid’;7. if z > 1 then8. print z;9. else print ‘Invalid’;

Test Paths:1, 2, 3, 4, 5, J1, 7, 8, J21, 2, 3, 4, 5, J1, 7, 9, J21, 2, 3, 6, J1, 7, 8, J21, 2, 3, 6, J1, 7, 9, J2

1,2,3

4,5

Join1

6

7

8

Join2

9

68

Linearly Independent Paths

It is not feasible to calculate the total number of paths. It is feasible to calculate the number of linearly

independent paths: A complete path which, disregarding back tracking (such as

loops), has an unique set of decisions in a program. Any path through the program that introduces at least one new

edge that is not included in any other linearly independent paths.

The number of linearly independent paths is the number of end-to-end paths required to touch every path segment at least once.

69

McCabe’s Cyclomatic Complexity Software metric for the logical complexity of a program Defines the number of independent paths in the basis

set of a program (basis set: maximal linearly independent set of paths).

Provides the upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once.

For edges (E) and nodes (N), cyclomatic complexity number V is:V = E – N + 2

70

Example: Complexity of CFG

1

2

3

1 3,4

5 6

Join

3,4

5

Join

N = 3E = 2V = E - N + 2 = 2 – 3 + 2 = 1

N = 1E = 0V = E - N + 2 = 0 - 1 + 2 = 1

N = 4E = 4V = E - N + 2 = 4 - 4 + 2 = 2

N = 3E = 3V = E - N + 2 = 3 - 3 + 2 = 2

71

Example

1

2,3

6

10

8

4,5

9

7

11

V = 11 – 9 + 2 = 4

Independent paths:1-111-2-3-4-5-10-1-111-2-3-6-8-9-10-1…-111-2-3-6-7-9-10-1…-11

72

Exercise: Cyclomatic Complexity

3

4,5

6

Join

9

7,8

1,2

1

2

3

Join

5

6

1,2

Join

3,4

5 6

Join

7

1, 2

3

Join

4

5

6

73

Linearly Independent Path Coverage

Cyclomatic-number criterion: construct a test set such that all linearly independent paths are covered.

1. read (x)2. read (z)3. if x 0 then {4. y x * z;5. x z; }6. else print ‘Invalid’;7. if z > 1 then8. print z;9. else print ‘Invalid’;

Cyclomatic complexity: 3 (= 9 – 8 + 2)Test Paths:

1, 2, 3, 4, 5, J1, 7, 8, J21, 2, 3, 4, 5, J1, 7, 9, J21, 2, 3, 6, J1, 7, 8, J2,

1,2,3

4,5

Join1

6

7

8

Join2

9

74

Outline


Control flow graph (CFG) Statement coverage Branch coverage Condition coverage Path coverage Def-use coverage

75

Data Flow Coverage

Consider how variables are treated along the various paths, a.k.a. data flow analysis, e.g., definitions and uses.

Many variations All-defs-uses All-defs All-uses All-C-uses/Some-P-uses All-P-uses/Some-C-uses All-P-uses

* P-uses are predicate uses, e.g., in conditions; all others are C-uses.

76

Def-Use Coverage Def-use coverage: every path

from every definition of every variable to every use of that definition is exercised in some test.

1. read (x);2. read (z);3. if x 0 then {4. y x * z;5. x z; }6. else print ‘Invalid’;7. if y > 1 then8. print y;9. else print ‘Invalid’;

Test Path: 1, 2, 3, 4, 5, 7, 8, J

1,2,3

4,5

Join

6

7

8

Join

9

D: x, zU: x

U: x, z D: y, x

U: none

U: y

U: y U: none

77

Strength of Coverage

Statement

Branch

Def-Use

Path

Arrows point from weaker to stronger coverage.

Stronger coverage requires more test cases.

Condition

78

Limitation of Paths

What they don’t tell you: Timing errors Unanticipated error conditions User interface inconsistency (or anything else) Configuration errors Capacity errors

Documents

1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13