Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING White-Box Testing Claus Brabrand [...

Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

White-Box Testing

Claus Brabrand[ brabrand@itu.dk ]

( “FÅP”: First-year Project Course, ITU, Denmark )

[ 2 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Outline

Motivation Why bother with testing?

What is Testing? Relation to other programming-related tasks?

White-box Testing Methodology: ”coverage testing”

Automation How to automate testing

Exercises Training for the exam

[ 3 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Learning & Exam Goals

”Product”:

”Oral Exam”:

Today, we’ll ”train for the exam”… :-)

[ 4 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Software Errors

Mobile Phones ’00-…

Freeze and odd behaviors (really annoying)!

Cruise Control System Model ’86 (Grady Booch)

Accellerated after car ignition car crashes

Baggage Handling System ’94-’95 (at Denver Int’l Airport)

$ 360,000,000 USD

[ 5 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

zzzz

Software Errors (cont’d)

Train Control System ’98 (Berlin)

Train cancellations

Mars Pathfinder July ’97

Periodic resets

Win95/98 w/ 3rd-Party Device Drivers late ’90es

Dysfunction (“blue screen of death”)!

...on mars!

[ 6 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Software Errors (cont’d2)

Therac-25 Radiation Therapy ’85-’87

Massive overdoses (6 deaths / amputations)!

Patriot Missile Guidance System ’91 (Gulf War 1.0)

Accumulating rounding errors deaths

Ariane V ’96 (one of the most expensive bugs, ever)

Conversion from 64-bit float to 16-bit signed int

[ 7 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

…and what about?!

Surgical Laser Control System

Oops…!

Air Plane Control System

Dysfunction (plane crash)!

Nuclear Powerplant Control System

Core melt-down (“China-syndrome”)!

[ 8 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Outline

Motivation Why bother with testing?

What is Testing? Relation to other programming-related tasks?

White-box Testing Methodology: ”coverage testing”

Automation How to automate testing

Exercises Training for the exam

[ 9 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Errors! (different kinds)

Syntactic errors: Mal-formed program:

Semantic errors: Symbol errors Type errors Other semantic errors:

(e.g. uninitialized vars)

Logical errors: Compiler: ”no errors”

int square(int x) { return x*x} *** syntax error at line 2

’;’ expected

int square(int x) { return n*n;} *** symbol error at line 2

undefined variable ”n”

int square(float x) { return x*x;}*** type error at line 2 function returns float, not int

int square(int x) { return x+x;} no errors found!!!

[ 10 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

ISO9126

Maintainability

ReliabilityUsability

Portability

How robust is the SW wrt. incorrect inputs, ’^C’, external netwk failures, ...?

How easy is the SW to understand?…and use?

How easy is it to transfer and adapt SW to new environment / platform?How easy is it to modify the SW?

And fix errors?

Int’l standard for evaluation of: Software Quality

Functionality Efficiency

ISO 9126

Does the SW do what it’s supposed to?

Does it work as intended?

How much time/memory/space/- bandwidth/… does the SW consume?

[ 11 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Testing vs. Debugging?

Testing vs. Debugging:

Functionality: Efficiency:

Quality Assurance:(Functionality)

Testing(Performance)

Testing

Diagnosis: Profiling

Purpose:

Regarding:

Debugging

[ 12 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Testing vs. Debugging (cont’d)

Evaluate test resultsFix problem(reprogram)

FunctionalityTesting

Quality assurance:

Is program ok?

Debugging

Diagnosis:

Determine problem?

01101021Program:

01101011

(greater confidence!)

SYSTEMATIC

Document test results

(confidence?!?)

Re-

01101011

[ 13 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Performance Testing vs. Profiling

Evaluate test resultsImprove program(reprogram)

Performance Testing

Quality assurance:

Efficient enough?

Profiling

Diagnosis:

Determine problem?

01101021Program:

01101011

(greater confidence!)Document test results

(confidence?!?)

Re-

SYSTEMATIC

01101011

[ 14 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Testing…:

”Testing is easy” (e.g., ”random experimentation”)

”Testing well is not easy” Requires SYSTEMATIC approach; test-case

production evaluation documentation

”Testing can never prove error absence” (i.e., testing is an ”incomplete process”)

[ 15 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Representative?

Comprehensive?

Quality?

Quantity?

…?

Appropriate Test Sampling?

[ 16 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

White-box vs. Black-box Test

White-box Testing: (aka., ”structural testing”) (aka., ”internal testing”)

Test focus: source code

Black-box Testing: (aka., ”behavioral testing”) (aka., ”external testing”)

Test focus: specification (manual)

Complementary Approaches!!!

n = in();

n = n/2;

odd(n)

n = 3*n+1;

out(n);

tt ff ~?program spec

[ 17 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

”Software Testing” (R. Patton)

Background reading: ”Software Testing”, Ron Patton, Sams Publishing, 2006 Part II (pp. 53 – 123); ”Testing Fundamentals”:

”Examining the Code”(chapter 6)

”Examiningthe Spec.”(chapter 4)

”Testing w/Blinders On”

(chapter 5)

”Testing w/X-ray Glasses”

(chapter 7)

Dynamic(at runtime)

Static(before runtime)

-testing -testingTime:

Type:

[ 18 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Outline

Motivation Why bother with testing?

What is Testing? Relation to other programming-related tasks?

White-box Testing Methodology: ”coverage testing”

Automation How to automate testing

Exercises Training for the exam

[ 19 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Test Coverage?

Method coverage: Does every method run (at least once)?

Statement coverage: Does every statement run (at least once)?

Branch coverage: Does every branch run (at least once)?

Path coverage: Does every path run (at least once)?

[ 20 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Statement coverage

Branch coverage: Does every branch run (at least once)?

-Box ”Branch Coverage Testing” is: Efficient (fast) ! Effective (thorough) !

Good for complicated program logic(esp. ”initialization errors”)

[ 21 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Control Structures

Control Structures: Statements (or Expr’s) that affect ”flow of control”:

if-else:

if:

if ( Exp ) Stm1 else Stm2

if ( Exp ) Stm

Stm1

Exptrue false

Stm2

confluence

Stm

Exptrue false

confluenceThe expression must be of type boolean; if it evaluates to true, the given statement is executed, otherwise not.

The expression must be of type boolean; if it evaluates to true, Statement-1 is executed, otherwise Statement-2 is executed.

[syntax]

[semantics]

[syntax]

[semantics]

[ 22 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Control Structures (cont’d)

while:

for:

while ( Exp ) Stm

for (Exp1 ; Exp2 ; Exp3) Stm

Equivalent to:

The expression must be of type boolean; if it evaluates to false, the given statement is skipped, otherwise it is executed and afterwards the expression is evaluated again. If it is still true, the statement is executed again. This is continued until the expression evaluates to false.

{ Exp1; while ( Exp2 ) { Stm Exp3; }}

Stm

Exptrue false

confluence

Exp1;

Exp2true false

Stm

confluence

Exp3;

[syntax]

[semantics]

[syntax]

[semantics]

[ 23 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Stm/Branch Coverage Testing

if: TEST condition true and false

if-else: TEST condition true and false

while: TEST zero, one, more-than-one iterations in loop

for: TEST zero, one, more-than-one iterations in loop

[ 24 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Example 1

public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma);}}

/* 1if-else */

/* 2for */

/* 3if-else */

/* 4if */

if

else

for

if

elseif

Choice points?

[ 25 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Control-Flow Graph

CFG:int mi, ma;

System.out.println("No numbers");

args.length == 0

mi = ma = Integer.parseInt(args[0]);

int i=1;

i < args.length

i++;

System.out.println(”min=" + mi + "," + "max=" + ma);

int obs = Integer.parseInt(args[i]);

obs > ma

ma = obs; mi < obs

mi = obs;

true false

true false

true false

true false

2

34

1

[ 26 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

”Coverage Table”:

Coverage Table

Input property

No numbers

At least one number

Exactly one number

Exactly two numbers

At least three numbers

N > current maxN current maxN cur max & N > cur minN cur max & N cur min

Data set

A

B

B

C

E

C

D

E (3rd num)

E (2nd num)

Choice

1ife true

false

2for zero-times

once

more-than-once

3ife true

false

4if true

false

[ 27 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

”Expectancy Table”:

Expectancy Table

Data set

A

B

C

D

E

Advice:Avoid expected 0’s (i.e., zeroes)

(Default value in many languages.)Advice:Avoid reusing same numbers in tests

(Data layout sometimes reuse old memory.)

Input

[17]

[27,29]

[39,37]

[49,47,48]

Expected output

”no numbers”

”min=17,max=17”

”min=27,max=29”

”min=37,max=39”

”min=47,max=49”

Actual output

”no numbers”

”min=17,max=17”

”min=27,max=29”

”min=39,max=39”

”min=49,max=49”

=

[ 28 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma);}}

Debugging ’ D ’ then reveals…

/* 1if-else */

/* 2for */

/* 3if-else */

/* 4if */

if

else

for

if

elseif

Should have been:

(obs < mi)

[ 29 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Re-Test !

…as debugging oftenintroduces new errors !

Fixed Program:Coverage Table:

Expectancy Table:

Recall: no guarantee!

[ 30 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Example 2public static void main ( String[] args ) { int mi1 = 0, mi2 = 0; if (args.length == 0) /* 1if-else */ System.out.println("No numbers"); else { mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2if-else */ System.out.println("Smallest = " + mi1); else { int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3if */ { mi2 = mi1; mi1 = obs; } for (int i = 2; i < args.length; i++) { /* 4for */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5if-else */ { mi2 = mi1; mi1 = obs; } else if (obs < mi2) /* 6if */ mi2 = obs; } System.out.println("The two smallest are: " + mi1 + " and " + mi2);} } }

[ 31 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Coverage Table (Ex. 2)

Choice Input property Data set

1ife true No numbers

1ife false At least one number

2ife true Exactly one number

2ife false At least two numbers

3if true 2nd number ≥ 1st number

3if false 2nd number < 1st number

4for zero-times Exactly two numbers

4for once Exactly three numbers

4for more-than-once At least four numbers

5ife true 3rd number < current min

5ife false 3rd number ≥ current min

6if true 3rd ≥ cur min & 3rd < 2nd least

6if false 3rd ≥ cur min & 3rd ≥ 2nd least

A

B

B

C

C

D

D

E

H

E

F

F

G

[ 32 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Expectancy Table (Ex. 2)

Data set Input Expected output = Actual output

A ”no numbers” ”no numbers” B [17] ”17” ”17”

C [27,29] ”27 and 29” ”27 and 0”

D [39,37] ”37 and 39” ”37 and 39”

E [49,48,47] ”47 and 48” ”47 and 48”

F [59,57,58] ”57 and 58” ”57 and 58”

G [67,68,69] ”67 and 68” ”67 and 0”

H [77,78,79,76] ”76 and 77” ”76 and 77”

Debugging reveals that variable

”mi2” erroneously retains initialization (0).

[ 33 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Debugging (Ex. 2)public static void main ( String[] args ) { int mi1 = 0, mi2 = 0; if (args.length == 0) /* 1if-else */ System.out.println("No numbers"); else { mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2if-else */ System.out.println("Smallest = " + mi1); else { int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3if */ { mi2 = mi1; mi1 = obs; } for (int i = 2; i < args.length; i++) { /* 4for */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5if-else */ { mi2 = mi1; mi1 = obs; } else if (obs < mi2) /* 6if */ mi2 = obs; } System.out.println("The two smallest are: " + mi1 + " and " + mi2);} } }

mi2 = obs;

Re-Test:

[ 34 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Control Structures (cont’d2)

switch:

do-while:

”?:”; ”conditional expression”:

”&&”; ”lazy conjunction” (aka., ”short-cut ”):

”||”; ”lazy disjunction” (aka., ”short-cut ”):

switch ( Exp ) { Swb* }

case Exp : Stm* break;

do Stm while ( Exp );

default : Stm* break;

Swb:

Exp1 ? Exp2 : Exp3

Exp1 && Exp2

Exp1 || Exp2

Choice points?

[ 35 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Control Structures (cont’d3)

try-catch-finally (exceptions):

return / break / continue:

”method invocation”: e.g.;

”recursive method invocation”: e.g.;

”virtual dispatching”: e.g.;

try Stm1 catch ( Exp ) Stm2 finally Stm3

f(x)

return ; return Exp ; break ; continue ;

f(x)

f(x)

[ 36 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Outline

Motivation Why bother with testing?

What is Testing? Relation to other programming-related tasks?

White-box Testing Methodology: ”coverage testing”

Automation How to automate testing

Exercises Training for the exam

[ 37 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Test Automation

(Re-)Running tests is boooring (& error prone) Thus, automate them ”once-and-for-all”

JUnit:

Can be run from Eclipse/JUnit: (if appropriately ”subclassing” TestCase)

public class MyTestCase extends TestCase {

/* ...other tests... */}

@Test // Testing if 3*2=6: public void testMultiplication() { assertEquals("Multiplication", 6, 3*2); }

[ 38 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Outline

Motivation Why bother with testing?

What is Testing? Relation to other programming-related tasks?

White-box Testing Methodology: ”coverage testing”

Automation How to automate testing

Exercises Training for the exam

[ 39 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Exercise: Part I:

Part II:

Program merge (in Java):

Test your merge method:a) Label choice pointsb) Build ”coverage table” & make data set (test suite)c) Build ”expectancy table”d) Run test suite (upon failure: fix and retest program)

Introduce subtle bug Run test to document presence of bug Submit “erroneous program” to class program pool

List<Integer> merge(List<Integer> list1, List<Integer> list2);

Pick “erroneous program” from class program pool Test merge program (and debug to find bug) Re-Test fixed merge program Write report (and send it to the teaching assistant)

(produce)

(consume)

Warm up exercise:Draw a control-flow diagram for the ”do-while” construction

[ 40 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Specification (merge)

Interface (for ”merge”):

I/O assumptions: Input: both lists are sorted:

…and, numbers occur maximum once (in each list)

Output: list must be sorted: …and, numbers occur maximum once

Programming constraints: no recursion ! no java.util.Iterator’s !

List<Integer> merge(List<Integer> list1, List<Integer> list2);

[ 41 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Report (ca. 3 pages): Must explain how you tested, (debugged), & re-tested

the erroneous ”merge” (it must include, at least): i) The erroneous program ii) a ”Control-Flow Graph” for the program [hand-drawn ok]

(incl. labelled ”choice points”) iii) ”Coverage Tables” iv) ”Expectancy Tables”

Submit the testing report to the T.A. (Anders) (deadline: Tuesday; March, 11 at 09:00 CET) Note: the report is only on Part II

(i.e., the program you didn’t write)

The Testing Report

[ 42 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING

Example 1

public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma);}}