Automatic Unit Tests Generation Tools SEPL Seminar Benny Pasternak December 2007

Automatic Unit Tests Generation Tools

SEPL Seminar

Benny Pasternak

December 2007

2

Agenda

Brief Introduction Automatic Generation Tools

JCrasher Symstra GenUTest

Summary

3

Definition

A unit is the smallest testable part of an application

In the object oriented paradigm it is a class A unit test consists of a fixed sequence of

method invocations with fixed arguments Unit test explores a particular aspect of the

behavior of the Class Under Test (CUT)

4

Yet Another Calculator Examplepublic class Calculator {

public int add(int a, int b) { return a + b; }

public int subtract(int a, int b) { return a - b; }

public int multiply(int a, int b) { return a * b; }

public int divide(int a, int b) { return a / b; }}

5

Small JUnit Testpublic class CalculatorTest {

@Test public void add3and5Equals8() {

Calculator calc = new Calculator();

int result = calc.add(4, 6);

assertEquals(result, 10);

}

@Test(expected= java.lang.ArithmeticException.class)

public void divideByZeroThrowsException() {


int result = calc.divide(3,0);

}

@Test public void subtract5from6Equals2()

{


int result = calc.subtract(6, 5);

assertEquals(result, 2);

}

}

6

Test run and feedback

7

Effective Unit Tests

Effective unit test should be: Comprehensive – unit test should exercise all

class methods and achieve high code coverage rate

Test in Isolation – CUT dependent objects (e.g., Logger, Serializer) should not be tested

8

Comprehensiveness

StackInt

push(int)

int pop()

int top()

bool empty()

reverse()

Unit Test

test1()

.

.

testn()

test2()

9

Isolation

Logger

Serializer

StackInt

push(int)

int pop()

int top()

bool empty()

reverse()

LinkedListMockLinkedList

MockLogger

MockSerializer

Unit Test

test1()

test2()

.

.

testn()

10

Benefits

Enables the immediate detection of bugs introduced into a unit whenever code changes occur Unit tests provide a safety net of regression tests

and validation tests which encourage developers to refactor existing code.

Elimination of uncertainty in units Simplifies integration

Provides a sort of living documentation of the system

11

However…

Writing effective unit tests is a hard and tedious task…

Lots of unit tests need to be written

(unit tests code may be larger than project code)

Provide tools to assist developers

12

Tool Classification

Frameworks – JUnit, NUnit Mock Frameworks – EasyMock, jMock Generation – Automatic generation of unit

tests Selection – Selecting a small set of unit tests

from a large one Prioritization – Deciding what the “best

order” to run the tests

13

Unit Test Generation

Test Generation – generate a sequence of CUT method-calls

Test Assertion – Provide an assertion to determine if the test result is correct

14

Test Generation Techniques

Random Execution random sequences of method calls with random values

Symbolic Execution method sequences with symbolic arguments builds constraints on arguments produces actual values by solving constraints

Capture & Replay capture real sequences seen in actual program runs or test

runs

15

Test Assertion Generation Techniques

Manual (Design by Contract) Uncaught Exceptions

Classifies a test as potentially faulty if it throws an unexpected exception

Operation Model Infer an operational model from manual tests Properties: objects invariants, method pre/post conditions Properties violation potentially faulty

Capture & Replay Compare test results/state changes to the ones captured

in actual program runs and classify deviations as possible errors.

16

Generation Tool Map

Random Execution

Symbolic Execution

Capture & Replay

Uncaught

Exceptions

Operational

Model

Generation Selection Prioritization

Eclat

Symclat

JCrasher

Symstra

Test Factoring

SCRAPE

Rostra

CR Tool

GenuTestSubstra

PathFinder

JartegeJTest

17

Tools Presented Today

JCrasher (Random & Uncaught Exceptions) Symstra (Symbolic & User provided

specifications) GenUTest (Capture & Replay)

18

JCrasher – An Automatic Robustness Tester for Java (2003)

Christoph Csallner Yannis Smaragdakis

Available at

http://www-tatic.cc.gatech.edu/grads/c/csallnch/jcrasher/

19

Goal Robustness quality goal – “A class should not

crash with an unexpected runtime exception, regardless of the parameters provided.”

Do not assume anything about domain Robustness goal applies to all classes Function to determine class under test

robustness: expected exception type passunexpected exception type fail

20

Parameter Space

Huge parameter space. Example: m(int,int) has 2^64 param combinations Covering all parameters combination is

impossible May not need all combinations to cover all

control paths that throw an exception Pick a random sample Control flow analysis on byte code could derive

parameter equivalence class

21

Architecture Overview

22

Type Inference Rules

Search class under test for inference rules Transitively search referenced types Inference Rules

Method Y.m(P1,P2,.., Pn) returns X:X Y, P1, P2, … , Pn

Sub-type Y {extend | implements } X:X Y

Constants: int -1, 0 ,1

Add each discovered inference rule to mapping:X inference rules returning X

23

Generate Test Cases For a Method

24

Exception Filtering

JCrasher runtime catches all exceptions Example generated test case:Public void test1() throws Throwable {

try { /* test case */ }catch (Exception e) {

dispatchException(e); // JCrasher runtime}

Uses heuristics to decide whether the exception is a Bug of the class pass exception on to JUnit Expected exception suppress exception

25

Exception Filter Heuristics

26

Symstra: Framework for Generating Unit Tests using Symbolic Execution (2005)

Tao Xie Darko Wolfram David

Marinov Schulte Notkin

27

Binary Search Tree Example

public class BST implements Set {Node root;int size;static class Node {

int value;Node left;Node right;

}public void insert (int value) { … }public void remove (int value) { … }public bool contains (int value) { … }public int size () { … }

}

28

Other Test Generation Approaches

Straight forward – generate all possible sequences of calls to methods under test

Cleary this approach generates too many and redundant sequences

BST t1 = new Bst(); BST t2 = new Bst();

t1.size(); t2.size();

t2.size();

29

Other Test Generation Approaches

Concrete-state exploration approach Assume a given set of method calls arguments Explore new receiver-object states with method

calls (BFS manner)

30

Exploring Concrete States

Method arguments: insert(1), insert(2), insert(3), remove(1), remove(2), remove(3)

new BST()

insert(1) insert(2) insert(3)

remove(1)remove(2)

remove(3)

1st Iteration

1 2 3

31

Exploring Concrete States

Method arguments: insert(1), insert(2), insert(3), remove(1), remove(2), remove(3)

new BST()


remove(1)remove(2)

remove(3)

2nd Iteration

1

2

1 2 3

1

3

insert(2) insert(3)

remove(1)

remove(2)

remove(3)

32

Generating Tests from Exploration

Collect method sequences along the shortest path

new BST()


remove(1)remove(2)

remove(3)

2nd Iteration

1

2

1 2 3

1

3

insert(2) insert(3)

remove(1)

remove(2)

remove(3)

BST t = new BST();

t.insert(1);

t.insert(3);

33

Exploring Concrete States Issues

Not solved state explosion problem Need at least N different insert arguments to

reach a BST with size N experiments shows memory runs out when N = 7

Requires given set of relevant arguments in our case insert(1), insert(2), remove(1), …

34

Concrete States Symbolic States

new BST()


remove(1)remove(2)

remove(3)

1

2

1 2 3

1

3

insert(2) insert(3)

remove(1)

remove(2)

remove(3)

new BST()

insert(x1)

x1

x1x2

X1<x2

insert(x2)

35

Symbolic Execution

Execute a method on symbolic input values Inputs: insert(SymbolicInt x)

Explore paths of the method

Build a path condition for each path Conjunct conditionals or their negations

Produce symbolic states (<heap, path condition>) For example

x1x2

X1<x2

36

Exploring Symbolic Statespublic void insert(SymbolicInt x) {

if (root == null) {root = new Node(x);

} else { Node t = root; while (true) {

if (t.value < x) { // explore rigtht subtree} else if (t.value > x) { // explore left subtree } else return;

} }

}size++;

}

new BST()

insert(x1)

x1

x1x2

X1<x2

insert(x2)

x1x2

X1>x2

x1X1= x2

S1

S2

S3 S4 S5

37

Generating Tests from Exploration

Collect method sequences along the shortest path

Generate concrete arguments by usinga constraint solver

new BST()

insert(x1)

x1

x1x2

X1<x2

insert(x2)

x1x2

X1>x2

S1

S2

S3 S4

BST t = new BST;()

t.insert(x1);

t.insert(x2);

BST t = new BST;()

t.insert(-1000000);

t.insert(-999999);

X1>x2

38

Results

39

Results

40

More Issues

Symstra uses specifications (pre, post, invariants) written in JML. These are transformed to run-time assertions

Limitations: can not precisely handle array indexes currently supports primitive arguments can generate non-primitive arguments as

sequence of method calls. These eventually boil down to methods with primitive arguments

GenUTest: A Unit Test and Mock Aspect Generation Tool

Benny Pasternak

Shmuel Tyszberowicz

Amiram Yehudai

42

Motivation

Writing effective unit tests is a hard and tedious process At maintenance phase, writing tests from scratch is

not considered cost effective Corollary: Maintenance remains a difficult process

Goal: Automatically generate unit tests for projects in maintenance phase

43

Example Developers are asked to create effective unit tests for an

existing software project

StackInt is an implementation of integers’ stack, with the operations: Push Pop Top Empty Reverse

Goal is to test StackInt effectively

44

Obtaining Test Cases From Existing Tests System/Module test

that exercises IntStack as follows:

Test can be used to obtain test cases for unit tests

stack:IntStack

new()

push(2)

push(3)

reverse()

addFirst(2)

addFirst(3)

lst:LinkedList

2

pop()removeFirst()

2

new()

…

45

GenUTest

Captures and records execution of IntStack during module/system tests in order to obtain test cases

Recorded events are used to generate unit tests for IntStack

Unit tests assist developers in the testing process

46

Example – Generated Unit Test

1 @Test public void testpop1() 2 {3 // test execution statements4 IntStack IntStack_2 = new IntStack(); // #1 5 IntStack_2.push(2);

// #2 6 IntStack_2.push(3);

// #3 7 IntStack_2.reverse();

// #4 8 int intRetVal6 = IntStack_2.pop();

// #5 910 // test assertion statements11 assertEquals(intRetVal6,2);11 }

Unit Test Code stack:IntStack

new()

push(2)

push(3)

reverse()

addFirst(2)

addFirst(3)

lst:LinkedList

2

pop()removeFirst()

2

new()

…

47

Example

Logger

Serializer

StackInt

push(int)

int pop()

int top()

bool empty()

reverse()

LinkedList

Unit Test

test1()

test2()

.

.

testn()

48

Aspect Oriented Programming

class StackInt {

void reverse() {

LinkedList newlst = new LinkedList();int size = lst.size();

for (int i = 0; i < size; i++) { int elem = lst.get(i);

newlst.addFirst(elem);

}

lst = newlst;

}

int pop() {

int elem = lst.removeFirst();

return elem;

}

}

Join points

• object instantiation

• method-calls

• field setter/getter

Pointcut 1

Pointcut 2

Around Advice

print(“Before”);

execute join point

print(“After”);

print(“Before”);

print(“After”);

49

AOP – Quick Summary

Join points – well defined execution points in the control flow of the program (object instantiation, method-calls, field member access)

Pointcut – expression that specifies a set of join points

Advices – code specified to execute before, after, or around pointcuts

Aspects – The equivalent to class. Holds pointcut declarations and advices

50

Example

Logger

Serializer

StackInt

push(int)

int pop()

int top()

bool empty()

reverse()

LinkedList

Unit Test

test1()

test2()

.

.

testn()

Pointcut 2

Pointcut 1

Mock Advice

mock object behavior code

Mock Advice

mock object behavior code

Implementation

51

52

Capture Phase

Inter-object interactions logs

Capture Code

Program Code AspectJCompiler

• Software is instrumented with capture functionality at constructor-calls, method- calls, field getter/setters

returnVal = targetObject.someMethod(arg1 , … argn );

signaturetarget object

signature

arguments’ valuesreturn value

, target object, arguments’ valuesreturn value/thrown exception

Instrumented System – P’

• Attributes of interactions captured:

• Inter-object interactions are captured and logged during runtime

53

Capture Phase

Instrumentation is performed using AspectJ More elegant and simpler mechanism However, it is a weaker mechanism than

conventional instrumentation techniques that directly access a program’s Java bytecode Requires the use of elegant workarounds to handle

special cases: non primitive arrays: obj1.peform(myArray[6]); string syntactic: String me = “Benny”;

54

Generation Phase – Step I

Given a testable event, a backtracking algorithm recursively generates the statements needed for executing the test

1 @Test public void testpop1() {2 // test execution statements3456789 1011 }

IntStack IntStack_2 = new IntStack(); // #1IntStack_2.push(2); // #2IntStack_2.push(3); // #3IntStack_2.reverse(); // #4int intRetVal6 = IntStack_2.pop(); // #5

55

Backtracking Algorithm

Generally, in order to execute a test, GenUTest needs to generate statements that replay the relevant sequence of recorded events in a correct manner Execution of: intRetVal1 = obj1.process(obj2)

Requires: obj1 and obj2 must be in the correct state

56


Object states are represented by method-calls sequences:

stateT(o) = (method , method , … method )

Time is represented by a sequence number incremented before a method begins execution and after it finishes execution

The interval [before, after] is called the method-interval

t1 t2 tn

t1< t2<..< tn<T

57


Logged interactions:

…

obj1.report()[65,80]

obj2.report()[51,64]

obj1.process(obj2)[31,50]

obj2.perform(obj3)[21,30]

obj3.initialize()[9,20]

obj2 = new Type2()[5,8]



obj3obj2obj1Method Interval

58

Backtracking Algorithm (cont) Generated statements:

Algorithm may need to remove redundant statements Static and dynamic types of objects are stored for:

casting – myObject = (MyObject)List.get(2); null values – obj1.process(null); static methods – System.out.println(“Hello World”); changes in modifier access policy – inner private class inheriting

from a public outer one

Type1 obj1 = new Type1();Type3 obj3 = new Type3();Type2 obj2 = new Type2();obj3.initialize();obj2.perform(obj3);int intRetVal1 = obj1.process(obj2);

59

Generation Phase – Step II

Case I – Value is returned from the call Generate statements that compare valuetest with valuecaptured.

Case II – An exception is thrown Generate statements that expect a particular exception

1 @Test public void testpop1() {2 // test execution statements3456789 // test assertion statements1011 }

IntStack IntStack_2 = new IntStack(); // #1 IntStack_2.push(2); // #2IntStack_2.push(3); // #3IntStack_2.reverse(); // #4int intRetVal6 = IntStack_2.pop(); // #5

assertEquals(intRetVal6,2);

60

Mock Aspect Generation

Definitions: Incoming method-calls – method-calls invoked by

the unit test on the Class Under Test (CUT) Outgoing method-calls – method-calls invoked by

the CUT on dependent objects

Class Under Test

Unit Test

Object #1

Object #2

`

Object #n

`

incoming method-calls outgoing method-calls

61


Definitions: mi(A()) – method interval of A() [BeforeA, AfterA] method A() contains method B() if mi(A()) contains mi(B())

[BeforeA, AfterA] [BeforeB, AfterB]

Observations: method B() resides in the control flow of method A() iff

method A() contains method B() An outgoing method-call of the CUT is contained in exactly

one incoming method-call∩

62


Last definition Outgoing(I()) is the sequence <Io1(), Io2(), …, Ion()>

I() is an incoming method call Io1(), Io2(), …, Ion() are all the outgoing method-calls

contained in I()

If method o() is contained in method I() and method o() is the jth element in Outgoing(I()) then method o() is uniquely identified by the pair (mi(I()), j)

63

Mock Aspect Generation – Needed Example Four outgoing method-calls to

addFirst() mi(push(2)) is [5,8] Outgoing(push(2)) =

<addFirst(2)> addFirst(2) is uniquely

identified by <[5,8],1>

mi(push(3)) is [9,12] Outgoing(push(3)) =

<addFirst(3)> addFirst(3) is uniquely

identified by <[9,12],1>

12

13

IntStack_2:IntStack

new()

push(2)

push(3)

reverse()

addFirst(2)

addFirst(3)

get(0)

lst:LinkedList

newlst:LinkedList

get(1)

addFirst(2)

2

pop()removeFirst()

2

5

8

9

67

1011

new()

1

4

new()23

addFirst(3)

3

2

Class Under Test

Unit Test

Object #1

`

Object #2

Object #n

`


24

64

Mock Aspect Generation – Needed Example Mi(reverse()) is [13,24] Outgoing(reverse()) =

<get(0), addFirst(3), get(1), addFirst(2)>

get(0) is uniquely identified by <[13,24],1>

addFirst(3) is uniquely identified by <[13,24],2>

get(1) is uniquely identified by <[13,24],3>

addFirst(2) is uniquely identified by <[13,24],4>

12

13

IntStack_2:IntStack

new()

push(2)

push(3)

reverse()

addFirst(2)

addFirst(3)

get(0)

lst:LinkedList

newlst:LinkedList

get(1)

addFirst(2)

2

pop()removeFirst()

2

5

8

9

67

1011

new()

1

4

new()23

addFirst(3)

3

2

Class Under Test

Unit Test

Object #1

`

Object #2

Object #n

`


24

65


Algorithm works as follows:1. For each incoming method-call I() of the CUT, outgoing(I())

is calculated2. Each outgoing method-call is uniquely identified3. For each incoming method-call I() different pointcut and

advice are generated4. A statement that sets method interval and clears the

element counter is added before the incoming method call is invoked in the unit test

5. Bookkeeping code is added in advice6. Backtracking algorithm is applied to mimic the behavior of

the dependent object in the advice

66

Mock Aspect Generation – Sample Code

Integer around(): call (Object java.util.LinkedList.get(int)) && restriction()

{

if (before == 13 && after == 24) { if (elementCounter == 1) {

elementCounter++;return 3;

}

if (elementCounter == 3) {elementCounter++;return 2;

}}thrown new RuntimeException(“Invalid method interval”);

}

void setMI(int b, int a){

before = b;after = a;elementCounter == 1;

}

@Test public void testpop1() {

// test execution statementsIntStack IntStack_2 = new IntStack(); IntStack_2.push(2);IntStack_2.push(3);

StackIntMockAspect.setMI(13,24);

IntStack_2.reverse();

int intRetVal6 = IntStack_2.pop();

// test assertion statementsassertEquals(intRetVal6,2);

}

StackIntMockAspect.ajStackIntTest.java

4

3

5

5

6

6

67

Implementation Overview

Inter-object interactions logs

Capture Code

Program Code AspectJCompiler

Instrumented System – P’

Capture Phase

Unit tests &Mock AspectsGenerator

Generation Phase Mock Aspects

Unit Tests

JUnit

Test Results

Unit Testing Phase

AspectJCompiler

CUT weaved with mock advices

68

Experimentation

Used on open source project JODE (Java Optimize and Decompile Environment)http://jode.sourceforge.net/

JODE is a medium sized project ~35K loc Executed JODE combined with GenUTest on

a chosen input GenUTest generated 592 unit tests from

recorded data captured during runtime

69

Experimentation Measured code coverage with EclEmma

(www.eclemma.org/):1. Execution of JODE on chosen input

Coverage is 25% of JODE’s lines of code

2. Execution of generated unit tests with JUnitCoverage is 5.2% of JODE’s lines of code

Current limitations and bugs may cause generation of invalid tests Primary reason for differences in loc coverage rate

70

Limitations

Partial support for inner classes and anonymous classes

Does not support multi-thread applications Support of arrays need to be improved Scalability and performance issues

71

Future Work

Handle limitations and extend support:Inner/Anonymous classes, multi-threaded support,

Optimize array handling, optimize performance Scalability – selective capturing, detect

redundant tests, discard non mutating events, make use of concrete object states

Research effectiveness in detecting regression bugs

72

More tools not mentioned…

JTest – commercial product by parasoft Jartege – operational model formed from JML PathFinder – symbolic execution tool by NASA Symclat – An evolution of Eclat & Symstra Rostra – Framework for detecting redundant object

oriented unit tests Orstra – Augmenting Generated Unit-Test Suites

with Regression Oracle Checking Agitator – www.agitar.com (Nice presentation

available on website)

73

Summary Quick unit testing Survey Lots of unit testing tools

Roughly categorized into frameworks, generation, selection and prioritization

Concentrated on various test and assertion generation techniques used in several tools: JCrasher Symstra GenUTest

74

Thank you for listening

Questions?

Documents

Automatic Unit Tests Generation Tools SEPL Seminar Benny Pasternak December 2007