Automating the Generation of Test Cases from Object-Z Specifications

Adnan Ashraf Aamer Nadeem Center for Software Dependability, Mohammad Ali Jinnah University,

Islamabad, Pakistan rareadnan@gmail.com, anadeem@jinnah.edu.pk

Abstract

In this paper, we propose a test case generation technique based on Object-Z specification of a class, based on formalization of the test case generation strategy. The proposed technique enables automatic generation of test cases from an Object-Z specification, under the specified strategy. We also propose to enhance the prototype tool TinMan, which was originally developed to automate derivation and management of testing information for specification-based class testing, by introducing semantics knowledge for the application of test case generation strategies in a format acceptable to the tool.

1. Introduction

Formal specification-based test case generation is a vastly explored area. However, most formal specification-based test case generation techniques are manual. Xie and Notkin [10] point out that manually generated test cases are often insufficient and the main alternative is automated test case generation. In this paper, we propose an automated test case generation technique for specification-based class testing. The test case generation technique builds upon earlier work in the area of test case generation from Object-Z specifications [8]. We build our technique upon object-oriented TTF technique [2], which is supported with (semi-) automated tools, TinMan [7], an automated oracle generator, and ClassBench.

TinMan [7] is a test derivation and management tool for specification-based class testing. It was originally developed to provide automated support for test derivation and management in object-oriented TTF. However, TinMan could provide automated support only for mundane tasks and it lacked semantics knowledge with respect to application of

test case generation strategies. We propose to extend TinMan by introducing semantics knowledge for the application of test case generation strategies in a format acceptable to the tool.

The rest of the paper is organized as follows. Section 2 discusses related work. Section 3 introduces our technique. Section 4 demonstrates the proposed technique on a small case study. Section 5 draws some conclusions.

2. Related Work

Stocks and Carrington [9] proposed test template framework (TTF) for test case generation from Z specifications of software. TTF extracts input predicate of an operation under test and calls it valid input space (VIS). VIS of an operation is partitioned by applying test case generation strategies to generate abstract level test cases called test templates (TTs), which are represented as Z schemas. TTs are arranged in tree-like hierarchical structures called test template hierarchies (TTHs). The tester may apply any suitable test case generation strategy for generating TTs, such as category partitioning, boundary value analysis, and type-based strategy.

Murray et al. [5], [6] extended TTF to incorporate capabilities for formal specification-based object-oriented testing. The extended TTF uses Object-Z specifications of the class under test for deriving testing information. A specification-based class testing technique [2], [4] was developed that generates test cases, executes tests, and evaluates test results. The technique uses Object-Z specifications of the class under test to generate test cases and to derive test oracle. The object-oriented TTF technique allows semi-automated generation of test cases. TinMan [7] is a prototype tool for facilitating the process of TTs generation using object-oriented TTF.

Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06)0-7695-2655-1/06 $20.00 © 2006

Liu et al. [3] proposed Test Class Framework (TCF) for automated class testing from Object-Z specifications. TCF includes three aspects of class testing: testing each method, testing method interactions, and testing inherited features.

Object-oriented TTF technique [2] is supported with (semi-) automated tools, which are TinMan, an automated oracle generator, and ClassBench. No such automation is currently available for TCF technique. We build our technique upon object-oriented TTF technique.

3. Enhancing TinMan to Automatically Generate Test Cases using a Specified Strategy

TinMan [7] provides partial automated support for object-oriented TTF. We enhance the efficacy of object-oriented TTF technique by introducing automated support for specification-based test generation.

3.1. Existing TinMan

TinMan imports Object-Z specification of software in LaTeX format, and outputs generated test templates arranged as a test template hierarchy. An abstract view of existing TinMan is shown in Figure 1.

Figure 1. An abstract view of existing TinMan

TinMan automates only mundane tasks in TTF and does not provide automated support for the application of test case generation strategies. It recognizes test case generation strategies only as names. The key reason for this is a lack of semantics knowledge of the test case generation strategies. We see a possibility to introduce automated support for the application of test case generation strategies in TinMan.

3.2. Proposed Enhancement

We propose to introduce semantics knowledge, in the form of formal specification of test case generation strategies, to TinMan. We do not restrict any test case generation strategy, which was originally allowed in the object-oriented TTF. We also support inclusion of customized strategies at later stages. Our technique enhances TinMan, such that, a tester only needs to select a predicate and a test case generation strategy,

and TinMan automatically applies the selected test case generation strategy to the selected predicate.

TinMan is modified to incorporate semantics knowledge of test case generation strategies. An abstract view of enhanced TinMan is shown in Figure2. The rectangle at the top represents Z (or Object-Z) specification of test case generation strategy. TinMan is modified, such that it accepts Z specification of a test case generation strategy and stores it in the repository of test case generation strategies.

Formal Specification of newtest case generation strategy

TinM anObject-Zspecification

Formal specification ofstrategies (repository)

TTs and OTs

Storing astrategy

Using astrategy

Adding anew strategy

Figure 2. An abstract view of enhanced TinMan

Our proposed enhancement requires TinMan to accept Z specification of test case generation strategies and to store it in the repository of test case generation strategies. The repository of test case generation strategies is the source of semantics knowledge required to generate TTs. We reuse the inputting functionality of TinMan by extending it to accept Z specifications of test case generation strategy in LaTeX format, to convert it into Java serialization format, and to store it into a file. We use .tgs (for test case generation strategy) file extension for files that contain a test case generation strategy. Such .tgs files, when accumulated, represent a repository of test case generation strategies. Once a test case generation strategy is stored as a .tgs file, enhanced TinMan allows a tester to use it for partitioning a predicate and to edit it. A benefit of the proposed extension is that it does not require any additional expertise or skill at the tester’s end. A tester familiar with Object-Z, TTF, and TinMan can easily adapt to the enhanced version of TinMan.

3.3. Formalizing Test Case Generation Strategies

Writing formal specification of test case generation strategies is not a new concept. Burton [1]

Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06)0-7695-2655-1/06 $20.00 © 2006

proposed a template for writing formal specification of test case generation strategies in Z, i.e.,

∀∀∀∀ Vars(P) ● P ⇔⇔⇔⇔ P1 ∨∨∨∨ … ∨∨∨∨ Pn

This states that the predicate P is equivalent to the disjunction of its n partitions, for all variables in the predicate P. Burton used this template for writing Z specifications of some traditional black-box test case generation heuristics and used these specifications for automated test case generation from Z specifications of the software under test.

Automation in our technique is facilitated by formally specifying the test case generation strategies in TinMan. We reuse the partitioning template proposed by Burton for our purposes.

4. Case Study

We explain our test case generation technique with the help of a simple case study. Inputs to the enhanced TinMan are Object-Z specification of the class under test and Z specifications of the required test case generation strategies.

4.1. Object-Z Specification

Consider the Object-Z specification of a stack class shown in Figure 3. A similar specification is used in [3]. Class invariant restricts the size (cardinality) of items to max. The two operation schemas in the stack class are Push and Pop.

Stack max : �

items : seq T

#items ≤ max items = ⟨⟩

Δ (items) item? : T

#items<max items’=⟨item?⟩ � items

PushΔ (items) item! : T

Pop

#items > 0 items=⟨item!⟩ � items’

INIT

Figure 3. Object-Z specifications of stack class

The precondition of Push operation ensures that the size of sequence items is less than max and its post-condition ensures that input variable item? is successfully concatenated to sequence items. The precondition of Pop ensures that the stack is non-empty and its post-condition ensures that item? is successfully removed from the stack.

4.2. Test Case Generation

Test cases are generated from the input predicates of each operation schema. Moreover, if an Object-Z class states some class invariants in its state schema, those class invariants are conjoined to the input predicates for the purposes of test case generation. The class invariant of stack class is #items ≤ max.

To generate test cases for Push operation, we extract its precondition, which is conjoined to the class invariant. The precondition of the Push operation is #items<max. The conjoined predicate for Pushoperation provides its VIS, i.e.

VISPush � #items ≤ max Λ pre Push VISPush � #items ≤ max Λ #items<max

This simplifies to VISPush � #items<max. The precondition of Pop operation schema checks that stack is non-empty, which is #items > 0. The conjoined predicate for Pop operation provides its VIS:

VISPop � #items ≤ max Λ pre Pop VISPop � #items ≤ max Λ #items > 0

One or more test case generation strategies can be used to partition VIS of an operation. Our technique requires Z specifications of test case generation strategies for automated generation of TTs. For this simple case study, we use some examples of boundary value analysis (BVA) strategy and type-based (TB) strategy, which are shown in Table 1, reproduced from [1].

Table 1. Z specifications of BVA and TB strategiesPredicate Partitions

BVA applied to < ∀ A,B : � ● A<B ⇔ A=B-1 ∨ A<B-1

BVA applied to > ∀ A,B : � ● A>B ⇔ A=B+1 ∨A>B+1

BVA applied to ≤

∀ A,B : � ● A≤B ⇔ A=B ∨ A=B-1 ∨ A<B-1

TB (0-1-many) ∀ X: seq T ● #X=0 ∨ #X=1 ∨ #X >1

VIS of Push operation is based on the cardinality of the sequence items. Thus 0-1-many strategy is applied to generate TTs from the VIS of Pushoperation. The resultant TTs and TTH are:

TTPush.1 � [VISPush | #items = 0] TTPush.2 � [VISPush | #items = 1] TTPush.3 � [VISPush | #items > 1]

TTHPush (VISPush, TB) = { TTPush.1, TTPush.2, TTPush.3}

Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06)0-7695-2655-1/06 $20.00 © 2006

TTPush.3 is further partitioned by applying BVA strategy, which results in:

TTPush.3.1 � [VISPush | #items>1 Λ #items<max-1] TTPush.3.2 � [VISPush | #items = max-1]

TTHPush (TTPush.3, BVA) = {TTPush.3.1, TTPush.3.2}

VIS of Pop operation is a conjunction of two predicates. Both the predicates are based on cardinality of sequence items, thus 0-1-many strategy is slightly modified and applied. The resultant TTs and TTH are:

TTPop.1 � [VISPop | #items = 1] TTPop.2 � [VISPop | #items = 2] TTPop.3 � [VISPop | #items > 2]

TTHPop (VISPop, TB) = {TTPop.1, TTPop.2, TTPop.3}

TTPop.3 can be further partitioned by applying BVA strategy, which results in:

TTPop.3.1 � [VISPop | #items>2 Λ #items<max-1] TTPop.3.2 � [VISPop | #items = max-1] TTPop.3.3 � [VISPop | #items = max]

TTHPop (TTPop.3, BVA) = {TTPop.3.1, TTPop.3.2, TTPop.3.3}

The process of TTs derivation may continue by applying more strategies, but we end it with these TTs as the final TTs. Figure 4 draws TTHPush and TTHPop.

VISPush TTPush.1

TTPush.2

TTPush.3.1

TTPush.3

VISPop

TTPop.2

TTPop.3.1

TTPop.3.3

TTPush.3.2

TTPop.3

TTPop.1

TTPop.3.2

Figure 4. Test template hierarchies for stack

operations

5. Conclusion and Future Work

In this paper, we propose an automated test case generation technique, which allows automated test case generation support for many traditional black-box and domain specific test case generation strategies, thus reducing human labor required. Our work builds upon the existing work in the area of test case generation from Object-Z specifications. We also

propose to enhance an existing tool, TinMan, by allowing it to accept formal specification of a test case generation strategy. We demonstrate our technique on a simple case study.

The future work includes looking into the possibility of automated support for testing strategies that cannot be easily specified in Z.

Acknowledgement

The authors wish to thank Dr. Muhammad Jaffar-ur-Rehman (late), former Prof. and Dean, Faculty of Engineering and Sciences, Mohammad Ali Jinnah University, Islamabad, Pakistan, for his valuable guidance during early phases of this work.

References

[1] S. Burton, “Automated Testing from Z Specifications,” Department of Computer Science, University of York, 2000.

[2] D. Carrington, I. MacColl, J. McDonald, L. Murray, P. Strooper, “From Object-Z Specifications to ClassBench Test Suites,”in Software Testing, Verification and Reliability, 2000, pp. 111-137.

[3] L. Liu, H. Miao, X. Zhan, “A Framework for Specification-Based Class Testing,” in Eighth IEEE International Conference on Engineering of Complex Computer Systems, 2002, pp. 153-162.

[4] I. MacColl, L. Murray, P. Strooper, D. Carrington, “Specification-based Class Testing: A Case Study,” Second IEEE International Conference on Formal Engineering Methods, December 1998, pp. 222-231.

[5] L. Murray, D. Carrington, I. MacColl, P. Strooper, “Extending Test Templates with Inheritance,” Australian Software Engineering Conference (ASWEC ’97), September 28 – October 02, 1997, IEEE, pp. 80-87.

[6] L. Murray, D. Carrington, I. MacColl, P. Strooper, “Inheritance and Specification-based Object-oriented Testing,” Software Verification Research Centre, The University of Queensland, Tech. Rep. 97-18, April 1997.

[7] L. Murray, D. Carrington, I. MacColl, P. Strooper, “TinMan – A Test Derivation and Management Tool for Specification-based Class Testing,” 32nd International Conference on Technology of Object-Oriented Languages, November 1999, IEEE, pp. 222-233.

[8] G. Smith, The Object-Z Specification Language, Kluwer Academic Publishers, 2000, America.

[9] P. Stocks, D. Carrington, “A Framework for Specification-Based Testing,” IEEE Transactions on Software Engineering, Vol. 22, No. 11, November 1996, pp. 777-793.

[10] T. Xie, D. Notkin, “Tool-Assisted Unit-Test Generation and Selection Based on Operational Abstractions,” Automated Software Engineering Journal, 2006.

Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06)0-7695-2655-1/06 $20.00 © 2006