Bllp04 Ij Author

SOFTWARE—PRACTICE AND EXPERIENCESoftw. Pract. Exper. 2003; 34:915–948 Prepared using speauth.cls [Version: 2002/09/23 v2.2]

Generation of test sequencesfrom formal specifications:GSM 11-11 standardcase-study

Eddy Bernard1†, Bruno Legeard∗2‡, Xavier Luck1§, Fabien Peureux2¶

1 SchlumbergerSema Montrouge Product Center – 50, avenue Jean Jaures - 92542 MontrougeCedex, France2Laboratoire d’Informatique de l’Universite de Franche-Comte (LIFC) - CNRS – 16, route deGray - 25030 Besancon cedex, France

SUMMARY

This paper presents results of a case-study on generating test cases for a fragment ofthe Smart Card GSM 11-11 standard. The generation method is based on an originalapproach using the B notation and techniques of Constraint Logic Programming withsets. The GSM 11-11 technical specifications were formalised with the B notation. Fromthis B specification, a system of constraints was derived, equivalent to this formal model.Using a set constraint solver, boundary states were computed, and test cases wereobtained by traversing the constrained reachability graph of the specifications.

The purpose of this project was to evaluate the contribution of this testing environment,called B-TESTING-TOOLS, in an industrial process on a real-life size application, bycomparing the generated test sequences with the already used and high quality manually-designed tests. This comparison enabled us to validate our approach, and showed itseffectiveness in the validation process of critical applications: the case-study gives a widecoverage (about 85%) of the generated tests compared to the pre-existing tests, and asaving of 30% of test design time.

key words: specification-based testing, B notation, Smart Card GSM 11-11 standard, Constraint

Logic Programming with sets

∗Correspondence to: Laboratoire d’Informatique de l’Universite de Franche-Comte - CNRS – 16, route de Gray- 25030 Besancon cedex, France (tel:(33) 381 666 663 – fax:(33) 381 666 450)†E-mail: bernard [email protected]‡E-mail: [email protected]§E-mail: [email protected]¶E-mail: [email protected]/grant sponsor: SchlumbergerSema Montrouge Product Center

Received 14 October 2003Copyright c⃝ 2003 John Wiley & Sons, Ltd. Revised 19 November 2003

916 E. BERNARD ET AL.

1. Introduction

Today, functional black-box testing still remains a significant stage of the software life cycle.The necessity of testing a product is bound to the need for checking that the featuresdescribed in the specifications are implemented exactly as expected. In application domains ofcritical software, i.e. software, avionics, railway signalling, automotive control and managementsystems, secure smart card applications..., more than 50% of the development is devoted tothe validation of the software product. Testing is performed by applying test cases to animplementation under test, by making observations during the execution of the tests, and bysubsequently assigning a verdict on the correct execution of the implementation. The designof black-box tests in industry is seldom supported by tools, and relies on the expertise ofvalidation engineers. Consequently, there is considerable subjectivity exercised in created testsets. The manual design of functional black-box tests is generally based around an informalreview of the system specification.

The need to offer better methods and tools for functional black-box testing has given rise toa large amount of research on generating tests from formal specifications e.g. [1, 2, 3]. Formalmethods of specification, and particularly model-oriented notations such Z [4], VDM [5] andB [6], allow a high-level abstract formalisation of the expected behaviour of the system undertest. These notations are well suited for test generation because the expressiveness of set-oriented logic constructs and the definition of an explicit model help both test case generationand oracle synthesis. Thus, these formal notations are the basis of various proposals to moreor less automatically generate tests from the formal model, see for example [7, 8, 9, 10, 11].

In [12, 13, 14], we presented a new method for automated test generation from B abstractmachines using Constraint Logic Programming [15], called b-testing-tools . From the Bformal model of the functional requirements of the system to be tested, an equivalent systemof constraints is derived, and then, each operation of the specification is partitioned througha translation into Disjunctive Normal Form. Using this partition, boundary goals and thenboundary states are computed with a specific solver, called CLPS-B [16, 17, 18], that usesConstraint Logic Programming with sets. The CLPS-B solver is then used to build test casesby traversing the constrained reachability graph of the specifications. This technique has theadvantage of avoiding as a preliminary the construction of the complete reachability graph.Finally, the formal specifications are used as oracles to determine the expected output for agiven input. This test generation approach is supported by an environment called b-testing-tools [19]. The b-testing-tools environment provides automated support for the designand construction of test sets. Also the rationale used to create specific test sets is made explicit.

The purpose of this paper is to show, in an industrial validation context, the practicabilityand effectiveness of the b-testing-tools approach to generating functional test cases. Thesubject of this industrial case study is the GSM 11-11 standard [20]. The GSM 11-11 standardapplication had the advantages, on the one hand, that it was nontrivial and common in thesmart card industry today and, on the other hand, that SchlumbergerSema† already had a

†The SchlumbergerSema company is the leading supplier of Smart Card technology.

Copyright c⃝ 2003 John Wiley & Sons, Ltd. Softw. Pract. Exper. 2003; 34:915–948Prepared using speauth.cls

GENERATION OF TEST SEQUENCES: GSM 11-11 STANDARD CASE-STUDY 917

mature, high-quality manually generated test set. The comparison of tests shows that morethan 85% of the tests used by the validation team are covered by those generated using theb-testing-tools environment, while nearly 50% of the generated test cases supplement thesepre-existing tests. Moreover, the test design time analysis reveals a saving of time about 30%.

Our presentation is organised as follows:

• Section 2 introduces the B method and notation.• Section 3 gives a short overview of the test generation method with the b-testing-toolsenvironment.

• Section 4 presents the informal specifications and the relevant B formal model of theGSM 11-11 standard.

• Section 5 describes the computation of the boundary goals and the constitution of theboundary states.

• Section 6 presents the generated test sequences computed using the boundary states andthe B model.

• Section 7 provides a summary of the results and discusses the main lessons learned fromthe industrial case study.

• Section 8 describes related work.• Section 9 summarises the contribution of the b-testing-tools environment andsuggests further work.

2. B formal method and notation

This section presents the B Abstract Machine Notation, used to formally specify the system’sfunctional requirements.

2.1. Presentation of the B method

Formal methods are mostly used in safety-critical domains to produce high quality softwareor hardware. The B method was designed for this purpose by Abrial in the late 1980s [6]. Itsbasic construct is the abstract machine which contains encapsulated data and operations tomanipulate these data. It extends earlier set-based specification notations such as V DM [5] andZ [4] by placing more emphasis on composition of machines, refinement [21], code generationand tool support [22]. B has already been successfully used for large industrial projects [23].Figure 1 shows the use of the B method in the software life cycle.

2.2. The data model of an abstract machine

The data of an abstract machine are specified in a data model by means of a number ofmathematical concepts such as sets, relations, functions, sequences and trees. This model, alsocalled the static model, presents the various data of the B machine, which are used by theoperations, and the invariant properties that the state variables must follow. The data model



Requirement Analysisand

Technical Specifications

Incr

emen

tal M

odif

icat

ions

Code Generation

Refinement

Abstract Machine

B Method

Formal

Safety Properties

Proofs

Figure 1. B method in the software life cycle

is also formally described by means of a number of clauses. In the GSM 11-11 B machine (seeAppendix A), we find eight clauses, which are the following:

• The machine clause introduces the name of the machine.• The sets clause introduces given sets. These sets, be they enumerated or abstract, alldenote independent types.

• The constants clause lists the various constants of the machine.• The definitions clause introduces some useful, possibly parameterised, aliases, as wellas useful definitions to facilitate the reading of the specifications.

• The properties clause introduces the properties of the machine, which take the formof various conjoined predicates involving the constants and the given sets.

• The variables clause introduces the state variables (components) of the machine.• The invariant clause introduces the invariant properties of the machine. The invariantconsists of a conjunction of a number of predicates.

• The initialisation clause makes possible the assignment of initial values to the variablesof the machine. The assignment must be proved to uphold the invariant. The initialisationtakes the form of a substitution.



2.3. The operation model of an abstract machine

The dynamics of an abstract machine is expressed through its operations. The role of anoperation, which is executed by the computer, is to modify the state of the abstract machineor to reveal part of that state to an observer. Each operation must maintain the state invariant.The specifications of the operations of an abstract machine are expressed in a last clause, calledoperations, as non-executing pseudo-code that does not contain any sequencing or looping. Inthis pseudo-code each operation has a pre-condition and an atomic action. The pre-condition isa necessary condition for invoking an operation. The final state of the variables after executionof the operation are defined using the notion of substitution.

2.4. Testing framework with B-TESTING-TOOLS

The next section introduces a short overview of the test generation method with the b-testing-tools environment. This method is based on models formalised by means of the Bnotation and the concept of abstract machine, but is different to the traditional process of theB method. The b-testing-tools approach consists of formalising the technical specificationsof requirements by using the most abstract level of B modelling. This abstract model is usedto generate functional tests for the validation of an implementation developed using a classical(design and coding) rather than a B method life cycle. The abstract model expresses in aprecise and detailed way the functional requirements of the product to be tested. The modelis developed from informal specifications with the help of the validation team. The model isanimated [17] to provide further confidence that it captures what was intended. For simplicity,the b-testing-tools environment do not deal with composition of B machines. This is nottoo restrictive; in various case-studies (process scheduler [17], protocol T=1 [24] or GSMapplication discussed in this paper), this practice was convenient and gave good results.

The objective of the case-study presented here was to evaluate the coverage and quality ofthe test generation process with the b-testing-tools environment. For that, the GSM 11-11application is very suitable because the SchlumbergerSema manually-designed tests constitutea very mature and high-quality test suite.

The manually-designed test suite was developed iteratively by the validation team usinga classical design process based on the informal review of the requirements documents (inour case, the GSM 11-11 Standard). As the project was the first trial of formal model-basedtest generation at SchlumbergerSema Smart Card R&D Division, we focused on the textualcomparison between the two test suites: manually-designed and automatically-generated.Translating the test cases generated by the b-testing-tools approach into executable testscripts is outside the scope of this paper.

3. Short overview of the test generation method

The first step of our test generation method is to elaborate a B formal model of the system to betested. The abstract machine, the basic element of the B specifications, characterises a machinewhich has a hidden state (consisting of several state variables) and a number of operations.



A user is able to activate these operations in order to modify the state of the machine. Thebehaviour of a B abstract machine can be described in terms of a sequence of operations,where the first is activated from the initial state of the machine. This sequence of operationsis called a trace. However, if the pre-condition of an operation is false, substitution cannotestablish any post-condition. The traces in this case are of no interest, since it is impossibleto determine the state of the machine. We define a test case to be any legal trace, i.e. tracewhere all preconditions are true. A test case corresponds to a sequence of states presentingthe value of each state variable after each operation invocation. The submission of a legaltrace is a success if all the values returned by the concrete implementation during the traceare equivalent (through a function of abstraction) to the values returned by its specificationduring the simulation of the same trace (or included in the possible values if the specificationis non-deterministic).

In order to select the traces to be submitted, our approach consists of testing the operationswith boundary inputs when the system is in a boundary state. A boundary state is a statewhere at least one state variable has a value at an extremum -minimum or maximum- of itssubdomains (see Section 3.1). The trace constituting the test case is also divided into fourparts. According to the ISO9646 standard [25], the four successive parts are defined as follows(see Figure 2):

1. Preamble: the part of a test case taking the system from its initial state to a state inwhich the test purpose can be achieved (i.e. from initial state to a boundary state).

2. Body: the part of a test case containing all the events essential to achieve some testpurpose (i.e. invocation of one operation that becomes the tested operation). In thisway, each operation is executed with input boundary values.

3. Identification: the part of a test case consisting of invocations whose aim is to determinecertain observable aspects of the system at the end of the test body. In addition to thepossible output data returned by the operation of the body, the output values returnedby this third part enable us to assign a verdict: the results obtained by simulation ofthe specifications are those expected during the execution of the same sequence on theimplementation.

4. Postamble: the part of a test case taking the system from the final state of theidentification part to the final state of the test case. This last part is used to reacha specific state in order to link several new test cases together.

The operations are partitioned into update and observation operations: update operationsare used in the preamble, body and postamble, and observation operations in the identificationpart. Update means that such an operation modifies state variables. Observation operationsreturn information about the state variables (but do not modify them). The test purposeis to test each update operation from each boundary state of the B abstract model. Thegeneration method is defined by the following algorithm, where {bound1, bound2, ..., boundn}and {op1, op2, ..., opm} respectively define the set of all boundary states and the set of all theupdate operations of the B model:



postamble

Preamble invocation

Body invocation

Identification invocation

Postamble invocation

preamble identification

body

Figure 2. Test case constitution

for i=1 to n % for each boundary statepreamble(boundi); % invocations to reach the boundary state from the initial statefor j=1 to m % for each update operation

body(opj); % computation of the operation as tested operationidentification; % invocations to observe the current statepostamble(boundi); % invocations to return to the current boundary state

endfor

postamble(init); % invocations to return to the initial stateendfor

Figure 3 shows how the state space is traversed during this generation method. Such acollection of test cases, that are linked together, defines a test sequence. The next subsectionsdescribe each step of the b-testing-tools generation method.

3.1. Boundary state and preamble

A boundary goal is a predicate which describes a subset of the system state, where at leastone variable has an extremum value (maximum or minimum). A boundary state is a stateverifying one or several boundary goals. Since the b-testing-tools method aims to test everyoperation at every boundary state, the boundary goals act as test objectives.

The state variables, that are not valued by the current boundary goal, take a value of theirdomain during a constrained simulation of the specifications. Indeed, the technique can beseen as a traversal of the reachability graph whose nodes represent the constrained statesbuilt during the simulation, and the transitions symbolise the activation of an operation. Thisprocessing is performed with a single specific solver, devoted to the B notation and usingConstraint Logic Programming with sets, called CLPS-B [16, 17, 18]. Simulation is performedfrom the initial state to a state satisfying the required boundary goal. In order to improveconvergence of the resolution procedure, a Best-First search [26, 27] is implemented.



BOUNDARYSTATE n

BOUNDARYSTATE 2

BOUNDARYSTATE 1

1.1FINAL STATEOF BODY

FINAL STATE OFIDENTIFICATION

...

...


FINAL STATEOF BODY


FINAL STATEOF BODY 1.2 1.m

preamble invocations

body invocation

identification invocations

postamble invocations

INITIAL STATE

Figure 3. Traversal of the state space during the test sequence generation

3.2. Body

The constrained simulation is also used to execute the operation to be tested at the boundarystate. A body is generated for each boundary value of input parameters. Thus, from the samepreamble and boundary state, several different bodies can be obtained with the same operation,but with different input values.

3.3. Identification

The operations selected to be called for the observation are then activated. This part is usedto verify the current value of state variables. The computed operations, that generally haveno decision points, already belong to the model and are available in the implemented system.



3.4. Postamble

When the test sequences are executed on the implementation, in order to avoid having tomanually initialise the system into a specific state (initial or boundary state) after each testcase, it is useful to compute a postamble to reach this kind of state. If no postamble is able toreach the initial or boundary state, a complete reset is performed, and the preamble is usedto reach the boundary state. This part is computed using Best-First search again.

4. GSM 11-11 standard: the B formal model

The GSM (Global System for Mobile communication) is the first of the second generationmobile phone systems: i.e. the first system involving fully digital features. It was launched in1979 by the World Administrative Radio Conference (WARC), and has evolved technicallyfor many years before the official opening of the first commercial GSM networks in Europearound 1993. From 1991 onwards, it has been adapted to create very similar new systems(DCS 1800 and then DCS 1900 in which only the radio interface differs). Nowadays, theGSM is a worldwide standard and the only one accepted in Europe. Its evolution is managedby the European Telecommunication Standard Institute (ETSI). Although most of the GSMoperators are european, the GSM is used in 109 countries worldwide, including Africa, Asia andAustralia. Its main competitors are the North American and Japanese mobile communicationsystems.

The main interest of mobile networks is of course the user’s mobility. To ensure security,GSM features an optimised radio interface with digital communications over it being securedby cryptographic algorithms. The SIM‡ contains the ciphering keys, authentication algorithmsand also information about the subscriber’s identity, and services available to him. This is acard belonging to and preset by the GSM operator, but remaining in the user’s mobile phone.The commands performed by the mobile phone to the SIM and all the features available fromit are specified in the GSM 11-11 standard [20].

The GSM 11-11 standard also defines the interface between the SIM and the ME§. Duringthe communication, the SIM is passive: it only answers the requests sent by the ME. Theapplications contained in the ME access and modify the files of the SIM through definedfunctions and by respecting access conditions of each file. The abstraction level of the testedkernel concerns only the functions of the GSM 11-11 without formalising the upper layer (theapplication layer), or the lower layer (the communication protocol between the SIM and theME). In the industrial case-study, we proceed with 11 operations and a subtree of 8 datafiles. Due to the industrial confidency, in this article, we only present a smaller fragment (6operations and 4 data files).

The files of the SIM and the selected functions are now described. These functions make itpossible to select a file and insert a personal identification number (PIN). According to the

‡Subscriber Identification Module.§Mobile Equipment.



Figure 4. Tree Structure of considered SIM files

validity of this code and the relevant access of files, the user may read the currently selectedfile. The rights corresponding to the PIN code may be blocked if the authentication processfails three times, and can be unblocked if a corresponding personal unblocking key (PUK) isentered.

4.1. Selected files

We consider two kinds of file defined in the GSM 11-11 standard:

• The transparent Elementary files (EF) are files which contain only data. An EF file witha transparent structure consists of a sequence of bytes¶.

• The Dedicated files (DF) are directories, which can contain other DF or EF. They definethe tree structure of the files. A special DF file, called Master File (MF), defines the rootof the tree.

The tree structure and the files considered in this case-study were limited to two DFs (ofwhich MF is one) and four EFs with the tree structure shown in Figure 4. The current directoryof the system is defined by a DF while its current file may be defined by an EF (in this case,the current directory is its DF father). Initially, the current directory of the system is MF andno current file is selected. It should be noted that there is no current file after the selection ofa DF. The selection of a file (DF or EF) is performed according to the following rules:

• Every file which is a child of the current directory can be selected.• Every DF file which is a sibling of the current directory can be selected.• The parent of the current directory can be selected.• The current directory can be selected.• The MF can always be selected.

¶The GSM 11-11 standard also includes sequences of records and cyclic EF, which are not considered in thisstudy.



Table I. Read access conditions

Files Read access conditions

ef iccid NEVer

ef lp ALWays

ef imsi CHV

ef ad ADM

4.2. Security aspects

Every EF has its own specific read access conditions (no file access condition is currentlyassigned by GSM to the DF). The relevant access condition must be fulfilled before therequested action can take place. These conditions are described in Table I. Four types ofaccess condition can be found in the standard:

• ALWays: the action can be performed without any restriction,• NEV er: the action cannot be performed over the SIM\ME interface (the SIM mayperform the action internally),

• CHV ∥: the action is possible only if one of the following conditions is fulfilled:

– a correct PIN code has already been presented to the SIM during the currentsession,

– the PUK code has been successfully presented during the session.

The CHV level, once satisfied, remains valid until the end of the GSM session as longas the corresponding PIN code remains unblocked, i.e. after three consecutive wrongattempts, not necessarily in the same card session, the access rights previously grantedby the PIN or PUK code are lost immediately.

• ADM : Allocation of these levels and respective requirements for their fulfilment are theresponsibility of the appropriate administrative authority. In the selected kernel of thestandard, ADMinistrator access is always denied.

4.3. Selected commands

We present six operations among those defined in the GSM 11-11 standard. This selection wasmotivated by the wish to generate test sequences to elucidate in more detail the security ofthe read access conditions. The chosen operations were the following:

∥Card Holder Verification.



• RESET : this function consists in selecting MF as the current file, and to suppress theaccess conditions granted since the beginning of the session. This function makes possiblethe simulation of consecutive card sessions.

• SELECT FILE: this function selects a new file according to the conditions defined inSection 4.1 above.

• READ BINARY : this function reads the data from the current EF. This function canbe performed only if the read access condition for this EF is satisfied.

• V ERIFY CHV : this function verifies the PIN code presented by the ME by comparingit with the relevant one stored in the SIM. The verification process is allowed only ifCHV is not blocked.If the presented PIN code is correct, the number of remaining CHV attempts shall bereset to its initial value of 3 and the CHV access condition is satisfied.If the presented PIN code is false, the number of remaining CHV attempts shall bedecremented. After 3 consecutive false PIN presentations, not necessarily in the samecard session, the CHV shall be blocked and the CHV access condition can never befulfilled until the UNBLOCK CHV function has been successfully performed.

• UNBLOCK CHV : this function, by verifying the presented PUK code, unblocks CHVwhich has been blocked by three consecutive wrong PIN code presentations. This functionmay be performed whether or not CHV is blocked, but is not allowed if UNBLOCK CHVis itself blocked.If the presented PUK code is correct, the value of the PIN code, presented together withthe PUK code, replaces the currect stored CHV PIN value, the number of remainingUNBLOCK CHV attempts is reset to its initial value of 10 and the number of remainingCHV attempts is reset to its initial value of 3. After a successful unblocking attempt,the CHV access condition level is satisfied.If the presented PUK code is false, the number of remaining UNBLOCK CHV attemptsshall be decremented. After 10 consecutive false PUK code presentations, not necessarilyin the same card session, the UNBLOCK CHV shall be blocked. A false PUK code shallhave no effect on the status of the CHV itself.

• STATUS: this function returns the value of the current file, the current directory, andthe current value of all the counters of remaining attempts.

Each function, except the STATUS operation, always returns a status word. This response,which has the form of a hexadecimal code, takes one of the values described in Table II.

4.4. The B formal model

The informal specifications of the GSM 11-11 kernel, introduced in the previous subsections,were used to write the B formal specifications. The relevant B abstract model, presented inAppendix A, is now briefly described.

4.4.1. Sets

Six sets are defined to model the simplified kernel:



Table II. Potential response codes to commands

Responses Descriptionscodes

9000 - Normal ending of the command

9400 - No EF selected

9404 - File not found

9804 - Access condition not fulfilled- Unsuccessful CHV verification, at least one attempt left- Unsuccessful UNBLOCK CHV verification, at least one attempt left

9840 - Unsuccessful CHV verification, at least one attempt left- Unsuccessful UNBLOCK CHV verification, at least one attempt left- CHV blocked- UNBLOCK CHV blocked

• PERMISSION : this represents the various read access rights.• V ALUE: this defines the two Boolean values true and false.• FILES: this defines the set of SIM files and directories to be considered.• BLOCKED STATUS: this is used to model the various states of blocking.• CODE: this represents the potential codes, which can be used to verify an accesscondition. The codes were limited to four possible values to facilitate the reading ofthe generated test sequences.

• DATA: this abstract set defines the set of all the data, which can be written in the files.

4.4.2. Constants

Five constants are defined in the B machine:

• MAX CHV : this is the maximum number of consecutive unsuccessful PIN attemptsthat will be tolerated.

• MAX UNBLOCK: this is the maximum number of consecutive unsuccessful PUKattempts that will be tolerated.

• FILES CHILDREN : this is used to define the tree structure of files with parent\childpairs.

• PERMISSION READ: this describes the read access condition for each file withfile\condition pairs.

• PUK: this defines the constant value of the PUK code.



4.4.3. Definitions

Five definitions complete the sets and constants already defined:

• COUNTER CHV : a range of non-negative numbers. The number of further consecutiveunsuccessful PIN attempts that will be tolerated is in this range.

• COUNTER UNBLOCK: a range of non-negative numbers. The number of furtherconsecutive unsuccessful PUK attempts that will be tolerated is in this range.

• MF : this set defines the master file.• DF : this contains the set of all the directories except the master file.• EF : this presents the set of all the transparent files.

4.4.4. Variables

Ten state variables are used in the formal model:

• current file: this indicates the current file.• current directory: this indicates the current directory.• counter chv: this defines the number of PIN attempts with a wrong code, which aretolerated to verify CHV.

• counter unblock: this defines the number of PUK attempts with a wrong code, whichare tolerated to verify UNBLOCK CHV.

• pin: this defines the PIN code.• permission session: this determines the access rights currently granted withcondition\Boolean pairs.

• blocked chv status: this defines the status of blocking of CHV.• blocked status: this defines the blocking status of UNBLOCK CHV.• data: this presents the data in each file with file\data pairs.

The invariant and initialisation clauses introduce respectively the invariant propertiesof the state of the machine and the initialisation of each state variable. The six operations,previously described, are presented in the operations clause. Table III shows, for eachoperation, the description of its input and output parameters.

4.5. Validation and verification of the B abstract model

During the formalisation process, we use the animation capabilities of the b-testing-toolsenvironment [17] to simulate the execution of the model. It allowed the interactive validationof the model with regard to the requirement specifications (i.e the informal specifications ofthe GSM 11-11 standard).

Moreover, the model was verified using the proof tools of the atelier b [22]. In the Bmethod, at the abstract level, proof obligations define what must be proved to ensure thatoperations preserve invariants (subject to operation pre-conditions being met). For the GSM11-11 B abstract machine shown in Appendix A, 377 proof obligations were generated: 376were automatically proved and 1 was satisfied using the interactive proof tool.



Table III. Input and output parameters of operations

Operations Input Parameters Output Parameters

RESET none none

SELECT FILE ff : file to be reached sw: status word

READ BINARY none dd: read datasw: status word

V ERIFY CHV code: code of CHV sw: status word

UNBLOCK CHV code unblock: sw: status wordcode of UNBLOCK CHV

new code: new code of CHV

STATUS none cd: current directorycf : current filecc: counter of CHVcuc: counter of UNBLOCK CHV

The two next sections describe the application of the test generation method on thestandard GSM 11-11 B specifications: firstly the computation of boundary states, and thenthe computation of the test sequences themselves.

5. Boundary state computation

To compute boundary goals, firstly a partition analysis is performed on each operation, bytransforming its postcondition into disjunctive normal form (DNF), then conjoining it withthe precondition and the state invariant. This ensures that each effect of each operation istested independently.

Secondly, boundary goals are obtained by searching for maximum and minimum solutionsto each of these effect predicates. The maximization/minimization is performed using a defaultordering which is chosen according to the types of the state variables. A boundary goal is alsoa predicate that describes a subset of the state space, represented by a set of constraints.

Finally, each boundary goal is instantiated into a boundary state obtained by traversal ofthe state space during the preamble computation. A more detailed presentation is given in [13].

5.1. Domain partitioning

For test generation purposes, we transform the postcondition of each operation of the B modelinto DNF (each before/after predicate of the DNF is conjoined with the precondition and thestate invariant). After this transformation, each disjunct corresponds to one behaviour of the



Table IV. Effect predicates of the READ BINARY operation

N◦ Before predicates After predicates

P1 current file ⊆ EF ∧ current file = ∅ sw := 9400dd := −1

P2

∃ dp · dp ∈ PERMISSION ∧ current file ⊆ EF ∧current file = ∅ ∧ permission session(dp) = true ∧PERMISSION READ[current file] = {dp}

sw := 9000 ∥ANY ff WHEREff ∈ current file THENdd := data(ff) END

P3

∀ dp · dp ∈ PERMISSION ∧ current file ⊆ EF ∧

sw := 9804dd := −1

current file = ∅ ∧ permission session(dp) = true ∧PERMISSION READ[current file] = {dp}

P4

∀ dp · dp ∈ PERMISSION ∧ current file ⊆ EF ∧current file = ∅ ∧ permission session(dp) = false ∧PERMISSION READ[current file] = {dp}

P5

∀ dp · dp ∈ PERMISSION ∧ current file ⊆ EF ∧current file = ∅ ∧ permission session(dp) = false ∧PERMISSION READ[current file] = {dp}

relevant operation. These disjuncts are called effect predicates. For each operation, the set ofall effect predicates corresponds to the set of all behaviours of the operation.

For example, if an operation contains a conditional substitution ‘IF condx THEN S1 ELSE

S2’ where condx is an atomic condition on the variable x, and S1 and S2 are two atomicsubstitutions, two effect predicates are created: condx ∧ S1 and ¬condx ∧ S2.

Table IV summarises the effect predicates obtained from the partitioning into DNFof the operation READ BINARY . The effect predicates computed from the operationsV ERIFY CHV UNBLOCK CHV and SELECT FILE are shown in Appendix B. Thetwo other operations (STATUS and RESET), presenting one simple (only substitutions) effectpredicate, are not shown. Although it does not appear in the various tables, the state invariantis assumed to hold for all the identified effect predicates. The 22 generated effect predicatesare then used to generate boundary goals and hence boundary states.

5.2. Boundary goal computation

On the basis of the partition analysis, one or more boundary goals are calculated fromthe before-state of each effect predicate. The boundary goals are obtained by searching formaximum and minimum solutions to each effect predicate. The maximization/minimization isperformed on the state variables of the predicates as follows:



• If the variable is a set of non-constant cardinality, two types of boundary values arecomputed: the first one is equal to the smallest cardinality set, and the other to thelargest cardinality set of each subdomain.

• If the variable is a set of constant cardinality, all the potential values are boundary values.If the variable is equal to a pair or a set of pairs, the boundary values are recursivelycomputed on the antecedent and on the image of each pair.

• If the variable is a numeric atom, the maximum and minimum values of each subdomainare boundary values.

• If the variable is an non-numeric atom, by default, all the potential values are boundaryvalues (a validation engineer can force the choice of a single value).

The generated boundary goals may be unreachable. Within this framework, a partial test ofreachability is used. On the one hand, it is based on checking the invariant properties of thespecifications: each boundary goal not satisfying the conditions of the invariant clause of thespecifications is not taken into account. On the other hand, during the preamble computation,the paths to reach a boundary state from the initial state are bounded by length.

The Boundary goals, generated from the effect predicates of the operation READ BINARYare shown in Table V. The other boundary goals (obtained from SELECT FILE,V ERIFY CHV and UNBLOCK CHV effect predicates) can be found in Appendix C.

Some boundary goals are found to be redundant and are withdrawn from the generationby the validation engineer (for example, boundary goals BG7, BG8 and BG9 are respectivelysubsumed by BG10, BG11 and BG12). Finally, 17 boundary goals are used to generate testcases (the selected goals are identified in the various tables by heavy type).

6. Test case computation

The test cases, introduced by the concept of traces, are built according to the constrainedreachability graph of the specifications and the collection of boundary goals. A trace isrepresented by a sequence of invocations. Each invocation is defined by the activation ofan operation of the formal specification. The values of input and output parameters of theoperation, and the values of state variables following its activation, are always indicated.

We now illustrate the various stages of the approach with a specific example. The particularboundary goal is current file = ef imsi ∧ (chv, true) ∈ permission session (BG3 in theTable V). The approach for other boundary goals follows a similar pattern.

6.1. Computation of the preamble

The first stage of the method is to generate a path from the initial state to each boundarystate of the specification. As shown previously, the computation of the invocations, making itpossible to reach a boundary state from the initialisation, is performed during the computationof the boundary state itself. This technique has the advantage of avoiding, as a preliminary,the construction of the complete reachability graph: the preamble is computed during theconstruction of the boundary state. Thus, for each boundary goal, a path from the initial state



Table V. Boundary goals from the READ BINARY operation

Predicates Boundary Goals N◦

P1 current file = ∅ BG1

P2current file = ef lp ∧ (always, true) ∈ permission session BG2

current file = ef imsi ∧ (chv, true) ∈ permission session BG3

P3

current file = ef iccid ∧ (never, false) ∈ permission session ∧BG4(chv, true) ∈ permission session ∧

(always, true) ∈ permission sessioncurrent file = ef imsi ∧ (chv, false) ∈ permission session ∧

BG5(always, true) ∈ permission sessioncurrent file = ef ad ∧ (adm, false) ∈ permission session ∧

BG6(chv, true) ∈ permission session ∧(always, true) ∈ permission session

P4

current file = ef iccid ∧ (never, false) ∈ permission session BG7

current file = ef imsi ∧ (chv, false) ∈ permission session BG8

current file = ef ad ∧ (adm, false) ∈ permission session BG9

P5

current file = ef iccid ∧ (never, false) ∈ permission session ∧BG10(chv, false) ∈ permission session ∧

(adm, false) ∈ permission sessioncurrent file = ef imsi ∧ (chv, false) ∈ permission session ∧

BG11(never, false) ∈ permission session ∧(adm, false) ∈ permission sessioncurrent file = ef ad ∧ (adm, false) ∈ permission session ∧

BG12(chv, false) ∈ permission session ∧(never, false) ∈ permission session

to the relevant boundary state is performed. A consequence of this preamble computationis that state variables which are not already assigned a value by the boundary goal, areassigned a reachable value of their domain. Only one instance is generated. A kind of uniformityhypothesis over the preamble domain is therefore assumed. As argued below, this may be toostrong and require revision in future work.

Table VI shows the path generated by CLPS-B to reach, from the initial state, a state ofthe specification that satisfies the boundary goal current file = ef imsi ∧ (chv, true) ∈permission session. Thus, from the initial state, the three consecutive invocations of theoperations V ERIFY CHV (a1), SELECT FILE(df gsm) and SELECT FILE(ef imsi)lead the machine into a state where the variables current file and permission session havethe expected values.



Table VI. Path from the initial state into the boundary state

Operations Input Output Modified datadata data from the previous state

V ERIFY CHV a1 9000 permission session = [..., (chv, true)]

SELECT FILE df gsm 9000 current directory = df gsm

SELECT FILE ef imsi 9000 current file = ef imsi

6.2. Computation of the body

The purpose of the body is to test, for a given boundary state, all the update operations,with all boundary values of their input variables. The process of boundary analysis for inputvariables is similar to that for state variables. The maximization/minimization is performedon the input variables of the predicates in a similar way to that for the state variables (seeSection 5.2).

This selection of input boundary values makes possible the exploration of all the decisionpoints of the operation (according to their possible different input boundary values, extractedfrom the effect predicate already generated and used to compute the boundary goals). In thisway, as many invocations were computed with the same operation as its input parameterspresent different boundary values.

From the boundary state generated from the boundary goal current file = ef imsi ∧(chv, true) ∈ permission session, each operation is executed as many times as it has differentpossible input boundary values, and each execution represents one test. Table VII lists all theoperations with their respective input boundary values to be activated from the boundarystate.

Table VIII shows the computation of one possible test : the execution of the operationV ERIFY CHV with a wrong code as input parameter.

6.3. Computation of the identification

We have chosen the STATUS and READ BINARY operations to be executed for theobservation (no operation is added to the B model to perform observations). The identificationpart can consist of a single operation or a sequence of operations.

In the GSM 11-11 standard case-study, observations are computed with the STATUSand READ BINARY operations. In the example (see Table IX), the computation of theSTATUS operation shows that the counter of attempts to verify the CHV access condition,was decremented, while the computation of the READ BINARY operation ensures that thepermission on the CHV access is always accepted.



Table VII. Presentation of the operations to be computed in the body

References Computed operations Input parameters Values

1 reset none

2 verify chv code correct code

3 verify chv code wrong code

4 unblock chv code unblock correct codenew code any code

5 unblock chv code unblock wrong codenew code any code

6 select file ff master file

7 select file ff current directory

8 select file ff DF sibling

9 select file ff EF child

Table VIII. Example of body

Reference Operations Input Output Modified datadata data from the boundary state

2 V ERIFY CHV a2 9804 counter chv = 2

Table IX. Invocations of the test case identification

Operation STATUS

Output variables Values

current directory df gsm

current file ef imsi

counter chv 2

counter unblock chv 10

Operation READ BINARY

Output variables Values

sw 9000

dd d3



Table X. Return path to the boundary state

Operations Input Output Modified datadata data from the last state

V ERIFY CHV a1 9000 counter chv = 3

Table XI. Return path from the boundary state into the initial state

Operations Input Output Modified datadata data from the boundary state

RESET none none current file = ∅current directory = mfpermission session = [..., (chv, false)]

6.4. Computation of the postamble

When the test sequences are executed on the concrete system, in order to avoid having toinitialise manually the system into the initial or boundary state after each test case (thetesting process would be seriously slowed down), it is useful to compute a path that returnsto this kind of state. So, after each identification part, a sequence of operations is computedto return to the relevant boundary state. Table X shows the sequence obtained on the currentexample to return to the boundary state from the current state.

In the same way, when all the bodies have been computed, a path to return to the initialstate from the current boundary state is performed. Table XI shows the postamble used inthis case on the current example.

7. Comparison results

This section deals with the comparison between the test cases generated with b-testing-tools and the existing manually-designed test cases. The comparison results were obtainedon a larger part of the GSM 11-11 standard than presented in this paper: the model contained11 operations and 8 data files with many access conditions of CHV type and CHV updaterights. This model gave rise to the generation of 58 boundary goals and 42 different boundary



states∗∗. From each boundary state, about 24 different invocations were performed in the body,and about 1000 test cases were generated [28]. Computing time was about two hours (on SunUltra Sparc - Processor 200MHz - Memory 256MB).

7.1. Test coverage

We denote by Man-T the set of all SchlumbergerSema manually-designed test cases for theGSM 11-11 standard considered kernel, and by Gen-T the test cases generated with b-testing-tools for the same subset of GSM 11-11 standard. Man-T is a very mature and high-quality test suite, used for several years by the SchlumbergerSema company, and continuouslyimproved. Each test of Man-T is, in general, multi-purpose, with at least twenty operations.Man-T includes about 100 different tests. A test case from Gen-T contains less operationinvocations than a test case from Man-T. Indeed, the length of Gen-T tests lies between 2and 13 operation executions (more generally, the Gen-T tests are built with an average of 7operations).

7.1.1. Comparison method

Three comparisons can be done between manually-designed (Man-T) and automatically-generated (Gen-T) tests:

• The first comparison is lead by the fine analysis of the GSM 11.11 technical specificationby people. This study gave out a number of functional basic requirements, i.e. sentencesor small sentence sets defining the expected behaviour of the system in given situations.The entire reading of the specification then results in a list of basic requirements, whichcan be used as coverage reference. The reading of the Man-T test plan enables one to tageach of theses requirements as ’manually tested’ (this is the classical method for manualconstruction of functional coverage matrices). Then, keeping in mind that requirementscan be formalised as (input/expected output) pairs, the reading of Gen-T tests enablesone to tag the requirements as ‘automatically tested’. At the end, the requirement listcontains four requirement types: never tested, manually only tested, automatically onlytested and tested by both methods. Coverage statistics are then computed with the basicrequirement as their unit.

• The second comparison is quite simple. It consists in listing all the automatically-generated behaviours and checking within the list for the ones also covered by the Man-Ttests. Statistics are then computed with the behaviour as their unit.

• The third comparison is the contrary of the second one: it consists in listing the manually-designed tests and checking within the list for the ones also covered by the Gen-T tests.Statistics are then computed with the behaviour as their unit.

For example, six basic requirements can be extracted from the section 8.9 (Verify CHV) ofthe GSM 11.11 version 7.2.0 standard [20]. They are all covered by 20 Man-T tests and by

∗∗A single boundary state may contain several boundary goals.



112 Gen-T tests. Although the number of tests is different, we can assert that the functionalcoverage is identical according to the GSM 11.11 reference. But among the 112 generatedbehaviours, only 28 (25%) are also covered by the Man-T tests. Among the 20 Man-T tests,16 (80%) are also covered by the Gen-T tests (the missing four tests are repetitions of othertests with command order changes).

More generally, the comparison of tests on the overall fragment of the GSM 11.11 standardshows that Man-T and Gen-T both cover the GSM 11.11 basic requirements, Gen-T covers85 % of Man-T tests and 50 % of Gen-T supplements Man-T. These comparison results arenow presented and discussed as follows:

• the automatically-generated test cases in common with manually-designed test cases,• the automatically-generated test cases not in the manually-designed test cases,• the manually-designed test cases not automatically-generated.

7.1.2. Gen-T test cases in common with Man-T

Each manually-designed test case that belongs to this class usually corresponds to severalautomatically-generated test cases. Performing only one operation after the boundary statemakes the generated test cases more modular and atomic than the manually-designed testcases, which are already optimised. So, several generated test cases correspond to one manually-designed test. This explains in particular why the automatic procedure can generate a numberof tests ten times as great as those manually-designed, and why about 500 of the (short) Gen-Ttests correspond to 87 of the (long) Man-T tests.

7.1.3. Gen-T test cases not in Man-T

The generation of test cases which do not appear in Man-T (about 500) has two main causes:

• Use of boundary values from a set with non-numeric values: when the boundary valuesof a variable are computed, if it is not a numeric atom, all these potential values areconsidered as boundary values, and each of these values is also used to generate boundarystates. Similar test cases are also performed, and, in most cases, only one appears in theMan-T tests. The typical example is the pin variable, which has each possible codefrom the B model as boundary value. Thus, each value is tested while, at first sight, itseems that they are all equivalent, and so only one test of the code would be sufficient.Nevertheless, this type of test makes it possible to eliminate any anomaly concerning thecommon properties of the values of the same variable.

• Execution of all the update operations from each boundary state: the operations thatdo not impact and use, directly or indirectly, the variables of the relevant boundarygoal are executed from the corresponding boundary state. In this case, the activationof these operations may be not needed. An obvious example is the execution of theUNBLOCK CHV operation after each boundary state, even if the correspondingboundary goal does not concern CHV or UNBLOCK CHV attempts. However, there



Table XII. Comparison of the design time

Humandesign

Test plan redaction 6 days

Test implementation 24 days

Total 30 days

Automaticdesign

B specifications writing 12 days

Test generation 6 daysand coding

Total 18 days

is a need to verify the real independence of the processed variables and to check any sideeffects.

7.1.4. Man-T test cases not in Gen-T

15 test cases from the Man-T were not generated with the automatic procedure. It is mainlybecause test cases involving consecutive sessions were not generated. On the one hand, thesearch heuristics, used to reach the boundary state from the initial state, gave only one pathand the RESET operation did not appear in this path. This operation is the only one makingit possible to simulate consecutive card sessions. As the RESET operation did not appear inthe generated test preamble, this kind of test case was not generated. On the other hand, inthis study, the number of operations executed after a boundary state was limited to one. Ifmore operations were executed, these kinds of test case could be generated by activating theRESET operation before the last operation of the body. More generally, the choice to takeinto account only one path to reach a boundary state and to perform only one operation fromthe boundary state, prevents us from obtaining some judicious test cases.

7.2. Design cost

Table XII presents an estimation of design time of both Man-T and Gen-T test suites. Acomparison between our method and a human design is shown.

The evaluation of the work duration to dedicate for the design of manual tests has been doneas for industrial feasibility study: statistics are computed based on actual projects managementcases using internal criteria. Then, based on it, it is possible to evaluate the workload inducedby the validation of the defined scope of the GSM 11.11 (a few man.days per command, forthis kind of simple command and for an experienced validation engineer).

This test design time analysis shows that the specification-based generation method savesabout 30% of time, including the formalization time.



7.3. Method assessment

The comparison between the Man-T and the Gen-T tests shows that a large coverage rate,about 85%, of Man-T are also generated by b-testing-tools, while nearly 50 % of Gen-T test cases supplement Man-T. Although the obtained results are very encouraging, it ispossible to increase the quality of the generated test sets. The reasons explaining the lackof automatically-generated test cases in relation to those manually-designed, show the mainimprovements:

• to handle the boundary goals: it would be necessary to be able to eliminate certainboundary goals, which prove in fact to have only little interest. The reverse isalso necessary: to have the possibility of adding boundary goals to those generatedautomatically.

• to propose several preambles: the choice of the path to reach a boundary state fromthe initial state is currently the first path discovered by the CLPS-B solver. However,according to the path, the direction of the test can be different. The assumption ofuniformity on the domain of path may be too strong. This proposition involves thecreation of testing objectives according to particular needs required by the system to betested.

• to test several operations activated consecutively: once the boundary state is reached,the body is built using only one update operation. However, certain test cases of Man-Trelate to the consecutive activation of two update operations.

7.4. Position in the industrial validation process

The B formal model of the GSM 11.11 standard was a collaborative design between validationengineers from SchlumbergerSema and researchers from the LIFC laboratory. This B formalmodel was validated using interactive model review and animation techniques with the b-testing-tools environment. In fact, the B notation appears easily understood by engineersand the use of the b-testing-tools simulation capabilities to improve the formal modelvalidation was practical. We believe that validation engineers should always be involved inthe generation process, and should be able to interact with the environment in the wayssummarised above.

8. Related work and discussion

The b-testing-tools approach is related to the emerging Model-Based Testingtechniques [29, 30]. Model based test automation is a very active research area. These toolsuse as input a formal model of the system under test and allow the validation engineer to drivethe test generation automated process.

Formal description languages based on Labelled Transition Systems (LTS), Extended FiniteState Machines (EFSM) or Abstract State Machines (ASM) have been widely considered [31].



In these approaches, the test generator may support a variety of existing transition basedstrategies like state coverage, path coverage, constrained path coverage, all transition pairs,etc. The main limitation of these approaches comes from the state explosion problem and thetest case explosion problem [32]. Several proposals address these limitations, for example onthe basis of the formalisation of reachable properties guiding the test generation or by reducingthe test suites using individual requirements selection [33].

On the other hand, set-oriented model based formal notations, like VDM, Z or B, have beenextensively studied for test generation purpose. Most approaches [7, 34, 10, 9] use a partitionanalysis of the operation to build a Finite State Automaton - FSA - corresponding to anabstraction of the reachability graph denoted by the specification. Test cases are then generatedusing the same kinds of coverage criteria as used by the LTS/EFSM/ASM approaches, with thesame limitations (test case explosion). Moreover, the transformation of the formal model intoan abstract FSA introduces several fundamental problems, such as the non-discovery problemand again the state explosion problem [31].

The b-testing-tools approach proposes another way, which is also based on the partitionanalysis of the operation, but avoids the a priori construction of the FSA. The test generationis then conducted by Boundary Goal calculation, guiding the computation of the preambles.Moreover, the Constraint Logic Programming technology [18], used as a basis for animationpurposes, allows reasoning on so called constraint states, represented by constraint stores,which denote sets of valued states. This reduces the combinatorial explosion.

Another unique feature of b-testing-tools is its strong boundary-testing orientation.Despite the enormous popularity of boundary-value testing strategy for black-box testing inthe software practitioner guides, currently this approach has not been widely investigated as anautomated specification-based test generation approach. In the b-testing-tools approach,boundaries are computed both for state variables for each effect predicate and for the inputvariables, in the computation of the body part of the test. In this, our approach is related todomain testing [35, 36].

In the GSM 11-11 case study, two kinds of variable make it possible to compute boundaries:integer variables and set variables on which we compute the set cardinality (the integervariables are counter chv, counter unblock and a set variable is current file). Actually, theboundary analysis process generalises the boundary computation and applies it to all kindsof variable: enumerated by taking a value, and boolean by considering 0-1 as bounds of aninterval.

Finally, the b-testing-tools approach is based on specific CLP techniques. There isan increasing use of constraint solving in software verification techniques as well in model-checking [37] and in test generation. For code-based test generation, Gotlieb et. al. [38] presenta framework where a system of constraints is built from a Static Single Assignment form ofthe source code (for C programs). Solving the generated constraints allows to identify codeelements (statements or decisions) to be exercised in particular ways. Meudec [39] uses CLPfor symbolic execution of Ada code in order to generate tests. In specification-based testgeneration, Marre et. al. [40] interpret LUSTRE specifications in terms of constraints overboolean and integer variables and solve them to generate test sequences. Pretschner et. al. [41]translate System Structure Diagrams and State Transition Diagrams into Prolog rules andconstraints to allow symbolic execution of the specifications and thus test generation. They



applied also their approach on smart card software [42]. Van Aertryck et. al. [10] use DNFand constraint solving to generate an FSA and test sequences from a B-like specification, butthis is not fully automated. Mostly, these techniques use existing constraint solvers (Booleanand finite domains in general). In b-testing-tools, due to the particular features of the set-oriented B notation, we developed our own solver to handle constraints over sets, relationsand mappings. This solver augments the capabilities of and co-operates with the integer finitedomain solver of SICStus Prolog [43].

9. Conclusions and future work

This paper summarises the results of a case-study. The purpose of this project consisted ingenerating test cases for the Smart Card GSM 11-11 standard, and was to evaluate, in anindustrial process, the contribution of the generation method on a real life size application.The test case generation technique is different from the traditional refinement process of the Bmethod. It is based on an extraction of boundary goals from the abstract model by automatedpartition analysis. A test case is composed of four parts:

• the preamble used to reach a boundary state (a state verifying a boundary goal) fromthe initial state of the B machine,

• the body composed of the invocation of the operation to be tested,• the identification part used to assign a verdict,• the postamble used to reach a specific state in order to link test cases together.

A specific solver of constraints, CLPS-B, reduces the B abstract machine to a constrainedsystem, calculates boundary goals and then boundary states, and finally computes thegeneration of test cases by constrained animation. The principal advantages and originality ofour approach lie in the B-specific constraint logic programming technology used in CLPS-B,in the computation of traces during the constrained simulation (not requiring the constructionof a reachability graph or finite state automaton of the specifications), and in the technique ofindirect observation of the states using the output values returned by the operations invokedin the identification part of the test case.

A thorough comparison of the generated test collection with the manually-designed testsenabled us to validate our approach, and showed its effectiveness in the validation processof critical applications. In the same way, this case-study shows possible improvements. Itdemonstrated that our procedure should not be completely automatic - human input is needed.The testing engineer should interact with the tool, in particular in the choice of the boundarygoals to be considered and in the choice of preamble.

Thus, our test generation method makes possible the improvement and partial automationof the generation of functional tests. The testing engineer is not excluded from the testingprocess; he writes the B formal model, and drives the test generation (choice of boundarygoals, choice of preambles, etc). This approach helps validation engineers efficiently generatebetter functional tests. Complete automation is not the aim. A project currently underway isinvestigating the generation of functional test sequences for a Java Card Virtual Machine [44].This should enable us to study the proposed improvements and evaluate their impact. A new



feature is being considered in this new case-study: the automatic production of executable testapplets from generated test sequences. This new experiment will contribute to the developmentof a prototype for distribution [45].

ACKNOWLEDGEMENTS

The authors wish to acknowledge Mark Utting from The University of Waikato (New-Zealand) andTim Miller from The University of Queensland (Australia) for their detailed reading and helpfulsuggestions in the preparation of this paper. We sincerely appreciate the clarifying comments of thereferees on the earlier version of this paper.

REFERENCES

1. M.J. Balcer, W.M. Hasling, and T.J. Ostrand. Automatic generation of test scripts from formaltest specifications. In Proceedings of 3rd Symposium on Software Testings, Analysis and Verification(TAV’89), pages 210–218, Key West, December 1989. ACM SIGSOFT.

2. D.A. Carrington and P. Stocks. A tale of two paradigms: formal methods and software testing. InProceedings of the 8th Z User Meeting (ZUM’94), pages 51–68, Cambridge, June 1994. Springer Verlag.

3. M-C. Gaudel. Testing can be formal too. In Proceedings of the 6th Joint Conference on Theory andPractice of Software Development (TAPSOFT’95), volume 915 of LNCS, pages 82–96, Aarhus, Denmark,May 1995. Springer Verlag.

4. J.M. Spivey. The Z notation: A Reference Manual. Prentice-Hall, 2nd edition, 1992. ISBN 0 13 9785299.

5. C.B. Jones. Systematic Software Development Using VDM. Prentice-Hall, 2nd edition, 1990.6. J-R. Abrial. The B-BOOK: Assigning Programs to Meanings. Cambridge University Press, 1996. ISBN

0 521 49619 5.7. J. Dick and A. Faivre. Automating the generation and sequencing of test cases from model-based

specifications. In Proceedings of the International Conference on Formal Methods Europe (FME’93),volume 670 of LNCS, pages 268–284. Springer Verlag, April 1993.

8. P. Stocks and D.A. Carrington. Test templates: a specification-based testing framework. In Proceedingsof the 15th International Conference on Software Engineering (ICSE’93), pages 405–414, Baltimore,Maryland, May 1993. IEEE Computer Society Press.

9. R. Hierons. Testing from a Z specification. The Journal of Software Testing, Verification and Reliability,7:19–33, 1997.

10. L. Van Aertryck, M. Benveniste, and D. Le Metayer. CASTING: a formally based software test generationmethod. In 1st IEEE International Conference on Formal Engineering Methods (ICFEM’97), pages 99–112, 1997.

11. S. Behnia and H. Waeselynck. Test criteria definition for B models. In Proceedings of the World Congresson Formal Methods (FM’99), volume 1708 of LNCS, pages 509–529, Toulouse, France, 1999. SpringerVerlag.

12. B. Legeard and F. Peureux. Generation of functional test sequences from B formal specifications -Presentation and industrial case-study. In Proceedings of the 16th International Conference on AutomatedSoftware Engineering (ASE’01), pages 377–381, San Diego, USA, November 2001. IEEE Computer SocietyPress.

13. B. Legeard, F. Peureux, and M. Utting. Automated Boundary Testing from Z and B. In Proceedings ofthe International Conference on Formal Methods Europe (FME’02), volume 2391 of LNCS, pages 21–40,Copenhagen, Denmark, July 2002. Springer Verlag.

14. B. Legeard, F. Peureux, and M. Utting. A comparison of the BTT and TTF test-generation methods.In Proceedings of the International Conference on Formal Specification and Development in Z and B(ZB’02), volume 2272 of LNCS, pages 309–329, Grenoble, France, January 2002. Springer Verlag.

15. J. Jaffar and J-L. Lassez. Constraint Logic Programming. In Proceedings of the 14th Symposium onPrinciples of programming Languages (POPL’87), pages 111–119, Munich, Germany, January 1987. ACMPress.



16. F. Ambert, B. Legeard, and E. Legros. Constraint Logic Programming on Sets and Multisets. InProceedings of the ILPS’94 Workshop on Constraint Languages/Systems and their Use in ProblemModeling, pages 151–165, Ithaca, New York, November 1994.

17. F. Bouquet, B. Legeard, and F. Peureux. Constraint Logic Programming with sets for animation of Bformal specifications. In Proceedings of the CL’00 Workshop on (Constraint) Logic Programming andSoftware Engineering (LPSE’00), London, UK, July 2000.

18. F. Bouquet, B. Legeard, and F. Peureux. CLPS-B – A constraint solver for B. In Proceedings ofthe ETAPS’02 International Conference on Tools and Algorithms for the Construction and Analysisof Systems (TACAS’02), volume 2280 of LNCS, pages 188–204, Grenoble, France, April 2002. SpringerVerlag.

19. F. Ambert, F. Bouquet, S. Chemin, S. Guenaud, B. Legeard, F. Peureux, N. Vacelet, and M. Utting.BZ-TT: A tool-set for test generation from Z and B using constraint logic programming. In Proceedingsof the CONCUR’02 Workshop on Formal Approaches to Testing of Software (FATES’02), pages 105–120,Brno, Czech Republic, August 2002. INRIA Technical Report.

20. European Telecommunications Standards Institute, F-06921 Sophia Antipolis cedex - France. GSM 11-11V7.2.0 Technical Specifications, 1999.

21. E. Sekerinski and K. Sere. Program development by refinement - Case studies using the B method.Springer Verlag, 1999. ISBN 1 85233 053 8.

22. Clearsy, Europarc de Pichaury 13856 Aix-en-Provence Cedex 3 - France. Atelier B Technical Supportversion 3, May 2001. http://www.atelierb.societe.com.

23. P. Behm, P. Desforges, and J.M. Meynadier. METEOR : An Industrial Success in Formal Development.In Proceedings of the 2nd International Conference on the B method (B’98), volume 1393 of LNCS, pages29–45, Montpellier, France, April 1998. Springer Verlag.

24. J. Julliand, B. Legeard, T. Machicoane, B.Parreaux, and B.Tatibouet. Specification of an IntegratedCircuit Card Protocol Applications using the B method and Linear Temporal Logic. In Proceedings of the2nd International Conference on the B method (B’98), volume 1393 of LNCS, pages 273–292, Montpellier,France, 1998. Springer Verlag.

25. ISO. Information Processing Systems, Open Systems Interconnection. OSI Conformance TestingMethodology and Framework – ISO 9646, 1998.

26. A. Pretschner. Classical search strategies for test case generation with Constraint Logic Programming.In Proceedings of the CONCUR’01 Workshop on Formal Approaches to Testing of Software (FATES’01),pages 47–60, Aalborg, Denmark, August 2001. BRICS.

27. S. Colin, B. Legeard, and F. Peureux. Preamble computation in automated test generation usingConstraint Logic Programming. In Proceedings of UK-Test Workshop, York, UK, September 2003.

28. B. Legeard, F. Peureux, and J. Vincent. Automatic generation of functional test patterns from a formalizedsmart card model - application to the GSM 11-11 specification (confidential). Technical Report TR-01/01,LIFC - University of Franche-Comte and Schlumberger Montrouge Product Center, 2001.

29. A. Hartman. AGEDIS - Model Based Test Generation Tools. http://www.agedis.de/documents/ModelBasedTestGenerationTools\_cs.pdf, 2002.

30. I.K. El-Far and J.A. Whittaker. Model-based software testing. Encyclopedia of Software Engineering,1:825–837, 2002.

31. W. Grieskamp, Y. Gurevich, W. Schulte, and M. Veanes. Generating Finite State machines from AbstractState Machines. In Proceedings of the International Symposium on Software Testing and Analysis(ISSTA’02), volume 27, pages 112–122, Rome, Italy, July 2002. ACM SIGSOFT.

32. G. Friedman, A. Hartman, K. Nagin, and T. Shiran. Projected State machine Coverage for SoftwareTesting. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’02),volume 27, pages 134–143, Rome, Italy, July 2002. ACM SIGSOFT.

33. B. Vaysburg, L.Tahat, and B. Korel. Dependence Analysis In Reduction of Requirement Based TestSuites. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’02),volume 27, pages 107–111, Rome, Italy, July 2002. ACM SIGSOFT.

34. H.M. Horcher and J. Peleska. Using formal specifications to support software testing. Software QualityJournal, 4(4):309–327, 1995.

35. E.J. Weyuker and T.J. Ostrand. Theories of program testing and the application of revealing subdomains.The journal of IEEE Transactions in Software Engineering, 6(3):236–246, May 1980.

36. B. Beizer. Black-Box Testing: Techniques for Functional Testing of Software and Systems. John Wiley& Sons, New York, USA, 1995.

37. Y. Choi, S. Rayadurgam, and M.P.E. Heimdahl. Automatic abstraction for model checking softwaresystems with interrelated numeric constraints. Software Engineering Notes, 26(5):164–174, September



2001.38. A. Gotlieb, B. Botella, and M. Rueher. A CLP framework for computing structural test data. In

Proceedings of the First International Conference on Computational Logic (CL’00), pages 399–413,London, UK, July 2000. Springer Verlag.

39. C. Meudec. ATGEN: Automatic test data generation using constraint logic programming and symbolicexecution. The Journal of Software Testing, Verification and Reliability, 11(2):81–96, 2001.

40. B. Marre and A. Arnould. Test Sequence generation from Lustre descriptions: GATEL. In Proceedingsof the 15th International Conference on Automated Software Engineering (ASE’00), pages 229–237,Grenoble, France, 2000. IEEE Computer Society Press.

41. A. Pretschner and H. Lotzbeyer. Model-Based Testing with Constraint Logic programming : First Resultsand Challenges. In Proceedings of the 2nd ICSE Workshop on Automated program Analysis, Testing andVerification (WAPATV’01), pages 1–9, Toronto, May 2001.

42. J. Philipps, A. Pretschner, O. Slotosch, E. Aiglstorfer, S. Kriebel, and K. Scholl. Model-Based Test CaseGeneration for Smart Cards. In Proceedings of the 8th International Workshop on Formal Methods forIndustrial Critical Systems (FMICS’03), pages 168–182, Trondheim, Norway, June 2003. Also ElectronicNotes in Theoretical Computer Science Volume 80, Elsevier Science.

43. Swedish Institute of Computer Sciences. SICStus Prolog 3.8.7 manual documents, October 2001.http://www.sics.se/sicstus.html.

44. Sun microsystems. Java Card 2.1.1 Virtual Machine Specification, May 2000. http://java.sun.com/products/javacard/javacard21.html\#specification.

45. The BZ-TT web site. http://lifc.univ-fcomte.fr/~bztt, 2005.



APPENDIX A. The B abstract model of a fragment of GSM 11-11 standard

MACHINE GSM

SETSFILES = {mf, df gsm, ef iccid, ef lp, ef imsi, ef ad};PERMISSION = {always, chv, never, adm};V ALUE = {true, false};BLOCKED STATUS = {blocked, unblocked};CODE = {a1, a2, a3, a4};DATA = {d1, d2, d3, d4}

CONSTANTSFILES CHILDREN,

PERMISSION READ,

MAX CHV ,

MAX UNBLOCK,

PUK

DEFINITIONSMF == {mf};DF == {df gsm};EF == {ef iccid, ef lp, ef imsi, ef ad};COUNTER CHV == 0..MAX CHV ;

COUNTER UNBLOCK CHV == 0..MAX UNBLOCK

PROPERTIESFILES CHILDREN ∈ (MF ∪DF )←→ FILES ∧FILES CHILDREN = {(mf, df gsm), (mf, ef iccid), (df gsm, ef lp), (df gsm, ef imsi),

(df gsm, ef ad)} ∧PERMISSION READ ∈ EF −→ PERMISSION ∧PERMISSION READ = {(ef iccid, never), (ef lp, always), (ef imsi, chv), (ef ad, adm)} ∧MAX CHV = 3 ∧MAX UNBLOCK = 10 ∧PUK ∈ CODE ∧PUK = a3

VARIABLEScurrent file,current directory,counter chv,counter unblock chv,blocked chv status,blocked status,permission session,pin,data

INVARIANTcurrent file ⊆ EF ∧card(current file) ≤ 1 ∧current directory ∈ DF ∪MF ∧counter chv ∈ COUNTER CHV ∧counter unblock chv ∈ COUNTER UNBLOCK CHV ∧pin ∈ CODE ∧permission session ∈ PERMISSION −→ V ALUE ∧(always, true) ∈ permission session ∧(adm, false) ∈ permission session ∧(never, false) ∈ permission session ∧blocked chv status ∈ BLOCKED STATUS ∧blocked status ∈ BLOCKED STATUS ∧data ∈ EF −→ DATA ∧(blocked chv status = blocked) =⇒ (chv, false) ∈ permission session ∧(counter chv = 0)⇐⇒ (blocked chv status = blocked) ∧(counter unblock chv = 0)⇐⇒ (blocked status = blocked) ∧(current file = ∅ ∨ (dom(FILES CHILDREN � current file) = {current directory}))



INITIALISATIONcurrent file := ∅ ∥current directory := mf ∥counter chv := MAX CHV ∥counter unblock chv := MAX UNBLOCK ∥blocked chv status := unblocked ∥blocked status := unblocked ∥permission session := {(always, true), (chv, false), (adm, false), (never, false)} ∥pin := a1 ∥data := {(ef iccid, d1), (ef lp, d2), (ef imsi, d3), (ef ad, d4)}

OPERATIONS

sw ←− SELECT FILE(ff) =PRE

ff ∈ FILESTHEN

IF (ff ∈ (DF ∪MF ))THEN

IF (((ff, current directory) ∈ FILES CHILDREN)∨ ((current directory, ff) ∈ FILES CHILDREN)∨ (∃ dp ∈ (DF ∪MF ) · ((dp, current directory) ∈ FILES CHILDREN ∧

(dp, ff) ∈ FILES CHILDREN))∨ (ff = mf)THEN

sw := 9000 ∥current directory := ff ∥current file := ∅

ELSEsw := 9404

ENDELSE

IF ((current directory, ff) ∈ FILES CHILDREN)THEN

sw := 9000 ∥current file := ff

ELSEsw := 9404

ENDEND

END;

sw, dd←− READ BINARY =PRE

current file ⊆ EFTHEN

IF (current file = ∅)THEN

sw := 9400 ∥dd := −1

ELSEIF (∃ dp ∈ PERMISSION · (PERMISSION READ[current file] = {dp} ∧

permission session(dp) = true))THEN

sw := 9000 ∥ANY ff WHERE (ff ∈ current file)THEN

dd := data(ff)END

ELSEsw := 9804 ∥dd := −1

ENDEND

END;

cd, cf, cc, cuc←− STATUS =BEGIN

cd := current directory ∥cf := current file ∥cc := counter chv ∥cuc := counter unblock chv

END;



sw ←− VERIFY CHV(code) =PRE

code ∈ CODETHEN

IF (blocked chv status = blocked)THEN

sw := 9840ELSE

IF (pin = code)THEN

counter chv := MAX CHV ∥permission session(chv) := true ∥sw := 9000

ELSEIF (counter chv = 1)THEN

counter chv := 0 ∥blocked chv status := blocked ∥permission session(chv) := false ∥sw := 9840

ELSEcounter chv := counter chv − 1 ∥sw := 9804

ENDEND

ENDEND;

sw ←− UNBLOCK CHV(code unblock, new code) =PRE

code unblock ∈ CODE ∧new code ∈ CODE

THENIF (blocked status = blocked)THEN

sw := 9840ELSE

IF (PUK = code unblock)THEN

pin := new code ∥blocked chv status := unblocked ∥counter chv := MAX CHV ∥counter unblock chv := MAX UNBLOCK ∥permission session(chv) := true ∥sw := 9000

ELSEIF (counter unblock chv = 1)THEN

counter unblock chv := 0 ∥blocked status := blocked ∥sw := 9840

ELSEcounter unblock chv := counter unblock chv − 1 ∥sw := 9804

ENDEND

ENDEND;

RESET =BEGIN

current file := ∅ ∥current directory := mf ∥permission session := {(always, true), (chv, false), (adm, false), (never, false)}

END;

END;



APPENDIX B. Effect predicates of GSM 11.11 formal model

N◦ Before predicates After predicates

SELECT FILE

P6∃ ff · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧

sw := 9000 ∥current directory := ff ∥current file := ∅

(ff, current directory) ∈ FILES CHILDREN

P7∃ ff · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧(current directory, ff) ∈ FILES CHILDREN

P8∃ ff, dp · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧dp ∈ (DF ∪MF ) ∧ (dp, ff) ∈ FILES CHILDREN ∧(dp, current directory) ∈ FILES CHILDREN

P9 ∃ ff · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧ ff = mf

P10

∃ ff ∀ dp · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧

sw := 9404

dp ∈ (DF ∪MF ) ∧ ff = mf ∧(dp, ff) ∈ FILES CHILDREN ∧(dp, current directory) ∈ FILES CHILDREN ∧(current directory, ff) ∈ FILES CHILDREN ∧(ff, current directory) ∈ FILES CHILDREN

P11

∃ ff ∀ dp · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧dp ∈ (DF ∪MF ) ∧ ff = mf ∧(dp, ff) ∈ FILES CHILDREN ∧(dp, current directory) ∈ FILES CHILDREN ∧(current directory, ff) ∈ FILES CHILDREN ∧(ff, current directory) ∈ FILES CHILDREN

P12

∃ ff ∀ dp · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧dp ∈ (DF ∪MF ) ∧ ff = mf ∧(dp, ff) ∈ FILES CHILDREN ∧(dp, current directory) ∈ FILES CHILDREN ∧(current directory, ff) ∈ FILES CHILDREN ∧(ff, current directory) ∈ FILES CHILDREN

P13∃ ff · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧ sw := 9000 ∥

current file := ff(current directory, ff) ∈ FILES CHILDREN)

P14∃ ff · ff ∈ FILES ∧ ff ∈ (DF ∪MF ) ∧

sw := 9404(current directory, ff) ∈ FILES CHILDREN)

VERIFY CHV

P15 blocked chv status = blocked sw := 9840

P16∃ code · code ∈ CODE ∧ blocked chv status = blocked∧ pin = code

counter chv := MAX CHV ∥permission session(chv) := true ∥sw := 9000

P17∃ code · code ∈ CODE ∧ blocked chv status = blocked ∧pin = code ∧ counter chv = 1

counter chv := 0 ∥blocked chv status := blocked ∥permission session(chv) := false ∥sw := 9840

P18∃ code · code ∈ CODE ∧ blocked chv status = counter chv := counter chv − 1 ∥blocked ∧ pin = code ∧ counter chv = 1 sw := 9804

UNBLOCK CHV

P19 blocked status = blocked sw := 9840

P20∃ code unblock, new code · code unblock ∈ CODE ∧new code ∈ CODE ∧ blocked status = blocked ∧ PUK = code unblock

pin := new code ∥blocked chv status := unblocked ∥counter chv := MAX CHV ∥counter unblock chv :=

MAX UNBLOCK ∥permission session(chv) := true ∥sw := 9000 ∥

P21

∃ code unblock, new code · code unblock ∈ CODE ∧ counter unblock chv := 0 ∥new code ∈ CODE ∧ blocked status = blocked ∧ blocked status := blocked ∥PUK = code unblock ∧ counter unblock chv = 1 sw := 9840

P22

∃ code unblock, new code · code unblock ∈ CODE ∧ counter unblock chv :=new code ∈ CODE ∧ blocked status = blocked ∧ counter unblock chv − 1 ∥PUK = code unblock ∧ counter unblock chv = 1 sw := 9804



APPENDIX C. Boundary goals of GSM 11.11 formal model

Predicates Boundary Goals N◦

SELECT FILE

P6 current directory = dfgsm BG13

P7current directory = mf BG14

current directory = dfgsm BG13

P8 no solution



P10 no solution

P11 no solution

P12 no solution





VERIFY CHV

P15 blocked chv status = blocked ∧ counter chv = 0 BG15

P16 blocked chv status = unblocked BG16

P17 blocked chv status = unblocked ∧ counter chv = 1 BG17

P18blocked chv status = unblocked ∧ counter chv = 2 BG18

blocked chv status = unblocked ∧ counter chv = 3 BG19

UNBLOCK CHV

P19 blocked status = blocked ∧ counter unblock chv = 0 BG20

P20 blocked status = unblocked BG21

P21 blocked status = unblocked ∧ counter unblock chv = 1 BG22

P22blocked status = unblocked ∧ counter unblock chv = 2 BG23

blocked status = unblocked ∧ counter unblock chv = 10 BG24


Documents

Bllp04 Ij Author