Object Oriented Mutation Testing: A Survey

Muhammad Bilal Bashir, Aamer Nadeem

Center for Software Dependability

Mohammad Ali Jinnah University

Islamabad, Pakistan

bilalbezar@gmail.com, anadeem@jinnah.edu.pk

Abstract— Mutation testing is a fault based testing technique that

measures the effectiveness of test case suite and helps to generate

test cases that have potential of identifying real faults in the

program. Faults are injected using a pre-defined set of mutation

operators. Object oriented paradigm is known for its salient

features that make it different from traditional programming

paradigms. We find traditional mutation operators insufficient

and incapable of introducing faults in object oriented features

like encapsulation, inheritance and polymorphism. This paper

presents a survey on object oriented mutation testing to find out

the issues mutation testing is currently facing and solutions that

have been proposed to cater those issues. We have studied

various techniques proposed till now and tools that are developed

for the aid of testers to perform mutation testing. We analyze

these techniques considering the issues in object oriented

mutation testing against the solutions they have offered. Our

survey concludes that selective mutation testing can reduce the

computational cost and experiments on large scale can help

identifying potential mutation operators to avoid generation of

equivalent mutants.

Keywords-object oriented paradigm; mutation testing; mutation

operators

I. INTRODUCTION

Mutation testing is a fault based testing technique that was first introduced by DeMillo, Lipton, and Sayward [1] in 1978. Since then a lot of research has been done and it has been an area of interest for the researchers during last two decades. The purpose of mutation testing is to measure effectiveness of test case suite. Figure 1 shows an iteration of mutation testing process.

Mutation testing process starts with the generation of mutants. A mutant is actually a copy of original program containing one permutation (fault) which is syntactically correct. These permutations or faults are introduced using a pre-defined set of permutations called mutation operators.After generating mutants, the next step is to execute all the test cases against all mutants and compare the outputs with the original program’s outputs. If a test case produces different output on a mutant then the mutant is said to be killed by the test case otherwise the mutant is said to be alive. A mutant is called equivalent mutant if no test case can distinguish its output from the original program’s output. It is impossible to kill an equivalent mutant because it is semantically equivalent to the original program. The ratio of killed and alive mutants is known as mutation score which indicates how effective a given

test case set is. Tester can re-generate test cases to kill the remaining alive mutants and to raise the mutation score because a test case set with higher mutation score is considered more effective.

In this paper we present a survey on the existing object oriented mutation techniques. We have critically reviewedthese techniques on the basis of evaluation criteria that we have designed by considering important aspects of mutation testing. These aspects can have their influence on mutation testing process. For example at what level of software size (class or inter-class) a given technique can be applied? How well a given technique covers the domain of object orientation? How the researchers have designed the mutation operators set and what evidence by means of empirical evaluation they have provided to show that their technique is good enough for object oriented mutation testing? Another contribution of our work is a survey of available mutation testing tools. We have listed them all in this work with their technical details.

The rest of this paper is organized as follow. Section 2 discusses object oriented mutation testing. Section 3 presents the surveyed techniques. Section 4 contains evaluation criteria and section 5 presents our analysis. In section 6 we discuss the available tools. We conclude the discussion in section 7 and provide directions for future work.

Mutant Executor

Test Case

Suite

Mutant Generator

Original

Program

Mutation Score

Figure 1: Mutation Testing Process Flow

Mutation

Operators

Mutants

978-1-4673-4451-7/12/$31.00 ©2012 IEEE

II. OBJECT ORIENTED MUTATION TESTING

Object oriented paradigm is different in nature as compared to conventional procedural paradigm in a variety of ways which includes;

It maps real world entities to software objects using templates (classes)

It groups the data members and applying constraints on their access (encapsulation)

It reuses the pre-written code (inheritance) and

It maintains the software by adding functionality without changing the design of software (polymorphism)

The mutation testing process in object oriented paradigm goes with similar steps as in conventional mutation testing. We have original program, mutation operators set and test case suite as the input, mutants are generated and then executed against test case suite. Mutation score is calculated to show the effectiveness of a test case set. The difference exists in test case suite and mutation operators set. Test case does not just contain input values rather it has object creation, calls to different methods to attain a certain state of an object and then call to the method under test plus the input values required to execute methods in a test case. We will not discuss structure of a test case in detail here because it is beyond the scope of this study. Mutation operator set in object oriented paradigm is large in numbers as compared to procedural paradigm. Figure 2 shows mutation operators set for object oriented paradigm in general.

There are 3 ovals shown in figure 2 named A, B, and C. Oval A represents mutation operators for procedural paradigm features like primitive data types, arithmetic operators, relational operators and so on, whereas oval C is set of mutation operators of language specific features like exception handling, input/output streams and so on. Oval B is set of all mutation operators that we require for object oriented mutation testing. It borrows operators from A and C as well it has its own set of operators devised from object oriented features. We can conclude that mutation operator set for object oriented paradigm is larger than the conventional mutation operator set.The reason is that Oval B not only imports mutation operators from Oval A (relational operators, arithmetic operators etc.) but also introduces operators for object-oriented features (encapsulation, inheritance and polymorphism) and borrows language specific features from Oval C (File handling, exception handling etc.).

III. SURVEYED TECHNIQUES

In this section we present critical review of all the surveyed techniques. These techniques are proposed for object oriented mutation testing.

A. S. Kim, J. Clark, and J. McDermid (1999)

Kim, Clark, and McDermid [5] propose a technique which is based on the concept of HAZOP (Hazard and Operability Study) [2]. Authors identify all the Java language constructs (elements) using Java specification 1.0 and list out all of their attributes. After that they use 10 guide words and pair them with all the attributes one by one to identify possible deviation and its consequence. The pair of guide word and attribute that gives meaningful interpretation is listed down as mutation operator. This technique can be applied at inter-class level mutation testing. Their technique looks good and practical because it covers all the object oriented features with a rigrous method instead of intuition. On the other hand the list of generated operators is too long that increases the computational cost of mutation testing if we use it to test a software. Authors have not applied their approach on a working software to show the results and its performance with respect to the number of mutants generated and the number of exuections run.

B. S. Kim, J. Clark, and J. McDermid (2000)

Kim, Clark, and McDermid [6] propose a technique to perform object oriented mutation testing using a new method they name as Class Mutation. This is an inter-class mutation testing technique to test object oriented software. Authors claim that their approach is less expensive and mutation operators can be selected according to the requirement. Although they cover object oriented features quite well but there is no evidence that they successfully cover all possible mutation operators. Although authors claim that they have applied Class Mutation on an IBM product Product Starter Kit(PSK) yet they have not included experimental results in this study. Authors also claim that their technique can be merged with languages that have both non-object oriented features as well as object oriented features like C++ and Java.

C. P. Chevalley (2001)

Chevalley [8] extends the work of Kim, Clark, and McDermid [6] by introducing 5 new mutation operators for object oriented mutation testing. These operators are for inter-class level mutation testing. In this approach he tries to reduce the set of mutation operators and comes up with a set that has almost same impact on the testing as of applying full set of mutation operators. Author has then provided his survey on related work and available tools for mutation testing. He hasdeveloped a GUI (Graphical User Interface) based tool calledJavaMut that supports 26 mutation operators; fifteen from Class Mutation [6], five new mutation operators they have devised and six selective mutation operators from traditional mutation testing. Author has presented the details of experiments performed on Flight Guidance System (FGS)using JavaMut. The main issue with this technique is the lackof confidence about object oriented features coverage because author has devised new operators on the basis of past experience rather than using some rigrous method.

Figure 2: Mutation Operators Set

A B C

978-1-4673-4450-0/12/$31.00 ©2012 IEEE

D. Y.-S. Ma, Y.-R. Kwon, and J. Offutt (2002)

Ma, Kwon, and Offutt [11] propose an inter-class level mutation testing approach, which is based on an object oriented fault model [16]. Gallagher, Offutt, and Cincotta [16] presentan object oriented fault model that lists all the faults that can appear in the collaboration of more than one classes. Authors analyze research of Kim, Clark, and McDermid [6] and Chevalley, and Thévenod-Fosse [7, 8] and state that mutation operators presented in these studies are not general rather they are application or problem specific. One major contribution in this study is that authors provide formuals to calculate number of mutants that a particular mutation operator can create when it is applied on the product under test. Authors have developed a tool for mutation testing using compile time reflective system following the guidelines of Chevalley and Thévenod-Fosse [7, 8]. Authors present results of their experiments on small scalein the paper.

E. R. T. Alexander, J. M. Bieman, S. Ghosh, and J. Bixia

(2003)

Alexander, Bieman, Ghosh, and Bixia [12] present a new technique that can be used for the mutation of Java objects. According to authors mutation operators are created to surve one of the two purposes; first to introduce faults in the program, second to attain traditional code coverage like branch coverage [12]. Authors claim that it is difficult to define a method to change states of user defined objects because it is impossible to know the semantics of an object without knowing the intention of its creation. Authors have made two contributions in this study; they have designed mutation operators for the Java APIs, and they have applied these mutants on objects to introduce faults. Authors have developed a Mutation Engine to geneate mutants for each type of mutation. This idea looks exciting for state based mutation testing but a lot of work needs to be done to bring it up and to apply it on real software because still it lacks the general idea of creating mutation operators for any kind of an object.

F. Offutt, Ma, and Kwon (2006)

Offutt, Ma, and Kwon [18] extend their earlier work [13] with the addition and change of some mutation operators for object oriented paradigm. In this work they split 3 mutation operators, merge 2 mutation operators into 1, and introduce 3 new mutation operators. Authors splits ISK (super keyword deletion) into ISD (super keyword deletion) and ISI (super keyword insertion), JSC (static modifier change) into JSI (static modifier insertion) and JSD (static modifier deletion), and JTD (Member variable initialization deletion) into JTD (this keyword deletion) and JTI (this keyword insertion). They merge two mutation operators OAO (Argument order change) and OAN (Argument number change) into OAC (Arguments of overloading method call change). The new additions include PCI (Type cast operator insertion:), PCC (Type cast operator deletion), and PCD (Cast type change) that are related to type conversion. Authors present some guidelines for the class-level mutation operators to prevent generation of equivalent mutants. They come up with some 'equivalency conditions' that explain scenarios when and how certain type of equivalent mutants for a set of sixteen mutation operators are usually created.

IV. EVALUATION CRITERIA

In this section we present evaluation criteria we use to analyze object-oriented mutation testing techniques.

A. Cost Effective

Mutation testing is computationally expensive so it is important to check if a technique by any means helps to reduce the computational cost. For example, by reducing number of mutants’ executions Table 1 describes evaluation criteria for this parameter.

TABLE 1. EVALUATION CRITERIA FOR COST EFFECTIVENESS

Value Criteria

Yes The value Yes means the technique offers selective muation

oerpators set.

No The value No means the technique does not offer selective muation oerpators set.

B. Equivalent Mutant Detection

Equivalent mutants increase the computational overhead because they cannot be killed by any test case. If a technique helps to avoid creation of equivalent mutants or helps detecting equivalent mutants, computational cost can be reduced Table 2 describes evaluation criteria for this parameter.

TABLE 2. EVALUATION CRITERIA FOR EQUIVALENT MUTANT DETECTION

Value Criteria

Yes The value Yes means the technique either helps avoiiding

creation of equivalnet mutants or can detect equivalent

mutants.

No The value No means the technique neither helps avoiiding

creation of equivalnet mutants nor can detect equivalent

mutants.

C. OO Feature Coverage

A good mutation testing technique should offer operators that cover all the object oriented features. Unfortunately there is no measure that can help to declare that a given set of mutation operators are providing full coverage. So we have chosen technique [18] as benchmark because of two reasons; it is the most recent study on object oriented mutation testing, it defines largest set of mutation operators.

TABLE 3. EVALUATION CRITERIA FOR OO FEATURE COVERAGE

Value Criteria

Full The value Full means the technique provides full support to all

the object oriented features.

Partial The value Partial means the technique does not provide full

support to all the object oriented features.

D. Level of Testing

An inter-class level technique can ensure complete feature coverage as compared to unit-level testing technique. So the technique should offer mutation operators that can introduce mutations at inter-class level. Table 4 describes evaluation criteria for this parameter.

978-1-4673-4450-0/12/$31.00 ©2012 IEEE

TABLE 4. EVALUATION CRITERIA FOR LEVEL OF TESTING

Value Criteria

AC The value AC means that this technique support mutation

testing at class (unit) level.

IC The value IC means that this technique support inter-class

(integration) level mutation testing.

E. State Mutation Support

Object has states and testing object's behavior in different states is critical. Mutation operators should be able to mutate object's state at runtime to test this aspect. Table 5 describes evaluation criteria for this parameter.

TABLE 5. EVALUATION CRITERIA FOR STATE MUTATION SUPPORT

Value Criteria

Yes If a technique can mutate object’s state at runtime then value

is Yes.

No If a technique cannot mutate object’s state at runtime then

value is No.

F. Potenial Mutation Operators

Potential mutation operators set can help revealing real faults in the program. To achieve the objective, a reduced set of potential mutation operators is required. Table 6 describes evaluation criteria for this parameter.

TABLE 6. EVALUATION CRITERIA FOR POTENTIAL MUTATION OPERATORS

Value Criteria

Yes If a technique provides list of potential mutant operators then

the value is Yes.

No If a technique does not provide list of potential mutant operators then the value is No.

G. Tool Support

To increase performance of mutation testing, automated tools can be used. A tool is a good mean to test practicality of an approach as it serves as proof of concept and experimental results can show effectiveness of an approach. Table 7 describes evaluation criteria for this parameter.

TABLE 7. EVALUATION CRITERIA FOR TOOL SUPPORT

Value Criteria

Yes The value Yes means technique provides tool support and the

whole mutation process is automated.

No The value No means technique does not provide tool support

and it is not automated.

V. ANALYSIS OF SURVEYED TECHNIQUES

In this section we present our analysis on the surveyed techniques. We analyze these techniques on the basis of our evaluation criteria. The table 8 presents the analysis in a row-column format

The biggest limitation of mutation testing is its high computational cost which hinders the testers and software

industry to use mutation testing practically and on testing large scale software products. Selective mutation operators can serve this purpose but none of the studies have been able to reduce the operators set for object oriented paradigm. In [4, 5, 6, 7, 11, 15, 18] we find some discussion on selective mutation testing but more experiments and work still need to be done in this area. In our opinion it is not quite possible to give a definite reduced set of operators because in object oriented paradigm a given user defined data type (class) can have variety of implementations depending upon the requirement and the personnel implementing the class.

Another issue with mutation testing is identification of equivalent mutants, which also increases computational cost of mutation testing. Although the problem of identifying equivalent mutant is un-decidable yet through experiments we can find out which mutation operators most of the times produces equivalent mutants. We find researchers in [7, 8, 11] perform some experiments and are able to identify some potential mutation operators like AMC that are found producing most of the equivalent mutants. In [7, 8] experiments show that there are few mutation operators like SMC (static Modifier Change) that produce equivalent mutants in testing Flight Guidance System (FGS) so these results can be used as justification for not using a particular mutation operators for mutation testing. Also in [11] we find operatorslike PNC (new Method Call with Child Class Type), PMD (Instance Variable Declaration with Parent Class Type), PPD (Parent Variable Declaration with Child Class Type), and IHI (Hiding Variable Insertion) produce equivalent mutants in the presence of method overriding.. In [18] authors have come up with some 'equivalency conditions' that explain scenarios when and how certain type of equivalent mutants for a set of sixteen mutation operators are usually created. Avoiding these cases can prevent generation of equivalent mutants.

A technique covering all the object oriented features can give more confidence to tester on the quality of software than the technique covering the features partially. These include features from encapsulation, inheritance, polymorphism and other features related to object oriented paradigm. For this it is important that mutation operators are created through well defined and rigorous method instead of using past experiences and intuition of the teeters. We find technique [18] as better because it is comprehensive and provides quite a few operators covering all the object oriented features.

A good and comprehensive mutation testing technique should be able to inject faults at any level and more importantly at inter-class level where more than one classes collaborate and interact with teach other. A unit-level technique cannot cover the inheritance and polymorphism features because it tests one class at a time. All the studied techniques [5, 6, 7, 8, 11, 12, 18] can be applied at inter-class level and covers all the features of object oriented paradigm.

Object has states and an object's state is changed when an operation is performed with the reference of that object at runtime. It is important to test the behavior of an object when its state changes to see if the object has successfully transits from one state to another. This phenomenon is called state based testing. In [12] authors are mutating the state of an object

978-1-4673-4450-0/12/$31.00 ©2012 IEEE

TABLE 8. ANALYSIS TABLE OF MUTATION TESTING TECHNIQUES FOR OBJECT-ORIENTED PROGRAMS

Techniques

Evaluation Parameters

Cost

Effective

Equivalent

Mutant Detection

OO Feature

Coverage

Level of

Testing

State Mutation

Support

Potential Mutation

Operators

Tool

Support

Kim et al. [5] No No Partial IC No No No

Kim et al. [6] No No Partial IC No No No

Chevalley [8] No Yes Partial IC No No Yes

Ma et al. [11] No Yes Partial IC No Yes Yes

Alexander et al. [12] No No Partial IC Yes No No

Offutt et al. [18] No Yes Full IC No No Yes

at runtime by manipulating its state values. The scope of their technique is quite limited as far as application is concerned as they just apply it on three APIs of Java. Other surveyed techniques [5, 6, 7, 8, 11, 18] do not deal with the state of an object rather these techniques focus on covering object oriented features and errors are injected before the execution of a program.

The objective of mutation testing is to find out real faults in the program by injecting plausible faults and trying to catch them. If the faults are not realistic the whole purpose is defeated as they may not have potential of finding real faults. It is important that mutation operators should be created by considering examples of real faults. We find only one technique [11] that considers an object oriented fault model to design mutation operators, whereas other techniques [5, 6, 7, 8, 12, 18] simply do not consider this concept and they may fail to gain the confidence of the tester.

Mutation testing is computationally expensive so it is very important to have tool support to aid the tester in order to increase the performance. Tool also plays the role of proof of concept because it gives the confidence that a technique is practical and cane be used. Out of six surveyed techniques we find [7, 8, 11, 18] have properly implemented a complete tool whereas [12] has implemented prototype a tool. In [5, 6] we do not find any discussion about the tool. Most of the techniques [7, 8, 11, 18] have performed experiments and present their results with analysis. In [6, 12] authors claim that they have performed experiments but they do not provide results of their experiments. In [5] authors have not performed any experiment at all.

VI. TOOL SUPPORT

In this section we present brief analysis on the mutation testing tools for object oriented paradigm that we find in literature. There are seven tools and their details are given below one by one.

A. JavaMut (2001)

Chevalley and Thévenod-Fosse [7] developed a tool to perform the mutation analysis and they name it JavaMut. It is a GUI (Graphical User Interface) based tool. The tool is implemented using compile time reflective system called OpenJava. One good feature of this tool is that tester can view the mutated code in the tool. JavaMut implements 26 mutation operators; six selective mutation operators [3, 10], fifteen from Class Mutation [6], and five new operators [7, 8] that authors

have created. Tool considers only those mutants that satisfy two conditions; they are successfully compiled, the byte-code of mutant is different than the byte code of original program.

B. Jester (2001)

Moore [9] has developed a tool called Jester for mutation testing. This tool provides mutation testing support for the programs written in Java programming language. Jester is quite simple in nature as it only supports mutation operators that can be run on a single Unit (class). It uses JUnit [21] as base that supports Unit testing of Java programs. Due to unit level testing this tool cannot apply mutation operators that are designed for other features like inheritance and polymorphism and work at integration (system) level.

C. Object Mutation Engine (2003)

Alexander, Bieman, Ghosh, and Bixia [12] have developed a tool called Object Mutation Engine (OME). It performs mutation testing on objects of Java APIs. The tool has quite complex architecture containing 7 different components. These components are OME Executor (controls the overall execution of tool), Test Manager (prepares for the test job to be executed), State Inspector and Results Collection (monitor the object's state and collect the values of objects), Result Rendering (renders the results in the required format), Test Drivers (establish required environmental conditions), Mutation Operators (classes containing one type of mutation operator), and Test Evaluation (evaluates results).

D. MuJava (2005)

Offutt, Ma, and Kwon [13], and Ma, Offutt, and Kwon [14] have developed a tool for mutation testing of Java programs and they name it MuJava (Mutation System for Java). This is a GUI (Graphical User Interface) based tool that allows both generation of mutants and execution of them automatically. The first version of tool implements 29 mutation operators in total; five selective traditional mutation operators [3], and twenty-four object oriented mutation operators from the work of [11]. There are three main modules in the MuJava tool. First is Mutants Generator (creates mutants), second module is Mutants Executor (executes mutants), and third module is Mutants Viewer (displays results). This tool has been used to perform experiments to evaluate the object oriented mutation testing and operators used and results are available in [15]. Authors have launched extended version of MuJava [18] with some changes, omissions and additions to class mutation operators (see Section III.F for details). Now MuJava supports

978-1-4673-4450-0/12/$31.00 ©2012 IEEE

up to 34 mutation operators including 5 mutation operators from conventional paradigm and 29 mutation operators from object oriented paradigm.

E. ExMan (2006)

Bradbury, Cordy, and Dingel [17] propose a generalized approach for experimental mutation analysis. In this approach artifacts and components can be interchanged with each other in order to compare quality assurance tools. To use this tool we have to pass through a setup phase that involves creation of profile that tells the tools about command-line usage and purpose of using this application. Secondly, it involves selection of project to be run. Then it selects original source code and generates mutants. After that mutants and original source code are compiled. The benchmarks to compare results or assertions are provided to the ExMan and finally it performs analysis on the basis of benchmarks and produces results.

F. Jumble (2007)

Just like Jester [9], Jumble [19] is mutation testing tool for Java programs. Jumble is quite simple in nature as it only supports mutation operators that can be run on a single Unit (class). It uses JUnit [20] as base that supports Unit testing of Java programs. Due to unit level testing this tool cannot apply mutation operators that are designed for other features like inheritance and polymorphism and work at integration (system) level.

G. JavaLanche (2009)

Grun, Schuler, and Zeller [20] propose a framework and have implemented in a tool called JavaLanche. The main purpose of the study is to check the impact of equivalent mutants in the mutation testing. JavaLanche implements selective mutation operators. This tools uses coverage data about a test set to run only those test cases that execute the mutated statement in the code.

VII. CONCLUSION

Mutation testing is computationally expensive method and suffers from the issue of identifying equivalent mutants, which is inherent from its nature. Although techniques have been proposed for compensating new features of object oriented paradigm yet the inherited issues of mutation testing need more attention. Our survey shows that although proposed techniques discuss these problems yet none of them has provided a practical solution to solve them. Selective mutation testing can significantly reduce the computational cost and researchers are trying to find out potential mutation operators but still more work needs to be done in this area. Identifying equivalent mutant problem is un-decidable but experiments can help identifying those mutation operators that can prevent or at least minimize generation of equivalent mutants. In future we are working to solve mutation testing issues inherited from its nature and focusing on combining evolutionary testing with mutation testing to automate test case generation. Evolutionary testing approaches like Genetic Algorithm can help increasing performance of overall process by automating test generation process.

REFERENCES

[1] R. A. DeMillo, R. J. Lipton, F. G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer 11, pp 34–41, April 1978

[2] T. A. Kletz, "HAZOP and HAZAN: Identifying and Assessing Process Industry Hazards," 3rd Edition, Hemisphere Publishers; ISBN: 1-560-32276-4, 1992

[3] J. Offutt, A. Lee, G. Rothermel, R. H. Untch, C. Zapf, "An experimental determination of sufficient mutant operators," ACM Trans Software Eng Methodol 5, pp 99–118, 1996

[4] S. Kim, J. Clark, and J. McDermid, "Assessing Test Set Adequacy for Object-Oriented Programs Using Class Mutation," 28 JAIIO: Symposium on Software Technology (SoST`99), September. 1999

[5] S. Kim, J. Clark, and J. McDermid, "The Rigorous Generation of Java Mutation Operators Using HAZOP," In 12th International Conference Software & Systems Engineering and their Applications, December 1999

[6] S. Kim, J. Clark, and J. McDermid, "Class Mutation: Mutation Testing for Object-Oriented Programs," In Proceedings of the Net.ObjectDays Conference on Object-Oriented Software Systems, Erfurt, Germany, Oct. 2000

[7] P. Chevalley and P. Thévenod-Fosse, "A Mutation Analysis Tool for Java Programs," LAAS Report No 01356, Toulouse, France, September 2001, M. Daran and P. Thévenod-Fosse. Software Error

[8] P. Chevalley, "Applying Mutation Analysis for Object Oriented Programs Using a Reflective Approach," In Proceedings of the 8th Asia-Pacific Software Engineering Conference, Macau SAR, China, December 2001

[9] I. Moore. Jester. http://jester.sourceforge.net/, 2001. accessed April 2012

[10] E. F. Barbosa, J. C. Maldonado, A. M. R. Vincenzi, "Toward the determination of sufficient mutant operators for C," Software TestingVerification & Reliability vol 11, pp 113–136, 2001

[11] Y.-S. Ma, Y.-R. Kwon, and J. Offutt, "Inter-class Mutation Operators for Java," In Proceedings of the 13th IEEE International Symposium on Software Reliability Engineering, pp 352-363, Annapolis MD, November 2002

[12] R. T. Alexander, J. M. Bieman, S. Ghosh, and J. Bixia, "Mutation of Java objects," In Proceedings of IEEE 13th International Symposium on Software Reliability Engineering, 2003

[13] J. Offutt, Y.-S. Ma, and Y.-R. Kwon, "An Experimental Mutation System for Java," ACM SIGSOFT Software Engineering Notes, vol.29 no.5, September 2004

[14] Y.-S. Ma, J. Offutt, and Y.-R. Kwon, "MuJava: an Automated Class Mutation System: Research Articles," Software Testing, Verification & Reliability, vol.15 no.2, p.97-133, June 2005

[15] Y.-S. Ma, M. J. Harrold, and Y.-R. Kwon, "Evaluation of Mutation Testing for Object-Oriented Programs," Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China

[16] L. Gallagher, J. Offutt, A. Cincotta, "Integration testing of object-oriented components using finite state machines," Research Articles, Software Testing, Verification & Reliability, v.16 no.4, pp 215-266, December 2006

[17] J.S. Bradbury, J.R. Cordy, and J. Dingel, "Exman: A generic and customizable framework for experimental mutation analysis", Mutation Analysis, Workshop on, 0:4, 2006

[18] J. Offutt, Y.S. Ma, and Y.R. Kwon. The class-level mutants of mujava. In AST ’06: Proceedings of the 2006 International Workshop on Automation of software test, pages 78–84, NY, USA, ACM 2006

[19] Sourceforge. Jumble. http://jumble.sourceforge.net/, 2007. accessed April 2012

[20] B. Grun, D. Schuler, A. Zeller, "The impact of equivalent mutants", In Proceedings of the 4th International Workshop on Mutation Testing, 2009

[21] Sourceforge. Junit. http://www.junit.org/. accessed April 2012

978-1-4673-4450-0/12/$31.00 ©2012 IEEE