Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Mutation Testing ToolsSoftware Analysis and TestingCu Nguyen Duy (cunduy at fbk dot eu)Alessandro Marchetto (marchetto at fbk dot eu)Paolo Tonella (tonella at fbk dot eu)Mariano Ceccato (ceccato at fbk dot eu)
Academic Year 2011-2012
Friday, November 25, 11
Introduction
• Mutation testing is expensive
• Compilation, execution, comparison cost
• Not many supported tools
• MuClipse/mujava, Jumblen
• Improving mutant score is time consuming
Friday, November 25, 11
Mutation testing with MuClipse
Friday, November 25, 11
MuClipse!An Eclipse plugin for MuJava (http://
cs.gmu.edu/~offutt/mujava/)!Highlight
! Traditional operators! OO operators! Java
!Functionality!Generate mutants!Run tests!Report mutant score
Friday, November 25, 11
MuClipse: generate mutants!Eclipse: Run > Run Configurations
Friday, November 25, 11
MuClipse: generate mutants!Eclipse: Run > Run Configurations
Friday, November 25, 11
MuClipse: generate mutants!Eclipse: Run > Run Configurations
Friday, November 25, 11
MuClipse: generate mutants!Eclipse: Run > Run Configurations
Friday, November 25, 11
MuClipse: run test!Eclipse: Run > Run Configurations
Friday, November 25, 11
MuClipse: run test!Eclipse: Run > Run Configurations
Friday, November 25, 11
MuClipse: run test!Eclipse: Run > Run Configurations
Friday, November 25, 11
MuClipse: run test!Eclipse: Run > Run Configurations
Friday, November 25, 11
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
Friday, November 25, 11
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
***********| Executing Mutants |*********
----------------------------------------------AOIS_11 of 15----------------------------------------------- - testAdd = junit.framework.AssertionFailedError : expected:<5> but was:<6> ------------0 live, 1 killed.
Friday, November 25, 11
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
----------------------------------------------AOIS_22 of 15----------------------------------------------- - testAdd = junit.framework.AssertionFailedError : expected:<5> but was:<4> ------------0 live, 2 killed.
Friday, November 25, 11
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
----------------------------------------------AOIS_33 of 15----------------------------------------------- - testAdd = true - testAdd_2 = true - testAdd_3 = true - testAdd_4 = true - testAdd_5 = true - testAdd_6 = true - testAdd_7 = true ------------1 live, 2 killed.
Friday, November 25, 11
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
----------------------------------------------AOIS_44 of 15----------------------------------------------- - testAdd = true - testAdd_2 = true - testAdd_3 = true - testAdd_4 = true - testAdd_5 = true - testAdd_6 = true - testAdd_7 = true ------------2 live, 2 killed.
Friday, November 25, 11
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
----------------------------------------------AOIS_55 of 15----------------------------------------------- - testAdd = junit.framework.AssertionFailedError : expected:<5> but was:<6> ------------2 live, 3 killed.
Friday, November 25, 11
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
-------------------------------Results for class math.Add-------------------------------Live mutants: 12Killed mutants: 31Mutation Score: 72.0Writing to file...
Friday, November 25, 11
Exercise 1
!Install MuClipse!http://selab.fbk.eu/swat/projects/muclipse.tgz
!Add java library to Exercise-3-MathOperators!http://selab.fbk.eu/swat/projects/mu-lib.tgz
!Generate mutants! Identify 10 changes in the generated mutants!Hints:
! Use eclipse comparison feature
Friday, November 25, 11
Exercise 2
!Continue exercise 1!Report the mutant score of the test case:
TestCase1_add_Junit3_a of the project! use only traditional mutation operators
!Improve the mutant score of the test case
Friday, November 25, 11
Exercise 3
!Compare the coverage of the test case: TestCase1_add_Junit3_a before and after being improved using mutant score
!Hints:!Enable Clover!Execute only the test case and report the
coverage
Friday, November 25, 11
Mutation testing with Jumble
Friday, November 25, 11
Installation
๏ Download Jumble from http://jumble.sourceforge.net
๏ Unzip the downloaded file to a folder, namely jumble_1_1_0
๏ Inside the folder eclipseplugin > plugin-export, Copy or Move the folder jumble to your Eclipse plugins folder: Eclipse > Plugins, or Eclipse > dropins > eclipse > plugins
๏ Restart Eclipse
2
Tuesday, November 9, 2010
Run default mutation testing
๏ Condition: Class-To-Be-Mutated and test class have the same package, and test class name = <Class-To-Be-Mutated>Test.java
๏ Steps: 1. Right click on <YourClass>2. Choose Jumble > Jumble Class 3. Look at the Console view to see the mutant
score
3
Tuesday, November 9, 2010
Run custom mutation testing
๏ Applied when <test class > # <class>Test.java, or in different packages
๏ Follow the step 1 and 2 of the default run (see previous section)
๏ Customize the execution arguments of Jumble to specify your test class. In Eclipse, go to Run > Run Configurations
4
Tuesday, November 9, 2010
Run custom mutation testing Select Run Jumble Configuration In the Arguments tab, put your <TestClass> after
<YourClass> Run
5
Tuesday, November 9, 2010
Mutation testing report Mutation points = number of mutants “M FAIL”: mutant was not killed by the current test
class, example: “M FAIL: math.Add:5: += -> -=” : line 5 of class math.Add,
“+=” is mutated to be “-=” Mutant score = (mutant points - “m-failed”) / mutant
points
6
Tuesday, November 9, 2010
Exercise 1
Apply Mutation testing with Exercise-3-MathOperators
Report the mutant score of the test case: TestCase1_add_Junit3_a of the project
Improve the mutant score of the test case
7
Tuesday, November 9, 2010
Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not
be killed, for example:
!Write more test cases to try to kill other mutants
public class Add { public static int sum (int a, int b){ return a++ +b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}
Friday, November 25, 11
Mutation analysis!Use traditional mutation operators!Generate mutants of one of your classes
(your choice)!Execute unit test cases that test the chosen
class!Report the mutant score of the test set w.r.t
the chosen class!Improve the mutant score of the test set to
at least 90%
Friday, November 25, 11