PIT: state of the art of mutation testing systemRevised edition

MUTANTS KILLER

by Tàrin Gamberìni - CC BY-NC-SA 4.0

Lavora da sempre con tecnologie Java-based free e Open Source presso la Regione Emilia-Romagna

Si interessa di Object Oriented Design, Design Patterns, Build Automation, Testing e Continuous Integration

TÀRIN GAMBERÌNI

Partecipa attivamenteal Java User GroupPadova

Unit Testing

Test-Driven Development (TDD)

Continuous Integration

Code Coverage

Mutation Testing

DEVELOPMENT

UNIT TESTING & TDD

The proof of working code

Refactor with confidence (regression)

Improve code design

Living documentation

TDD extremely valuable in some context

CONTINUOUS INTEGRATION

Maintain a single source repository

Automate the build

Cheap statics for code quality

Automate unit & integration tests

Catch problems fast

LINE COVERAGE

Is a measure of tested source code

line, statement, branch, … , coverage

LINE COVERAGE 100%

Only able to identify not tested code

Doesn't check tests are able to detect faults

Measures only which code is executed by tests

WEAK TESTS

public static String foo( boolean b ) { if ( b ) { performVitallyImportantBusinessFunction(); return "OK"; } return "FAIL";}

@Testpublic void shouldFailWhenGivenFalse() { assertEquals("FAIL", foo( false ));}

@Testpublic void shouldBeOkWhenGivenTrue() { assertEquals("OK", foo( true ));}

The untested side effect

WEAK TESTS

public static String foo( boolean b ) { if ( b ) { performVitallyImportantBusinessFunction(); return "OK"; } return "FAIL";}

@Testpublic void shouldFailWhenGivenFalse() { assertEquals("FAIL", foo( false ));}

@Testpublic void shouldBeOkWhenGivenTrue() { assertEquals("OK", foo( true ));}

The untested side effect

WEAK TESTS

public static String foo( int i ) { if ( i >= 0 ) { return "foo"; } else { return "bar"; }}

@Testpublic void shouldReturnFooWhenGiven1() { assertEquals("foo", foo( 1 ));}

@Testpublic void shouldReturnBarWhenGivenMinus1() { assertEquals("bar", foo( -1 ));}

The missing boundary test

WEAK TESTS

public static String foo( int i ) { if ( i >= 0 ) { return "foo"; } else { return "bar"; }}

@Testpublic void shouldReturnBarWhenGiven1() { assertEquals("bar", foo( 1 ));}

@Testpublic void shouldReturnFooWhenGivenZero() { assertEquals("foo", foo( 0 ));}

@Testpublic void shouldReturnFooWhenGivenMinus1() { assertEquals("foo", foo( -1 ));}

The missing boundary test

WEAK TESTS

public static String foo(Collaborator c, boolean b) { if ( b ) { return c.performAction(); } return "FOO";} @Test

public void shouldPerformActionWhenGivenTrue() { foo(mockCollaborator,true); verify(mockCollaborator).performAction(); }

@Testpublic void shouldNotPerformActionWhenGivenFalse() { foo(mockCollaborator,false); verify(never(),mockCollaborator).performAction(); }

The myopic mockist

WEAK TESTS

public static String foo(Collaborator c, boolean b) { if ( b ) { return c.performAction(); } return "FOO";} @Test

public void shouldPerformActionWhenGivenTrue() { String result = foo(mockCollaborator,true); verify(mockCollaborator).performAction(); assertEquals("MOCK", result);}

@Testpublic void shouldNotPerformActionWhenGivenFalse() { String result = foo(mockCollaborator,false); verify(never(),mockCollaborator).performAction(); assertEquals("FOO", result);}

The myopic mockist

WHO TESTS THE TESTS ?

MUTATION TESTING

● originally proposed by Richard Lipton as a student in 1971

● 1st implementation tool was a PhD work in 1980

● recent availability of higher computing power has led to a resurgence

● expose weaknesses in tests (quality)

MUTATION TESTING

● originally proposed by Richard Lipton as a student in 1971

● 1st implementation tool was a PhD work in 1980

● recent availability of higher computing power has led to a resurgence

● expose weaknesses in tests (quality)

MUTATION

Is a small change in your program

original mutated

MUTANT

A

mutated

version of

your

program

MUTATION TESTING

generate lots of mutants

run all tests against all mutants

check mutants: killed or lived?

MUTANT KILLED

Proof of detected mutated code Test is testing source code properly

If the unit test fails

MUTANT LIVED

Proof of not detected mutated code

If the unit test succeed

Test isn't testing source code

PIT: BASIC CONCEPTS

Applies a configurable set of mutators

Generates reports with test results

Manipulates the byte code generated by compiling your code

PIT: MUTATORSConditionals Boundary

< → <=<= → <> → >=>= → >

PIT: MUTATORSNegate Conditionals

== → !=!= → ==<= → >>= → << → >=> → <=

PIT: MUTATORSMath Mutator

+ → -- → +* → // → *% → *

& → || → &^ → &<< → >>>> → <<>>> → <<

PIT: MUTATORSRemove Conditionals

if (a == b) { // do something}

if (true) { // do something}

→

PIT: MUTATORSReturn Values Mutator Conditionals

public Object foo() { return new Object();}

public Object foo() { new Object(); return null;}

→

PIT: MUTATORSVoid Method Call Mutator

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

→public int foo() { int i = 5;

return i;}

public void doSomething(int i) { // does something}

PIT: MUTATORSMany others

Activated by defaultConditionals Boundary Mutator - Increments Mutator - Invert Negatives Mutator - Math Mutator - Negate Conditionals Mutator - Return Values Mutator - Void Method Calls Mutator

Deactivated by defaultConstructor Calls Mutator - Inline Constant Mutator - Non Void Method Calls Mutator - Remove Conditionals Mutator - Experimental Member Variable Mutator - Experimental Switch Mutator

PIT: MUTATORSApplication field

Traditional and classPIT is suitable for "ordinary" java programs because mutators change "ordinary" java language instructions.

The right mutator for the right application fieldNot the best option if you are interesting testing:● Aspect Java programs, try AjMutator● Concurrent programs, try ExMan● Mutators for: reflection, java annotations, generics, … ?

HOW PIT WORKSRunning the tests

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

Your program

HOW PIT WORKSRunning the tests

PIT runs traditional line coverage analysis

HOW PIT WORKSRunning the tests

Generates mutant by applaying mutators

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

mutant 2

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

mutant 1

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

mutant 3

HOW PIT WORKSBrute force

x = 9

100 tests x 1.2ms = 0.12s

0.12s x 10000 mutants = 20min !!!1000 class x 10 mutants per class = 10000 mutants

HOW PIT WORKSRunning the tests

PIT leverages line coverage to discard tests that do not exercise the mutated line of code

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

public int foo() { int i = 5; doSomething(i); return i;}

public void doSomething(int i) { // does something}

mutant 3

HOW PIT WORKSExtremely fast

x = 1

x = 1

x = 2 4 < 9

HOW PIT WORKSOutcomes

● Killed, Lived● No coverage: the same as Lived except there were no

tests that exercised the mutated line of code● Non viable: bytecode was in some way invalid● Timed Out: infinite loop● Memory error: mutation that increases the amount of

memory used● Run error: a large number of run errors probably

means something went wrong

PIT ECOSYSTEMTools

Command line

Other IDEs as a Java application

( )

EXAMPLEclass Ticket

EXAMPLEtestGetPrice_withAge11

EXAMPLEtestGetPrice_withAge71

EXAMPLEtestGetPrice_withAge18

EXAMPLEmvn org.pitest:pitest-maven:mutationCoverage

>> Generated 8 mutations Killed 6 (75%)>> Ran 11 tests (1.38 tests per mutation)=================================================================- Mutators=================================================================> org.pitest...ConditionalsBoundaryMutator>> Generated 2 Killed 0 (0%)> KILLED 0 SURVIVED 2 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 -----------------------------------------------------------------> org.pitest...ReturnValsMutator>> Generated 3 Killed 3 (100%)> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0

EXAMPLE

EXAMPLE

EXAMPLEtestGetPrice_withAge12

EXAMPLEHelps discovering better test data

Well known techniques(1) for creating dataInput space partitioning, boundary values, error guessing.Cartesian product of test data leads to a combinatorial explosion.

PIT helps but YOU have to findPIT tells you when a mutant has survived, but you are in charge to find out the right test data which will kill it.

(1) Introduction to Software Testing, 2008 – by P. Ammann, J. Offutt

EXAMPLEmvn org.pitest:pitest-maven:mutationCoverage

>> Generated 8 mutations Killed 7 (88%)>> Ran 10 tests (1.25 tests per mutation)=================================================================- Mutators=================================================================> org.pitest...ConditionalsBoundaryMutator>> Generated 2 Killed 1 (50%)> KILLED 1 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 -----------------------------------------------------------------> org.pitest...ReturnValsMutator>> Generated 3 Killed 3 (100%)> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0

EXAMPLE

EXAMPLE

EXAMPLEtestGetPrice_withAge70

EXAMPLE

?!?!?!?!?

EXAMPLEtestGetPrice_withAge70

QUESTIONS ?PIT: state of the art of mutation testing system

byTàrin GamberìniCC BY-NC-SA 4.0

CONTACT

Tàrin Gamberìni

tarin.gamberini@jugpadova.it

www.taringamberini.com

SOURCE CODE

https://github.com/taringamberini/linuxday2015.git

CREDITSMinion with gun - by jpartwork (no license available)https://nanookofthenerd.wordpress.com/2015/01/30/the-magnificent-maddening-marvelous-minion-episode-032/

Minion Hands on - by Ceb1031 (CC BY-SA 3.0)http://parody.wikia.com/wiki/File:Bob_minions_hands.jpg

Canon XTi components after disassembly - by particlem (CC BY 2.0)https://www.flickr.com/photos/particlem/3860278043/

Continuous Integration - by New Media Labs (no license available)http://newmedialabs.co.za/approach

Rally car - by mmphotography.it (CC BY-NC 2.0)https://www.flickr.com/photos/grantuking/9705677549/

Crash Test dummies for sale - by Jack French (CC BY-NC 2.0)https://www.flickr.com/photos/jackfrench/34523572/

Minion (no license available)http://thisandthatforkidsblog.com/2013/09/17/easy-to-make-diy-minion-halloween-pumpkin/

Images

Mutant - by minions fans-d6txvph.jpg (no license available)http://despicableme.wikia.com/wiki/File:Evil_minions_by_minions_fans-d6txvph.jpg

Mutant killed - by Patricia Uribe (no license available)http://weheartit.com/entry/70334753

Mutant lived - by Carlos Hernández (no license available)https://gifcept.com/VUQrTOr.gif

PIT logo - by Ling Yeung (CC BY-NC-SA 3.0)http://pitest.org/about/

Circled Minions - by HARDEEP BHOGAL - (no license available)https://plus.google.com/109649247879792393053/posts/ghH6tiuUZni?pid=6004685782480998482&oid=109649247879792393053

Finger puppets – by Gary Leeming (CC BY-NC-SA 2.0)https://www.flickr.com/photos/grazulis/4754781005/

Questions - by Shinichi Izumi (CC BY-SA)http://despicableme.wikia.com/wiki/File:Purple_Minion.png

CREDITSImages