MDroid+: A Mutation Testing Framework for AndroidKevin Moran

College of William and MaryUnited States

Michele TufanoCollege of William and Mary

United States

Carlos Bernal-CárdenasCollege of William and Mary

United States

Mario Linares-VásquezUniversidad de los Andes

Colombia

Gabriele BavotaUniversità della Svizzera italiana

Switzerland

Christopher VendomeCollege of William and Mary

United States

Massimiliano Di PentaUniversity of Sannio

Italy

Denys PoshyvanykCollege of William and Mary

United States

ABSTRACTMutation testing has shown great promise in assessing the e�ec-tiveness of test suites while exhibiting additional applications totest-case generation, selection, and prioritization. Traditional mu-tation testing typically utilizes a set of simple language speci�csource code transformations, called operators, to introduce faults.However, empirical studies have shown that for mutation testingto be most e�ective, these simple operators must be augmentedwith operators speci�c to the domain of the software under test.One challenging software domain for the application of mutationtesting is that of mobile apps. While mobile devices and accom-panying apps have become a mainstay of modern computing, theframeworks and patterns utilized in their development make testingand veri�cation particularly di�cult. As a step toward helping tomeasure and ensure the e�ectiveness of mobile testing practices,we introduce MD����+, an automated framework for mutationtesting of Android apps. MD����+ includes 38 mutation operatorsfrom ten empirically derived types of Android faults and has beenapplied to generate over 8,000 mutants for more than 50 apps.Video URL: https://youtu.be/yzE5_-zN5GA

CCS CONCEPTS• Software and its engineering → Software veri�cation andvalidation;

KEYWORDSMutation testing, Operators, Android

ACM Reference Format:Kevin Moran, Michele Tufano, Carlos Bernal-Cárdenas, Mario Linares-Vásquez, Gabriele Bavota, Christopher Vendome, Massimiliano Di Penta,and Denys Poshyvanyk. 2018. MDroid+: A Mutation Testing Framework forAndroid. In ICSE ’18 Companion: 40th International Conference on SoftwareEngineering , May 27-June 3, 2018, Gothenburg, Sweden. ACM, New York,NY, USA, 4 pages. https://doi.org/10.1145/3183440.3183492

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor pro�t or commercial advantage and that copies bear this notice and the full citationon the �rst page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden© 2018 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-5663-3/18/05.https://doi.org/10.1145/3183440.3183492

1 INTRODUCTIONMobile devices have become a mainstay of the modern computinglandscape. The increasing popularity of these devices is drivenprimarily by a rich ecosystem of applications, commonly referred toas “apps". Strong user demand for mobile apps has driven increasedcompetition on storefronts such as Google Play [10] and Apple’sApp Store [4]. These competitive marketplaces necessitate thatmobile developers devote time to extensively test and validate theirapps to ensure a positive user experience, as new app releases havebeen shown to a�ect users’ perception via marketplace ratings [21].Android is currently the most popular OS in the world [1], andhence garners a large amount of interest from developers.

However, e�ectively testing Android apps is di�cult, as devel-opers face constraints speci�c to the domain of mobile softwareincluding change-prone APIs [5, 16], platform fragmentation [12],and a lack of automated testing tools that meet developer needs [19].Furthermore, when devising testing strategies or writing test cases,their e�ectiveness needs to be evaluated. To this aim, mutationanalysis has been de�ned as a process for evaluating the e�cacy ofa test suite and functions by injecting small changes, meant to rep-resent common bugs, into a “correct” program and then evaluatingthe number of injected faults uncovered by the test suite [8, 11]. Thehigher the ratio of uncovered bugs, ormutants, the more e�ective atest suite is said to be. However, mutation analysis typically relieson a concept called the coupling e�ect that states that “complexfaults are coupled to simple faults in such a way that a test data setthat detects all simple faults in a program will detect most complexfaults” [23]. The validity of this e�ect has come under scrutiny fromthe software testing research community [2, 3, 7, 15, 20], and whilestudies have generally indicated a correlation between mutants andreal faults, they also point out that non-negligible portion of realfaults do not e�ectively map to mutants [15]. The number of simplemutants mapping to real faults in the mobile domain is likely tobe lower, due to heavy usage of APIs and frameworks that enablethe varied feature sets of mobile devices. Thus, in order to developan e�ective mutation testing tool for Android, one must take intoaccount the software domain, and model operators according tofaults that naturally occur in mobile software development.

In this paper, we describeMD����+, a mutation testing frame-work for Android apps that aims to support developers in writingmobile tests.MD����+ includes 38 Android and Java speci�c mu-tation operators that were designed according to an empirically

ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden K. Moran, M. Tufano, C. Bernal Cardenas, M. Linares Vasquez et al.

derived taxonomy of common, naturally occurring faults in An-droid applications [17]. The main contributions ofMD����+ canbe summarized as follows:• A set of empirically derived mutation operators designed to sim-ulate commonly observed faults in Android apps;

• An e�cient, automated methodology for deriving a PotentialFault Pro�le (PFP) from subject apps, dictating possible locationsfor mutant injection;

• A process for applying operator transformations to locationsdictated by the PFP to create mutant apps;

• Built-in extensibility that facilitates new operator de�nitions.

2 APPROACHIn order to ensure that MD����+ is an e�ective, practical, and�exible/extensible tool for mutation testing, it takes into accountthe following design considerations: (i) an empirically derived setof mutation operators; (ii) a design embracing the open/closedprinciple (i.e., open to extension, closed to modi�cation); (iii) visitorand factory design patterns for deriving the Potential Failure Pro�le(PFP) and applying operators, (iv) parallel computation for e�cientmutant seeding. MD����+ is written in Java and available as anopen source project [18]. In the following sections, we describeMD����+ according to its work�ow described in Figure 1.

2.1 Implemented OperatorsAs mentioned in Section 1, in order to create a mutation testingframework for Android, mutation operators must be de�ned ac-cording to naturally occurring faults to ensure a strong couplingbetween operators and faults likely to befall an Android project. Tothis end, the authors undertook an extensive empirical study fol-lowing a procedure inspired by open coding [22]. More speci�cally,the authors analyzed 2,007 documents, assigning a label describingan observed bug, from the following sources: (i) bug reports ofopen source Android apps, (ii) bug-�xing commits of open sourceAndroid apps, (iii) Stack-Over�ow discussions, (iv) exception hi-erarchies of Android APIs, (v) bugs described in previous studies,and (vi) reviews from Google Play. The open-coding procedurewas supported by a custom designed web-application, where eachdocument was tagged by at least two authors, and labeling con�ictswere resolved. For the complete methodology, we refer readers toour previous technical paper [17].

The study described above resulted in a taxonomy of faults forAndroid apps that spans 14 di�erent categories. We implemented38 operators corresponding to ten di�erent categories, excludingcertain categories of operators such as hardware con�gurations, asthey do not map well to the process of mutation testing. A detailedlist and description of all implemented operators is available on theMD����+ website [18].

2.2 Derivation of the Potential Fault Pro�leIn the context ofMD����+, we de�ne a PFP that stipulates locationsin analyzed apps—which can correspond to source code statements,XML tags, or locations in other resource �les—that coincide with po-tential fault locations, given the empirically derived taxonomy [17].Consequently, the PFP stipulates the locations where our de�nedmutation operators can be applied. To extract the PFP,MD����+statically analyzes a target mobile app, looking for locations where

App Src

1 PFP Derivation

AST Analysis Text Analysis

Activity Not DefinedInvalid LabelNull GPS Location

Invalid URIInvalid Date

…

Android Operators

Potential Fault Profile

Activity Not Defined

Invalid Label

Null GPS Location

Invalid URI

InvalidDate

2 Mutant GenerationParallel Project Cloning& Operator Application

Mutant Projects + Log

Mutant Log

Figure 1: Overview tool Work�ow forMD����+operators can be implemented. The locations are detected auto-matically by parsing XML �les in the case of resources or throughAbstract Syntax Tree (AST) based analysis for Java code.

For implemented operators targeting one of the Android resourceXML �les (i.e., archives in the /res folder), the structure of eachXML �le is analyzed and a pattern matching process for di�erentattributes within the XML is used. However, for operators thatare applied to the Java source code, a two-phase AST-based andtext-based analysis is utilized that is capable of identifying thelocation of target API calls. The identi�cation of API call sitesis implemented utilizing the visitor design pattern, allowing forextensible, decoupled operations to be performed on the AST ofa target app. This helps to ensure that MD����+ is capable ofsupporting additional operators in the future that may require moreadvanced AST analysis. After the AST-based location of speci�c APIcalls, a �ne-grained, text-based pattern matching is performed onidenti�ed API calls to derive the precise textual location where themutation operator transformation will be applied. The end result ofthe PFP derivation process is a list stipulating all potential injectionpoints in code of the Android-speci�c mutation operators.

2.3 Mutant CreationGiven an automatically derived PFP for an app and the catalog ofAndroid-speci�c operators, MD����+ generates a mutant for eachlocation in the PFP. This process is performed using text or ASTmanipulation rules speci�c to each implemented operator. Thus, foreach location related to an operator, the text/AST transformationis applied to the speci�ed location in either the code or .xml �le.

Due to the event driven nature of Android applications, testingis generally performed at the GUI-level and is centered around appuse-cases [19]. Therefore, in order to operationalizeMD����+ to�t in a typical testing work�ow, the mutated applications must becompilable to an .apk �le that can be run on an emulator or realdevice. Thus, during the mutant creation process,MD����+ createsa project-level clone of a target and applies a single mutation toa speci�ed location in the cloned project, resulting in one mutantproject for each seeded instance of a mutation operator. It is impor-tant to note that the cost of applying the transformations to thecloned projects is trivial in practice, and a majority of the executiontime cost is related to the cloning of the app project, and is thusI/O bound. To ensure the cloning and mutant generation processproceeds as e�ciently as possible,MD����+ implements an option

MDroid+: A Mutation Testing Framework for Android ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden

to parallelize the process, utilizing the multi-core architecture ofmost modern hardware. The end result of the mutation creationprocess is the set of cloned mutant projects and a log that delin-eates the operator and location of injection for each created mutant.Currently, we do not o�er a universal interface for the compilationof mutants, as the parameters and build systems used by Androidapps can vary dramatically. However, the compilation process canbe easily scripted using the CLI support o�ered by ant, gradle,and the Android SDK1.

2.4 Tool Usage and ExtensibilityMD����+ is implemented as a Java command-line utility in whichthe user can select the speci�c set of mutation operators to beapplied during mutant generation, as well as an option for enablingmulti-threading. A R����� and user guide can be found on theproject repository, which is accessible from the tool website [18].

Given that the Android platform is prone to rapid evolution [5,16], it is important that MD����+ allows for easy modi�cation/ex-tension of the operators list, in order to keep pace with rapid evo-lution. To add a new operator to MD����+, there are two majorcomponents that must be implemented: (i) an operator locator/de-tector, and (ii) the operator transformation rules. If a proposedoperator applies to the source code of an application, then eitherthe target API-call of the operator must be added to an existing ASTvisitor or a new visitor to identify the AST pattern required by theoperator must be de�ned. Next, an operator speci�c pattern locatormust be implemented to derive the precise operator location afterAPI-calls in question have been detected via AST-analysis. To allowfor streamlined extensibility,MD����+ o�ers a generic Locatorinterface that includes a findExactLocations() method thatcan be implemented to support additional operators. An exampleof this interface implemented for the BuggyGUIListener opera-tor is given in Listing 1. This locator simply returns the start andend of the API-call location passed in the MutationLocation ob-ject so that it can be manipulated later. Note, that this is a simpleexample, and most of MD����+’s pre-de�ned operators need toparse particular properties or patterns from identi�ed API calls.If the proposed operator applies to the resource .xml, MD����+implements a TextBasedDetector abstract class that can beextended to account for matching the patterns of xml attributes.

Listing 1: Example of Operator Locator1 public class BuggyGUIListenerLocator implements Locator {23 private void findExactLocation(MutationLocation loc) {4 //Fix start column5 loc.setStartColumn(loc.getStartColumn()+1);6 //Build exact mutation location7 loc.setEndColumn(loc.getStartColumn()+loc.

getLength());8 }9 }

Once the precise locator/detector for a proposed mutation oper-ator has been implemented, the actual transformation rule must bedelineated so that it can be applied to any detected injection pointsduring theMD����+ analysis. To facilitate this process,MD����+includes a MutationOperator interface and implements the fac-tory design pattern for managing and instantiating operators. To

1https://developer.android.com/studio/build/building-cmdline.html

add an additional transformation rule for a desired operator, theperformMutation()method of the interfacemust be implementedaccording to the MutationLocation. An example of theNullInputStream operator that sets an inputStream to null before itis closed is shown in Listing 2.

Listing 2: Example of Operator De�nition1 public class NullStream implements MutationOperator{2 @Override3 public boolean performMutation(MutationLocation

location) {4 ObjectMutationLocation mLocation = (

ObjectMutationLocation) location;5 List<String> newLines = new ArrayList<String>();6 List<String> lines = FileHelper.readLines(

location.getFilePath());7 for(int i=0; i < lines.size(); i++){8 String currLine = lines.get(i);9 //Null object10 if(i == location.getLine()){11 String newLine = mLocation.getObject() +

" = null;";12 newLines.add(newLine);13 }14 newLines.add(currLine);15 }16 FileHelper.writeLines(location.getFilePath(),

newLines);17 return true;18 }19 }

3 EVALUATION3.1 Study ContextTo evaluate MD����+, we conducted a study with the followinggoals: (i) understand and compare the applicability of MD����+and other currently available mutation testing tools to Androidapps; (ii) to understand the underlying reasons for non-compilableor non-executable (i.e., trivial) mutants. To accomplish this, wecompareMD����+ with two popular open source mutation testingtools (Major [14] and PIT [6]), which we adapted to be applicableto Android apps, and with one context-speci�c mutation testingtool for Android called muDroid [9]. We chose these tools becauseof their diversity (in terms of functionality and mutation opera-tors), their compatibility with Java, and their representativenessof tools working at di�erent representation levels: source code,Java bytecode, and smali bytecode (i.e., Android-speci�c bytecoderepresentation). The modi�cations to the PIT and Major tools canbe accessed using our replication package [18]. To compare theapplicability of each mutation tool, we used the Androtest suite ofapps [24], which includes 68 Android apps from 18 Google Playcategories. The mutation testing tools exhibited issues in 13 of theconsidered 68 apps, i.e., the mutant-injection process rendered themnon-compilable. Thus, in the end, we considered 55 subject apps inour study. For more information see our replication package [18].

To quantitatively assess the applicability and e�ectiveness of theconsidered mutation tools to Android apps, we de�ned three met-rics: Total Number of Generated Mutants (TNGM), Non-CompilableMutants (NCM), and Trivial Mutants (TM). In this paper, we con-sider non-compilable mutants as those that are syntactically incor-rect and cause compilation/assembly errors, and trivial mutants asthose that are killed arbitrarily by most test cases (e.g., crashing onlaunch). The trivial mutant study was supported by a large-scaledynamic analysis framework [17].

ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden K. Moran, M. Tufano, C. Bernal Cardenas, M. Linares Vasquez et al.(a) %Non-Compilable Mutants

●●

●

●

●●●

Major MuDroid MDroid+

020

4060

80100

(b) %Trivial Mutants

●

●

●

●●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●●

Major Pit MuDroid MDroid+0

2040

6080

100

(c) # of Mutant Generated per App

●●● ●

●● ●● ●● ●●

● ● ●●

●● ●●● ●●

MajorPit

MuDroidMDroid+

0 5000 10000 15000

Figure 2: Non-compilable & trivial mutants per app.3.2 Study ResultsFigure 2 reports the results of (i) the percentage of non-compilablemutants (NCM), (ii) the percentage of trivial mutants (TM), and(iii) the total number of generated mutants per app. On average,167, 904, 2.6k+, and 1.5k+ mutants were generated by MD����+,Major, PIT, and muDroid, respectively for each app. MD����+ hadan average runtime of 19 seconds per app on server-class hardware.While the number of mutants generated is high for other tools, thisis mostly due to the fact they implement far more general opera-tors with a higher number of potential injection points. However,MD����+’s operators are more speci�c, and coupled to features thatmay or may not be implemented by a given app, resulting in fewer,but stronger, mutants. The average percentage of non-compilablemutants (NCM) generated by MD����+, Major and muDroid overall the apps is 0.56%, 1.8%, and 53.9%, respectively, while no NCMare generated by PIT due to its mutant assembly process (Figure 2a).Thus,MD����+ achieves the lowest ratio of non-compilable mu-tants by a statistically signi�cant margin (Wilcoxon paired signedrank test p-value< 0.001 for Major and muDroid – adjusted withHolm’s correction [13], Cli�’s d=0.59 - large for Major, and Cli�’sd=0.35 - medium for muDroid), allowing for a mutation testing pro-cedure more capable of evaluating the e�cacy of a set of tests . Theoverall rate of NCM is very low for MD����+, and most instancespertain to edge cases requiring more robust static analyses.

All four tools generated trivial mutants, and the mean percent-age of their distribution over all apps for MD����+, Major, PIT,and muDroid is 2.42%, 5.4%, 7.2%, and 11.8%, respectively (Fig-ure 2b). MD����+ generates signi�cantly fewer TM than muDroid(Wilcoxon paired signed rank test adjusted p-value=0.04, Cli�’sd=0.61 - large) and PIT (adjusted p-value=0.004, Cli�’s d=0.49 -large), while there is no statistically signi�cant di�erence with Ma-jor (adjusted p-value=0.11). While these percentages may appearsmall, the raw values show that TMs can comprise a large set ofinstances for tools that can generate thousands of mutants per app.For example, for the Translate app, 518 out of the 1,877 mutantsgenerated by PIT were TM. For the same app, muDroid creates 348TM out of the 1,038 it generates. The biggest underlying reasonfor TM inMD����+ results from null objects or references in theMainActivity, causing a crash on startup. Future improvementsto the tool could avoid mutants seeded in components related tothe MainActivity.

4 DEMO REMARKS & FUTUREWORKIn this demo, we have presented MD����+, a mutation testing toolfor Android applications that supports 38 empirically derived muta-tion operators, automates the process of detecting potential mutantlocations and generating mutants, and facilitates the addition ofnew operators and the maintenance of existing operators throughan extensible architecture.MD����+ was evaluated against otherpopular mutation testing tools for the Java language and was shownto generate fewer non-compilable and trivial mutants. In the future,we plan to add to the functionality of MD����+ by allowing usersto stipulate speci�c activities for mutant injection and supportingthe increasingly popular Kotlin language.

Acknowledgments. Bavota was supported in part by the SNFproject JITRA, No. 172479. We would like to thank Sebastián Sierraand Camilo Escobar from Universidad de los Andes for their helptestingMD����+.

REFERENCES[1] Google’s Android just beat Microsoft Windows as most popular OS http://fortune.

com/2017/04/03/google-android-microsoft-os/, 2017.[2] J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for

testing experiments? In ICSE’05, pages 402–411, 2005.[3] J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin. Using mutation analysis

for assessing and comparing testing coverage criteria. IEEE Trans. Software Eng.,32(8):608–624.

[4] Apple. App store. https://itunes.apple.com/us/genre/ios/id36?mt=8, 2017.[5] G. Bavota, M. Linares-Vásquez, C. Bernal-Cárdenas, M. Di Penta, R. Oliveto, and

D. Poshyvanyk. The impact of API change- and fault-proneness on the userratings of Android apps. IEEE Trans. Software Eng., 41(5):384–407, 2015.

[6] H. Coles. Pit. http://pitest.org/, 2017.[7] M. Daran and P. Thévenod-Fosse. Software error analysis: A real case study

involving real faults and mutations. In ISSTA’96, pages 158–171, 1996.[8] R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help

for the practicing programmer. Computer, 11(4):34–41, April 1978.[9] L. Deng, N. Mirzaei, P. Ammann, and J. O�utt. Towards mutation analysis of

Android apps. In ICSTW’15, pages 1–10, April 2015.[10] Google. Google play. https://play.google.com/store, 2017.[11] R. G. Hamlet. Testing programs with the aid of a compiler. IEEE Trans. Software

Eng., 3(4):279–290, July 1977.[12] D. Han, C. Zhang, X. Fan, A. Hindle, K. Wong, and E. Stroulia. Understanding

Android fragmentation with topic analysis of vendor-speci�c bugs. In WCRE’12,pages 83–92, 2012.

[13] S. Holm. A simple sequentially rejective Bonferroni test procedure. ScandinavianJournal on Statistics, 6:65–70, 1979.

[14] R. Just. The Major mutation framework: E�cient and scalable mutation analysisfor java. In International Symposium on Software Testing and Analysis, ISSTA’14.

[15] R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Aremutants a valid substitute for real faults in software testing? In ESEC/FSE’14.

[16] M. Linares-Vásquez, G. Bavota, C. Bernal-Cárdenas, M. Di Penta, R. Oliveto, andD. Poshyvanyk. API change and fault proneness: A threat to the success ofAndroid apps. In ESEC/FSE’13, pages 477–487, 2013.

[17] M. Linares-Vásquez, G. Bavota, M. Tufano, K. Moran, M. Di Penta, C. Vendome,C. Bernal-Cárdenas, and D. Poshyvanyk. Enabling mutation testing for androidapps. In ESEC/FSE’17, pages 233–244, 2017.

[18] M. Linares-Vásquez, G. Bavota, M. Tufano, K. Moran, M. Di Penta, C. Vendome,C. Bernal-Cárdenas, and D. Poshyvanyk. Online appendix for “enabling mutationtesting for android apps". http://android-mutation.com, 2017.

[19] M. Linares-Vásquez, C. Bernal-Cárdenas, K. Moran, and D. Poshyvanyk. How dodevelopers test android applications? In ICSME’17, pages 613–622, Sept 2017.

[20] Q. Luo, K. Moran, and D. Poshyvanyk. A large-scale empirical comparison ofstatic and dynamic test case prioritization techniques. In Proceedings of the2016 24th ACM SIGSOFT International Symposium on Foundations of SoftwareEngineering, FSE 2016, pages 559–570, New York, NY, USA, 2016. ACM.

[21] W. Martin, F. Sarro, and M. Harman. Causal impact analysis for app releases ingoogle play. In ESEC/FSE’16, pages 435–446, 2016.

[22] M. B. Miles, A. M. Huberman, and J. Saldaña. Qualitative Data Analysis: A MethodsSourcebook. SAGE Publications, Inc, 3rd edition, Apr 2013.

[23] A. J. O�utt and R. H. Untch. Mutation 2000: Uniting the Orthogonal, pages 34–44.Springer US, Boston, MA, 2001.

[24] S. Roy Choudhary, A. Gorla, and A. Orso. Automated test input generation forandroid: Are we there yet? (E). In ASE’15, pages 429–440, 2015.