Multi-objective Test Report Prioritization usingImage Understanding

Yang Feng†∗, James A. Jones†, Zhenyu Chen∗, Chunrong Fang∗†Department of Informatics, University of California, Irvine, USA

∗State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China{yang.feng, jajones}@uci.edu, zychen@nju.edu.cn

ABSTRACTIn crowdsourced software testing, inspecting the large num-ber of test reports is an overwhelming but inevitable soft-ware maintenance task. In recent years, to alleviate thistask, many text-based test-report classification and prioriti-zation techniques have been proposed. However in the mo-bile testing domain, test reports often consist of more screen-shots and shorter descriptive text, and thus text-based tech-niques may be ineffective or inapplicable. The shortage andambiguity of natural-language text information and the welldefined screenshots of activity views within mobile applica-tions motivate our novel technique based on using image un-derstanding for multi-objective test-report prioritization. Inthis paper, by taking the similarity of screenshots into con-sideration, we present a multi-objective optimization-basedprioritization technique to assist inspections of crowdsourcedtest reports. In our technique, we employ the Spatial Pyra-mid Matching (SPM) technique to measure the similarityof the screenshots, and apply the natural-language process-ing technique to measure the distance between the text oftest reports. Furthermore, to validate our technique, an ex-periment with more than 600 test reports and 2500 imagesis conducted. The experimental results show that image-understanding techniques can provide benefit to test-reportprioritization for most applications.

CCS Concepts•Software and its engineering → Maintaining soft-ware;

KeywordsCrowdsourced Testing; Test Report Prioritization; ImageUnderstanding; Multi-Objective Optimization

1. INTRODUCTIONCrowdsourced techniques have recently gained wide popu-

larity in the software-engineering research domain [20]. One

of the key advantages of crowdsourced techniques is thatthey can provide engineers with information and operationsof real users, and those users provide data from tasks per-formed on real, diverse software and hardware platforms.For example, crowdsourced testing (e.g., beta testing) pro-vides validation data for a large population of varying users,hardware, and operating systems and versions. Such ben-efits are particularly ideal for mobile application testing,which often needs rapid development-and-deployment iter-ations and support many mobile platforms. In addition,crowdsourced mobile testing can provide developers withreal users’ feedback, new feature requests, and user-experienceinformation, which can be difficult to obtain through conven-tional software testing practices. For these reasons, severalsuccessful crowdsourcing mobile testing platforms (such asuTest,1 Testin,1 and AppStori1) have emerged in the pastfive years [20].

Typically, crowdsourced workers provide information fordevelopers in the form of test reports, which may consist ofscreenshots and textual content. Due to the inherent na-ture of crowdsourced testing, which usually involves a largenumber of users, the number of test reports can be greatand the resulting task of inspecting those test reports canbe quite time-consuming and expensive. As such, it is nat-ural for developers to seek methods to assist in identifyingand prioritizing new and useful information.

In the past decades, to alleviate tedious test-report inspec-tion, researchers have proposed many full- or semi-automaticmethods [6, 9, 11,26,30,34,35,37,39], in which, they mainlyfocused on the problems of duplicate-report identification,report classification, and report prioritization. To reduce thecosts of inspecting duplicate test reports, techniques havebeen proposed and widely used, such as Bugzilla1 and Man-tis.1 Similarly, report-classification techniques have beenproposed to group similar reports, so that ideally an ex-pert would only need to sample some reports from eachgroup to gain a sufficient understanding of the bugs thatthey represent [6, 26, 35]. Finally, prioritization techniqueshave gained wide attention in the software-testing domain.Feng et al. [9] first proposed the concept of prioritizationof crowdsourced testing reports. Instead of attempting toreduce the time-cost of inspecting, the basic assumption ofprioritization techniques is “the earlier a bug is detected, thecheaper it is to remedy,”which implies that all of the reportswill be eventually inspected.

In almost all such techniques, the test reports are cap-tured and analyzed based on their textual similarity (e.g.,

1utest.com, itestin.com, appstori.com, bugzilla.org, mantisbt.org

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from Permissions@acm.org.

ASE’16, September 3–7, 2016, Singapore, Singaporec© 2016 ACM. 978-1-4503-3845-5/16/09...$15.00

http://dx.doi.org/10.1145/2970276.2970367

202

[9,11,30,34,35,37]) or based on their execution traces (e.g.,[6, 26, 39]). For software designed for a desktop computer,such techniques are likely sufficient. However, for mobilesoftware, writing long and descriptive test reports may bemore challenging on a mobile-device keyboard. In fact, testreports written from mobile devices tend to be shorter andless descriptive, but also to include more screenshots (pri-marily due to the ease of taking such screenshots on mobileplatforms). Due to this paucity of textual information fortest reports, and also due to the ambiguity of natural lan-guage and prevalence of badly written reports [41], utilizingthe screenshots to assist with such mobile crowdsourced test-ing techniques is appealing. Moreover, the activity views oftypical mobile applications often provide distinguishable as-pects of the software interface and feature set, and providemore motivation for utilizing such screenshots.

In this paper, we proposed an approach to test-report pri-oritization that utilizes a hybrid analysis technique, which isboth text-based and image-based. This approach is a fullyautomatic diversity-based prioritization technique to assistthe inspection of crowdsourced mobile application test re-ports. To facilitate this, we capture textual and image in-formation and measure the similarity among these artifacts.For the image analyses, we employed the Spatial PyramidMatching (SPM) [17] technique to measure the similarityof screenshots. For the textual analyses, we used natural-language textual analysis techniques to measure the simi-larity of textual descriptions within test reports. Finally,we combine these similarity results using a multi-objectiveoptimization algorithm to produce a hybrid distance matrixamong all test reports. Based on these results, we priori-tize the test reports for inspection using a diversity-basedapproach, with the goal of assisting developers of finding asmany unique bugs as possible, as quickly as possible.

To evaluate this proposed hybrid test-report prioritizationtechnique, we implemented the technique and conductedan experiment. The experiment was conducted with threecompanies and more than 300 students, who simulated thecrowdsourcing of testing of five widely-used mobile applica-tions. In all, we received and analyzed 686 crowdsourced testreports from the crowd workers. We assessed effectivenessof our technique using the Average Percentage of Faults De-tected (APFD) [28] metric and the fault detection rate. Toserve as our baseline effectiveness results, we calculated theresults of two strategies: an Ideal strategy, which is a best-case ordering to find all bugs in the shortest order possible,and a Random strategy, which is a random ordering.

The results of our empirical study shows that: (1) Screen-shots are critical in the test report of mobile application,which could significantly improve the effectiveness of the pri-oritization technique and the efficiency of test-report inspec-tion; (2) For certain classes of mobile applications, our multi-objective optimized prioritization technique can outperformthe single image-based optimized technique, the text-basedoptimized technique, as well as the random technique.

The main contributions of this paper are as follows:

• To the best of our knowledge, this is the first work totake the image information as well as the text infor-mation of test reports into consideration to assist theinspection procedure.

• A novel multi-objective optimization-based techniqueis proposed to combine the image similarity and text

similarity, which improves the effectiveness and effi-ciency of test-report maintenance.

• Five mobile applications with more than 2500 screen-shots are used to evaluate our test prioritization tech-niques. Based on the experimental results and ourexperiences, we provide some practical guidance forcrowdsourced mobile test-report prioritization.

2. BACKGROUNDTest and Bug Report Resolution. Software-maintenanceactivities are known to be generally expensive and challeng-ing. One of the most important maintenance tasks is bug-report resolution. However, current bug-tracking systemssuch as Bugzilla, Mantis, the Google Code Issue Tracker,the GitHub Issue Tracker, and commercial solutions such asJIRA rely mostly on unstructured natural-language bug de-scriptions. These descriptions can be augmented with filesuploaded by the reporters (e.g., screenshots).

Although test descriptions and execution traces are cur-rently used to characterize and analyze test reports, howto involve screenshots remains unsolved. Specifically formobile crowdsourced testing, the reporters often prefer toprovide only short text descriptions along with necessaryscreenshots. In this situation, how to combine short textprocessing with image processing is important for test-reportprioritization.

Artificial-intelligence and computer-vision researchers cre-ated a class of analyses classified as image understanding,which extracts features from images and uses them for analy-sis. Within the software-engineering research domain, image-understanding techniques have been used in cross-browser is-sues for web applications. Cai et al. propose the VIPS algo-rithm [2], which segments a web page’s screenshot into visualblocks to infer the hierarchy from the visual layout, ratherthan from the DOM. Choudhary et al. proposed a tool calledWEBDIFF to automatically identify cross-browser issues inweb applications. Given a page to be analyzed, the compar-ison is performed by combining a structural analysis of theinformation in the page’s DOM and a visual analysis of thepage’s appearance, obtained through screen captures.

However, to date, there has been no work that addressesthe use of screenshot images for use with test reports, par-ticularly for mobile test reports produced by crowd workersin crowdsourced testing. Unfortunately, the crowd workerstend to describe bugs with a direct screenshot and shortdescriptions rather than verbose and complex text descrip-tions. At the same time, the developers are also interestedin screenshots rather than inspecting the workers’ long natu-ral language descriptions. But, due the complexity of imageunderstanding, there is a paucity of study on automatedprocessing of screenshots in crowdsourcing testing.

In this paper, we overcome the difficulties in understand-ing the screenshots by applying advanced image matchingtechniques.

Image Understanding. Image matching is an importantproblem in the area of computer vision. Matching images ofreal world objects is particularly challenging as a matchingalgorithm must account for factors such as scaling, lighting,and rotation. Fortunately, the images that we compare inthis work are screen captures of application views renderedby different devices by different workers for different apps. In

203

Figure 1: Test-report processing framework

this context, the above issues are ameliorated, and the mainproblems are, for instance, the shifting of GUI elements orthe fact that some elements are not displayed at all.

A basic technique for comparing two images is to comparetheir histograms, where an image histogram represents thedistribution of the value of a particular feature in the image[1]. In particular, a color histogram of an image representsthe distribution of colors in that image (i.e., the number ofpixels in the image whose color belongs in each of a fixed listof color ranges, or “bins”). Obviously, if two images are thesame, their color distributions will also match. Although theconverse is not true, and two different images can have thesame histogram, this issue is again not particularly relevantin our problem domain.

3. TECHNIQUE FRAMEWORKThis section elaborates the details of our method. We

assume the test reports only consist of two parts: text de-scription and screenshots, which we will handle separatelyand finally generate the balanced distance. Figure 1 showsthe framework of calculating the distance between the testreports, which mainly contains three steps: (1) screenshot-set distance calculation, (2) test-description distance calcu-lation, and (3) distance balancing. After we compute thedistance matrix from the test-report set, we apply variousstrategies to prioritize test reports.

3.1 PreliminaryEven though, in practice, there could be other multi-

media information that exists in the mobile test reports,such as the short operation videos and voice messages, ourexperience indicates that text descriptions and screenshotsare the most widely used types of information. In this pa-per, we focus on the processing of mobile screenshots toassist the test-report prioritization procedure. We assumeeach of the test reports only consists of two parts: a textdescription and a set of screenshots, i.e., the test report setR(r) = {r(Si, Ti)|i = 0...n}, in which, S denotes the screen-shots (i.e., images) containing the views that may capturesymptoms of the bug being reported, and T denotes the textdescribing the buggy behavior.

3.2 Test-Description ProcessingThe processing of text consists of two steps: (1) keywords

set building and (2) distance calculation. Because natural-language-processing (NLP) techniques have been widely usedto assist various software engineering tasks (e.g., [9, 10, 30,38]), we focus our description below on the distinguishingfeatures and implementation choices of our approach.

Keywords Set Building. In order to extract the keywordsfrom the natural-language description, we first need to seg-ment the text. Fortunately, word segmentation is a basicNLP task, and as such many efficient tools for word seg-mentation for different natural languages have been imple-mented [13]. In our method, we adopted the Language Tech-nology Platform (LTP)2 [4], which is the most widely usedChinese NLP cloud platform, to process the Chinese textdescriptions. LTP segments the Text parts of test reportsand marks each word with its Part-of-Speech (POS) for itscontext. In this procedure, LTP used the Conditional Ran-dom Fields (CRF) [15] model to segment Chinese words andadopted the Support Vector Machine (SVM) approach fortagging the POS. After we compute the segmentation re-sults with the POS tags, we filter out relatively meaninglesswords that could negatively impact the distance calculation.According to prior works (e.g., [27,33]), verbs and nouns canreveal the main information of a document. So, to simplifythe technique, we extract only the nouns and verbs to buildthe keywords sets.

It is worth noting that our technique should not be lim-ited to only the Chinese language. By applying other NLPtools, such as the Stanford NLP toolkit,3 similar text modelscan be built for text descriptions written in other languages,such as English, French, or German. However, different nat-ural languages have different characteristics, and may needspecial accommodations. For example, languages with rel-atively more prevalent polysemy (i.e., many possible mean-ings for a word or phrase) and synonyms may require specialprocessing, such as synonym detection and replacement, toavoid negative impacts on analyses.

Distance Calculation. Our method focuses on processingmobile test reports. Compared with the test reports of desk-top or web applications, one characteristic of typical mobiletest reports, and based on our experience, is that their textdescriptions are shorter and contain more screenshots. Assuch, we treat all of the words in the text description equally,and we adopted the Jaccard Distance to measure the differ-ence between the text descriptions Ti in the test-report setR(r). The definition of Jaccard Distance used in our tech-nique is presented in the following equation, in which, Ki

denotes the keyword set of test report Ti, and DT (ri, rj)denotes the distance between the text portion of the testreports ri and report rj .

DT (ri, rj) = 1− |Ki ∩Kj ||Ki ∪Kj |

2http://www.ltp-cloud.com/3http://nlp.stanford.edu/software/

204

(a) Playing-1 (b) Playing-2 (c) Lyrics-1 (d) Lyrics-2

Figure 2: Four example screenshots from the test reportsof the CloudMusic application. (a) and (b) are screenshotsof the playing view, and (c) and (d) are screenshots of thelyrics view of two different songs.

3.3 Screenshot ProcessingCompared with NLP techniques, image understanding tech-

niques are relatively less studied and used in the software-engineering domain. One of our motivations of conduct-ing this research is to proposed a method to extract the in-formation from images to assist software-engineering tasks.The workflow of processing screenshots S is presented inthe top branch of Figure. 1. The process is composed ofthree key steps to build up the distance between screen-shot sets: (1) building feature histograms, (2) calculatingdistance between individual screenshots, and (3) computingthe distance between screenshot sets.

Feature Histogram Building. In order to compute thedifference between the screenshots, we convert the screen-shots into feature vectors. Bug screenshots provide not onlyviews of buggy symptoms, but also app-specific visual ap-pearances. We hope to automatically identify applicationbehaviors based on their visual appearance in the screen-shots. However, the screenshots often have variable reso-lution and complex backgrounds. Therefore, modeling thesimilarity between the screenshots merely based on RGBis not an approach that is well suited for our task. To ad-dress the challenges, we apply the Spatial Pyramid Matching(SPM) [17] to build a global representation of screenshots.Since the details of SPM are beyond this paper’s topic, weonly briefly introduce it here.

Given an image, SPM partitions it into sub-regions ina pyramid fashion. At each pyramid level, it computesan orderless histogram of low-level features in each sub-region. After decomposition, it concatenates statistics oflocal features over sub-regions from all levels. After build-ing the “Spatial Pyramid” representation, we apply kernel-based pyramid matching scheme to compute feature corre-spondence in two images.

Figure 2 presents four original and actual screenshots fromfour test reports of a popular Chinese music-playing app,CloudMusic. Figures 2a and 2b show the music-playingview of the application, and Figures 2c and 2d show thelyrics view. Note that in each screenshot, the details of theview differ: e.g., different music is playing, different back-ground images appear, different lyrics are shown, and eventhe screen size is different for the last image. The layout ofscreenshots and background colors differ and provide chal-lenges for correct matching: although Figures 2a and 2bhave the same view layout, Figures 2b and 2d share a sim-ilar background color. If we were to directly calculate dis-tance based on the RGB histograms, we would incorrectly

500 1000 1500 2000 2500 3000 3500 40000

0.005

0.01

0.015

0.02

0.025

Feature

Val

ue

(a) Playing-1

500 1000 1500 2000 2500 3000 3500 40000

0.005

0.01

0.015

0.02

0.025

Feature

Val

ue

(b) Playing-2

500 1000 1500 2000 2500 3000 3500 40000

0.005

0.01

0.015

0.02

0.025

Feature

Val

ue

(c) Lyric-1

500 1000 1500 2000 2500 3000 3500 40000

0.005

0.01

0.015

0.02

0.025

Feature

Val

ue

(d) Lyric-2

Figure 3: The corresponding feature histograms of thescreenshots in Figure 2.

Table 1: Distance between screenshots of Figure 2

Playing-1 Playing-2 Lyrics-1 Lyrics-2Playing-1 0 0.38957 0.40255 0.45109Playing-2 0.38957 0 0.51161 0.51873Lyrics-1 0.40255 0.51161 0 0.32029Lyrics-2 0.45109 0.51873 0.32029 0

get a closer distance between Figures 2b and 2d. Neverthe-less, the image-understanding technique should be able tocapture the similarities of the the similar views. Intuitively,Figures 2a and 2b should be identified as similar views, andFigures 2c and 2d should be identified as similar views.

Based on the four images, SPM first builds the histogramsof features for each of image. The resulting histograms forthese images are shown in Figure 3.

Screenshot Distance Calculation. Using the screenshotfeature histograms, a distance is computed for each pair ofimages. To compute such distances between feature his-tograms, we adopt the chi-square distance metric [29]. Thechi-square metric is generally used to compute the distancebetween two normalized histogram vectors, i.e., their ele-ments sum to 1. Also, both of the pairwise histograms beingcompared should contain the same number of bins (i.e., thevectors should have the same number of dimensions).

We use Hi(x1, x2, ..., xn) to denote the feature histogramof screenshot si, and Hi(xk) to denote the value of kth fea-ture of si. The formula used to calculate chi-square distanceDs(si, sj) between screenshot si and sj is defined as follows:

Ds(si, sj) = χ2(Hi, Hj)

=1

2

d∑

k=1

(Hi(xk)−Hj(xk))2

Hi(xk) +Hj(xk)

(1)

Based on Equation 1, we obtain the distance matrix shownin Table 1 from the feature histograms of Figure 3.

These results show that the calculated distance betweenthe same views (Playing-1 and Playing-2, and Lyrics-1 andLyrics-2) have relatively shorter (i.e., smaller) distances (0.389between playing screenshots and 0.320 between lyrics screen-shots) than the across-view distances.

Screenshot Sets Distance Calculation. The previousstep uses the chi-square distance metric to compute dis-tances between pairs of screenshots. However, in practice,

205

each test report may contain more than one screenshot. So,in this step, we compute the distance between screenshotsets. To account for the diversity of display resolutions ofmobile devices and user content (e.g., songs, backgrounds),we set a threshold γ to assess screenshots that match. Theγ threshold is first used to find representative members fromwithin the same screenshot set (i.e., from the same test re-port). Screenshot subsets whose histograms produce chi-square distances that are below the distance threshold (i.e.,assessed as representing the same situation) are first repre-sented as an aggregated, summary histogram which is com-puted as the mean of the feature histograms from the con-stituent members.

Once the representative set of screenshots are selectedfrom each test report, the chi-squared metric with the γ met-ric is again used to compute the across-test-report screenshotsimilarity between the representative screenshots. Again,for screenshots (i.e., their representative histograms) whosedistance is less than γ, they are assessed as representing thesame view, and as such, the similar and non-similar screen-shots from each test report can be used to calculate the inter-test-report screenshot set distance for a pair of reports. Forthis calculation, we use the Jaccard distance metric. For thetest reports ri and rj and their respective screenshot sets Si

and Sj , the distance metric is defined as:

DS(ri, rj) = 1− |Si ∩ Sj ||Si ∪ Sj |

Note that in the special case where both Si and Sj arethe empty set (i.e., no screenshots were included for eithertest report), we assess DS to be zero.

3.4 Balanced FormulaBased on above distance computations for both the tex-

tual descriptions and the screenshot sets, we combine thesedistances to produce a hybrid distance. We present Equa-tion 2 to combine these differing distance values. Equation 2is a step-wise formula, where the first condition holds forwhen the textual descriptions are assessed to be identicalby way of the text distance formula DT . In this case, weassess the balanced distance metric to be similarly identi-cal. In the next step, where DS = 0, where typically noscreenshots were included for either test report, the textualdifference is used and scaled to make them more similar, andthus less diverse. This diversity adjustment will make theseless descriptive test reports less likely to be highly prioritizedin the next prioritization step. In the final step, which holdsin all other cases, the harmonic mean is calculated betweenthe textual distance DT and screenshot set distance DS.The resulting balanced distance BD is used to represent thepairwise distance of the corresponding test reports.

BD(ri, rj) =

⎧⎪⎪⎨

⎪⎪⎩

0, if DT (ri, rj) = 0

α × DT (ri, rj), if DS(ri, rj) = 0

(1 + β2)DS(ri,rj)×DT (ri,rj)

β2DS(ri,rj)+DT (ri,rj), otherwise

(2)

3.5 Diversity-Based PrioritizationUsing the computed balanced distance measures for all

test reports, we can prioritize the test reports for inspectionby developers. The guiding principle of our prioritizationapproach is to promote diversity of test reports that get

inspected. In other words, when a developer inspects onetest report, the next test reports that she inspects should beas different as possible to allow her to witness as many di-verse behaviors (and bugs) as possible in the shortest order.This diversity-based prioritization strategy has been used byother software-engineering researchers for test prioritization(e.g., [5, 12, 32]). The goal is for software engineers to findas many bugs as possible in a limited time budget.

Given Q denotes the result queue, the distance between atest report r and Q, denoted by D(r,Q), is defined by theminimal distance between r and each ri in Q, i.e., D(r,Q) =Minri∈Q{D(r, ri)}. The algorithm of BDDiv is shown inAlgorithm 1. In the beginning, Q is empty, we first initializethe algorithm by randomly choosing one report from R andappend it to Q. The second step is to calculate the distancebetween each test report ri ∈ R and Q. As soon as we getthe distance values, we choose the largest one to append toQ. The whole procedure completes when |R| = 0.

Algorithm 1: BDDiv(BD, R)

1: Q = ∅

2: Randomly choose a test report rk from R, append rc to Q3: R := R − {rk}4: while |R| �= 0 do5: maxDis := −1, rc = NULL6: for all ri ∈ R do7: minDis := 28: for all rj ∈ Q do9: if BD(ri, rj) < minDis then10: minDis = BD(ri, rj)11: end if12: end for13: if minDis > maxDis then14: maxDis = minDis15: rc = ri16: end if17: end for18: Append rc to Q19: R := R − {rc}20: end while

21: return Q

4. EXPERIMENTIn our experiment, we propose the following three research

questions:

RQ1: Can test-report prioritization substantially im-prove test-report inspection to find more uniquebuggy reports earlier?

RQ2: To what extent can the image-based approachesimprove the effectiveness of the text-only-basedapproach?

RQ3: How much improvement is further possible,compared to a best-case ideal prioritization?

If the software engineers have no test report prioritiza-tion technique, they may randomly inspect test reports, ina non-systematic order. RQ1 is designed to inform whetherprioritization of test reports is, in fact, advantageous. Toaddress the RQ1, we conduct the experiment to evaluatethe effectiveness of our prioritization techniques alongsidea Random-based strategy. RQ2 is designed to investigatewhether image-understanding techniques can assist the in-spection procedure compared with the text-only-based tech-nique. RQ3 is designed to investigate the gap between theperformance of our techniques and the theoretical Ideal pri-oritization technique, which could be helpful to engineers in

206

Table 2: Experimental Software Subjects

Name Version |R| |F | |S| |Rs| |Rf |SE-1800 2.5.1 192 7 856 164 99

CloudMusic 2.5.1 96 16 272 70 40Ubook 2.1.0 99 22 719 90 99

iShopping 1.3.0 209 73 581 160 130JustForFun 1.8.5 90 9 109 69 90

Totals 686 127 2537 553 458

selecting proper techniques in practice and inform the futureresearch in this field.

4.1 Software Subject ProgramsFrom November 2015 to January 2016, we collaborated

with three companies and more than 300 students to simu-late a crowdsourced testing process. The five applications onwhich we simulated crowdsourced testing are as follows:

JustForFun: A picture editing and sharing application,produced by Dynamic Digit.

iShopping: A shopping application for Taobao, producedby Alibaba.4

CloudMusic: An application for free-sharing music as wellas a music player, produced by NetEase.5

SE-1800: A monitoring application for a power supplycompany, produced by Panneng.

Ubook: An application for online education, produced byNew Oriental.6

Testing for all of these applications was crowdsourced toworkers on Kikbug.net. For these five apps, more than 300students were involved. To perform crowdsourced testing,each student installed a Kikbug-Android app, chose testingtasks, and completed testing tasks on their own phone. Dur-ing the testing process, workers performed testing tasks ac-cording to some guidelines, specified by the app developers.During task performance, the workers could take screenshotsif necessary, such as experiencing some unexpected behav-ior. After the testing task was completed, the worker couldprovide a brief description on bug phenomenon on his ownphone. Finally, the student submitted a test report, includ-ing the short descriptions and possible screenshots.

Then all the test reports are submitted to app develop-ers, and the developers can inspect the reports and beginthe debugging process. With the help of the developers’ in-spection, Kikbug obtained ground truth assessments for thestudents’ reports. The detailed information of the applica-tions is shown in Table 2, in which, the |R| denotes the num-ber of reports, |F | denotes the number of faults revealed inthe test reports, |S| denotes the number of screenshots con-tained in the test reports, |Rs| denotes the number of testreports containing at least one screenshot, and |Rf | denotesthe number of test reports that revealed faults.

4.2 Prioritization Strategies

Technique 1: Ideal. The best result in theory to inspecttest reports in such a way as to demonstrate the mostunique bugs as early as possible. Represented as Ideal.

Technique 2: TextDiv. The prioritization strategy basedonly on the distance between test reports’ text de-scriptions, i.e., in this strategy DT will replace BD

4https://guang.taobao.com5http://music.163.com6http://www.pgyer.com/y44v

as the first parameter of Algorithm 1. Represented asTextDiv.

Technique 3: ImageDiv. The prioritization strategy basedonly on the distance between test reports’ screenshots,i.e., in this strategy DS will replace BD as the firstparameter of Algorithm 1. Represented as ImageDiv.

Technique 4: Random. The random prioritization strat-egy, which is used to simulate the situation withoutany prioritization technique. Represented as Random.

Technique 5: Text&ImageDiv. Our prioritization strat-egy that balances the distance of screenshot sets andtext descriptions. Represented as Text&ImageDiv.

4.3 Evaluation MetricsWe employed the APFD (Average Percentage of Fault De-

tected) metric [28], which is the most widely-used evaluationmetric for test-case prioritization techniques, to measure theeffectiveness of our techniques. For each fault, APFD marksthe index of the first test report revealing it. We presentthe formula to compute the AFPD value in Equation 3, inwhich, n denotes the number of test reports, M denotes thetotal number of faults revealed by all test reports, Tfi is theindex of the first test report that reveals fault i.

APFD = 1− Tf1 + Tf2 + ...+ TfM

n×M+

1

2× n(3)

In our experiment, a higher APFD value implies a betterprioritization result. That is, it can reveal more faults earlierthan the other methods do.

Although the APFD values reflect the global performanceof prioritization techniques, in practice developers often can-not inspect all reports in a limited time budget. Thus, wealso provide a metric to reveal the percentage of bugs thatwould be found at certain milestones of inspection. For this,we use linear interpolation [18] to evaluate the partial per-formance of each prioritization technique. We define linearinterpolation as following:

• Qp = M×p, which is the number of faults correspond-ing to a percentage p. Let int(Q) and frac(Q) be theinteger part and fractional part of Q, respectively. Iffrac(Q) �= 0, the linear interpolation is needed.

• i, j are the indexes of reports that reveal at least Qand Q+1 faults respectively. The linear interpolationvalue Vp is calculated as Vp = i+ (j − i)× frac(Q)

In our experiment, we set the p ∈ {25%, 50%, 75%, 100%}.

4.4 Experimental SetupIn order to ensure the correctness of the implementation of

SPM, we directly used the MATLAB code provided by theinventors of SPM. There are some key parameters affectingthe performance of SPM, which are the size of the descriptordictionary DictSize, number of levels of the pyramid L, andnumber of images to be used to create the histogram binsHistBin. In our experiment, as the recommended valuesof the SPM inventor, we set DictSize = 200, L = 3, andHistBin = 100. For the NLP technique, because all of testreports in our experiment are in Chinese, we employed theLTP platform to assist the text description analysis.

Moreover, the size of screenshots (i.e., image resolution)submitted by the crowd workers was not fixed; in fact, theyvaried widely. In order to apply the SPM technique, weresize all screenshots to 480 × 480 pixels. Given the way

207

that the SPM technique focuses on detecting features withinimages, resizing the images should not produce a substantialimpact to the distance calculation.

In this experiment, we implemented all of the strategiespresented in Section 4.2. Particularly for theText&ImageDiv

strategy, we set the threshold of determining the identity ofscreenshots γ to 0.1, the factor α that is used to weaken theweight of test reports without any screenshots to 0.75, andthe parameter β used to balanced the text-based distanceand screenshot-set distance to 1, which means, we weigh thetwo kinds of distance equally.

5. RESULTS, ANALYSIS AND DISCUSSIONIn this section we present the results of our experiment,

then interpret those results to attempt answers to our re-search questions, and finally discuss the overall results. Inorder to reduce the bias that was introduced by the ran-dom initialization of the algorithm and the tie-breaking, weconducted the experiment 30 times and present the resultin Figure 4. Figures 4 (a, c, e, g, and i) show the boxplotsof the APFD results for the five projects, respectively, eachaggregated over the 30 experimental runs. Figures 4 (b, d, f,h, and j) show the average fault detection rate curves. Theexact mean value of APFD is shown in Table 3, which alsoincludes the result of one-way ANOVA tests of all strategies:the improved extent over Random, and the gap between ourstrategies and Ideal. Furthermore, we present the meanlinear interpolation value over the 30 experiment runs inTable 4 to demonstrate the performance of our techniquesin limited time budgets.

5.1 Answering Research Question RQ1RQ1: Can test-report prioritization substantially improvetest-report inspection to find more unique bugs earlier?

Based on the results shown in Figure 4 (a, c, e, g, i) andin the third column of Table 3, we find, to different extents,all of the three diversity-based prioritization strategies out-perform Random. Furthermore, in Table 3, all F-values arerelatively large and the p-values ≤ 0.001, which means theAPFD values of the four strategies are significantly different.Compared with the Random strategy, the percentage of im-provement of Text&ImageDiv ranges 9.93% – 24.95%.

Summary: All of the diversity-based prioritization meth-ods can improve the effectiveness of test report inspectionover Random, and thus test-report prioritization can sub-stantially, and significantly, find more unique buggy reportsearlier in the prioritized order.

5.2 Answering Research Question 2RQ2: To what extent can the image-based approaches im-prove the effectiveness of the text-only-based approach?

Figure 4 reveals that, except on the “JustForFun”project,the Text&ImageDiv outperforms the TextDiv, Image-

Div and Random strategies, which means, the image-un-derstanding technique improves the performance of the text-only-based technique. We did a deeper investigation on thisproblem and found what we speculate to be the reason forthe different result for the “JustForFun” project. JustFor-Fun is an image editing and sharing application, and assuch, the inherent functionality is to process various user-provided photos. The screenshots for this app largely con-sist of user content, with relatively few app-specific featuresin those screenshot images. Thus, the various screenshots

of “JustForFun” make the screenshot sets distance calculat-ing procedure generate large distances, even between thesame activity views, which leads to a negative impact onthe image-based strategies. In contrast, based on Table 4,Text&ImageDiv outperformed the single text-based pri-oritization techniques on inspecting different percentage oftest report of “SE-1800”, “CloudMusic” and “Ubook.”

Summary: Generally, compared with the text-only-basedprioritization strategy, the image-understanding technique isable to improve the performance of prioritizing test reports,both globally (i.e., APFD) and partially (i.e., linear interpo-lation at many level). However, we found that some classesof apps are naturally less suited for image-understandingtechniques — namely apps where the bulk of the views arecomposed of user contect.

5.3 Answering Research Question 3RQ3: How much improvement is further possible, comparedto a best-case ideal prioritization?

The fourth column of Table 3 shows the gap between ourstrategies and the theoretical Ideal. We found the gap be-tween Text&ImageDiv and Ideal vary from 15.21% to31.98%. For more details, we can observe the growth curvesin Figure 4. The curve of Ideal grows at a fast rate. Thebest situation reached top while the Text&ImageDiv onlystayed around 35%.

Summary: We find that our prioritization methods canprovide a reasonable small gaps for the theoretical Ideal re-sult, particularly for some subjects. However, there is roomfor future work to continue to improve the prioritization or-dering of test reports.

5.4 DiscussionMethod Selection. Reflecting on all of our experimentalresults, we find that image-understanding techniques canprovide benefits to test-report prioritization, and that thearea of such hybrid text-and-image approaches demonstratespromise. That said, we also observed that different tech-niques may be more or less applicable for different types ofapplications. Specifically, we observed that the image editorapp produced the worst results for the image-based and hy-brid techniques, compared to text-only. In such cases, wherethe screenshots mainly represent user content, image-basedtechniques may be less applicable. However, in applicationsin which little user or external content is displayed, image-based or hybrid techniques may be more applicable.

One noteworthy point is that both the TextDiv andText&ImageDiv are full-automated, which we believe aremore applicable in practice than the semi-automated Di-vRisk and Risk techniques [9] that require the users to in-put the inspection result to prioritize the crowdsourced testreports dynamically.

Mobile Application Testing. All of our experimentationwas conducted on mobile applications, and thus we cannotstate with certainty that such results would hold for othertypes of GUI software, such as desktop or web applications.However, we speculate that while there will likely be newand unique challenges in these domains, the basic conceptswould likely hold, at least for the class of applications withrelatively less user content. Desktop and web applicationshave the potential for even more differing screen and win-dow sizes, as well as multiple windows and pop-up dialogwindows, and each of these unique aspects would likely need

208

Table 3: One-way ANOVA Tests

Method APFD Improvement Gap

Means X−RandomRandom

Best−XX

SE-1800: F (3, 119) = 54.966, p-value ≤ 0.001Ideal 0.982 37.47% —

Text&ImageDiv 0.852 19.32% 15.21%TextDiv 0.817 14.46% 20.10%ImageDiv 0.836 17.04% 17.45%Random 0.714 — 37.47%

CloudMusic: F (3, 119) = 73.170, p-value ≤ 0.001Ideal 0.917 58.65% —

Text&ImageDiv 0.722 24.95% 26.97%TextDiv 0.664 14.98% 37.98%ImageDiv 0.641 10.99% 42.94%Random 0.578 — 58.65%Ubook: F (3, 119) = 84.167, p-value ≤ 0.001Ideal 0.889 40.92% —

Text&ImageDiv 0.750 18.95% 18.47%TextDiv 0.735 16.57% 20.88%ImageDiv 0.686 8.69% 29.65%Random 0.631 — 40.92%

iShopping: F (3, 119) = 73.178, p-value ≤ 0.001Ideal 0.825 45.08% —

Text&ImageDiv 0.625 9.93% 31.98%TextDiv 0.614 7.88% 34.48%ImageDiv 0.586 2.98% 40.89%Random 0.569 — 45.08%

JustForFun: F (3, 119) = 94.482, p-value ≤ 0.001Ideal 0.950 45.89% —

Text&ImageDiv 0.784 20.41% 21.16%TextDiv 0.842 29.28% 12.85%ImageDiv 0.681 4.54% 39.55%Random 0.651 — 45.89%

Table 4: Linear Interpolation (average number of inspectedtest reports)

Program Strategy 25% 50% 75% 100%Ideal 1.75 3.50 5.25 7.00

Text&Image 3.51 12.32 31.30 91.70SE- TextDiv 6.98 23.27 43.27 112.671800 ImageDiv 4.21 16.38 50.38 86.47

Random 4.79 31.05 79.36 145.57Ideal 4.00 8.00 12.00 16.00

Text&Image 13.10 24.30 39.33 62.00Cloud- TextDiv 16.10 34.57 44.57 59.00Music ImageDiv 11.07 26.53 49.77 85.83

Random 14.10 33.97 59.20 88.83Ideal 5.50 11.00 16.50 22.00

Text&Image 7.33 18.67 44.17 64.43Ubook TextDiv 10.52 23.67 35.17 78.03

ImageDiv 8.05 20.20 50.73 95.40Random 9.35 29.03 57.82 93.13Ideal 18.25 36.50 54.75 73.00

Text&Image 37.16 66.42 119.27 201.23iShop- TextDiv 52.89 82.82 111.30 160.07ping ImageDiv 32.60 75.88 134.59 206.30

Random 37.20 83.72 144.13 207.13Ideal 2.25 4.50 6.75 9.00

Text&Image 2.94 9.32 18.13 64.83Just- TextDiv 2.88 8.07 17.28 45.23

ForFun ImageDiv 3.16 18.12 39.01 79.47Random 2.88 22.25 49.88 80.17

to be addressed. Overall, we speculate that the success ofsuch image-understanding-assisted test-report prioritizationtechniques would likely depend on the visual complexity ofthe application views.

5.5 Threats to ValidityThere are some general threats to validity in our experi-

mental results. For example, we need more projects and dif-ferent parameter values combinations to reduce the threatto external validity and to better generalize our results.

Crowd Workers. Due to a monetary limitation, we “simu-

lated”the crowdsourced mobile testing procedure to validateour techniques, in which, we invited the students to work ascrowd workers. Such a choice means that our population ofworkers may be less diverse than the population of crowd-sourced workers from the general populace. Theoretically,“crowdsourcing” requires workers come from a large pool ofindividuals that one has no direct relationship with the oth-ers [20], which implies that our result may be different ifthe crowd workers were from the internet with open calls.However, according to the study of Salman et al. [31], if atechnique or task is new to both students and professionals,similar performance can be expected to be observed. Basedon this study, we believe this threat may not be the keyproblem for our validation procedure.

Subject Program Selection. The cost of conducting thiskind of experiment is quite expensive (involved more than300 people), the monetary budget is limited, so we con-ducted the experiment on only five applications. However,these five applications are widely used and publicly acces-sible. The functionalities of our subject applications varywidely, including music player, video player, picture editor,power monitor, and online shopping assistant. Thus, webelieve these applications can be used to validate the ourmethods, at least to give initial indications of effectivenessand applicability.

Natural Language Selection. Admittedly, in our experi-ment, all of the test reports were written in Chinese, whichcould threaten the generalizability to other natural languages.However, the NLP techniques and text-based prioritizationtechnique are not the focus of this work. Even though weused text-based techniques as one of our baselines, whatmatters to the performance of these technique is the dis-tance built from keywords set but not the languages. As forthe keyword-extraction technique, different languages havetheir own inherent characteristics, and thus NLP researchershave proposed keywords-extraction techniques for differentlanguages. In future research, we will validate our techniquewith test reports written in English. Moreover, the focus ofthis work is to study the potential for image-understandingtechniques to augment text-only-based techniques.

6. RELATED WORKBug Report Triage. As a large number of bug reports willbe submitted in the software testing phase, manually triag-ing each of these reports will become an effort-consumingtask. Bug report triage is a process that includes: priori-tizing bug reports, filtering out duplicate reports, and as-signing reports to the proper bug fixer. Among the variousbug report triaging techniques, we address two highly rele-vant research areas: bug report prioritization and duplicateidentification techniques.

Yu et al. [40] used neural networks to predict the priorityof bug reports. Their technique also employs the reused dataset from similar systems to accelerate the evolutionary train-ing phase. Kanwal et al. [13] used SVM and Naive Bayesclassifiers to assist bug priority recommendation. Tian etal. [36] predicted the priority of bug reports by presenting amachine learning framework that takes multiple factors in-cluding temporal, textual, author, related-report, severity,and product into consideration. By analyzing the textualdescription from bug reports and using text mining algo-rithms, Lamkanfi et al. [16] conducted case studies on threelarge-scale open source projects, and based on the result,

209

0.4

0.5

0.6

0.7

0.8

0.9

1

Text&Image Text Image Random

AP

FD

Ideal

(a) APFD on SE-1800

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Percentage of Inspected Reports

Per

cent

age

of F

ound

Fau

lts

IdealText&ImageTextImageRandom

(b) Average Fault Detection Rates on SE-1800

0.4

0.5

0.6

0.7

0.8

0.9

1

Text&Image Text Image Random

AP

FD

Ideal

(c) APFD on CloudMusic

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Percentage of Inspected Reports

Per

cent

age

of F

ound

Fau

lts

IdealText&ImageTextImageRandom

(d) Average Fault Detection Rates on CloudMusic

0.4

0.5

0.6

0.7

0.8

0.9

1

Text&Image Text Image Random

AP

FD

Ideal

(e) APFD on Ubook

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Percentage of Inspected Reports

Per

cent

age

of F

ound

Fau

lts

IdealText&ImageTextImageRandom

(f) Average Fault Detection Rates on Ubook

0.4

0.5

0.6

0.7

0.8

0.9

1

Text&Image Text Image Random

AP

FD

Ideal

(g) APFD on iShopping

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Percentage of Inspected Reports

Per

cent

age

of F

ound

Fau

lts

IdealText&ImageTextImageRandom

(h) Average Fault Detection Rates on iShopping

0.4

0.5

0.6

0.7

0.8

0.9

1

Text&Image Text Image Random

AP

FD

Ideal

(i) APFD on JustForFun

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Percentage of Inspected Reports

Per

cent

age

of F

ound

Fau

lts

IdealText&ImageTextImageRandom

(j) Average Fault Detection Rates on JustForFun

Figure 4: Experiment Results (averaged over 30 runs)

210

concluded that the technique is able to predict the severitywith a reasonable accuracy.

By applying natural language processing techniques, Rune-son et al. [30] took more textual features, including softwareversions, tester information, and submission date into con-sideration to detect duplicate bug reports. They validatedthis technique by conducting a large-scale experiment on in-dustrial projects. Jalbert et al. [11] used the surface features,textual semantics, and graph clustering to identify the du-plicate status. Besides duplicate detection, their techniqueis also able to rank the existing reports that are more similarwith the new one. By measuring the textual semantic simi-larity between the test reports, Nyugen et al. [24] applied thetopic model to detect duplicate bug reports. Podgurski et al.proposed the first approach to categorizing software failurereports by applying the undirected clustering on executiontraces. Feng and Xin et al. [8, 39] adopted the multi-labelclassification technique to assign the bug reports into morethan one classes based on the execution traces.

To assist test report prioritization in crowdsourced soft-ware testing, Feng et al. [9] proposed two strategies: Divand Risk. Both of the two strategies are text-based. Divis a fully automated technique, which aims at assisting thedevelopers inspect a wide variety of test reports and to avoidduplicates and wasted effort on falsely classified faulty be-havior. In this paper, we denote Div technique as TextDiv,and treat it as a baseline. Risk is designed to assist devel-opers to identify test reports that may be more likely to befault-revealing based on past observations. Because Risk re-quires the users to input the inspection result, i.e., it is asemi-automated technique (and not fully automatic), it is adistinct category of technique, and thus we did not employit as a baseline for evaluation.

Crowdsourced Software Testing. Mao et al. conducteda comprehensive survey on the crowdsourced software engi-neering [20], in which, they defined the crowdsourced soft-ware engineering as “the act of undertaking any externalsoftware engineering tasks by an undefined, potentially largegroup of online workers in an open call format.” In fact,crowdsourced techniques have been widely used in indus-trial software testing and gained popularity in usability test-ing, localization testing, GUI testing, user-experience test-ing, and stress&performance testing. However, it is a fairlynew research topic in the software-engineering research com-munity. Liu et al. [19] investigated both methodologicaldifferences and empirical contrasts of the crowdsourced us-ability testing and traditional face-to-face usability testing.To solve the oracle problem, Pastore et al. [25] applied thecrowdsourcing technique to generating test inputs depend-ing on a test oracle that requires human input in one formor another. Dolstra et al. [7] used virtual machines to runthe system under test and enable the crowd workers to ac-complish expensive and semi-automatic GUI testing tasks.By introducing crowdsourced testing, Nebeling et al. [23]conducted an experiment to evaluate the usability of websites and web-based services, the result of which showed thatcrowdsourcing testing is an effective method to validate theweb interfaces.

Application of Image Understanding on Testing. In[21], Michail et al. proposed a static approach, GUISearch,to guide search and browsing of its source code by using theGUI of an application. They further proposed a dynamic ap-

proach to obtain an explicit mapping from high-level actionsto low-level implementation by identifying execution trig-gered by user actions and visually describing actions froma fragment of the application displayed [3]. Kurlander etal. [14] introduced the notion of an editable graphical his-tory that can allow the user to review and modify the actionsperformed with a graphical interface. Similarly, Michail andXie [22] used before/after screenshots to visually describeapplication state at a very high level of abstraction to helpusers avoid bugs in GUI applications. However, images inthese work are provided to developers or users directly with-out machine understanding.

Image-understanding techniques have been used in cross-browser issues for web applications. Cai et al. propose theVIPS algorithm [2], which segments a web page’s screen-shot into visual blocks to infer the hierarchy from the vi-sual layout, rather than from the DOM. Choudhary et al.proposed a tool called WEBDIFF to automatically identifycross-browser issues in web applications. Given a page to beanalyzed, the comparison is performed by combining a struc-tural analysis of the information in the page’s DOM and avisual analysis of the page’s appearance, obtained throughscreen captures.

7. CONCLUSIONIn this work, we proposed a novel technique to priori-

tize test reports for inspection by software developers byusing image-understanding techniques to assist traditionaltext-based techniques, particularly in the domain of crowd-sourced testing of mobile applications. We proposed ap-proaches for prioritizing based on text descriptions, basedon screenshot images, and based on a hybrid of both sourcesof information. To our knowledge, this is the first work topropose using image-understanding techniques to assist intest-report prioritization. In order to evaluate the promise ofusing image understanding of screenshots to augment text-based prioritization, we implemented our hybrid approach,as well as a text-only- and image-only-based approaches, andtwo baselines: an ideal best-case and a random average-casebaseline. We found that prioritization, in almost all cases,is advantageous as compared to test-report inspection basedon an unordered process. We also found that for most soft-ware applications that we studied, there was a benefit tousing the screenshot images to assist prioritization. How-ever, we also found that there exist a class of applicationsfor which image-understanding may not be as applicable,and found room for improvement to narrow the gap to thehypothetical best-case ideal result.

As such, in future work, we will investigate ways to helpprioritize for those classes of applications, and also iden-tify application classes that are best suited for each type oftechnique. Finally, in future work we will extend the set ofsoftware systems that we use and the natural language usedto write the test reports.

8. ACKNOWLEDGEMENTSWe would like to express gratitude to Peiyun Hu for valu-

able suggestions for this paper. This work is supportedby the National Science Foundation under award CAREERCCF-1350837, and is partly supported by National Basic Re-search Program of China (973 Program 2014CB340702), Na-tional Natural Science Foundation of China (No. 61373013).

211

9. REFERENCES[1] G. Bradski and A. Kaehler. Learning OpenCV:

Computer vision with the OpenCV library. ” O’ReillyMedia, Inc.”, 2008.

[2] D. Cai, S. Yu, J.-R. Wen, and W.-Y. Ma. Vips: avisionbased page segmentation algorithm. Technicalreport, Microsoft technical report, MSR-TR-2003-79,2003.

[3] K. Chan, Z. C. L. Liang, and A. Michail. Designrecovery of interactive graphical applications. InProceedings of the 25th international conference onSoftware engineering, pages 114–124. IEEE ComputerSociety, 2003.

[4] W. Che, Z. Li, and T. Liu. Ltp: A chinese languagetechnology platform. In Proceedings of the 23rdInternational Conference on ComputationalLinguistics: Demonstrations, pages 13–16. Associationfor Computational Linguistics, 2010.

[5] T. Y. Chen, F.-C. Kuo, R. G. Merkel, and T. Tse.Adaptive random testing: The art of test casediversity. Journal of Systems and Software,83(1):60–66, 2010.

[6] Y. Dang, R. Wu, H. Zhang, D. Zhang, and P. Nobel.Rebucket: a method for clustering duplicate crashreports based on call stack similarity. In Proceedings ofthe 34th International Conference on SoftwareEngineering, pages 1084–1093. IEEE Press, 2012.

[7] E. Dolstra, R. Vliegendhart, and J. Pouwelse.Crowdsourcing gui tests. In Software Testing,Verification and Validation (ICST), 2013 IEEE SixthInternational Conference on, pages 332–341. IEEE,2013.

[8] Y. Feng and Z. Chen. Multi-label software behaviorlearning. In Proceedings of the 34th InternationalConference on Software Engineering, pages 1305–1308.IEEE Press, 2012.

[9] Y. Feng, Z. Chen, J. A. Jones, C. Fang, and B. Xu.Test report prioritization to assist crowdsourcedtesting. In Proceedings of the 10th Joint Meeting onFoundations of Software Engineering. New York:ACM, pages 225–236, 2015.

[10] M. Ilieva and O. Ormandjieva. Automatic transitionof natural language software requirementsspecification into formal presentation. In NaturalLanguage Processing and Information Systems, pages392–397. Springer, 2005.

[11] N. Jalbert and W. Weimer. Automated duplicatedetection for bug tracking systems. In DependableSystems and Networks With FTCS and DCC, 2008.DSN 2008. IEEE International Conference on, pages52–61. IEEE, 2008.

[12] B. Jiang, Z. Zhang, W. K. Chan, and T. Tse.Adaptive random test case prioritization. InAutomated Software Engineering, 2009. ASE’09. 24thIEEE/ACM International Conference on, pages233–244. IEEE, 2009.

[13] A. Kao and S. R. Poteet. Natural language processingand text mining. Springer Science & Business Media,2007.

[14] D. Kurlander and S. Feiner. Editable graphicalhistories. In IEEE Workshop on Visual Languages,pages 127–134. Citeseer, 1988.

[15] J. Lafferty, A. McCallum, and F. C. Pereira.Conditional random fields: Probabilistic models forsegmenting and labeling sequence data. 2001.

[16] A. Lamkanfi, S. Demeyer, E. Giger, and B. Goethals.Predicting the severity of a reported bug. In 2010 7thIEEE Working Conference on Mining SoftwareRepositories (MSR 2010), pages 1–10. IEEE, 2010.

[17] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags offeatures: Spatial pyramid matching for recognizingnatural scene categories. In Computer Vision andPattern Recognition, 2006 IEEE Computer SocietyConference on, volume 2, pages 2169–2178. IEEE,2006.

[18] Y. Ledru, A. Petrenko, and S. Boroday. Using stringdistances for test case prioritisation. In AutomatedSoftware Engineering, 2009. ASE’09. 24thIEEE/ACM International Conference on, pages510–514. IEEE, 2009.

[19] D. Liu, R. G. Bias, M. Lease, and R. Kuipers.Crowdsourcing for usability testing. Proceedings of theAmerican Society for Information Science andTechnology, 49(1):1–10, 2012.

[20] K. Mao, L. Capra, M. Harman, and Y. Jia. A surveyof the use of crowdsourcing in software engineering.RN, 15:01, 2015.

[21] A. Michail. Browsing and searching source code ofapplications written using a gui framework. InProceedings of the 24th International Conference onSoftware Engineering, pages 327–337. ACM, 2002.

[22] A. Michail and T. Xie. Helping users avoid bugs in guiapplications. In Proceedings of the 27th internationalconference on Software engineering, pages 107–116.ACM, 2005.

[23] M. Nebeling, M. Speicher, M. Grossniklaus, and M. C.Norrie. Crowdsourced web site evaluation withcrowdstudy. Springer, 2012.

[24] A. T. Nguyen, T. T. Nguyen, T. N. Nguyen, D. Lo,and C. Sun. Duplicate bug report detection with acombination of information retrieval and topicmodeling. In Automated Software Engineering (ASE),2012 Proceedings of the 27th IEEE/ACMInternational Conference on, pages 70–79. IEEE, 2012.

[25] F. Pastore, L. Mariani, and G. Fraser. Crowdoracles:Can the crowd solve the oracle problem? In SoftwareTesting, Verification and Validation (ICST), 2013IEEE Sixth International Conference on, pages342–351. IEEE, 2013.

[26] A. Podgurski, D. Leon, P. Francis, W. Masri,M. Minch, J. Sun, and B. Wang. Automated supportfor classifying software failure reports. In SoftwareEngineering, 2003. Proceedings. 25th InternationalConference on, pages 465–475. IEEE, 2003.

[27] A.-M. Popescu and O. Etzioni. Extracting productfeatures and opinions from reviews. In Naturallanguage processing and text mining, pages 9–28.Springer, 2007.

[28] G. Rothermel, R. H. Untch, C. Chu, and M. J.Harrold. Prioritizing test cases for regression testing.Software Engineering, IEEE Transactions on,27(10):929–948, 2001.

[29] Y. Rubner, C. Tomasi, and L. J. Guibas. The earthmover’s distance as a metric for image retrieval.

212

International journal of computer vision,40(2):99–121, 2000.

[30] P. Runeson, M. Alexandersson, and O. Nyholm.Detection of duplicate defect reports using naturallanguage processing. In Software Engineering, 2007.ICSE 2007. 29th International Conference on, pages499–510. IEEE, 2007.

[31] I. Salman, A. T. Misirli, and N. Juristo. Are studentsrepresentatives of professionals in software engineeringexperiments? In Proceedings of the 37th InternationalConference on Software Engineering-Volume 1, pages666–676. IEEE Press, 2015.

[32] Q. Shi, Z. Chen, C. Fang, Y. Feng, and B. Xu.Measuring the diversity of a test set with distanceentropy.

[33] E. Shutova, L. Sun, and A. Korhonen. Metaphoridentification using verb and noun clustering. InProceedings of the 23rd International Conference onComputational Linguistics, pages 1002–1010.Association for Computational Linguistics, 2010.

[34] C. Sun, D. Lo, S.-C. Khoo, and J. Jiang. Towardsmore accurate retrieval of duplicate bug reports. InProceedings of the 2011 26th IEEE/ACM InternationalConference on Automated Software Engineering, pages253–262. IEEE Computer Society, 2011.

[35] C. Sun, D. Lo, X. Wang, J. Jiang, and S.-C. Khoo. Adiscriminative model approach for accurate duplicatebug report retrieval. In Proceedings of the 32ndACM/IEEE International Conference on Software

Engineering-Volume 1, pages 45–54. ACM, 2010.

[36] Y. Tian, D. Lo, and C. Sun. Drone: Predictingpriority of reported bugs by multi-factor analysis. In2013 IEEE International Conference on SoftwareMaintenance, pages 200–209. IEEE, 2013.

[37] Y. Tian, C. Sun, and D. Lo. Improved duplicate bugreport identification. In Software Maintenance andReengineering (CSMR), 2012 16th EuropeanConference on, pages 385–390. IEEE, 2012.

[38] X. Wang, L. Zhang, T. Xie, J. Anvik, and J. Sun. Anapproach to detecting duplicate bug reports usingnatural language and execution information. InProceedings of the 30th international conference onSoftware engineering, pages 461–470. ACM, 2008.

[39] X. Xia, Y. Feng, D. Lo, Z. Chen, and X. Wang.Towards more accurate multi-label software behaviorlearning. In Software Maintenance, Reengineering andReverse Engineering (CSMR-WCRE), 2014 SoftwareEvolution Week-IEEE Conference on, pages 134–143.IEEE, 2014.

[40] L. Yu, W.-T. Tsai, W. Zhao, and F. Wu. Predictingdefect priority based on neural networks. In AdvancedData Mining and Applications, pages 356–367.Springer, 2010.

[41] T. Zimmermann, R. Premraj, N. Bettenburg, S. Just,A. Schroter, and C. Weiss. What makes a good bugreport? Software Engineering, IEEE Transactions on,

36(5):618–643, 2010.

213