Mukul R. Prasad

Fujitsu Labs. Of America USA

A Grey-Box Approach for Automated GUI-Model Generation of Mobile

Applications

Wei YangNC State University

USA

Tao XieNC State University

USA

Growing Popularity of Smartphone

2 Source: IDC

Growing Demand for High-Quality Apps

Automated Testing Tools!Solution? Source: Compuware

Sample App: Simple TippyTipperUsed to calculate tip amount for a meal

MENU

MENU

4

GUI Model for Simple TippyTipper

1 2

3

5a

4

5c 5d

5b

00

10

01

Calculate

Menu Menu

About Settings

a1

a1

a2a2

a1: Toggle exclude tax rate optiona2: Toggle round up option

11

（ 1） （ 2） （ 4）

（ 3） （ 5）（ 3）

5

Automated Testing with GUI Model

• Classical model-based testing–Generation of system-level test

cases–Driver synthesis for symbolic

execution

• Recent “crawl-and-compare” testing–Cross-platform compatibility testing–Regression testing across different

versions

6

ORBIT GUI Crawler: Overview

7

Automated Program Analysis (Static + Dynamic)

Android Apps

GUI Model

ORBIT GUI Crawler: Overview

8

Android Apps

Action inference Dynamic Crawler

GUI ModelORBITAction

Mapping

Proposed GUI Model • Visual observable state

–Inspired by the UI-design principles “make places in the app look distinct” by the Android team

–Hierarchy GUI-component tree of a UI screen

• Model: a finite-state machine• States: visual observable states

• Transitions: user actions

9

GUI Model for Simple TippyTipper

1 2

3

5a

4

5c 5d

5b

00

10

01

Calculate

Menu Menu

AboutSettings

a1

a1

a2a2

a1: Toggle exclude tax rate option.a2: Toggle round up option.

11

（ 1） （ 2） （ 4）

（ 3） （ 5）（ 3）

10

Examples of States

11

Two different states of Screen # 5b & 5d

These two states differ in the state of this widget

（ 5b） （ 5d）

Action Inference

R.Id.javaView btn_delete = findViewById(R.id.btn_delete);

Btn_delete.setOnClickListener(new onClickListener() { public void onClick(View v) { removeBillAmount(); FlurryAgent.onEvent(“Delete Button”); } });

Btn_delete.setOnClickListener(new onLongClickListener() { public void onLongClick(View v) { clearBillAmount(); return true; } });

……

TippyTipper.java

Inference: Widget btn_delete with Id = 0x7f0000a supports actions click and longClick

• Locate all instances of event handler definitions in the code (sinks)

• Associate each with the corresponding widget instantiations (sources)

12

Action Inference Algorithm

13

Algorithm 1. registeredActionDetectionInput : A: app source codeOutput: E: action map1 begin 2 ActionSet ← getAllActions()3 EntryPoints ← getAllEntryPoints()4 foreach P ∈ EntryPoints do5 CG ← makeCallGraph(A, P)6 foreach X ∈ ActionSet do7 L ← getEventRegMethod(X)8 PNodeSet ← getParentNode(CG, L) // Get all L’s callers9 foreach PNode ∈ PNodeSet do10 s ← findCallT o(PNode,L)11 v ← getCallingObject(s)12 i ← backLocate(v,A)13 ID ← getParameter(i)14 E.add(ID,X)15 end16 end17 end18 end

Simple TippyTipper: Actions Found

• ScreenActionMapping– (net.mandaria.tippytipper.activities.TippyTipp

er, Menu)– (net.mandaria.tippytipper.activities.Total,

Menu)• WidgetActionMapping

– (R.id.btn_delete, click)– (R.id.btn_delete, longclick)– (R.id.btn_clear, click)– (R.id.btn_clear, longclick)

14

MENUBACK

Activity Total

Dynamic Crawling

15

Activity TippyTipper

MENU

Activity Setting

SETTINGS

BACK

Toggle

Toggle

Toggle

Toggle

Toggle

Calculate

Toggle

BACK

ToggleToggle

1-9，DEL, CLEAR

Dynamic Crawling: AlgorithmAlgorithm: crawlappInput : A: app under test, E: action mapOutput: M: crawled model1 begin2 M←∅; s ← getOpeningScreen(A)3 while s = null do4 s ← forwardCrawlFromState(s,A,M,E) // forward crawl from s5 s ← backtrack(s,A)6 if isInitialState(s) then s ← findNewOpenState(s,M,A)7 end8 end

1 begin2 sx ← sc3 while sx = null do4 s ← sx5 if isNewState(s) then6 initActions(s,E,A)7 addToModel(s,M)8 end9 e ← getNextOpenAction(s)10 if e = null then sx ← null11 else12 sx ← execute(s,e,A)13 updateOpenActions(s,e)14 addToModel(s,e,sx,M)15 end16 end17 return s18 end

Algorithm: forwardCrawlFromStateInput : sc: state to crawl from, A: app under test,M: crawled model being generated, E: action mapOutput: s: current state at end of crawling

16

Tool Implementation

FwdCrawl Algorithm

Robotium

Android Runtime

Dynamic Crawler

Action DetectorWALA

Intent Passing Logic

Sub-CallGraph

Partial Connected Call Graph

Inference Algorithm

Action MappingAndroid App

sour

ce c

ode

ORBIT

GUI Modelde

ploy

17

• Event and component based model (not suitable for stateful behavior)• Black-box & dynamic (not effective for complex events)

 Related Work• GUITAR: GUI Ripping for Windows, Java-based GUI

(desktop) apps, Memon et al. 2003

18

Click 1 Click

2Click

5Click

4

Click 3

Click 6

Click 9

Click 8

Click 7

Click 0

Click Calculat

e

Click DEL

Click CLEAR

1 2

35a

4

5c 5d

5b

00

10

01

Calculate

Menu Menu

AboutSettingsa1

a1a2a2

a1: Toggle exclude tax rate optiona2: Toggle round up option

11

A simplified event-flow model of simple tippytipper Our GUI model of simple tippytipper

 Related Work(Cont.)• Concolic Testing of Android Applications

Anand et al. 2012

• Random Testing of Android ApplicationsHu and Neamtiu 2011

• AndroidRipper: Stress Testing of Android ApplicationsAmaltano et al. 2012

• Crawljax: Crawling of AJAX Web ApplicationsMesbah et al. 2012 19

Research Questions• RQ1: Is the proposed GUI crawling

algorithm more efficient than a standard depth-first state-traversal algorithm?

• RQ2: Are the widget and screen actions inferred by static analysis effective in enhancing the behavior covered by the generated model?

• RQ3: Can our tool generate a higher-quality model, more efficiently, compared to other state-of-the-art techniques?

20

Evaluation SubjectsSubject #LOC #

ACategory Purpose

TippyTipper 2238 5 Tool Dining tip calculatorOpenManager 1595 6 Business File manager for AndroidNotepad 322 3 Productivity Note creation and

managementTomDroid 3711 3 Business Online note-readingAarddict 4518 4 Books Aard dictionary for AndroidHelloAUT 234 1 Entertainme

ntColor shapes

ContactManager

497 2 Productivity Contacts manager

ToDoManager 323 2 Productivity Create and manage task lists

21

• Our evaluation is based on subjects drawn from existing related work• Avoid bias by including all subjects used to

evaluate Android GUITAR and GUI Ripper in previous work.

Efficiency of Crawling Algorithms

Subjects FwdCrawl DFSTime(sec)

Cov. (%)

#Fwd

Back

Time(sec)

Cov. (%)

#Fwd

Back

TippyTipper 198 78 61 15 512 82 134 52OpenManager 480 63 92 18 822 56 209 29Notepad2 102 82 25 4 147 83 39 12Notepad0 80 78 18 2 75 71 15 2TomDroid 340 70 78 23 459 58 61 8AardDict 173 65 15 2 397 60 20 8HelloAUT 156 86 46 0 278 85 61 0ContactManager

125 91 20 1 137 92 22 2

ToDoManager 178 75 60 2 294 74 84 4

22

• DFS takes 70% more time to traverse all 9 subjects together

Effectiveness of Action Inference

Subjects #clicks

#longClicks

#menu #States

Coverage(%)

C C+I

C C+I C C+I C C+I

C C+I

TippyTipper 21 55 - 2 - 4 3 9 47 78OpenManager 50 67 - 19 - 4 10 20 39 63Notepad2 2 13 - 3 - 9 2 7 39 82Notepad0 0 8 - 1 - 9 0 7 14 78TomDroid 3 52 - 0 - 26 2 9 36 70AardDict 4 15 - 0 - 7 3 7 43 64HelloAUT 15 34 - 0 - 12 4 8 53 86ContactManager

20 20 - 0 - 0 5 5 92 92

ToDoManager 60 60 - 0 - 0 7 7 76 76

23

• Firing non-click actions during crawling increases the coverage by 34% on average.

• Our crawler is 32%-75% faster while constructing a 5%-140% more complete model than Android GUITAR and Android GUI Ripper.

Comparison with Other ToolsSubjects Monkey Android

GUITARAndroid GUI

RipperORBIT

Time(sec)

Cov.(%)

Time (sec)

Cov. (%)

Time (sec)

Cov.(%)

Time (sec)

Cov.(%)

TippyTipper 83 41 322 47 - - 198 78OpenManager 90 29 - - - - 480 63Notepad2 127 60 - - - - 102 82Notepad0 122 59 - - - - 80 78TomDroid 69 46 - - 529 40 340 70AardDict 124 51 - - 694 27 173 65HelloAUT 98 71 117 51 - - 156 86ContactManager 90 53 247 61 - - 125 91ToDoManager 115 71 194 71 - - 178 75

Conclusion

ORBIT: a grey-box approach for automated GUI-model generation of mobile applications

Evaluation shows that ORBIT• RQ1: performs more efficiently than a standard

DFS algorithm• RQ2: enhances effectively behavior coverage w/

static analysis• RQ3: generates a higher-quality model, more

efficiently than existing state-of-the-art techniques25

Android Apps

Action inference Dynamic Crawler

GUI ModelORBITAction

Mapping

Thanks!

Acknowledgment: NSF grants CCF-0845272, CCF-0915400, CNF-0958235, CNS-1160603, and an NSA Science of Security Lablet Grant.

Conclusion

ORBIT: a grey-box approach for automated GUI-model generation of mobile applications

Evaluation shows that ORBIT• RQ1: performs more efficiently than a standard

DFS algorithm• RQ2: enhances effectively behavior coverage w/

static analysis• RQ3: generates a higher-quality model, more

efficiently than existing state-of-the-art techniques27

Android Apps

Action inference Dynamic Crawler

GUI ModelORBITAction

Mapping

Future Work• Crawling algorithm

– Incorporate the context of event call to prune the order of the sequence.

• GUI model– Explore the possibility to generate a

complete specification or test oracle by using our driver and partial specification.

• Selection of subjects– Inter-application communication of

Android application.• Manual effort

28