1

Program Testing and Analysis:

GUI Testing

Dr. Michael Pradel

Software Lab, TU Darmstadt

2

Warm-up Quiz

var a = (0.1 + 0.2) + 0.3;var b = 0.1 + (0.2 + 0.3);console.log(a === b);

What does the following code print?

Something elsefalsetrue

2

Warm-up Quiz

var a = (0.1 + 0.2) + 0.3;var b = 0.1 + (0.2 + 0.3);console.log(a === b);

What does the following code print?

Something elsefalsetrue

2

Warm-up Quiz

var a = (0.1 + 0.2) + 0.3;var b = 0.1 + (0.2 + 0.3);console.log(a === b);

What does the following code print?

Something elsefalsetrue

Floating point numbers are representedwith finite precision(not only in JavaScript)

2

Warm-up Quiz

var a = (0.1 + 0.2) + 0.3;var b = 0.1 + (0.2 + 0.3);console.log(a === b);

What does the following code print?

Something elsefalsetrue

0.30000000000000004(due to rounding)

3

GUI Testing

� Test application via its graphical userinterface (GUI)

� Possible approaches� Manual testing� Semi-automated, e.g., Selenium

� Here: Automated GUI testing� Purely random testing, e.g., Monkey for

Android� Today: Sophisticated approaches beyond

purely random testing

4

Challenges

� Two input dimensions� Sequences of events, e.g., clicks, mouse

movements (focus of today’s lecture)� Input values, e.g., strings entered into form

� Not all events are known ahead of time� Web apps load content dynamically

� Need a test oracle� When does an execution expose a bug?

� Huge search space

5

Huge Search Space

Challenge: Huge search space� Too large to explore exhaustively

......

...

...

......

.........

......

5

Huge Search Space

......

...

...

......

.........

...

Approach:Steer search toward potential bugsor not yet explored behavior

...

6

Outline

� Feedback-directed GUI testingBased on A Framework for Automated Testing ofJavaScript Web Applications, Artzi et al., ICSE 2011

� Model inference-based testingBased on Guided GUI Testing of Android Apps withMinimal Restart and Approximate Learning , Choi et al.,OOPSLA 2013

� Responsiveness testingBased on EventBreak: Analyzing the Responsivenessof User Interfaces through Performance-Guided TestGeneration, Pradel et al., OOPSLA 2014

7

Artemis

Feedback-directed generation of GUItests

� Start with randomly selected events

� Gather feedback from execution

� Steer toward particularly interesting behavior

� Implemented for web applications

� Test oracle: JavaScript exceptions and invalidHTML

8

Example

Prepare1

Prepare2

Process

Application with 3 buttons:

prepared1 = true;

prepared2 = true;

if (prepared1)if (prepared2)window.alert("hi");

Initially, prepared1=prepared2=false

9

Gathering Feedback

Feedback gathered while executinggenerated sequences of events:

� Available events

� Source code of handlers attached to events

� Memory locations read & written

� Branch coverage

10

Artemis: Algorithm

� Input: URL u

� Add sequence [load u] to worklist

� While worklist not empty

� Execute next sequence and gather feedback

� Add new sequences to worklist∗ Modify inputs of existing sequence∗ Extend sequence with additional event∗ Create new sequence with new start URL

� Prioritize worklist

11

Prioritization by Coverage

Coverage-guided prioritization

� Keep track of branch points in each handler� Branch here means entry of handler or control flow

branch

� Prioritize sequences that trigger handlers with lowcoverage

P (e1, ..., ek) = 1− cov(e1) · ... · cov(ek)

where cov(e) =covered branches of e’s handler

all discovered branches of e’s handler

12

Example

Prepare1

Prepare2

Process

Application with 3 buttons:

prepared1 = true;

prepared2 = true;

if (prepared1)if (prepared2)window.alert("hi");

Initially, prepared1=prepared2=false

1

14

Priorization by Memory Accesses

Prioritize based on read/write sets

� Keep track of memory locations read/written byeach handler

� Prioritize sequences where some handlers writevalues read by a subsequence handler

� Intuition: Can cover interesting behavior only aftersome handlers have set the right pre-conditions

15

Outline

� Feedback-directed GUI testingBased on A Framework for Automated Testing ofJavaScript Web Applications, Artzi et al., ICSE 2011

� Model inference-based testingBased on Guided GUI Testing of Android Apps withMinimal Restart and Approximate Learning , Choi et al.,OOPSLA 2013

� Responsiveness testingBased on EventBreak: Analyzing the Responsivenessof User Interfaces through Performance-Guided TestGeneration, Pradel et al., OOPSLA 2014

16

SwiftHand

� Challenge: Restarting the applicationis expensive

� Learn finite-state model of applicationwhile exploring it� Explore states with unknown outgoing

transitions� Continuously refine model by splitting states

� Explores application with smallnumber of restarts

17

Outline

� Feedback-directed GUI testingBased on A Framework for Automated Testing ofJavaScript Web Applications, Artzi et al., ICSE 2011

� Model inference-based testingBased on Guided GUI Testing of Android Apps withMinimal Restart and Approximate Learning , Choi et al.,OOPSLA 2013

� Responsiveness testingBased on EventBreak: Analyzing the Responsivenessof User Interfaces through Performance-Guided TestGeneration, Pradel et al., OOPSLA 2014

18

Motivation

single thread of execution

Event-based UI applications should beresponsive

18

Motivation

click

single thread of execution

Event-based UI applications should beresponsive

18

Motivation

click

single thread of execution

Event-based UI applications should beresponsive

click

18

Motivation

click

single thread of execution

Event-based UI applications should beresponsive

click

18

Motivation

single thread of execution

Event-based UI applications should beresponsive

19

Real-World Example

Joomla bug 30274

19

Real-World Example

Joomla bug 30274

19

Real-World Example

Joomla bug 30274

19

Real-World Example

Joomla bug 30274

19

Real-World Example

Joomla bug 30274

19

Real-World Example

Joomla bug 30274

19

Real-World Example

Joomla bug 30274

19

Real-World Example

Joomla bug 30274

Unresponsive

19

Real-World Example

Joomla bug 30274

Cost plot for responsiveness problem

19

Real-World Example

Joomla bug 30274

Unboundedgrowth:Unresponsiveapplication

Cost plot for responsiveness problem

20

EventBreak: Idea

Analyze responsiveness of webapplications through automated testing

Focus: Slowdown pairs

Event Ecause increases cost of event Eeffect

21

Overview

Event-cost history

Dynamic analysis of application

Infer potentialslowdown pairs

Infer finite statemodel of application

Slowdown pairs with cost plots

Targeted test generation:Verify slowdown pairs

21

Overview

Event-cost history

Dynamic analysis of application

Infer potentialslowdown pairs

Infer finite statemodel of application

Slowdown pairs with cost plots

Targeted test generation:Verify slowdown pairs

22

Event-Cost History

Sequence of event-cost pairs

� DOM element� Type of event� Pre-state� Post-state

Number ofconditionals evaluatedin event handler

23

Potential Slowdown Pairs

Event CostB 5A 3B 10B 12A 3B 12

Does A increase cost of B?

Supportingevidence S

Refutingevidence R

Supp = |S| = 1 Conf =|S|

|S|+ |R|= 33%

24

Algorithm

� Input: History of event-cost pairs (e, c) withunique events E

� For each (e1, e2) ∈ E × E

� Initialize S(e1, e2) = 0 and R(e1, e2) = 0

� For each minimal subsequence (e2, c12), ..., (e2, c

22) in

history∗ If c12 < c22 and e1 in subsequence, S(e1, e2)++∗ If c12 < c22 xor e1 in subsequence, R(e1, e2)++

� Output (e1, e2) if S(e1, e2) > minsupp andS(e1,e2)

S(e1,e2)+R(e1,e2)> minconf

25

Quiz

What are support andconfidence for thepotential slowdownpair (A,B)?

Event CostB 14A 4C 7B 18A 4B 18A 4C 9B 22

2

26

Targeted Test Generation

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:

current state target event

Ecause = Save new item

Eeffect =

Save menu

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:

current state target event

Ecause = Save new item

Eeffect =

Save menuEeffect =

Save menu

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:

current state target event

Ecause = Save new item

Eeffect =

Save menuEeffect =

Save menu

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:5

current state target event

Ecause = Save new item

Eeffect =

Save menu

Ecause = Save new item

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:5

current state target event

Ecause = Save new item

Eeffect =

Save menu

Ecause = Save new item

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:5

current state target event

Ecause = Save new item

Eeffect =

Save menu

Ecause = Save new item

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:5

current state target event

Ecause = Save new item

Eeffect =

Save menu

Ecause = Save new item

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:5

current state target event

Ecause = Save new item

Eeffect =

Save menuEeffect =

Save menu

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

26

Targeted Test Generation

Costs of Eeffect:5, 10, 15, etc.

current state target event

Ecause = Save new item

Eeffect =

Save menu

Confirm or reject slowdown pair:Alternate between Eeffect and Ecause

27

Navigation: Challenge

Shortest event sequence in model maybe infeasible in application

27

Navigation: Challenge

Ecause = Save new item

Eeffect =

Save menucurrent state target event

Shortest event sequence in model maybe infeasible in application

27

Navigation: Challenge

Ecause = Save new item

Eeffect =

Save menu

Insert text,Save new item

current state target event

Shortest event sequence in model maybe infeasible in application

27

Navigation: Challenge

Ecause = Save new item

Eeffect =

Save menu

Insert text,Save new item

current state target event

Ecause = Save new item

Shortest event sequence in model maybe infeasible in application

28

Navigation: Approach

Ecause = Save new item

Eeffect =

Save menu

Insert text,Save new item

current state target event

Ecause = Save new item

� Compute sequences to target event� Randomly pick from set of first steps

29

Evaluation

Find responsiveness problems?� 2 known + 4 previously unknown

Effectiveness of targeted testgeneration?� Reaches 89% of all target events� Invalidates > 99% of all potential

slowdown pairs

30

Results: Joomla

Joomla bug 30274

30

Results: Joomla

Joomla bug 30274

Several similar examples:� Ecause accumulates data items� Eeffect processes all of them and

has unbounded cost

31

Summary: GUI Testing

� Automated system-level testing

� Black-box and white-box approaches to explorehuge search space� Artemis: Whitebox� SwiftHand and EventBreak: Mostly blackbox

� Different test oracles possible� Application crashes (robustness testing)� Consistency criterion, e.g., HTML validation� Responsiveness (performance testing)