A Comprehensive Framework for Testing Graphical User Interfaces

1

A Comprehensive A Comprehensive Framework Framework for Testing for Testing

Graphical User InterfacesGraphical User InterfacesAtif M. MemonAtif M. Memon

[email protected]@cs.pitt.edu

Dept. of Computer ScienceDept. of Computer Science

University of PittsburghUniversity of Pittsburgh

Advisors: Mary Lou Soffa & Martha E. Advisors: Mary Lou Soffa & Martha E. PollackPollack

2

GUI

UnderlyingCode

Interactions between theGUI and the underlying code

Research focusResearch focus

50% of code

3

Challenges of GUI TestingChallenges of GUI Testing• What is the test input to the GUI? What is the test input to the GUI?

– (Test case generation)(Test case generation)

• How much testing is enough?How much testing is enough?– (Test coverage)(Test coverage)

• Is the GUI executing correctly during testing?Is the GUI executing correctly during testing?– (Test oracles)(Test oracles)

• What test results can be salvaged from the What test results can be salvaged from the previous test runs to test a new version?previous test runs to test a new version?– (Regression testing)(Regression testing)

• How to represent the GUI to handle all the How to represent the GUI to handle all the above?above?– (Representation)(Representation)

4

Current TechniquesCurrent Techniques

• Capture/replay toolsCapture/replay tools• GUI test case generationGUI test case generation

– FSMFSM [Esmelioglu and Apfelbaum] and [Esmelioglu and Apfelbaum] and VFSMVFSM [Shahady and Siewiorek] [Shahady and Siewiorek] modelsmodels

– Genetic algorithmGenetic algorithm technique [Kasik technique [Kasik and George]and George]

– Visual TDE for GUIsVisual TDE for GUIs [Foster, Goradia, [Foster, Goradia, Ostrand, and Szermer] Ostrand, and Szermer]

5

Goals of My ResearchGoals of My Research

• Develop a comprehensive environment Develop a comprehensive environment that integrates techniques (tools) to that integrates techniques (tools) to address various aspects of GUI testingaddress various aspects of GUI testing

• Goals of developing the techniquesGoals of developing the techniques– IntegratedIntegrated: for compatibility: for compatibility– AutomatedAutomated: avoid manual repetition of tasks: avoid manual repetition of tasks– EfficientEfficient: to meet tight testing deadlines: to meet tight testing deadlines– RobustRobust: to recognize a GUI error as soon as : to recognize a GUI error as soon as

it occurs and report the problemit occurs and report the problem– PortablePortable: apply to multiple platforms: apply to multiple platforms– GeneralGeneral: apply to a wide variety of GUIs: apply to a wide variety of GUIs

6

FrameworkFramework

• New representation for GUIsNew representation for GUIs• Test case generation based on AI Test case generation based on AI

planning planning [ICSE 1999, AIPS 2000, IEEE TSE Feb [ICSE 1999, AIPS 2000, IEEE TSE Feb 2001]2001]

• New event-based coverage criteria New event-based coverage criteria [FSE 2001][FSE 2001]

• Unique test oracles Unique test oracles [FSE 2000][FSE 2000]

• Repairing test cases for automated Repairing test cases for automated regression testingregression testing

• Automated test executor Automated test executor [QW 2000][QW 2000]

• A comprehensive framework that A comprehensive framework that integrates all the above componentsintegrates all the above components

**

*

*

7

Test CaseGenerator

RegressionTester

TestCoverageEvaluator

Test Oracle

GUI Representation

Overview of the FrameworkOverview of the Framework

ExecutingGUI

GUI Implementation:Tools (Languages/Toolkits)

GUI Implementer

GUI Specifications

Test Executor

8

Test CaseGenerator

RegressionTester


Test Oracle

GUI RepresentationGUI Representation

ExecutingGUI


GUI Implementer

GUI Specifications

GUI ModelTest Designer

Test Executor

9

Creating the GUI ModelCreating the GUI Model

• ModelingModeling– GUI’s state in terms of objects & their GUI’s state in terms of objects & their

propertiesproperties– Events as state transducersEvents as state transducers– GUI’s hierarchical structureGUI’s hierarchical structure

• GUI componentsGUI components• Event-flow graphsEvent-flow graphs• Integration treeIntegration tree• Classifying eventsClassifying events

10

Modeling the GUI’s StateModeling the GUI’s State

• A GUI at time T is modeled using:A GUI at time T is modeled using:– Objects O = {oObjects O = {o11, o, o22, o, o33, …, o, …, omm}}

– Properties P = {pProperties P = {p11, p, p22, p, p33, …, p, …, pll}, }, where pwhere pii is an n is an nii-ary (n-ary (nii >= 1) >= 1) BooleanBoolean relation of the form: relation of the form:

Property(oj, oa, ob, …,ox, value)

Property NameObject

Optional value of Property

Optional Objects

GUI’s state: S = {pGUI’s state: S = {p11,, pp22,, pp33,,…,p…,pnn}}

True/False

Caption(Button, “Cancel”)

11Example: Modeling the GUI’s Example: Modeling the GUI’s StateState

Button1

Form1

Label1Align(Label1,

Caption(Label1,Color(Label1,Font(Label1,

alNone)“Files of type:”)clBtnFace)(tfont))

Caption(Button1,Enabled(Button1,

Visible(Button1,Height(Button1,

WState(Form1,Width(Form1,Scroll(Form1,

“Cancel”)TRUE)TRUE)65)

wsNormal)1088)TRUE)

12

All Properties of All Properties of Button1Button1

TabOrder 0

Caption Cancel

13Determining Objects & Determining Objects & PropertiesProperties

• Manual ExaminationManual Examination• Specifications (reduced set)Specifications (reduced set)

– GUI being testedGUI being tested

• Toolkit/language (complete set)Toolkit/language (complete set)– All available propertiesAll available properties

Now we know how to represent the GUI’s state

14

GUI EventsGUI Events

• GUI’s state is not staticGUI’s state is not static• Events change the GUI’s stateEvents change the GUI’s state

• Events E = {eEvents E = {e11, e, e22, e, e33, …, e, …, enn}, }, associated with a GUI are functions associated with a GUI are functions from one GUI state Sfrom one GUI state Sii to another to another state Sstate Sjj

• Notation: SNotation: Sjj = e = eii(S(Sii))

15

Example: An EventExample: An Event

set-background-color(w19, yellow)

State: Si

State: Sj

Event: e

Background color is yellow

Background color is not yellow

w19

Sj = e(Si)

16

Representing EventsRepresenting Events

• Infeasible to give exhaustive Infeasible to give exhaustive specifications of the state mapping specifications of the state mapping for each eventfor each event

• No set limit to the number of objects No set limit to the number of objects a GUI can contain at any point in timea GUI can contain at any point in time

• There can be infinitely many states of There can be infinitely many states of the GUIthe GUI

• Model the GUI events using Model the GUI events using operators, which specify their operators, which specify their preconditions and effectspreconditions and effects

17

OperatorsOperators• An Operator is a triple An Operator is a triple

– <Name, Precondition, Effects><Name, Precondition, Effects>– Name identifies an event and its parametersName identifies an event and its parameters– Precondition is a set of positive literalsPrecondition is a set of positive literals– Effects is a set of positive or negative literalsEffects is a set of positive or negative literals

• Operator Op is applicable in any state SOperator Op is applicable in any state Sii in in which:which:– All the literals in Precondition(Op) are TRUEAll the literals in Precondition(Op) are TRUE

• The resulting state SThe resulting state Sjj is determined by using is determined by using Effects(Op) Effects(Op) – All positive literals in Effects(Op), andAll positive literals in Effects(Op), and

– All literals that were TRUE in SAll literals that were TRUE in Sii, except, except

– Those that are negative in Effects(Op)Those that are negative in Effects(Op)

18

Operator :: set-background-colorParameters: wX: window; col: color;Precondition:

isCurrent(wX),background-color(wX, oldColor),oldColor != col.

Effects:background-color(wX, col).

Operator ExampleOperator Example

Now we know how to represent the events of the GUI

Primitive Operator


19

Model GUI HierarchicallyModel GUI Hierarchically

• HierarchyHierarchy– GUIs are designed as a hierarchy of GUIs are designed as a hierarchy of

componentscomponents– Hierarchical model makes testing Hierarchical model makes testing

intuitive and efficientintuitive and efficient

20

Modal Windows in GUIsModal Windows in GUIs

Main

21


Main

Print

invokes

22


Main

Print

Properties

Components

23

Integration TreeIntegration TreeMain

Properties

FileNew FileOpen Print FormatFontFileSave PageSetup ViewOptions

DefinitionDefinition: : Integration treeIntegration tree is a triple < is a triple <NN, , RR, , BB>>• NN is the set of components in the GUI is the set of components in the GUI• RR є є NN is a designated component called the is a designated component called the MainMain

component component • BB is the set of directed edges showing the invokes is the set of directed edges showing the invokes

relation between components, i.e., (Crelation between components, i.e., (Cxx, C, Cyy) є ) є BB iff C iff Cxx invokes Cinvokes Cyy..

24

Open …Save

Representing a ComponentRepresenting a Component

File Edit HelpOpen …Save

Cut CopyPaste

About …Contents …

Event-flow Graph

follows

DefinitionDefinition: Event e: Event exx follows e follows eyy iff e iff exx can be can be performed immediately after eperformed immediately after eyy..

25

Event-flow GraphEvent-flow Graph

FileEdit

Help

Open Save

Cut Copy Paste

About Contents

To File, Edit and Help

To File, Edit and Help

DefinitionDefinition: Event-flow graph is a 4-tuple <: Event-flow graph is a 4-tuple <V, E, B, IV, E, B, I>>• VV is the set of vertices, representing events, is the set of vertices, representing events,• EE is the set of directed edges, showing the follows is the set of directed edges, showing the follows

relationship,relationship,• BB is the set of events first available ( is the set of events first available (shown in redshown in red),),• II is the set of events that invoke other components is the set of events that invoke other components

((dotted lines).).

26

Classifying EventsClassifying Events

• Opening modal windows Opening modal windows – Restricted-focus eventsRestricted-focus events

• Closing modal windows Closing modal windows – Termination eventsTermination events

• Opening modeless windows Opening modeless windows – Unrestricted-focus eventsUnrestricted-focus events

• Opening menusOpening menus– Menu-open eventsMenu-open events

• Interacting with underlying software Interacting with underlying software – System-interaction eventsSystem-interaction events

•ClassificationClassification

–A new classification of events aids in A new classification of events aids in creating the hierarchical model of the GUIcreating the hierarchical model of the GUI

27

GUI Modeling StepsGUI Modeling Steps

• From the GUI’s specifications From the GUI’s specifications (formal/informal),(formal/informal),– Identify objects and propertiesIdentify objects and properties– Create operators for GUI eventsCreate operators for GUI events– Using the event classification, identify Using the event classification, identify

GUI components, create event-flow GUI components, create event-flow graphs and integration treegraphs and integration tree

28

RegressionTester


Test Oracle

Test Case GeneratorTest Case Generator

ExecutingGUI


GUI Implementer

GUI Specifications


Test-CaseGenerator

Test Case

Tasks

Test Executor

29

GUI Test CasesGUI Test Cases

• Individual user eventsIndividual user events– Not enough !Not enough !– Sequences of user events lead to Sequences of user events lead to

different statesdifferent states

• Test case: contains sequence of Test case: contains sequence of user eventsuser events

30

Definition: GUI Test CaseDefinition: GUI Test Case

• legal event sequencelegal event sequence – ee11;e;e22;e;e33;… ;e;… ;enn is a legal event sequence is a legal event sequence

• if (eif (eii, e, ei+1i+1) is an edge in an event-flow graph) is an edge in an event-flow graph

• or eor eii invokes component C invokes component Cxx and e and ei+1i+1 is an is an event in Cevent in Cxx

• A GUI test case is a tripleA GUI test case is a triple– (S(S00, e, e11;e;e22;e;e33;…;e;…;enn, S, S11; S; S22;… ;S;… ;Snn))

• SS00 is a GUI state, and is a GUI state, and

• ee11;e;e22;e;e33;…;e;…;enn is a legal event sequence is a legal event sequence

• SSii = e = eii(S(Si-1i-1), 1 <= i <= n), 1 <= i <= n

31

A Test Case for WordPadA Test Case for WordPad

SelectText(“This”)

Format Font 18 OKSelectText

(“text”)

Format Font Underline OK

This is the text.

This is the text.

S0

Expected State Sequence in Test Oracles

Expected State Sequence in Test Oracles

32

Selecting Event SequencesSelecting Event Sequences• Infinitely manyInfinitely many• ManualManual

– Expert chooses sequencesExpert chooses sequences

• AutomatedAutomated– Randomly choose sequencesRandomly choose sequences– Structural (event-flow graph & integration tree)Structural (event-flow graph & integration tree)– Automatically generate events for COMMONLY USED TASKSAutomatically generate events for COMMONLY USED TASKS

Initial State Goal State

MultipleEvent SequencesThis is the text. This is the text.

33

A Plan for a GUI TaskA Plan for a GUI Task


SelectText(“text”)

SetFontSize(18)

MouseClick(U)

Initial State

This is the text.

Goal State

This is the text.

is theThis text.

is theThis text.

34Overview of Test Overview of Test GenerationGeneration

Phase Step Test Designer Framework

Setup 1 Obtain planning operators f rom GUI model

2 Code preconditions and eff ects of operators in model

Test Case Generation

3 Specif y a task (I nitial and Goal States)

4 Generate Test Cases

35

Using Primitive OperatorsUsing Primitive Operators• One operatorOne operator for each event for each event

Operator :: set-background-colorParameters: wX: window; col: color;Precondition:

isCurrent(wX),background-color(wX, oldColor),oldColor != col.

Effects:background-color(wX, col).


36

Exploit the GUI’s StructureExploit the GUI’s Structure

• Reduce the number of operatorsReduce the number of operators– System more efficientSystem more efficient– Easier for the test designerEasier for the test designer

37

Operator AbstractionsOperator Abstractions

Two types of abstractionsTwo types of abstractions– Combine buttons Combine buttons create create system-system-

interactioninteraction operators operators– Decompose GUI hierarchicallyDecompose GUI hierarchically

create create componentcomponent operators operators• Key idea: the user interacts with one Key idea: the user interacts with one

component at a time.component at a time.

38Create System-Interaction Create System-Interaction OperatorsOperators

File_SendTo_MailRecipientFile_SendTo_MailRecipient

= <File + SendTo + = <File + SendTo + MailRecipient >MailRecipient >

Sys-Interaction Operator:

File

Send To

Mail Recipient

39Create Component Create Component Operators Operators

SelectFromList()Default

OKCancel

Language Window’s Operator Set

Main GUI’sOperator Set

…Set Language


OKCancel

...

English (United States)

OK Cancel Default...

Using Primitive Operators Only

Set Language

…Set Language

...

Main GUI’sOperator Set

Using Abstraction

40

Using Component Operators Using Component Operators

SetLanguage()

SelectFromList(“English(US)”)

OK

ComponentOperator

Planner


OKCancel

Language Window’s Operator Set

SetLanguage()

... ...High Level Plan

Sub PlanDefineAbstraction

AdvantageAlternative decomposition

leads to a new test case

41Effects of Exploiting the GUI’s Effects of Exploiting the GUI’s StructureStructure

• Reduction in planning operatorsReduction in planning operators– 362 operators 362 operators 32 operators 32 operators– Ratio 10:1 for MS WordpadRatio 10:1 for MS Wordpad– 20:1 for MS Word20:1 for MS Word

• EfficiencyEfficiency

TaskNo.

PlanLength

Time(sec.)

Hier PlanLength

Time(sec.)

1 18 8.93 3 0.112 20 47.62 4 0.183 24 189.87 5 0.144 26 3312.72 6 7.185 - - 3 0.106 - - 4 13.01

Single Level Hierarchical

42

Initial State

Goal State

This is the text.

This is the text.

43

Test CaseTest Case


FormatFont(“This”, 18pt)

FormatFont(“text”, Underline)


PrimitiveOperator

PrimitiveOperator

ComponentOperator

INITIAL

GOAL

FormatFont 18 OK

Planner

FormatFont Underline OK

Planner

Format Font Format Font

Mapping Mapping


Format Font 18 OKSelectText

(“text”)


ComponentOperator

44

Different from HTN PlanningDifferent from HTN Planning





PrimitiveOperator

PrimitiveOperator

ComponentOperator

ComponentOperator

INITIAL

GOAL

FormatFont 18 OK FormatFont Underline OK

Planner Planner

No Interactions

45

Alternative Alternative Test CaseTest Case


SelectFromList(18)



PrimitiveOperator

PrimitiveOperator

PrimitiveOperator

ComponentOperator


SelectFromList(18)







PrimitiveOperator

PrimitiveOperator

ComponentOperator

ComponentOperator

INITIAL

GOAL

46

ExperimentsExperiments• Purpose Purpose

– To determine whether planning is a feasible To determine whether planning is a feasible approach for GUI test case generationapproach for GUI test case generation• Execution timeExecution time• Human effortHuman effort

• Experimental designExperimental design– GUIGUI:: our version of MS Wordpad (36 modal our version of MS Wordpad (36 modal

windows, 362 events)windows, 362 events)– TasksTasks:: 50 tasks (initial & goal states) 50 tasks (initial & goal states)– Test casesTest cases:: generated 290 test cases (6-56 generated 290 test cases (6-56

events) using the IPP plannerevents) using the IPP planner– Hardware platformHardware platform:: 350 MHz Pentium based 350 MHz Pentium based

machine, 256 MB RAMmachine, 256 MB RAM

47

Test Case GenerationTest Case Generation

Generating Test Cases

00.10.20.30.40.50.60.70.80.9

1 6 11 16 21 26 31 36 41 46 51 56

Test-Case Length

Tim

e (

sec.

)

Test Case

48

Automated ExecutionAutomated Execution

Executing Test Cases

0

0.1

0.2

0.3

0.4

0.5

1 6 11 16 21 26 31 36 41 46 51 56

Test-Case Length

Tim

e (

sec.

)

Test Case

49

RegressionTester


Test OraclesTest Oracles

ExecutingGUI


GUI Implementer

GUI Specifications


Test-CaseGenerator

Test Case

TasksExpected-state

Generator

Expected-statesequence

Oracle

Verifier

Verdict

Actual StateInformation

ExecutionMonitor

GUI State(run-time)

Test Executor

50

Determine Correct BehaviorDetermine Correct Behavior• To check the To check the GUI’s stateGUI’s state after each after each

eventevent• ApproachesApproaches

– ManualManual– AutomatedAutomated Actual State Expected State

Automated Verifier

Verdict

• ChallengesChallenges– Generating expected stateGenerating expected state– Extracting actual stateExtracting actual state– Comparing expected & actual statesComparing expected & actual states

51

Overview of GUI OracleOverview of GUI Oracle

Event Sequence

Expected-stateGenerator

Verifier

Expected State

ExecutionMonitor

Oracle

ActualState

Run-timeinformation from

executing GUI

Verdict

GUIModel

52

Deriving Expected StateDeriving Expected State

• Obtain Obtain SS11 = e = e11(S(S00))

• And And SSii = e = eii(S(Si-1i-1))

e1 e2 e3 en

S0S0 S1

S1 S2S2 S3

S3 SnSn

53Obtaining Actual GUI’s Obtaining Actual GUI’s StateState

• Execution monitorExecution monitor– Compatible with expected stateCompatible with expected state– Returns property(objects…, value)Returns property(objects…, value)

e.g., Caption(button1, “Cancel”)e.g., Caption(button1, “Cancel”)– Actual state can be obtained byActual state can be obtained by

• Screen scrapingScreen scraping• QueriesQueries

54


Expected StateExpected State

Event Executor

GUI Under Test

Execution MonitorACTUAL STATE:(isCurrent ROOT)(Contains ROOT D(Contains ROOT D

Verifier

Event SequenceEvent Sequence

Level of checkingLevel of checking

55

ExperimentsExperiments• Purpose: determinePurpose: determine

– Time to derive expected stateTime to derive expected state– Time to execute monitor and verifier Time to execute monitor and verifier

• Experimental designExperimental design– GUIGUI:: our version of MS Wordpad (36 modal our version of MS Wordpad (36 modal

windows, 362 events)windows, 362 events)– Test casesTest cases:: generated 290 test cases (6-56 generated 290 test cases (6-56

events) using an AI plannerevents) using an AI planner– Hardware platformHardware platform:: 350 MHz Pentium based 350 MHz Pentium based

machine, 256 MB RAMmachine, 256 MB RAM– PropertiesProperties:: reduced set reduced set– Level of checkingLevel of checking:: GUI relevant property set GUI relevant property set

56

Deriving Expected StateDeriving Expected State

Generating Test Cases and Deriving Expected State

00.10.20.30.40.50.60.70.80.9

1 6 11 16 21 26 31 36 41 46 51 56

Test-Case Length

Tim

e (

sec.

)

Expected State

Test Case

Test Case + Expected State

Total CPU time (all 290 test cases and expected states)

75.84 sec.

57


Executing Test Cases, Verifier and Execution Monitor

0

1

2

3

4

5

1 6 11 16 21 26 31 36 41 46 51 56

Test-Case Length

Tim

e (

sec.

)

Test Case

Verifier + Execution Monitor

Test Case + Verifier + Execution Monitor

GUI Relevant-properties checkingTotal running time < 10 minutes

58

Putting it All TogetherPutting it All Together

ExecutingGUI


GUI Implementer

GUI Specifications


Test-CaseGenerator

Test Case

TasksExpected-state

Generator

Expected-statesequence

Oracle

Verifier

Verdict

Actual StateInformation

ExecutionMonitor

GUI State(run-time)

CoverageReport

Coverage Criteria

Coverage Evaluator

RegressionTester

Test Executor

GUI Specifications

59

Future WorkFuture Work

• ExtensionsExtensions– Web testingWeb testing– Interface & Underlying codeInterface & Underlying code

• RelationshipRelationship• SeparationSeparation

– GUI Specifications & TestingGUI Specifications & Testing– Prioritizing GUI test casesPrioritizing GUI test cases

• New DirectionsNew Directions– Object-oriented & Component-basedObject-oriented & Component-based– ReactiveReactive– Execution ProfilesExecution Profiles

Documents

A Comprehensive Framework for Testing Graphical User Interfaces