20041221 gui testing survey

GUI Testing GUI Testing ApproachesApproaches

Will ShenWill Shen

2004/12/212004/12/21

ReferencesReferences

Hierarchical GUI Test Case Hierarchical GUI Test Case Generation Using Automated Generation Using Automated PlanningPlanning Atif M. MemonAtif M. Memon, Student Member, IEEE, , Student Member, IEEE,

Martha E. PollackMartha E. Pollack, and , and Mary Lou SoffaMary Lou Soffa, , Member, IEEEMember, IEEE

Plan Generation for GUI TestingPlan Generation for GUI Testing Atif M. MemonAtif M. Memon and and Martha E. PollackMartha E. Pollack and and

Mary Lou SoffaMary Lou Soffa Dept. of Computer Science Dept. of Computer Science University of Pittsburgh Pittsburgh, PA 15260 University of Pittsburgh Pittsburgh, PA 15260 USA fatif, pollack, soffag@cs.pitt.edu USA fatif, pollack, soffag@cs.pitt.edu

Coverage Criteria for GUI Testing Atif M. Memon Dept. of Computer

Science University of Pittsburgh Pittsburgh, PA 15260 atif@cs.pitt.edu

Mary Lou Soffa Dept. of Computer Science University of Pittsburgh Pittsburgh, PA 15260 soffa@cs.pitt.edu

Martha E. Pollack Dept. of Electrical Engineering and Computer Science, University of Michigan Ann Arbor, MI 48109pollackm@eecs.umich.edu

OutlineOutline

1.1. IntroductionIntroduction

2.2. Overview of PATHSOverview of PATHS

3.3. Coverage Criteria for GUI TestingCoverage Criteria for GUI Testing

4.4. ConclusionsConclusions

1. Introduction1. Introduction

Testing GUI is difficult:Testing GUI is difficult: The space of possible interactions with a The space of possible interactions with a

GUI is enormous.GUI is enormous. In that each sequence of GUI commands can In that each sequence of GUI commands can

result in a different stateresult in a different state A GUI command may need to be evaluatedion A GUI command may need to be evaluatedion

all of these states.all of these states. Determining the coverage of a set of test Determining the coverage of a set of test

casescases.. No only how much the code is tested, but in No only how much the code is tested, but in

how many different possible states of the how many different possible states of the software each piece of code is tested.software each piece of code is tested.

An important aspect of GUI testing is An important aspect of GUI testing is verificationverification of its state at each step of of its state at each step of test case execution.test case execution. The execution of the test case must be The execution of the test case must be

terminated as soon as an error is detected.terminated as soon as an error is detected. Regression testing presents special Regression testing presents special

challenges for GUIs.challenges for GUIs. The input-output mapping does not remain The input-output mapping does not remain

constant across successive versions of the constant across successive versions of the software.software.

2. Overview of PATHS2. Overview of PATHS

PPlanning lanning AAssited ssited TTester for grapester for grapHHical ical user interface user interface SSystems.ystems.

A new approach to automatic testing of A new approach to automatic testing of GUIs that builds on AI planning GUIs that builds on AI planning techniques.techniques.

Given a specification of initial and goal Given a specification of initial and goal states for a GUI, a planner is used to states for a GUI, a planner is used to generate sequences of GUI actions that generate sequences of GUI actions that lead from the initial state to the goal lead from the initial state to the goal state.state.

2.1 The Example GUI2.1 The Example GUI

Microsoft Microsoft WordPadWordPad

(a) the Initial States

(b) The Goal State

2.2 Test case generation 2.2 Test case generation processprocess

Two phasesTwo phases Setup phaseSetup phase

1.1. PATHS creates a hierarchical model of the PATHS creates a hierarchical model of the GUI and returns a list of operators from GUI and returns a list of operators from the model to the test designerthe model to the test designer

2.2. By using knowledge of the GUI, the test By using knowledge of the GUI, the test designer then defines the preconditions designer then defines the preconditions and effects of the operators in a simple and effects of the operators in a simple language provided by the planning system.language provided by the planning system.

Plan-generationPlan-generation1.1. The test designer describes scenarios The test designer describes scenarios

(tasks) by defining a set of initial and goal (tasks) by defining a set of initial and goal states for test case generation.states for test case generation.

2.2. PATHS generates a test suite for the PATHS generates a test suite for the scenarios.scenarios.

2.3 Deriving GUI 2.3 Deriving GUI operatorsoperators

2.3.1 Events2.3.1 Events 2.3.2 Operators2.3.2 Operators 2.3.3 Operator-Event mapping2.3.3 Operator-Event mapping 2.3.4 The Step1 of example GUI2.3.4 The Step1 of example GUI 2.3.5 The Step2 of EDIT_CUT2.3.5 The Step2 of EDIT_CUT 2.3.6 The Step3 initial state and goal 2.3.6 The Step3 initial state and goal

statestate 2.3.7 The Step4 generate test case2.3.7 The Step4 generate test case

2.3.1 Events2.3.1 Events

Three classes of GUI eventThree classes of GUI event Unrestricted-focus eventsUnrestricted-focus events open GUI open GUI

windows that do not restrict the userwindows that do not restrict the user’’s focuss focus Restricted-focus eventsRestricted-focus events open GUI windows open GUI windows

that have the special property that once that have the special property that once invoked, they monopolize the GUI invoked, they monopolize the GUI interaction.interaction.

System-interaction eventsSystem-interaction events interact with the interact with the underlying software to perform some action.underlying software to perform some action.

2.3.2 Operators2.3.2 Operators

The setup phase starts creating a list The setup phase starts creating a list of a operators to be used during of a operators to be used during planning.planning.

Exploiting the GUI structure to Exploiting the GUI structure to derive hierarchical operators that derive hierarchical operators that are decomposed during planning.are decomposed during planning. System-Interaction OperatorsSystem-Interaction Operators Abstract OperatorsAbstract Operators

System-Interaction System-Interaction OperatorsOperators

To represent sequences of GUI actions To represent sequences of GUI actions that a user might perform to eventually that a user might perform to eventually interact with the underlying software.interact with the underlying software.

A sequence of zero or more A sequence of zero or more unrestricted-focus eventsunrestricted-focus events, followed by , followed by a a system-interaction eventsystem-interaction event..

Example:Example: Edit_Cut = <Edit, Cut>Edit_Cut = <Edit, Cut> Edit_Paste = <Edit, Paste>Edit_Paste = <Edit, Paste>

Abstract OperatorsAbstract Operators

Created from the restricted-focus Created from the restricted-focus events, which contain two parts:events, which contain two parts: The The prefixprefix of an abstract operator is of an abstract operator is

the sequence of the sequence of unrestricted-focus unrestricted-focus eventsevents that lead to that lead to restricted-focus restricted-focus eventevent..

The The suffixsuffix of an abstract operatorof an abstract operator represents the represents the restricted-focus restricted-focus user user interaction.interaction.

2.3.3 Operator-Event 2.3.3 Operator-Event mappingmapping

In order to keep a correspondence In order to keep a correspondence between the original GUI events and between the original GUI events and these high-level operators.these high-level operators.

2.3.4 The Step1 of 2.3.4 The Step1 of example GUIexample GUI

(a) Original GUI Events

(b) Planning operators derived by PATHS

2.3.5 The Step2 of 2.3.5 The Step2 of EDIT_CUTEDIT_CUT

2.3.6 The Step3 initial state 2.3.6 The Step3 initial state and goal stateand goal state

2.3.7 The Step4 generate 2.3.7 The Step4 generate test casetest case

High level plan that must be decomposition

decomposition

decomposition

An alternative An alternative decompositiondecomposition

A new test caseA new test case

3. Coverage Criteria for GUI 3. Coverage Criteria for GUI TestingTesting

3.1 What is Coverage Criteria3.1 What is Coverage Criteria 3.2 Event-flow Graphs3.2 Event-flow Graphs 3.3 Integration Tree3.3 Integration Tree 3.4 Intra-component Coverage 3.4 Intra-component Coverage

CriteriaCriteria 3.5 Inter-component Coverage 3.5 Inter-component Coverage

CriteriaCriteria

3.1 What is Coverage 3.1 What is Coverage CriteriaCriteria

Coverage criteria are sets of rules to Coverage criteria are sets of rules to help determine whether a test suite help determine whether a test suite has adequately tested a program and has adequately tested a program and to guide the testing process.to guide the testing process.

Important rules that provide an Important rules that provide an objective measure of test quality.objective measure of test quality.

This paper define a new class of This paper define a new class of coverage criteria called coverage criteria called event-based event-based coverage criteriacoverage criteria to determine the to determine the adequacy of tested event sequences, adequacy of tested event sequences, focus on GUIs.focus on GUIs.

The key idea is to define the The key idea is to define the coverage of a test suite in terms of coverage of a test suite in terms of GUI events and their interactions.GUI events and their interactions.

A GUI component is represented by A GUI component is represented by a new structure called an a new structure called an event-flow event-flow graphgraph that identifier events within a that identifier events within a component.component.

The interactions among GUI The interactions among GUI components are captured by a components are captured by a representation called the representation called the integration integration treetree..

Intra-component coverage criteriaIntra-component coverage criteria for events within a componentfor events within a component eventevent, , event-interactionevent-interaction and and length-n length-n

event-sequenceevent-sequence coverage. coverage. Inter-component coverage criteriaInter-component coverage criteria

for events among components.for events among components. invocationinvocation, , invocation-terminationinvocation-termination and and

length-n event-sequencelength-n event-sequence coverage coverage

3.2 Event-flow Graphs3.2 Event-flow Graphs

A part of the MainA part of the Main** component of component of MS WordPadMS WordPad

* We assume that all GUIs have a Main component, that is presented to the user when the GUI is first invoked.

III

B

3.3 Integration Tree3.3 Integration Tree

An integration tree for a part of An integration tree for a part of MS WordPadMS WordPad

3.4 Intra-component 3.4 Intra-component Coverage CriteriaCoverage Criteria

3.4.1 Event Coverage3.4.1 Event Coverage 3.4.2 Event-interaction Coverage3.4.2 Event-interaction Coverage 3.4.3 Length-n Event-sequence 3.4.3 Length-n Event-sequence

CoverageCoverage

3.4.1 Event Coverage3.4.1 Event CoverageEvent-

sequence:

3.4.2 Event-interaction 3.4.2 Event-interaction CoverageCoverage

3.4.3 Length-n Event-3.4.3 Length-n Event-sequence Coveragesequence Coverage

3.5 Inter-component 3.5 Inter-component Coverage CriteriaCoverage Criteria

3.5.1 Invocation Coverage3.5.1 Invocation Coverage 3.5.2 Invocation-termination 3.5.2 Invocation-termination

CoverageCoverage 3.5.3 Inter-component Length-n 3.5.3 Inter-component Length-n

Event-sequence CoverageEvent-sequence Coverage

3.5.1 Invocation 3.5.1 Invocation CoverageCoverage

3.5.2 Invocation-3.5.2 Invocation-termination Coveragetermination Coverage

3.5.3 Inter-component 3.5.3 Inter-component Length-n Event-sequence Length-n Event-sequence

CoverageCoverage

4. Conclusion4. Conclusion

Automatic testing of GUIs that builds Automatic testing of GUIs that builds on AI planning techniqueon AI planning technique

Coverage criteria for GUI testingCoverage criteria for GUI testing We also plan to explore the possibility We also plan to explore the possibility

of using the event-based coverage of using the event-based coverage criteria for software other than GUIs.criteria for software other than GUIs. Object-oriented softwareObject-oriented software Networking softwareNetworking software The broader class of reactive softwareThe broader class of reactive software

Any questions?