Hierarchical GUI Test Case Generation Using Automated

Planning

Atif M. Memon, Student Member, IEEE, Martha E. Pollack, and Mary Lou Soffa, Member, IEEE

IntroductionIntroduction

Graphical User Interfaces – important in interacting with today’s software

Makes software easier to use Makes software testing more difficult than

conventional software - must find a PATH from some initial state to a solution

Consists of labels, buttons, menus, and pop-up lists

User interacts with components to create events

What is a GUI?What is a GUI?

GUI Window

Object Window

Microsoft WordPad

Challenges in Testing GUI Challenges in Testing GUI ApplicationsApplications

GUI’s becoming more complex Complexity in testing correctness and underlying software

Enormous amounts of possible interaction Determining coverage of Test Cases Regression testing

Goal: Automatic generation of test cases

Solution: Planning Assisted Tester for

grapHical User interface Systems

State of the ArtState of the Art

A new technique both scalable and partially automatedthat reduces dependence on user expertise

Pros Cons

Mimic noviceusers

Not scalable

ScalableSemi-automated

Test case qualitydepends on userexpertise

Automated testcase generation

Finite statemachine models

35,000 foot View of the Mountain35,000 foot View of the Mountain

Performs hierarchical structure of GUI create hierarchical operators

Test designers describe operators(preconditions, effects)

Takes goals as input and generates multiple sequences of events (“plans”) automatically

Plans becomes the test cases for GUI

PATHS…

…easier to specify goals than all the possible PATHS toward the

goal

PATHS: AI PlanningPATHS: AI Planningwith Operatorswith Operators

Operator (preconditions, effects)

Given to the AI planning algorithm Guaranteed to result in the goal state when

executed in the initial state

Initial State Goal State

The PATHS Test CaseThe PATHS Test CaseGeneration ProcessGeneration Process

Setup Phase – PATHS creates a hierarchical model of the GUI and returns list of operators from the model to test designer

Design preconditions and effects in simple, planning system language

Plan Generation Phase – Test designer describes scenarios by defining a set of initial and goal states for test case generation

PATHS generates a test suite for test case generation

Phase 1: SetupPhase 1: Setup

PATHS partitions GUI into several classes using structural properties.

Menu-open Events – expand set of GUI events available to user

Unrestricted-focus Events – open GUI windows that do not restrict user’s focus, expand set of GUI events available to user (ex. Toolbar)

Restricted-focus Events – open GUI windows that have special property once invoked, monopolize GUI interaction, restrict user view of GUI to specific range until window is terminate (ex. Pop-up window)

System-interaction Events – interact with underlying software to perform some action

Phase 1: Operator TypesPhase 1: Operator Types

System-Interaction Operators – derived from those GUI events that generate interactions with the underlying software

Ex. Prefix_Suffix File_Open = <File, Open>

Abstract Operators – represents a sequence of GUI events that invoke a window which monopolizes the GUI interaction

Treat this restricted-focus window as a separate planning problem and decompose

Phase 1: Setup ExamplePhase 1: Setup Example

GUI EventsGUI Events

FileFile++

EditEdit++

NewNewOpen*Open*SaveSaveSave As*Save As*CutCutCopyCopyPastePaste

Open.UpOpen.UpOpen.SelectOpen.SelectOpen.CancelOpen.CancelOpen.OpenOpen.OpenSaveAs.UpSaveAs.UpSaveAs.SelectSaveAs.SelectSaveAs.CancelSaveAs.CancelSaveAs.SaveSaveAs.Save

Planning OperatorsPlanning OperatorsFile_File_File_File_File_File_File_File_

Edit_Edit_Edit_Edit_Edit_Edit_

NewNewOpen*Open*SaveSaveSaveAs*SaveAs*

CutCutCopyCopyPastePaste

Phase 1: Operator Event MappingPhase 1: Operator Event Mapping

Transforms GUI menu events into planning operators

These results Returned as Planning operators to the Designer…who then specifies preconditions and effects for each planning operator

Phase 2: Test Case GenerationPhase 2: Test Case Generation

Test case designer inputs defined operators into PATHS and identifies a task (initial state, goal state)

Planning Operators PATHSPATHS

Defined Task

Tests Cases Achieving Goal

Abstract Operator DecompositionAbstract Operator Decomposition

Operators from the higher level need to be decomposed into lower level operations by

Inserting expansion from operator-event mappings, or

Making an additional call to the planner

Hierarchical mechanism aids in regression testing since changes made to one component to do not necessarily invalidate all test cases.

Plan Generation: AI Planning Plan Generation: AI Planning ProblemProblem

Planning Problem – P(Λ, D, I G) P(Operator Set, Finite Set of Objects, Initial State, Goal

State)

Solution – <S, O, L, B> <Set of Plan Steps, Ordering constraints on S, Causal Links,

Binding constraints on S>

Si < Sj – Step i must occur before Step j (not necessarily

right before)

Causal Links -- <Si, c, Sj>

Ensure no step threatens a required link

Partial Order Plans…Partial Order Plans…

…aids in deriving total order plan Add ordering constraints i.e. Linearization of partial-order plan

… are solutions to a planning problem iff every consistent linearization of the partial-order plan meets the solution conditions

Want to learn more about how the Authors implemented AI Planning…?We invite you to read about it in the paper.

(IPP and HTN)

Evaluation: Empirical ApproachEvaluation: Empirical Approach

A PATHS prototype was developed based Microsoft WordPad

325 GUI events Reduced to 32 system-interaction and abstract

operators

Evaluation: Empirical ApproachEvaluation: Empirical ApproachEvaluation: Empirical ApproachEvaluation: Empirical Approach

10-fold reduction of operators10-fold reduction of operatorsstreamline plan generationstreamline plan generation

(phase 2 of PATHS)(phase 2 of PATHS)

Evaluation: Run Two ExperimentsEvaluation: Run Two Experiments

Two experiments were run using the PATHS prototype

Experiment 1: Generate test cases for multiple tasks

Is PATHS practical and efficient?

Experiment 2: Compare hierarchical vs. single-level test case generation

Is the hierarchical modeling approach more efficient than single-level?

Experiment 1: GExperiment 1: Generating Test enerating Test Cases for Multiple TasksCases for Multiple Tasks

Use PATHS prototype to generate a suite of test cases for 9 WordPad tasks

Deploy experiment on Pentium-based computer running Linux on 200Mb RAM

Track times required to generate high-level plans and subplans

PATHS is practical and efficientPATHS is practical and efficient

EExperiment 2: Hierarchical vs. xperiment 2: Hierarchical vs. Single-Level Test Case GenerationSingle-Level Test Case Generation

Use PATHS prototype to generate test cases for 6 WordPad tasks

Deploy experiment on same computer (Pentium-based running Linux on 200Mb RAM)

Track plan lengths

Track times required to generate a single-level and a hierarchical test case

Hierarchical test case generation Hierarchical test case generation is more efficientis more efficient

ConclusionsConclusions

Applying AI planning to the problem is a good match

Hierarchy of the model is efficient in generating test cases

4- to 6-fold saving in plan length Test case generated in seconds rather than hours

New technique is scalable when software releases new version, operators can be recycled

New technique is highly automated

PATHS: ContributionsPATHS: Contributions

Use well-known and used AI planning technique capable of solving large-space problems

Exploits structural features present in GUIs to reduce model size & complexity, and efficiently generate test cases

Hierarchical Planning makes regression testing easier

Increased portability – platform specific details incorporated last in test case generation

Reuse of commonly-appearing operator definitions

Open QuestionsOpen Questions

PATHS is practical, but is it useful? Preliminary exploration achieves 20-fold reduction in GUI events

How does one measure usefulness?

How well does PATHS scale? Manual analysis of operators will vary from application to

application

How good are the test cases generated by PATHS? Coverage

Negative test cases

Related / Future WorkRelated / Future Work

Use library of operators to minimize human intervention

How to automatically generate preconditions and effects of operators