Test Automation: From Record/Playback to Frameworks · 2019-07-25 · 9. Arnold Thomas, R.II: Building an Automation Framework with Rational Visual Test, ST Labs Report 1997 10. Buwalda,

Test Automation: From Record/Playback to

Frameworks John Kent,

Simply Testing, UKTh14

© Simply Testing Ltd 2007 Slide 2

Test Automation:

From Record/Playback

to FrameworksJohn Kent: [email protected]


Test Automation:

From Record/Playback

to Frameworks

1. Introduction

2. Record/Playback

3. Basic Data-Driven

4. Limitations of Basic Approaches

5. Advanced Frameworks

6. Summary


1. Introduction

• What Type of Automation?

• What Type of Tools?

• How the Tools Work

© Simply Testing Ltd 2007

Not:

x Performance Testing

x Test Harnesses/Drivers for Unit Test

x NUnit/JUnit frameworks

But:

Simulating User Interaction via User Interface

Automated Functional Testing via User Interface

What Type of Automation?


Script-based tools for User Interface (UI) testing

Script-based means that they are programming

development environments

E.g.

• WinRunner, QuickTest Pro (Mercury HP)

• QARun, TestPartner (Compuware)

• Robot, Functional Tester (IBM Rational)

• Others

What Type of Tools?


How Automation Tools Work

Actions Played back

Actions Recorded


User Interface Objects

• Called Controls in Windows

• Called Widgets in Unix

• Objects with Standard Window Classes

– Text boxes

– Combo boxes

– Buttons

– Option Buttons

– Check boxes

– Menu items

– List boxes

– Non-Standard


User Interface Object

Functions• Functions in the tool language

• Provide actions on UI objects

E.g.Button: button_press ( buttonID [, mouse_button ] );

E.g. Edit Box object:edit_set ( editboxID, text );

Generally of Form:

function (Object Identifier, Data)


UI Object Maps• GUI Map in WinRunner

• Object Repository in QTP

• Robot does not have one but can build one

• QARun has one

A UI Object Map lists

the objects in each

window/page. Each UI

Object is given a

logical name. The

map contains the

information that the

test tool needs to

identify each object

uniquely. The test tool

script uses the logical

name to perform

actions on UI Objects


2. Record/Playback

• Script is recorded whilst user performs

actions on SUT

• Can modify scripts with programming

(scripting)

• Test : test program ratio = 1:1


Record/PlaybackFor Each Test

There is an Automation program


3. Basic Data-Driven

Test scripts

Reads data file,

navigates to

screen/windows

and inputs data

Test ToolAdd Accounts

Data File


Basic Data-Driven

• Can create more tests by adding data – not code

• Test : test program ratio not 1:1

• More maintainable than Record/Playback method

• Automation code becomes more and more tested

with use


4. Limitations of

Basic Approaches

• Key features of test automation:– Automation is software engineering

– Maintenance of automation code is critical

– Testing of test automation code

• Test scripts must be– Developed

– Maintained

– Tested


Limitations of

Record/Playback

• We are interested in what may happen,

not what has happened

• It produces ‘difficult’ code

• May not re-run due to synchronisation

problems

• The script (code) is not tested rigorously


The Main Limitation of

Record/Playback

Number of Tests

Automated

Number of Automation

Programs

(Lines of Automation

Code)

stsNumberOfTedetomationCoLinesOfAmo

• Test : test program ratio =1:1

• The more the tests the larger the maintenance task

• Reaches limits of maintainability


Limitations of

Data-Driven• It provides repetitive actions not tests

• The data does not look like a test

– Can add lots of (e.g.) accounts

– Good for repetitive actions

• Pumps data into SUT in unnatural way -e.g. all customers added then all accounts added

• Generally used for data load - i.e. not actually a test


5. Advanced Frameworks

• What is Required?

• What is a Framework?

• Keyword Driven Automation– Business Scenario Level

– Window Level

• Object Mapping Approach

• Test Re-use


What is Required?

• A software engineering approach to the

automation code

• Maintainable automation code

• An ‘elegant’ way to create tests

– A way to automate tests easily


What is a Framework?

• Software Architecture

• Testware Architecture (Fewster, Graham)

• Software Structure

“The test framework is like an application architecture” Linda Hayes in The Automated Testing Handbook

“The software architecture of a program or computing system is the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationships among them.”

Bass, Clements, and Kazman of Carnegie Mellon’s SEI - Software Architecture in Practice see http://www.sei.cmu.edu/architecture/


Examples of Advanced

Automation

Frameworks/Architectures

• TestFrame from LogicaCMG

• EMOS_GPL open source

• SAFS open source

• ATAA from Simply Testing


Keyword Driven Automation

For Each

Window/screen/page

or Business scenario

There is only one

automation

program/function

- a Wrapper

Two levels of

wrapper/keywords

Business scenario

KeywordsWindow Level

Keywords

Keyword Driven – the wrapper is the function which ‘performs’ the keyword


Driver

WrappersData File Test Tool

Keyword Driven

Keyword Supplier_Add

wrapper

Function Supplier_Add

Function Loc_Add

Function Stock_Item

Driver reads the keyword

and calls the function for

that keyword


Actions in Test Data

Some Navigation in Test Data (Keywords)

Data is abstracted business process

Business Scenario

Level Keywords

#Supplier_Add Sup Name Description Type External Flag

Supplier_Add SC-SUP00 SC-SUP00 Supplier Company OFF

Supplier_Add SC-SUP01 SC-SUP01 Supplier Company ON

# Loc_Add Loc Name Description Corp Corporate Y/N

Loc_Add SC-SXX-1 Test LO Stock Room only ON OFF

# SI Name Item Description Name External Flag

Stock_Item_Add SC-STCK00 CABLE COAX Test SI.

Stock_Item_Add 99 PUBLICATION BOOK Wind in the ...

# SI Name Description Type Quantity

Stock_Loc_Add SC-STCK00 SC-SXX-1 1-01 10

Stock_Loc_Add 99 SC-SXX-1 1-01 10


Business Scenario WrappersFor Each Business

Scenario

There is only one

automation

program/function


Business Scenario

Level Keywords

• Business tasks to be automated are defined

• Wrappers (functions) are programmed manually

• Each business task has test code ‘Wrapper’

– Wrapper is passed test data

– Wrapper performs all the actions

• Tests specified at business process level using keywords

• Test data specifies the keyword to perform


Window Level Keywords

• Window name is the Keyword

• Totally Data Driven – all actions in data

• The excel data is the script

Window User Interface

Object

Action

Data


Window Level WrappersFor Each

Window/screen/page

There is only one

automation program


Window Wrappers

• Windows/screens/pages are ‘wrapped’ by

test programs

• Wrapper may have identification of UI

objects

• Wrapper is passed test data and actions


Wrappers Approach

Number of Tests

AutomatedAll Windows/Business

tasks automated


Programs

(Lines of Code -LOC)

{Automation

Framework/Architecture

ndowsNumberOfWiLOC

OR

zScenariosNumberOfBiLOC


5. UI Object Map Approach

• At the Window level, scripting is unnecessary

• When automation is running it needs:

– The UI Object it must act on

– The action to perform

– The data required

• This is all contained in the data

• In Object Map based frameworks, wrappers

are built for each UI Object Type – Control

Wrappers


UI Object Map Approach• No system specific code is built

• Only a map

• Framework has Control Wrappers for each class of UI Object

• Only need to maintain the map - not code


UI Object Map ApproachFor Each

Window/screen/page

There is only a map in the

Object repository


UI Object Map Approach

• Data specified at UIObject level

• The excel data is the script

WindowUser Interface

Object

Action

Data

Allowed Actions are

dependent upon UI

Object Type


UI Object Map Approach• UI Object Map built

• Control Wrappers are built for each UI Object type (if they do not already exist)

• Can use same control wrappers in all projects with that type of control

• The Control Wrappers implement the actions

MyAccount is a link UI Object.

This line of data is passed to

the Link Control Wrapper which

performs the CLICK action

firstName is a textbox object.

This line of data is passed to

the TextBox Control Wrapper

which performs the SET action



No User Interface Specific

Automation Code.

Only UI Object Maps

Number of Tests

Automated


Programs


{Automation


• Can create more tests without increasing code maintenance task


Write

Automation

Scripts

Create

Test

Data

Maintain

Automation

Scripts

Maintain

Data

Test Automation

Project Tasks


Object

Mapping

reduces

build

Create

Test

Data

Test Automation

Project Tasks – reducing

costs

Maintain

Data

Automated

Impact

Analysis

Object

Mapping

reduces

maintenance


Automated Mapping• Can generate the UI Object maps automatically from SUT:

– From source code

– From .exe

– .Net Reflection

• Only for Non-Dynamic User Interfaces


Test Re-Use • Can re-use tests and substitute values

• Higher level tests are at Business Language Level – abstracted from system UI

RegisterQUIT RTO – Re-usable Test Object

These tests specify that

the RegisterQUIT test is

run substituting in the

DOBYear values

High Level Tests


4. Summary


The Main Limitation of

Record/Playback

Number of Tests

Automated


Programs

(Lines of Automation

Code)

stsNumberOfTedetomationCoLinesOfAmo



No User Interface Specific

Automation Code.

Only UI Object Maps

Number of Tests

Automated


Programs


{Automation



Summary

UI Object Map Automation Approach:

• Removes need for scripting

• Reduces automation development time

• Vastly reduced automation maintenance

• Reduces risk: tried and tested automation code

• Independence from tool vendors: For all major automation tools: - WinRunner, QTP, QARun, TestPartner, Robot, Functional Tester


Programming

Object Mapping

2 Basic Data Driven

3 Wrapper Based

4 UI Object Mapping

5 Automated Mapping

1 Record/Playback

Automation Maturity Model

Keyword

Driven

Business Task Level or

Window Level

Wrappers


References1. Fewster, Mark and Graham, Dorothy : Software Test Automation, 1999 Addison-Wesley

2. Kent, John: Ghost in the Machine Part 5, Professional Tester, Oct 2004

3. Nagle, Carl J: Test Automation Frameworks (Web).

4. Zambelich, Keith: Totally Data-Driven Automated Testing, http://www.sqa-test.com/

5. Kent, John: Overcoming the Problems of Automated GUI Testing, presentations to STAR

1994 and British Computer Society SIGST Dec 1993.

6. Kent, John: An Automated Testing Architecture, presentation to the British Computer

Society SIGST July 1997.

7. Kent, John: Generation of Automated Test Programs Using System Models, Quality

Week 1999, EuroSTAR 1999.

8. Hayes G. Linda: The Automated Testing Handbook

9. Arnold Thomas, R.II: Building an Automation Framework with Rational Visual Test, ST

Labs Report 1997

10. Buwalda, Hans Testing with Action Words, From Automating Software Testing (Chapter

22), Addison Wesley Longman 1999

11. Ottensooser, Avner: Data Independent Test Scripts, paper presented at STAR 1997.

12. Michael Kelly: Choosing a Test Automation Framework - (Web)

13. Kaner, Cem: Improving the Maintainability of Automated Test Suites, Quality Week 1997

14. Dwyer, Graham and Graham Freeburn: Business Object Scenarios: a fifth-generation

approach to automated testing, From Automating Software Testing (Chapter 22), Addison

Wesley Longman 1999

http://www.sqa-test.com/



Documents

Test Automation: From Record/Playback to Frameworks · 2019-07-25 · 9. Arnold Thomas, R.II: Building an Automation Framework with Rational Visual Test, ST Labs Report 1997 10. Buwalda,